Merge pull request #43082 from kubernetes/dev-1.29

Official v1.29 Release Docs

Author: katcosgrove

assets/scss/_custom.scss

@@ -392,52 +392,63 @@ footer {
 }
 
 main {
-  .td-content table code,
-  .td-content>table td {
-    word-break: break-word;
-  }
 
-/* SCSS Related to the Metrics Table */
+/* SCSS Related to the Metrics list */
+
+  div.metric:nth-of-type(odd) { // Look & Feel , Aesthetics
+    background-color: $light-grey;
+  }
 
-    @media (max-width: 767px) { // for mobile devices, Display the names, Stability levels & types
+  div.metrics {
 
-     table.metrics { 
-      th:nth-child(n + 4),
-      td:nth-child(n + 4) {
+    .metric {
+      div:empty{
         display: none;
       }
 
-      td.metric_type{
-        min-width: 7em;
+      display: flex;
+      flex-direction: column;
+      flex-wrap: wrap;
+      gap: .75em;
+      padding:.75em .75em .75em .75em;
+  
+      .metric_name{
+        font-size: large;
+        font-weight: bold;
+        word-break: break-word;
       }
-      td.metric_stability_level{
-        min-width: 6em;
+  
+      label{
+        font-weight: bold;
+        margin-right: .5em;
       }
-    }
-    }
-
-    table.metrics tbody{ // Tested dimensions to improve overall aesthetic of the table
-      tr {
-        td {
-          font-size: smaller;
-        }
-        td.metric_labels_varying{
-          min-width: 9em;
-        }
-        td.metric_type{
-          min-width: 9em;
+      ul {
+        li:empty{
+          display: none;
         }
-        td.metric_description{
-          min-width: 10em;
+        display: flex;
+        flex-direction: column;
+        gap: .75em;
+        flex-wrap: wrap;
+        li.metric_labels_varying{
+          span{
+                display: inline-block;
+                background-color: rgb(240, 239, 239);
+                padding: 0 0.5em;
+                margin-right: .35em;
+                font-family: monospace;
+                border: 1px solid rgb(230 , 230 , 230);
+                border-radius: 5%;
+                margin-bottom: .35em;
+          }
         }
-    
+  
       }
+      
     }
 
-  table.no-word-break td,
-  table.no-word-break code {
-    word-break: normal;
-}
+
+  }
 }
 
 // blockquotes and callouts

content/en/docs/concepts/architecture/garbage-collection.md

@@ -137,6 +137,20 @@ collection, which deletes images in order based on the last time they were used,
 starting with the oldest first. The kubelet deletes images
 until disk usage reaches the `LowThresholdPercent` value.
 
+#### Garbage collection for unused container images {#image-maximum-age-gc}
+
+{{< feature-state for_k8s_version="v1.29" state="alpha" >}}
+
+As an alpha feature, you can specify the maximum time a local image can be unused for,
+regardless of disk usage. This is a kubelet setting that you configure for each node.
+
+To configure the setting, enable the `ImageMaximumGCAge`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) for the kubelet,
+and also set a value for the `ImageMaximumGCAge` field in the kubelet configuration file.
+
+The value is specified as a Kubernetes _duration_; for example, you can set the configuration
+field to `3d12h`, which means 3 days and 12 hours.
+
 ### Container garbage collection {#container-image-garbage-collection}
 
 The kubelet garbage collects unused containers based on the following variables,
@@ -178,4 +192,4 @@ configure garbage collection:
 
 * Learn more about [ownership of Kubernetes objects](/docs/concepts/overview/working-with-objects/owners-dependents/).
 * Learn more about Kubernetes [finalizers](/docs/concepts/overview/working-with-objects/finalizers/).
-* Learn about the [TTL controller](/docs/concepts/workloads/controllers/ttlafterfinished/) that cleans up finished Jobs.
+* Learn about the [TTL controller](/docs/concepts/workloads/controllers/ttlafterfinished/) that cleans up finished Jobs.

content/en/docs/concepts/cluster-administration/flow-control.md

@@ -7,7 +7,7 @@ weight: 110
 
 <!-- overview -->
 
-{{< feature-state state="beta"  for_k8s_version="v1.20" >}}
+{{< feature-state state="stable"  for_k8s_version="v1.29" >}}
 
 Controlling the behavior of the Kubernetes API server in an overload situation
 is a key task for cluster administrators. The {{< glossary_tooltip
@@ -45,30 +45,27 @@ are not subject to the `--max-requests-inflight` limit.
 
 ## Enabling/Disabling API Priority and Fairness
 
-The API Priority and Fairness feature is controlled by a feature gate
-and is enabled by default.  See [Feature
-Gates](/docs/reference/command-line-tools-reference/feature-gates/)
-for a general explanation of feature gates and how to enable and
-disable them.  The name of the feature gate for APF is
-"APIPriorityAndFairness".  This feature also involves an {{<
-glossary_tooltip term_id="api-group" text="API Group" >}} with: (a) a
-`v1alpha1` version and a `v1beta1` version, disabled by default, and
-(b) `v1beta2` and `v1beta3` versions, enabled by default.  You can
-disable the feature gate and API group beta versions by adding the
+The API Priority and Fairness feature is controlled by a command-line flag
+and is enabled by default.  See 
+[Options](/docs/reference/command-line-tools-reference/kube-apiserver/options/)
+for a general explanation of the available kube-apiserver command-line 
+options and how to enable and disable them.  The name of the 
+command-line option for APF is "--enable-priority-and-fairness".  This feature
+also involves an {{<glossary_tooltip term_id="api-group" text="API Group" >}} 
+with: (a) a stable `v1` version, introduced in 1.29, and 
+enabled by default (b) a `v1beta3` version, enabled by default, and
+deprecated in v1.29.  You can
+disable the API group beta version `v1beta3` by adding the
 following command-line flags to your `kube-apiserver` invocation:
 
 ```shell
 kube-apiserver \
---feature-gates=APIPriorityAndFairness=false \
---runtime-config=flowcontrol.apiserver.k8s.io/v1beta2=false,flowcontrol.apiserver.k8s.io/v1beta3=false \
+--runtime-config=flowcontrol.apiserver.k8s.io/v1beta3=false \
  # …and other flags as usual
 ```
 
-Alternatively, you can enable the v1alpha1 and v1beta1 versions of the API group
-with `--runtime-config=flowcontrol.apiserver.k8s.io/v1alpha1=true,flowcontrol.apiserver.k8s.io/v1beta1=true`.
-
 The command-line flag `--enable-priority-and-fairness=false` will disable the
-API Priority and Fairness feature, even if other flags have enabled it.
+API Priority and Fairness feature.
 
 ## Concepts
 
@@ -178,14 +175,12 @@ server.
 ## Resources
 
 The flow control API involves two kinds of resources.
-[PriorityLevelConfigurations](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#prioritylevelconfiguration-v1beta2-flowcontrol-apiserver-k8s-io)
+[PriorityLevelConfigurations](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#prioritylevelconfiguration-v1-flowcontrol-apiserver-k8s-io)
 define the available priority levels, the share of the available concurrency
 budget that each can handle, and allow for fine-tuning queuing behavior.
-[FlowSchemas](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#flowschema-v1beta2-flowcontrol-apiserver-k8s-io)
+[FlowSchemas](/docs/reference/generated/kubernetes-api/{{< param "version" >}}/#flowschema-v1-flowcontrol-apiserver-k8s-io)
 are used to classify individual inbound requests, matching each to a
-single PriorityLevelConfiguration.  There is also a `v1alpha1` version
-of the same API group, and it has the same Kinds with the same syntax and
-semantics.
+single PriorityLevelConfiguration.
 
 ### PriorityLevelConfiguration
 

content/en/docs/concepts/cluster-administration/system-metrics.md

@@ -202,10 +202,23 @@ Here is an example:
 --allow-label-value number_count_metric,odd_number='1,3,5', number_count_metric,even_number='2,4,6', date_gauge_metric,weekend='Saturday,Sunday'
 ```
 
+In addition to specifying this from the CLI, this can also be done within a configuration file. You
+can specify the path to that configuration file using the `--allow-metric-labels-manifest` command
+line argument to a component. Here's an example of the contents of that configuration file:
+
+```yaml
+allow-list:
+- "metric1,label2": "v1,v2,v3"
+- "metric2,label1": "v1,v2,v3"
+```
+
+Additionally, the `cardinality_enforcement_unexpected_categorizations_total` meta-metric records the
+count of unexpected categorizations during cardinality enforcement, that is, whenever a label value
+is encountered that is not allowed with respect to the allow-list contraints.
+
 ## {{% heading "whatsnext" %}}
 
 * Read about the [Prometheus text format](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-based-format)
   for metrics
 * See the list of [stable Kubernetes metrics](https://github.com/kubernetes/kubernetes/blob/master/test/instrumentation/testdata/stable-metrics-list.yaml)
-* Read about the [Kubernetes deprecation policy](/docs/reference/using-api/deprecation-policy/#deprecating-a-feature-or-behavior)
-
+* Read about the [Kubernetes deprecation policy](/docs/reference/using-api/deprecation-policy/#deprecating-a-feature-or-behavior)

content/en/docs/concepts/containers/container-lifecycle-hooks.md

@@ -55,12 +55,15 @@ There are two types of hook handlers that can be implemented for Containers:
 * Exec - Executes a specific command, such as `pre-stop.sh`, inside the cgroups and namespaces of the Container.
 Resources consumed by the command are counted against the Container.
 * HTTP - Executes an HTTP request against a specific endpoint on the Container.
+* Sleep - Pauses the container for a specified duration. 
+  The "Sleep" action is available when the [feature gate](/docs/reference/command-line-tool-reference/feagure-gates/)
+  `PodLifecycleSleepAction` is enabled.
 
 ### Hook handler execution
 
 When a Container lifecycle management hook is called,
 the Kubernetes management system executes the handler according to the hook action,
-`httpGet` and `tcpSocket` are executed by the kubelet process, and `exec` is executed in the container.
+`httpGet` , `tcpSocket` and `sleep` are executed by the kubelet process, and `exec` is executed in the container.
 
 Hook handler calls are synchronous within the context of the Pod containing the Container.
 This means that for a `PostStart` hook,

content/en/docs/concepts/containers/images.md

@@ -159,6 +159,17 @@ that Kubernetes will keep trying to pull the image, with an increasing back-off
 Kubernetes raises the delay between each attempt until it reaches a compiled-in limit,
 which is 300 seconds (5 minutes).
 
+### Image pull per runtime class
+
+{{< feature-state for_k8s_version="v1.29" state="alpha" >}}
+Kubernetes includes alpha support for performing image pulls based on the RuntimeClass of a Pod.
+
+If you enable the `RuntimeClassInImageCriApi` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/),
+the kubelet references container images by a tuple of (image name, runtime handler) rather than just the
+image name or digest. Your {{< glossary_tooltip text="container runtime" term_id="container-runtime" >}}
+may adapt its behavior based on the selected runtime handler.
+Pulling images based on runtime class will be helpful for VM based containers like windows hyperV containers.
+
 ## Serial and parallel image pulls
 
 By default, kubelet pulls images serially. In other words, kubelet sends only

content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md

@@ -159,8 +159,8 @@ The general workflow of a device plugin includes the following steps:
    {{< note >}}
    The processing of the fully-qualified CDI device names by the Device Manager requires
    that the `DevicePluginCDIDevices` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
-   is enabled for the kubelet and the kube-apiserver. This was added as an alpha feature in Kubernetes
-   v1.28.
+   is enabled for both the kubelet and the kube-apiserver. This was added as an alpha feature in Kubernetes
+   v1.28 and graduated to beta in v1.29.
    {{< /note >}}
 
 ### Handling kubelet restarts

content/en/docs/concepts/scheduling-eviction/assign-pod-node.md

@@ -358,6 +358,108 @@ The affinity term is applied to namespaces selected by both `namespaceSelector`
 Note that an empty `namespaceSelector` ({}) matches all namespaces, while a null or empty `namespaces` list and
 null `namespaceSelector` matches the namespace of the Pod where the rule is defined.
 
+#### matchLabelKeys
+
+{{< feature-state for_k8s_version="v1.29" state="alpha" >}}
+
+{{< note >}}
+<!-- UPDATE THIS WHEN PROMOTING TO BETA -->
+The `matchLabelKeys` field is a alpha-level field and is disabled by default in
+Kubernetes {{< skew currentVersion >}}.
+When you want to use it, you have to enable it via the
+`MatchLabelKeysInPodAffinity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
+{{< /note >}}
+
+Kubernetes includes an optional `matchLabelKeys` field for Pod affinity
+or anti-affinity. The field specifies keys for the labels that should  match with the incoming Pod's labels, 
+when satisfying the Pod (anti)affinity.
+
+The keys are used to look up values from the pod labels; those key-value labels are combined
+(using `AND`) with the match restrictions defined using the `labelSelector` field. The combined
+filtering selects the set of existing pods that will be taken into Pod (anti)affinity calculation. 
+
+A common use case is to use `matchLabelKeys` with `pod-template-hash` (set on Pods
+managed as part of a Deployment, where the value is unique for each revision).
+Using `pod-template-hash` in `matchLabelKeys` allows you to target the Pods that belong
+to the same revision as the incoming Pod, so that a rolling upgrade won't break affinity.
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: application-server
+...
+spec:
+  template:
+    affinity:
+      podAffinity:
+        requiredDuringSchedulingIgnoredDuringExecution:
+        - labelSelector:
+            matchExpressions:
+            - key: app
+              operator: In
+              values:
+              - database
+          topologyKey: topology.kubernetes.io/zone
+          # Only Pods from a given rollout are taken into consideration when calculating pod affinity.
+          # If you update the Deployment, the replacement Pods follow their own affinity rules
+          # (if there are any defined in the new Pod template)
+          matchLabelKeys: 
+          - pod-template-hash
+```
+
+#### mismatchLabelKeys
+
+{{< feature-state for_k8s_version="v1.29" state="alpha" >}}
+
+{{< note >}}
+<!-- UPDATE THIS WHEN PROMOTING TO BETA -->
+The `mismatchLabelKeys` field is a alpha-level field and is disabled by default in
+Kubernetes {{< skew currentVersion >}}.
+When you want to use it, you have to enable it via the
+`MatchLabelKeysInPodAffinity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
+{{< /note >}}
+
+Kubernetes includes an optional `mismatchLabelKeys` field for Pod affinity
+or anti-affinity. The field specifies keys for the labels that should **not** match with the incoming Pod's labels, 
+when satisfying the Pod (anti)affinity.
+
+One example use case is to ensure Pods go to the topology domain (node, zone, etc) where only Pods from the same tenant or team are scheduled in.
+In other words, you want to avoid running Pods from two different tenants on the same topology domain at the same time.
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+  labels:
+    # Assume that all relevant Pods have a "tenant" label set
+    tenant: tenant-a
+...
+spec:
+  affinity:
+    podAffinity:      
+      requiredDuringSchedulingIgnoredDuringExecution:
+      # ensure that pods associated with this tenant land on the correct node pool
+      - matchLabelKeys:
+          - tenant
+        topologyKey: node-pool
+    podAntiAffinity:  
+      requiredDuringSchedulingIgnoredDuringExecution:
+      # ensure that pods associated with this tenant can't schedule to nodes used for another tenant
+      - mismatchLabelKeys:
+        - tenant # whatever the value of the "tenant" label for this Pod, prevent  
+                 # scheduling to nodes in any pool where any Pod from a different
+                 # tenant is running.
+        labelSelector:
+          # We have to have the labelSelector which selects only Pods with the tenant label,
+          # otherwise this Pod would hate Pods from daemonsets as well, for example, 
+          # which aren't supposed to have the tenant label.
+          matchExpressions:
+          - key: tenant
+            operator: Exists
+        topologyKey: node-pool
+```
+
 #### More practical use-cases
 
 Inter-pod affinity and anti-affinity can be even more useful when they are used with higher

content/en/docs/concepts/scheduling-eviction/dynamic-resource-allocation.md

@@ -162,6 +162,17 @@ gets scheduled onto one node and then cannot run there, which is bad because
 such a pending Pod also blocks all other resources like RAM or CPU that were
 set aside for it.
 
+{{< note >}}
+
+Scheduling of pods which use ResourceClaims is going to be slower because of
+the additional communication that is required. Beware that this may also impact
+pods that don't use ResourceClaims because only one pod at a time gets
+scheduled, blocking API calls are made while handling a pod with
+ResourceClaims, and thus scheduling the next pod gets delayed.
+
+{{< /note >}}
+
+
 ## Monitoring resources
 
 The kubelet provides a gRPC service to enable discovery of dynamic resources of