Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 30 additions & 6 deletions content/en/docs/concepts/workloads/pods/pod-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -958,17 +958,41 @@ phase. Also, PodGC adds a Pod disruption condition when cleaning up an orphan Po
See [Pod disruption conditions](/docs/concepts/workloads/pods/disruptions#pod-disruption-conditions)
for more details.

### Pod status during kubelet restart {#kubelet-restart-pod-status}
## Pod behavior during kubelet restarts {#kubelet-restarts}

{{< feature-state feature_gate_name="ChangeContainerStatusOnKubeletRestart" >}}
If you restart the kubelet, Pods (and their containers) continue to run
even during the restart.
When there are running Pods on a node, stopping or restarting the kubelet
on that node does **not** cause the kubelet to stop all local Pods
before the kubelet itself stops.
To stop the Pods on a node, you can use `kubectl drain`.

If the node's `KubeletReady` condition remains unchanged, meaning its `Ready` condition does not transition from true to false, we consider it a kubelet restart.
### Detection of kubelet restarts
{{< feature-state feature_gate_name="ChangeContainerStatusOnKubeletRestart" >}}
When the kubelet starts, it checks to see if there is already a Node with
bound Pods.
If the Node's [`Ready` condition](/docs/reference/node/node-status/#condition) remains unchanged,
in other words the condition has not transitioned from true to false, Kubernetes detects this a _kubelet restart_.
(It's possible to restart the kubelet in other ways, for example to fix a node bug,
but in these cases, Kubernetes picks the safe option and treats this as if you
stopped the kubelet and then later started it).

When the kubelet restarts, the container statuses are managed differently based on the feature gate setting:

* With the `ChangeContainerStatusOnKubeletRestart` feature gate disabled (the default), the kubelet does not change container statuses after a restart. Containers that were in a Ready state remain Ready.

* With the feature gate enabled, and containers within the pod have a readiness probe, the kubelet will modify the container's Ready status after restart to be not ready. This behavior was the default for a long time and caused issues in large scale deployments. While the feature gate allows reverting to this legacy behavior temporarily, it is recommended to file a bug report if you encounter problems. Note that this feature gate will be removed in the future.
* By default, the kubelet does not change container statuses after a restart.
Containers that were in set to `ready: true` state remain remain ready.

If you stop the kubelet long enough for it to fail a series of
[node heartbeat](/docs/concepts/architecture/leases/#node-heart-beats) checks,
and then you wait before you start the kubelet again, Kubernetes may begin to evict Pods from that Node.
However, even though Pod evictions begin to happen, Kubernetes does not mark the
individual containers in those Pods as `ready: false`. The Pod-level eviction
happens after the control plane taints the node as `node.kubernetes.io/not-ready` (due to the failed heartbeats).

* In Kubernetes {{< skew currentVersion >}} you can opt in to a legacy behavior where the kubelet always modify the containers `ready` value, after a kubelet restart, to be false.
This legacy behavior was the default for a long time, but caused issue for people using Kubernetes, especially in large scale deployments.
Althought the feature gate allows reverting to this legacy behavior temporarily, the Kubernetes project recommends that you file a bug report if you encounter problems.
The `ChangeContainerStatusOnKubeletRestart` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/#ChangeContainerStatusOnKubeletRestart) will be removed in the future.

## {{% heading "whatsnext" %}}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ stages:
defaultValue: false
fromVersion: "1.35"
---
When disabled, kubelet restarts will not change the status of Pods already running on the node,
This Feature Gate was introduced to allow revert the behavior to previously used default.
Enable legacy writes to update container `ready` status after the kubelet detects a
[restart](/docs/concepts/workloads/pods/pod-lifecycle/#kubelet-restarts).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The link is pointing to a section that does not exist on the page.

Suggested change
[restart](/docs/concepts/workloads/pods/pod-lifecycle/#kubelet-restarts).
[restart](/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, but it will exist.


This feature gate was introduced to allow you revert the behavior to a previously used default.
If you are satisfied with the default behavior, you do not need to enable this
feature gate.