Change semantics of kubelet resources reservation

[VannTen]

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
This is extracted from #10714, with only the parts related to Node Allocatable.
This clarify the API change the API for users which happened in #10643 (see #12438):
Dinstinguish between:

1. reserving resources for kube / system daemon -> this impacts the calculation of Allocatable
2. enforcing those limits -> this translate the limits in cgroups terms on the node

The current variables (kube_reserved / system_reserved) were used, before #10643, to enable 1 and activate 2.
#10643 decoupled 1 and 2, but those variable and the documentation are still quite confusing. This PR align instead the variables with upstream Kubernetes terminology (enforce_node_allocatable_*).

Which issue(s) this PR fixes:

Fixes #12438

Special notes for your reviewer:
@rptaylor would appreciate your thoughts on this
@tico88612 I wonder if we should revert the change in #10643 which set resources unconditionnaly, and reapply them on master only, thoughts ?

Does this PR introduce a user-facing change?:

action-required
`kube_reserved` is renamed to `enforce_allocatable_kube_reserved`
`system_reserved` is renamed to `enforce_allocatable_system_reserved`

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: VannTen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [VannTen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rptaylor]

typo

[rptaylor]

"Enforcing" a reservation doesn't seem entirely clear to me.
If I recall correctly, the point is that the reservation should always be in effect as a matter of best practice, and it happens via scheduling considerations.
You can optionally also run the components in cgroups to ensure they don't exceed the amount of reserved resources.
In short I think it is clearer to say "enforce limits" rather than "enforce reservations", because the cgroup limits are a separate matter from the reservations which should always apply.
Basically similar to the idea of a k8s request vs limit.

[VannTen]

Yes, the wording is a bit poor you're right I'll fix that.

Yes, the various *ReservedSettings by themselves only have an effect because kubelet use them to calculate Allocatable (which is used by kube-scheduler).
To make them limiting (with cgroups) you need the corresponding ReservedCgroups + including them in enforceNodeAllocatable (which, btw, is a bit poorly worded IMO 😆 )

Maybe "You can optionally make the kubelet enforce the reservations as cgroups limits" ? 🤔

[rptaylor]

Sure sounds good, maybe something like:
"In addition, you can optionally make the kubelet enforce the reservations as cgroup limits, but beware that this can cause system instability if the limits are not set appropriately, because the system daemons will be terminated if their resource usage exceeds the reserved amount. (link to docs)"

[rptaylor]

Background: I believe there was a misunderstanding in the implementation of #9209 that lead to some confusion. I tried to fix this in #11367 , adding some clarifying comments but without breaking changes. This is also discussed in #9692

I took a quick look in this PR but didn't check all the details and logic. But @VannTen in my opinion the important thing is to understand the concepts and have clear terminology , then I am confident you will get the details right :) Hopefully the above background info helps clarify if needed.

[VannTen]

Yeah I read those comments some months ago, when doing #10714 (unfortunately the remaining bits of that needs a newer version of community.general, so we get rid of the weird cgroups variable).

[rptaylor]

Suggestion:

"Node Allocatable" is the amount of resources on a node that can be used by pods. It is calculated by subtracting the following quantities from the physically available resources of a node:

[rptaylor]

"when resource usage exceeds the Allocatable amount"

[tico88612]

@VannTen What is mean set resources unconditionnaly? I wasn't involved in this PR (might need some time to read through it), as I had just joined 1-2 months ago at that time. 😆

[VannTen]

Basically before #10643, kube_reserved and system_reserved had two effects:

1. Use the per resources *_kube/sytem_reserved reservation variables.-> which deduces them from Allocatable, aka, the resources available to schedule pods (without the main switch kube/system_reserved, those variables had no effect.
2. enable the enforcement of the reservation variables as cgroup limits (by adding them to kubelet setting enforceNodeAllocatable and specifying the cgroups on which limits should be applied.

#10643 removed 1., instead reserving the resources unconditionally based on kubespray variables.
Thus, for a cluster using system_reserved: false, for instance, with keeping the default value, the amount of resources in Allocatable would decrease (since the reservations are done when they weren't before).