Add log info when EC2NodeClass contains a Bottlerocket attribute which is not supported by Karpenter

[youwalther65]

Description

What problem are you trying to solve?

Karpenter supports only a subset of available K8s related Bottlerocket API attributes using a struct BottlerocketKubernetes as defined in pkg/providers/amifamily/bootstrap/bottlerocketsettings.go.

Currently if the EC2NodeClass contains an attribute in Bottlerocket userData which is not supported it will be silently ignored without any log info in Karpenter logs, see Karpenter docs here.

Example:
Bottlerocket supports settings.kubernetes.cpu-manager-policy and settings.kubernetes.cpu-manager-policy-options but using it in a EC2NodeClass, attribute cpu-manager-policy-options will be silently ignored, just check control container with apiclient get settings.kubernetes:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
...
spec:
  amiFamily: Bottlerocket
...
  userData: |
    [settings.kubernetes]
    cpu-manager-policy = ["static"]
    cpu-manager-policy-options = ["full-pcpus-only"]
...

Karpenter uses the go-toml v2 which supports a strict mode to throw a toml.StrictMissingError error.
I developed a small sample Go program in Go Playground here to show a possible implementation.

A stricter approach would not just log the issue, but rather lead to a failed EC2NodeClass validation by returning type: ValidationSucceeded with status: "False".

Note: Karpenter team is open and supports Pull Requests to enhance the Karpenter BottlerocketKubernetes struct, see example PR Enabled pass-through of kubelet log level for debugging purposes #8635.

How important is this feature to you?
This is misleading to users thinking that Karpenter supports all possible Bottlerocket API values.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ryan-mist]

Does logging this error and possibly adding an event on EC2NodeClass give enough observability? I'm not sure if throwing an error here and having the NodeClass go not ready is the best course since it is a breaking change.

[youwalther65]

@ryan-mist thank you.

It is a breaking change only to a certain degree, because it just extends an already existing but not yet documented behaviour of go-toml v2 in Karpenter use case:

If you specify a supported attribute from BottlerocketKubernetes but supply a wrong data type, it will already mark the EC2NodeClass as Reday = Unknown.

Example:
Use string instead of int for log-level

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
...
  userData: |
    [settings.kubernetes]
    log-level = "2"
...

Corresponding log line:

{"level":"ERROR","time":"2025-12-05T09:19:29.429+0100","logger":"controller","caller":"controller/controller.go:474","message":"Reconciler error","controller":"nodeclass","controllerGroup":"karpenter.k8s.aws","controllerKind":"EC2NodeClass","EC2NodeClass":{"name":"default"},"namespace":"","name":"default","reconcileID":"4ea4843f-45fe-4ed1-8fdf-98f1f8955c06","error":"validating ec2:CreateLaunchTemplate authorization, creating launch template, invalid UserData toml: cannot decode TOML string into struct field bootstrap.BottlerocketKubernetes.VerbosityLevel of type *uint32"}

Probably I can do a PR to update the Karpenter docs describing this behaviour.

See my corresponding PR Use go-toml v2 in strict mode to reject unknown Bottlerocket k8s parameters" #8745 for an implementaion