Skip to content

feat(charts/cozystack): add recommended sysctl and etcd defaults #131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
IvanHunters wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(charts/cozystack): add recommended sysctl and etcd defaults #131

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions charts/cozystack/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,17 @@ machine:
net.ipv4.neigh.default.gc_thresh1: "4096"
net.ipv4.neigh.default.gc_thresh2: "8192"
net.ipv4.neigh.default.gc_thresh3: "16384"
# TCP orphan handling
net.ipv4.tcp_orphan_retries: "3"
net.ipv4.tcp_fin_timeout: "30"
# Network backlog
net.core.netdev_max_backlog: "5000"
net.core.netdev_budget: "600"
net.core.netdev_budget_usecs: "8000"
# TCP keepalive (early detection of dead connections)
net.ipv4.tcp_keepalive_time: "600"
net.ipv4.tcp_keepalive_intvl: "10"
net.ipv4.tcp_keepalive_probes: "6"
kernel:
modules:
- name: openvswitch
Expand Down Expand Up @@ -120,6 +131,9 @@ cluster:
etcd:
advertisedSubnets:
{{- toYaml .Values.advertisedSubnets | nindent 6 }}
extraArgs:
quota-backend-bytes: "8589934592" # 8GiB - prevent etcd running out of space with large LINSTOR CRD datasets
max-request-bytes: "10485760" # 10MiB - allow larger CRD objects to be stored
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Increasing max-request-bytes in etcd is a necessary step for handling large objects, but it is often insufficient on its own. To fully support larger CRD objects (such as large LINSTOR datasets), the Kubernetes API server should also be configured with a matching --max-resource-write-bytes limit in its extraArgs. Without this corresponding change, the API server may reject large requests before they reach etcd, rendering the increased etcd limit ineffective for those operations.

Comment on lines +134 to +136
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🌐 Web query:

Does kube-apiserver in Kubernetes 1.30+ require any flag (e.g., --max-request-bytes) to be coordinated when etcd's --max-request-bytes is raised to 10MiB, or does the apiserver pass requests through transparently up to etcd's limit?

💡 Result:

No, kube-apiserver in Kubernetes 1.30+ does not require any flag like --max-request-bytes to be coordinated when etcd's --max-request-bytes is raised to 10MiB. The apiserver passes etcd requests transparently up to etcd's limit, but enforces its own independent limit on incoming client request body sizes for write operations. kube-apiserver has a hardcoded default MaxRequestBodyBytes of 3MiB (3145728 bytes) for decoding write request bodies (e.g., CREATE/UPDATE/PATCH), which fails requests larger than this with "Request entity too large: limit is 3145728" before reaching etcd. This limit accounts for JSON-to-protobuf conversion expansion when storing in etcd (etcd default 1.5MiB, apiserver allows ~2x). It is not exposed as a command-line flag in standard kube-apiserver; users of the generic apiserver library can configure it programmatically. etcd's --max-request-bytes independently limits gRPC requests from kube-apiserver (e.g., Put,Txn). Raising it to 10MiB allows larger backend requests without apiserver changes, as apiserver does not impose additional byte limits on outgoing etcd calls beyond what fits its object model. apiserver transparently forwards transformed storage requests to etcd. No changes in 1.30+ alter this; official docs list no --max-request-bytes flag for kube-apiserver, and source code confirms the client-side body limit is separate from etcd's server-side limit.

Citations:

Etcd defaults align with upstream guidance — note the kube-apiserver request-size boundary.

Both values are correctly placed inside the eq .MachineType "controlplane" block, so they apply only to control-plane nodes where etcd runs. The values match etcd's documented recommendations:

  • quota-backend-bytes: 8589934592 (8 GiB) is the upstream-suggested maximum (per the etcd "How to debug large db size issue" blog post).
  • max-request-bytes: 10485760 (10 MiB) sits at etcd's documented soft cap; exactly at the 10 MB boundary and safe from WAL-decode failure risk (Restriction on MaxRequestSize and maxWALEntrySize etcd-io/etcd#14114).

Important clarification: kube-apiserver (1.30+) has a hardcoded 3MiB limit on incoming client request bodies for write operations and does not expose this as a configurable flag. This means API clients cannot send requests larger than 3MiB, even though etcd can accept 10MiB. The 10MiB etcd setting still provides value for direct etcd operations or future use cases, but no apiserver flag coordination is needed.

Operational considerations to keep in mind:

  • Memory footprint: etcd (boltDB) mmaps the backend file, so raising quota-backend-bytes from the default 2 GiB to 8 GiB allows the resident-set on each control-plane node to grow up to ~8 GiB just for etcd. Ensure control-plane VMs are sized accordingly.
  • Defragmentation: Larger backends grow fragmentation. Confirm that periodic etcdctl defrag (or equivalent) is in place; otherwise the 8 GiB quota can still be hit by fragmented free pages.
  • Alerting: Recommend adding/keeping alerts on etcd_mvcc_db_total_size_in_bytes and etcd_server_quota_backend_bytes so operators see growth trends well before hitting the new ceiling.
  • LINSTOR CRD objects: The 10MiB etcd setting is sized to handle LINSTOR CRD workloads. Document that this setting exists to support that use case and not as a general invitation to store large blobs in arbitrary CRs (which remain bound by the apiserver's 3MiB client limit).
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@charts/cozystack/templates/_helpers.tpl` around lines 134 - 136, The etcd
extraArgs (quota-backend-bytes and max-request-bytes) are fine but need an
in-template clarification: add a comment next to quota-backend-bytes and
max-request-bytes (the keys under extraArgs) stating these apply only to
control-plane nodes and that kube-apiserver enforces a 3MiB hard client request
body limit (so the 10MiB etcd setting is for direct etcd or special use cases
like LINSTOR CRDs), and mention operational considerations (memory
sizing/defrag/alerts) so operators understand sizing tradeoffs and why these
values were chosen.

{{- end }}
{{- end }}

Expand Down