Skip to content

EAI-8208 Give the JVM the memory limit of its container - #814

Merged
pre merged 1 commit into
mainfrom
EAI-8208-jvm-maxram
Aug 22, 2026
Merged

EAI-8208 Give the JVM the memory limit of its container#814
pre merged 1 commit into
mainfrom
EAI-8208-jvm-maxram

Conversation

@pre

@pre pre commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Related:

What

Keycloak gets the memory limit of its own container through -XX:MaxRAM, and
the value comes from the Downward API, so the limit stays in one place.

Why

On Ubuntu 26.04 the Keycloak pod is OOMKilled again and again at its 2Gi
limit. The crash loop blocks airm-configure, which never creates the secret
airm-rabbitmq-common-vhost-user, so airm-agent and airm-api fail as well.
One JVM stops four workloads. The same chart, the same limits and the same image
are healthy on Ubuntu 24.04.

Ubuntu 26.04 has kernel 7.0, built without CONFIG_MEMCG_V1 and
CONFIG_CPUSETS_V1. That kernel does not list the memory and cpuset
controllers in /proc/cgroups. JDK 21 reads that file to decide whether it runs
in a container. It finds no entry, decides it is not containerized, and sizes the
heap from the memory of the host.

With one pod manifest, limits.memory: 2Gi, and the container reading
/sys/fs/cgroup/memory.max = 2147483648 on both releases:

Release Kernel JVM max heap
24.04.4 6.8.0-124-generic 512.00M
26.04 7.0.0-27-generic 29.50G

java -Xlog:os+container=trace -version on 26.04 gives the reason:

[debug][os,container] controller cpu is not enabled
[debug][os,container] controller memory is not enabled
[debug][os,container] One or more required controllers disabled at kernel level.

Why this option and not another

Measured in the Keycloak image with JDK 21.0.6, at a 2Gi limit:

Options Max heap Correct
none 29.50G no
-XX:MaxRAMPercentage=70 176.19G no, the percentage applies to the host
-Xmx1g 1.00G works, but stays at the value written
-XX:MaxRAM=2147483648 512.00M yes, and the JVM sizing rules work again

-Xmx writes the heap size in a second place, so it does not follow a later
change of limits.memory. -XX:MaxRAM from the Downward API follows it.

JAVA_OPTS_APPEND and not JAVA_OPTS, because the image sets its own defaults
in JAVA_OPTS.

Why not a newer image

No released Keycloak image carries a fixed JDK yet. Measured on a 26.04 node:

Image JDK Max heap at a 2Gi limit
quay.io/keycloak/keycloak:26.0.0 (in use) 21.0.6 29.50G
quay.io/keycloak/keycloak:26.0.8 21.0.6 29.50G
quay.io/keycloak/keycloak:26.2 21.0.7 29.50G
quay.io/keycloak/keycloak:26.4 (newest) 21.0.9 29.50G
eclipse-temurin:21-jre 21.0.11 512.00M
eclipse-temurin:25-jre 25.0.3 512.00M

Remove the two entries when the image carries JDK 21.0.11 or later. Check with:

kubectl -n keycloak exec deploy/keycloak -- java -XshowSettings:vm -version 2>&1 | grep -i "heap size"

Test

Found while testing cluster-bloom PR 306 (EAI-8203) with cluster-forge v2.2.2 on
two two-node clusters, one per Ubuntu release.

The change was applied to the running deployment on the 26.04 cluster:

  • The rollout finished and the pod is 1/1 Running with 0 restarts.
  • The heap in the pod is 512.00M, the same value that 24.04 produces without any
    option, so the change does nothing on 24.04 and needs no release condition.
  • airm-configure completed, the secret airm-rabbitmq-common-vhost-user
    appeared, and airm-agent and airm-api started. The 26.04 cluster then had
    the same 53 of 54 ArgoCD applications Synced and Healthy as the 24.04 cluster.

The stack holds one JVM, so this is the only workload that needs the change
today:

ps -eo comm --no-headers | grep -cx java   # 1 on the whole cluster

Jira: https://amd.atlassian.net/browse/EAI-8208

Ubuntu 26.04 has a kernel built without CONFIG_MEMCG_V1 and CONFIG_CPUSETS_V1,
so it does not list the memory and cpuset controllers in /proc/cgroups. JDK 21
reads that file to know whether it runs in a container. It finds no entry,
decides it is not containerized, and sizes the heap from the memory of the
host. Keycloak then passes its 2Gi limit and the kernel kills it. The crash
loop blocks airm-configure, which never creates the secret
airm-rabbitmq-common-vhost-user, so airm-agent and airm-api fail as well.

MaxRAM tells the JVM how much memory to assume, and the Downward API reads the
value from the limit of this container, so the limit stays in one place.

The heap becomes 512.00M on Ubuntu 26.04, which is the value that Ubuntu 24.04
already produces without any option, so the change does nothing there.
@pre
pre requested a review from a team as a code owner August 21, 2026 06:03
@pre
pre merged commit 62de809 into main Aug 22, 2026
7 checks passed
@pre
pre deleted the EAI-8208-jvm-maxram branch August 22, 2026 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants