grafana dashboard "ShinyProxy App Resources” not working with latest docker

[pescobar]

We have deployed the shinyproxy operator in a single host with docker backend as documented in Docker | ShinyProxy and we enabled monitoring as documented in Docker | ShinyProxy

We are using docker images openanalytics/shinyproxy-operator:2.3.1 and openanalytics/shinyproxy:3.2.4 and many of the grafana dashboard work fine but dashboard “ShinyProxy App Resources” was not showing any data.

I debugged this issue with the help of an LLM and the problem seems to be that the current setup of the shinyproxy operator only works with the docker overlay2 driver but new docker installations use overlayfs driver. See the notes in OverlayFS storage driver | Docker Docs and Storage drivers | Docker Docs

I have switched to overlay2 driver in my docker daemon by now so workaround the issue and now the grafana dashboards for shinyapps resources work fine.

I thought I would post here the details so the shinyproxy developers are aware of the issue. I paste below more details provided by the LLM

Storage Driver: overlayfs is not the classic Docker overlay2 driver. It’s the marker for the containerd image store (the “containerd snapshotter” feature) that Docker 25+ ships and that is enabled by default in Docker 28’s fresh installs. Confirmation: /var/lib/docker/image/contains only identity-cache.db — no overlay2/ or overlayfs/ subdirectory with a layerdb/. Under the containerd snapshotter, image and layer metadata live under /var/lib/docker/containerd/… (containerd’s content store), not the classic image//layerdb/mounts//mount-idlayout that cAdvisor was written against.

cAdvisor doesn’t support the containerd snapshotter when talking to Docker — it unconditionally goes looking for image//layerdb/mounts//mount-id, that file doesn’t exist, container init fails, and every per-container metric gets dropped. Tracking issues:google/cadvisor#3624 and friends. It’s a Docker‑side feature change, not a cAdvisor version regression.

Two real options

Option A (recommended): disable the containerd snapshotter

This reverts Docker to the classic overlay2 graph driver layout that cAdvisor understands. Edit /etc/docker/daemon.json and either remove the feature flag or set it to false:

{“features”: { “containerd-snapshotter”: false }}

Then:

sudo systemctl restart docker

Important caveat: switching image stores invalidates the current store — you’ll see your image list become empty. Pull the images you need again (docker pull openanalytics/shinyproxy:3.2.3, etc.). Running container instances of those images may keep working until you restartthem, but plan a maintenance window: re-pull, restart ShinyProxy and any apps, and you’re back. After the restart, docker info | grep “Storage Driver” should read overlay2, and /var/lib/docker/image/overlay2/layerdb/ will exist. The cAdvisor errors will stop and the twodashboards will populate within one scrape interval.

Option B: keep the containerd snapshotter, replace cAdvisor’s data source

cAdvisor can monitor containerd directly via --containerd=/run/containerd/containerd.sock --containerd-namespace=moby (Docker uses the moby namespace) instead of Docker. That avoids the broken layerdb lookup. But the operator hardcodes the cAdvisor flags in CAdvisorConfig.kt:91,so you’d need a wrapper image:

FROM gcr.io/cadvisor/cadvisor:v0.49.1ENTRYPOINT [“/usr/bin/cadvisor”, “–containerd=/run/containerd/containerd.sock”, “–containerd-namespace=moby”]

…also bind‑mount /run/containerd into the cAdvisor container — which the operator doesn’t do, so you’d additionally have to either patch the operator or run cAdvisor outside the operator (SPO_ENABLE_MONITORING=false and bring your own shinyproxy-monitoring stack). This is morework and the container-label translation under the containerd path is less battle-tested than the Docker path.

My recommendation

Go with A. One config change, one daemon restart, one round of docker pull, and you’re using the same setup the operator’s monitoring was designed and tested against. The containerd snapshotter is forward-looking but the surrounding tooling (cAdvisor in particular) hasn’t caughtup.

I have also posted this info in https://support.openanalytics.eu/t/grafana-resources-dashboards-not-showing-any-data/3222

[pescobar]

just to clarify because I realized that the title of the issue is not totally accurate.

with this docker driver it works:

docker info 2>/dev/null | grep -E "Storage Driver|Docker Root Dir"
 Storage Driver: overlay2
 Docker Root Dir: /var/lib/docker

with this docker driver it doesn't work

docker info 2>/dev/null | grep -E "Storage Driver|Docker Root Dir"
   Storage Driver: overlayfs
   Docker Root Dir: /var/lib/docker

if you do a clean installation of latest docker you will get the non working driver overlayfs

but if you installed and older docker version and then upgraded to latest docker you will still use the working driver overlayfs2

the issue only affects new installations of latest docker version.