feat: support distroless images in version probe

[ashishch432]

Why

The operator generates probe jobs using below which will fail with distroless images.

sh -c "<binary> --version > /dev/termination-log 2>&1"

What

Replace the command with:

/usr/bin/clickhouse local --query "INSERT INTO FUNCTION file('/dev/termination-log', 'RawBLOB', 'version String') SELECT version()"

• Use /usr/bin/clickhouse for both ClickHouseCluster and KeeperCluster probes.
  The server and keeper images use ClickHouse’s multicall binary model, and clickhouse local is available through the main binary.

• Use INSERT INTO FUNCTION file(...) instead of INTO OUTFILE.
  Manual Kubernetes validation showed INTO OUTFILE '/dev/termination-log' fails due to ClickHouse temp-file/rename behavior against the Kubernetes termination log path, while file('/dev/termination-log', ...) writes successfully.

Related Issues

We are planning to switch to distroless images once ClickHouse/ClickHouse#105677 is fixed.

[ashishch432]

Hi @GrigoryPervakov ,

I opened this draft PR to make version probe Jobs compatible with shell-free/distroless ClickHouse images.

The initial plan was straightforward: replace the current shell-dependent command:

sh -c "<binary> --version > /dev/termination-log 2>&1"

with a shell-free ClickHouse-native command:

/usr/bin/clickhouse local --query \
  "INSERT INTO FUNCTION file('/dev/termination-log', 'RawBLOB', 'version String') SELECT version()"

That works for the official distroless server and keeper images I tested, and this is what the draft PR intitially implemented (current version has the 2 path implementation). While ClickHouse/ClickHouse#105677 means the current published distroless images are not fully shell-free yet, this command should continue to work once that image issue is fixed.

However, the assumption that standard images have a superset of distroless tooling turned out to be backwards here. This command breaks on standard clickhouse-keeper images: they expose /usr/bin/clickhouse-keeper, but not necessarily /usr/bin/clickhouse.

Based on ClickHouse/ClickHouse#98664, ClickHouse/ClickHouse@739b60df502, and ClickHouse/ClickHouse@a05ed90e43c, this appears intentional. Distroless Keeper includes clickhouse-common-static and /usr/bin/clickhouse; standard Keeper uses the standalone Keeper package.

I also looked for a more universal ClickHouse-native way to write the version directly to /dev/termination-log. The closest option I found is still clickhouse local with the file() table function. --version itself only writes to stdout, and I did not find a supported --version-output-path style flag. INTO OUTFILE was also tested, but failed around /dev/termination-log file handling.

Alternatives

Option	Shape	Pros	Tradeoff
Read probe output from pod logs	Run clickhouse-server --version / clickhouse-keeper --version directly, then parse bounded logs from the completed version-probe container. Keep termination-message parsing as a backward-compatible fallback for old Jobs.	Same logic for standard and distroless images; no image flavor detection.	Requires get on pods/log; read-only, but security-sensitive with the manager ClusterRole.
Keep termination-log flow and select by image flavor	Use the current shell command for standard images and the clickhouse local ... file('/dev/termination-log') command for distroless images.	Smaller change; no new RBAC.	Needs either flavor detection, a user-provided flavor, or command/args override support.

For the second option, I see a few possible variants:

• Auto-detect flavor from the image name/tag, for example distroless.
  This is convenient, but brittle for mirrored images or digest-pinned images.
• Add an explicit user input such as imageFlavor: Standard | Distroless | Auto.
  This is clearer, but requires an API/CRD change.
• Expose version probe command/args overrides and document the distroless command for users.
  This is a useful, but it would not fix official distroless images by default unless users configure it.

My preference is the pod-log option if pods/log access is acceptable. It gives us a cleaner default behavior and avoids both shell dependence and image-flavor guessing.