feat: Replace credentials-secret with db/broker connection(s) #743

adwk67 · 2026-02-05T13:50:11Z

Description

Fixes #64
Decision: https://github.com/stackabletech/decisions/issues/72

Notes for reviewers

This is large PR although most of the files that have been changed are due to tests and docs: the main changes are in database.rs, queue.rs and connection.rs.
We don't want the operator to have to read the secrets, and so the connection code (which will eventually be moved to the framework) returns a connection string containing embedded variables.
These can be resolved in one of the following ways:
- by shell variable substitution (if the variables are presented as normal env-vars e.g. $FOO) by processes started from the shell
- by config-utils templating: this assumes that the variable is presented in the form of ${env:...} and has been written to a config file
- N.B. templating of env-vars themselves was considered but rejected as (re-)writing env-vars from Rust using the env library is not safe in a multithreaded environment
The operator is responsible for passing env-var names in the form that matches one of the above.
Some connections contain a driver-dependent prefix, depending on the context. The operator makes the decision which type of connection string to retrieve (e.g. postgresql/sqlalchemyuri or postgresql/celery).
Queues/brokers could be considered as being a subset of backends but have been kept separate as they were not specifically covered by ADR 29.

Jenkins 🟢 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/75/
Openshift 🟢 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/76/

Important

The logging test has been deactivated temporarily while testing this PR. Revert the setting before final approval/merging.

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

docs/modules/airflow/examples/getting_started/code/airflow.yaml

maltesander

First batch of comments.

docs/modules/airflow/pages/usage-guide/db-connect.adoc

rust/operator-binary/src/connections/database.rs

maltesander · 2026-02-09T17:25:30Z

rust/operator-binary/src/connections/database.rs

+pub struct GenericDb {
+    pub uri_secret: String,
+}


The generic connection offers (generic) per product options (with more than a connection string) in the ADR29?
https://docs.stackable.tech/home/stable/contributor/adr/adr029-database-connection/#_product_specific_manifests

In the decision we kind of decided to leave the generic option as simple as possible (at least for version 1), so I went with the single field we had in the proposal. Do you think that is OK? For the airflow impl, we don't need the other fields (driver, secret etc.), at least not until we have implemented templating for the generic option.

Ah did not see that one. Im fine with it, the question is only if we move that to op-rs, we could make it generic already if we see use cases for it. But referencing a secret and connection string is probably sufficient for most cases.

Wouldn't we need templating to be able to do something useful with the secret?

Templating where? Could you elaborate?

What I mean is: what can the "framework" code usefully do with the secret, other than supply its name to the operator? - it could read the secret (which we want to avoid), but other than that the secret has to be mounted (should be done in the operator) and the name of the associated env-var included in the connection string ~~(but the name of the env-var should be provided by the operator, since it is doing the mounting)~~ The name can be a constant coming from the framework code, which is only a small change from the current implementation.

But then it is the operator that has to decide how this env-var name should be passed ($FOO or ${env:FOO}).

For me this could be any arbitrary struct (just a secret reference or more fields). Even the secret referenced can have different entries that only the operator then knows about? It can mount it to pods etc, no need to read that?

rust/operator-binary/src/connections/database.rs

rust/operator-binary/src/connections/queue.rs

tests/templates/kuttl/versioning/30-install-airflow-cluster.yaml.j2

maltesander · 2026-02-10T07:16:17Z

tests/test-definition.yaml

+  # TODO revert this before merging!
+  # - name: logging
+  #   dimensions:
+  #     - airflow
+  #     - openshift
+  #     - executor


CHANGELOG.md

docs/modules/airflow/pages/getting_started/first_steps.adoc

maltesander · 2026-02-10T07:44:02Z

docs/modules/airflow/pages/usage-guide/db-connect.adoc

+        uriSecret: redis-celery # <3>
+----
+
+<1> A reference to a secret which must contain the single fields `uri` e.g.


I feel URI is very generic. I think connectionString is a better description?

No strong feelings - I just went with what we had in the decision.

Same, @sbernauer any opinions?

Co-authored-by: Sebastian Bernauer <[email protected]>

Co-authored-by: Malte Sander <[email protected]>

…cluster.yaml.j2 Co-authored-by: Malte Sander <[email protected]>

…uster.yaml.j2 Co-authored-by: Malte Sander <[email protected]>

…ml.j2 Co-authored-by: Malte Sander <[email protected]>

…r.yaml.j2 Co-authored-by: Malte Sander <[email protected]>

maltesander

Some more doc reviews. Tests look good.

docs/modules/airflow/pages/getting_started/first_steps.adoc

docs/modules/airflow/pages/usage-guide/db-connect.adoc

tests/templates/kuttl/triggerer/30-install-airflow-cluster.yaml.j2

Co-authored-by: Malte Sander <[email protected]>

…l.j2 Co-authored-by: Malte Sander <[email protected]>

rust/operator-binary/src/connections/queue.rs

rust/operator-binary/src/connections/database.rs

adwk67 · 2026-02-11T14:26:14Z

Re-ran tests after latest changes 🟢 https://testing.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/airflow-operator-it-custom/77/

adwk67 added 9 commits January 26, 2026 15:31

wip

11da381

added initial tests

032bf2f

Merge branch 'main' into feat/split-credentials-secret

e053806

adapted tests to use new structs

06c6151

Merge branch 'main' into feat/split-credentials-secret

685f66b

added some comments

d543b23

added test for generic URIs

9ea139c

typo

d210148

code style

50a2c5c

adwk67 mentioned this pull request Feb 6, 2026

tracking: Remove (non-docs) examples folders stackabletech/issues#823

Open

13 tasks

adwk67 added 3 commits February 6, 2026 12:56

changelog and doc updates

751fb68

apply cluster secrets separately and remove assert on generations

69ce878

docs

965176e

adwk67 self-assigned this Feb 6, 2026

Merge branch 'main' into feat/split-credentials-secret

173b6a2

adwk67 marked this pull request as ready for review February 9, 2026 07:52

adwk67 moved this to Development: Waiting for Review in Stackable Engineering Feb 9, 2026

adwk67 added this to Stackable Engineering Feb 9, 2026

maltesander self-requested a review February 9, 2026 16:44

maltesander moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Feb 9, 2026

sbernauer reviewed Feb 10, 2026

View reviewed changes

maltesander requested changes Feb 10, 2026

View reviewed changes

adwk67 and others added 8 commits February 10, 2026 09:06

Update docs/modules/airflow/examples/getting_started/code/airflow.yaml

93e3a8e

Co-authored-by: Sebastian Bernauer <[email protected]>

Update docs/modules/airflow/examples/getting_started/code/airflow.yaml

3c4e66d

Co-authored-by: Sebastian Bernauer <[email protected]>

Update docs/modules/airflow/examples/getting_started/code/airflow.yaml

07cca3d

Co-authored-by: Sebastian Bernauer <[email protected]>

Update docs/modules/airflow/examples/getting_started/code/airflow.yaml

f588994

Co-authored-by: Sebastian Bernauer <[email protected]>

Update docs/modules/airflow/pages/usage-guide/db-connect.adoc

784d5c8

Co-authored-by: Malte Sander <[email protected]>

Update rust/operator-binary/src/connections/database.rs

e9fcbf4

Co-authored-by: Malte Sander <[email protected]>

Update CHANGELOG.md

b4847c0

Co-authored-by: Malte Sander <[email protected]>

Update docs/modules/airflow/pages/getting_started/first_steps.adoc

ce50743

Co-authored-by: Malte Sander <[email protected]>

adwk67 and others added 8 commits February 10, 2026 09:10

Update rust/operator-binary/src/connections/queue.rs

f690097

Co-authored-by: Malte Sander <[email protected]>

Update tests/templates/kuttl/ldap/60-install-airflow-cluster.yaml.j2

652a1bb

Co-authored-by: Malte Sander <[email protected]>

Update tests/templates/kuttl/logging/41-install-airflow-cluster.yaml.j2

cf5ab70

Co-authored-by: Malte Sander <[email protected]>

Update tests/templates/kuttl/mount-dags-configmap/30-install-airflow-…

fe3ac91

…cluster.yaml.j2 Co-authored-by: Malte Sander <[email protected]>

Update tests/templates/kuttl/mount-dags-gitsync/30-install-airflow-cl…

20f12e9

…uster.yaml.j2 Co-authored-by: Malte Sander <[email protected]>

Update tests/templates/kuttl/versioning/30-install-airflow-cluster.ya…

322a7c5

…ml.j2 Co-authored-by: Malte Sander <[email protected]>

Update tests/templates/kuttl/remote-logging/40-install-airflow-cluste…

dbe5c94

…r.yaml.j2 Co-authored-by: Malte Sander <[email protected]>

consistent naming of credentials references, removed unused secret

1c04edc

sbernauer self-requested a review February 10, 2026 08:46

adwk67 added 4 commits February 10, 2026 10:09

remove suffixes and renaming

6580b9c

rework env-var maps

5f916a6

removed unused result

3462855

remove unused error

b6cf1b2

maltesander reviewed Feb 10, 2026

View reviewed changes

adwk67 and others added 4 commits February 10, 2026 16:57

Apply suggestions from code review

6e94e7b

Co-authored-by: Malte Sander <[email protected]>

Update tests/templates/kuttl/triggerer/30-install-airflow-cluster.yam…

8f6f461

…l.j2 Co-authored-by: Malte Sander <[email protected]>

review feedback: docs

6430271

make redis DB nr. configurable

6c626d1

maltesander reviewed Feb 11, 2026

View reviewed changes

rust/operator-binary/src/connections/queue.rs Show resolved Hide resolved

rust/operator-binary/src/connections/database.rs Outdated Show resolved Hide resolved

change databaseName to database

a462a97

Uh oh!

feat: Replace credentials-secret with db/broker connection(s) #743

Are you sure you want to change the base?

feat: Replace credentials-secret with db/broker connection(s) #743

Uh oh!

Conversation

adwk67 commented Feb 5, 2026 • edited by maltesander Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Notes for reviewers

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adwk67 Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adwk67 Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adwk67 Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maltesander left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adwk67 commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

adwk67 commented Feb 5, 2026 •

edited by maltesander

Loading

adwk67 Feb 10, 2026 •

edited

Loading

adwk67 Feb 11, 2026 •

edited

Loading

adwk67 Feb 11, 2026 •

edited

Loading