Skip to content

DOC-5279 started redesign of secrets pages #1634

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andy-stark-redis wants to merge 3 commits into
base: main
Choose a base branch
from
Open

DOC-5279 started redesign of secrets pages #1634

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
de08ba8
DOC-5279 started redesign of secrets pages
andy-stark-redis May 29, 2025
095cf30
Merge origin/main into DOC-5279-rdi-external-secrets
andy-stark-redis Jun 26, 2026
e7c32e1
DOC-5279 draft Vault and AWS secret-provider sections
andy-stark-redis Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
242 changes: 6 additions & 236 deletions content/integrate/redis-data-integration/data-pipelines/deploy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,241 +16,11 @@ type: integration
weight: 50
---

The sections below explain how to deploy a pipeline after you have created the required
[configuration]({{< relref "/integrate/redis-data-integration/data-pipelines" >}}).

## Set secrets

Before you deploy your pipeline, you must set the authentication secrets for the
source and target databases. Each secret has a name that you can pass to the
[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}})
command (VM deployment) or the `rdi-secret.sh` script (K8s deployment) to set the secret value.
You can then refer to these secrets in the `config.yaml` file using the syntax "`${SECRET_NAME}`"
(the sample
[config.yaml file]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config#example" >}})
shows these secrets in use).

The table below lists all valid secret names. Note that the
username and password are required for the source and target, but the other
secrets are only relevant for TLS/mTLS connections.

| Secret name | Description |
| :-- | :-- |
| `SOURCE_DB_USERNAME` | Username for the source database |
| `SOURCE_DB_PASSWORD` | Password for the source database |
| `SOURCE_DB_CACERT` | (For TLS only) Source database CA certificate |
| `SOURCE_DB_CERT` | (For mTLS only) Source database client certificate |
| `SOURCE_DB_KEY` | (For mTLS only) Source database private key |
| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password |
| `TARGET_DB_USERNAME` | Username for the target database |
| `TARGET_DB_PASSWORD` | Password for the target database |
| `TARGET_DB_CACERT` | (For TLS only) Target database CA certificate |
| `TARGET_DB_CERT` | (For mTLS only) Target database client certificate |
| `TARGET_DB_KEY` | (For mTLS only) Target database private key |
| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password |

{{< note >}}
{{< embed-md "rdi-tls-secrets.md" >}}
{{< /note >}}

### Set secrets for VM deployment

Use [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}})
to set secrets for a VM deployment.

The specific command lines for source secrets are as follows:

```bash
# For username and password
redis-di set-secret SOURCE_DB_USERNAME yourUsername
redis-di set-secret SOURCE_DB_PASSWORD yourPassword

# With source TLS, in addition to the above
redis-di set-secret SOURCE_DB_CACERT /path/to/myca.crt

# With source mTLS, in addition to the above
redis-di set-secret SOURCE_DB_CERT /path/to/myclient.crt
redis-di set-secret SOURCE_DB_KEY /path/to/myclient.key
# Use this only if SOURCE_DB_KEY is password-protected
redis-di set-secret SOURCE_DB_KEY_PASSWORD yourKeyPassword
```

The corresponding command lines for target secrets are:

```bash
# For username and password
redis-di set-secret TARGET_DB_USERNAME yourUsername
redis-di set-secret TARGET_DB_PASSWORD yourPassword

# With target TLS, in addition to the above
redis-di set-secret TARGET_DB_CACERT /path/to/myca.crt

# With target mTLS, in addition to the above
redis-di set-secret TARGET_DB_CERT /path/to/myclient.crt
redis-di set-secret TARGET_DB_KEY /path/to/myclient.key
# Use this only if TARGET_DB_KEY is password-protected
redis-di set-secret TARGET_DB_KEY_PASSWORD yourKeyPassword
```

### Set secrets for K8s/Helm deployment using the rdi-secret.sh script

Use the `rdi-secret.sh` script to set secrets for a K8s/Helm deployment. To use this script, unzip the archive that contains the RDI Helm chart and navigate to the resulting folder. The `rdi-secret.sh` script is located in the `scripts` subfolder. The general pattern for using this script is:

```bash
scripts/rdi-secret.sh set <SECRET-NAME> <SECRET-VALUE>
```

The script also lets you retrieve a specific secret or list all the secrets that have been set:

```bash
# Get specific secret
scripts/rdi-secret.sh get <SECRET-NAME>

# List all secrets
scripts/rdi-secret.sh list
```

The specific command lines for source secrets are as follows:

```bash
# For username and password
scripts/rdi-secret.sh set SOURCE_DB_USERNAME yourUsername
scripts/rdi-secret.sh set SOURCE_DB_PASSWORD yourPassword

# With source TLS, in addition to the above
scripts/rdi-secret.sh set SOURCE_DB_CACERT /path/to/myca.crt

# With source mTLS, in addition to the above
scripts/rdi-secret.sh set SOURCE_DB_CERT /path/to/myclient.crt
scripts/rdi-secret.sh set SOURCE_DB_KEY /path/to/myclient.key
# Use this only if SOURCE_DB_KEY is password-protected
scripts/rdi-secret.sh set SOURCE_DB_KEY_PASSWORD yourKeyPassword
```

The corresponding command lines for target secrets are:

```bash
# For username and password
scripts/rdi-secret.sh set TARGET_DB_USERNAME yourUsername
scripts/rdi-secret.sh set TARGET_DB_PASSWORD yourPassword

# With target TLS, in addition to the above
scripts/rdi-secret.sh set TARGET_DB_CACERT /path/to/myca.crt

# With target mTLS, in addition to the above
scripts/rdi-secret.sh set TARGET_DB_CERT /path/to/myclient.crt
scripts/rdi-secret.sh set TARGET_DB_KEY /path/to/myclient.key
# Use this only if TARGET_DB_KEY is password-protected
scripts/rdi-secret.sh set TARGET_DB_KEY_PASSWORD yourKeyPassword
```

### Set secrets for K8s/Helm deployment using Kubectl command

In some scenarios, you may prefer to use [`kubectl create secret generic`](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_create/kubectl_create_secret_generic/)
to set secrets for a K8s/Helm deployment. The general pattern of the commands is:

```bash
kubectl create secret generic <DB> \
--namespace=rdi \
--from-literal=<SECRET-NAME>=<SECRET-VALUE>
```

Where `<DB>` is either `source-db` for source secrets or `target-db` for target secrets.

If you use TLS or mTLS for either the source or target databases, you also need to create the `source-db-ssl` and/or `target-db-ssl` K8s secrets that contain the certificates used to establish secure connections. The general pattern of the commands is:

```bash
kubectl create secret generic <DB>-ssl \
--namespace=rdi \
--from-file=<FILE-NAME>=<FILE-PATH>
```

The specific command lines for source secrets are as follows:

```bash
# Without source TLS
# Create or update source-db secret
kubectl create secret generic source-db --namespace=rdi \
--from-literal=SOURCE_DB_USERNAME=yourUsername \
--from-literal=SOURCE_DB_PASSWORD=yourPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With source TLS
# Create of update source-db secret
kubectl create secret generic source-db --namespace=rdi \
--from-literal=SOURCE_DB_USERNAME=yourUsername \
--from-literal=SOURCE_DB_PASSWORD=yourPassword \
--from-literal=SOURCE_DB_CACERT=/etc/certificates/source_db/ca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update source-db-ssl secret
kubectl create secret generic source-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With source mTLS
# Create or update source-db secret
kubectl create secret generic source-db --namespace=rdi \
--from-literal=SOURCE_DB_USERNAME=yourUsername \
--from-literal=SOURCE_DB_PASSWORD=yourPassword \
--from-literal=SOURCE_DB_CACERT=/etc/certificates/source_db/ca.crt \
--from-literal=SOURCE_DB_CERT=/etc/certificates/source_db/client.crt \
--from-literal=SOURCE_DB_KEY=/etc/certificates/source_db/client.key \
--from-literal=SOURCE_DB_KEY_PASSWORD=yourKeyPassword \ # add this only if SOURCE_DB_KEY is password-protected
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update source-db-ssl secret
kubectl create secret generic source-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--from-file=client.crt=/path/to/myclient.crt \
--from-file=client.key=/path/to/myclient.key \
--save-config --dry-run=client -o yaml | kubectl apply -f -
```

The corresponding command lines for target secrets are:

```bash
# Without target TLS
# Create or update target-db secret
kubectl create secret generic target-db --namespace=rdi \
--from-literal=TARGET_DB_USERNAME=yourUsername \
--from-literal=TARGET_DB_PASSWORD=yourPassword \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With target TLS
# Create of update target-db secret
kubectl create secret generic target-db --namespace=rdi \
--from-literal=TARGET_DB_USERNAME=yourUsername \
--from-literal=TARGET_DB_PASSWORD=yourPassword \
--from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update target-db-ssl secret
kubectl create secret generic target-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--save-config --dry-run=client -o yaml | kubectl apply -f -

# With target mTLS
# Create or update target-db secret
kubectl create secret generic target-db --namespace=rdi \
--from-literal=TARGET_DB_USERNAME=yourUsername \
--from-literal=TARGET_DB_PASSWORD=yourPassword \
--from-literal=TARGET_DB_CACERT=/etc/certificates/target_db/ca.crt \
--from-literal=TARGET_DB_CERT=/etc/certificates/target_db/client.crt \
--from-literal=TARGET_DB_KEY=/etc/certificates/target_db/client.key \
--from-literal=TARGET_DB_KEY_PASSWORD=yourKeyPassword \ # add this only if TARGET_DB_KEY is password-protected
--save-config --dry-run=client -o yaml | kubectl apply -f -
# Create or update target-db-ssl secret
kubectl create secret generic target-db-ssl --namespace=rdi \
--from-file=ca.crt=/path/to/myca.crt \
--from-file=client.crt=/path/to/myclient.crt \
--from-file=client.key=/path/to/myclient.key \
--save-config --dry-run=client -o yaml | kubectl apply -f -
```

Note that the certificate paths contained in the secrets `SOURCE_DB_CACERT`, `SOURCE_DB_CERT`, and `SOURCE_DB_KEY` (for the source database) and `TARGET_DB_CACERT`, `TARGET_DB_CERT`, and `TARGET_DB_KEY` (for the target database) are internal to RDI, so you *must* use the values shown in the example above. You should only change the certificate paths when you create the `source-db-ssl` and `target-db-ssl` secrets.

## Deploy a pipeline

When you have created your configuration, including the [jobs]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}), you are
ready to deploy. Use [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}})
When you have created your configuration, including the
[jobs]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}),
and [set the authentication secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/set-secrets" >}})
for your source and target databases, you are ready to deploy. Use
[Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}})
to configure and deploy pipelines for both VM and K8s installations.

For VM installations, you can also use the
Expand All @@ -259,4 +29,4 @@ command to deploy a pipeline:

```bash
redis-di deploy --dir <path to pipeline folder>
```
```
6 changes: 3 additions & 3 deletions content/integrate/redis-data-integration/data-pipelines/pipeline-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ if you want to apply custom transformations to the captured data.

Below is an example of a `config.yaml` file. Note that the values of the
form "`${name}`" refer to secrets that you should set as described in
[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}).
[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/set-secrets" >}}).
In particular, you should normally use secrets as shown to set the source
and target username and password rather than storing them in plain text in this file.

Expand Down Expand Up @@ -210,7 +210,7 @@ configuration contains the following data:
to the source database, you may need to specify additional properties in the
`advanced` section with references to the corresponding certificates depending
on the source database type. Note that these properties **must** be references to
secrets that you should set as described in [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}}).
secrets that you should set as described in [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/set-secrets" >}}).
- `databases`: List of all databases to collect data from for source database types
that support multiple databases, such as `mysql` and `mariadb`.
- `schemas`: List of all schemas to collect data from for source database types
Expand Down Expand Up @@ -253,7 +253,7 @@ or [mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS) to connect
to the target database, you must specify the CA certificate (for TLS),
and the client certificate and private key (for mTLS) in `cacert`, `cert`, and `key`.
Note that these certificates **must** be references to secrets
that you should set as described in [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets" >}})
that you should set as described in [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/set-secrets" >}})
(it is not possible to include these certificates as plain text in the file).

{{< note >}}If you specify `localhost` as the address of either the source or target server during
Expand Down
2 changes: 1 addition & 1 deletion content/integrate/redis-data-integration/data-pipelines/prepare-dbs/spanner.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ RDI supports two authentication methods for accessing Spanner:

Before deploying the RDI pipeline, you need to configure the necessary secrets for the target
database. Instructions for setting up the target database secrets are available in the
[RDI deployment guide]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}).
[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/set-secrets#set-secrets-for-k8shelm-deployment-using-kubectl-command" >}}).

**Optional**: If you prefer to use a service account credentials file instead of Workload Identity
authentication, you'll need to create a Spanner-specific secret named `source-db-credentials`.
Expand Down
Loading
Loading