From de08ba810bf6d558d2d0f1a358264ded03e3379f Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 29 May 2025 16:09:57 +0100 Subject: [PATCH 1/2] DOC-5279 started redesign of secrets pages --- .../data-pipelines/data-pipelines.md | 27 ++++++++++-- .../data-pipelines/secret-providers.md | 40 ++++++++++++++++++ .../{deploy.md => set-secrets.md} | 42 +++++++------------ 3 files changed, 79 insertions(+), 30 deletions(-) create mode 100644 content/integrate/redis-data-integration/data-pipelines/secret-providers.md rename content/integrate/redis-data-integration/data-pipelines/{deploy.md => set-secrets.md} (89%) diff --git a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md index bad74e5dc9..b3b86543a3 100644 --- a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md +++ b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md @@ -416,11 +416,32 @@ find the preparation guides for the databases that RDI supports in the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) section. +## Provide authentication secrets + +You must provide authentication secrets for your source and target databases +before deploying a pipeline. You can supply the secrets to RDI directly +or for K8s deployments, you can also use an external secret provider, such as +[Vault](https://developer.hashicorp.com/vault) or +[AWS Secrets Manager](https://aws.amazon.com/secrets-manager/). + +See [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/set-secrets" >}}) and +[Using an external secret provider]({{< relref "/integrate/redis-data-integration/data-pipelines/secret-providers" >}}) +for more information. + + ## Deploy a pipeline -When your configuration is ready, you must deploy it to start using the pipeline. See -[Deploy a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) -to learn how to do this. +When you have created your configuration, including the [jobs]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#job-files" >}}), you are +ready to deploy. Use [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}}) +to configure and deploy pipelines for both VM and K8s installations. + +For VM installations, you can also use the +[`redis-di deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}}) +command to deploy a pipeline: + +```bash +redis-di deploy --dir +``` ## Pipeline lifecycle diff --git a/content/integrate/redis-data-integration/data-pipelines/secret-providers.md b/content/integrate/redis-data-integration/data-pipelines/secret-providers.md new file mode 100644 index 0000000000..b43883e85c --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/secret-providers.md @@ -0,0 +1,40 @@ +--- +Title: Using an external secret provider +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: | + Configure RDI to obtain authentication secrets for your source and target databases + from an external provider. +group: di +linkTitle: External secret providers +summary: Redis Data Integration keeps Redis in sync with the primary database in near + real time. +type: integration +weight: 3 +--- + +For K8s deployments, you can use an external secret provider, such as +[Vault](https://developer.hashicorp.com/vault) or +[AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) to provide +the authentication secrets for your source and target databases. +See the sections below to learn how to do this. If you prefer to set the secrets for RDI manually, see +[Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/set-secrets" >}}) +for more information. + +## Configure an external provider + +### Vault + +### AWS Secret Manager + +## Secret rotation + +*Secret rotation* is a technique where secrets are changed automatically +by the provider according to a schedule. +RDI versions 1.10.0 and above let you configure the pipeline to +restart the appropriate K8s pods automatically whenever a secret rotates in +the external provider that you have configured. diff --git a/content/integrate/redis-data-integration/data-pipelines/deploy.md b/content/integrate/redis-data-integration/data-pipelines/set-secrets.md similarity index 89% rename from content/integrate/redis-data-integration/data-pipelines/deploy.md rename to content/integrate/redis-data-integration/data-pipelines/set-secrets.md index e3c8e01074..1da4349a27 100644 --- a/content/integrate/redis-data-integration/data-pipelines/deploy.md +++ b/content/integrate/redis-data-integration/data-pipelines/set-secrets.md @@ -1,32 +1,34 @@ --- -Title: Deploy a pipeline -aliases: /integrate/redis-data-integration/ingest/data-pipelines/data-type-handling/ +Title: Set secrets +aliases: +- /integrate/redis-data-integration/ingest/data-pipelines/data-type-handling/ +- /integrate/redis-data-integration/data-pipelines/deploy/ alwaysopen: false categories: - docs - integrate - rs - rdi -description: Learn how to deploy an RDI pipeline +description: Set authentication secrets for your source and target databases. group: di -linkTitle: Deploy +linkTitle: Set secrets summary: Redis Data Integration keeps Redis in sync with the primary database in near real time. type: integration weight: 2 --- -The sections below explain how to deploy a pipeline after you have created the required -[configuration]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}). - -## Set secrets - -Before you deploy your pipeline, you must set the authentication secrets for the +Before you +[deploy]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#deploy-a-pipeline" >}}) +your pipeline, you must set the authentication secrets for the source and target databases. Each secret has a name that you can pass to the [`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}) -command (VM deployment) or the `rdi-secret.sh` script (K8s deployment) to set the secret value. -You can then refer to these secrets in the `config.yaml` file using the syntax "`${SECRET_NAME}`" -(the sample [config.yaml file]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}}) shows these secrets in use). +command (VM deployment) or the `rdi-secret.sh` script (K8s deployment) to set the secret value. +For K8s, you can also configure RDI to obtain the secrets from an +[external provider]({{< relref "/integrate/redis-data-integration/data-pipelines/secret-providers" >}}). + +The `config.yaml` file accesses these secrets with the syntax "`${SECRET_NAME}`" +(the sample [config.yaml file]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}}) shows the secrets in use). The table below lists all valid secret names. Note that the username and password are required for the source and target, but the other @@ -249,17 +251,3 @@ kubectl create secret generic target-db-ssl --namespace=rdi \ ``` Note that the certificate paths contained in the secrets `SOURCE_DB_CACERT`, `SOURCE_DB_CERT`, and `SOURCE_DB_KEY` (for the source database) and `TARGET_DB_CACERT`, `TARGET_DB_CERT`, and `TARGET_DB_KEY` (for the target database) are internal to RDI, so you *must* use the values shown in the example above. You should only change the certificate paths when you create the `source-db-ssl` and `target-db-ssl` secrets. - -## Deploy a pipeline - -When you have created your configuration, including the [jobs]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#job-files" >}}), you are -ready to deploy. Use [Redis Insight]({{< relref "/develop/tools/insight/rdi-connector" >}}) -to configure and deploy pipelines for both VM and K8s installations. - -For VM installations, you can also use the -[`redis-di deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}}) -command to deploy a pipeline: - -```bash -redis-di deploy --dir -``` \ No newline at end of file From e7c32e1a265224571433f693baba93db32dcf778 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 26 Jun 2026 15:06:40 +0100 Subject: [PATCH 2/2] DOC-5279 draft Vault and AWS secret-provider sections Drafted the Vault and AWS sections of secret-providers.md from the secret-provider request-body example in reference/api-reference/openapi.json, because the config-yaml-reference page and the live docs only document the bare skeleton (type plus an objects array whose items have "no properties"). The Vault params (vaultAddress, roleName, and the objectName/secretPath/secretKey object fields) come from that API example, which also carries a placeholder "someField", so the field list is provisional; AWS Secrets Manager has no documented params at all, so its section is just a question block. Open questions are left as visible blockquote TODOs so the RDI team can comment inline on the PR. Learned: concrete secret-provider params came from openapi.json, not the config-yaml-reference page or live docs, which are sparse. Directive: the TODO blockquotes are open questions for the RDI team - do not remove or fill them without their confirmation. Ticket: DOC-5279 Co-Authored-By: Claude Opus 4.8 (1M context) --- .../data-pipelines/secret-providers.md | 106 +++++++++++++++++- 1 file changed, 104 insertions(+), 2 deletions(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/secret-providers.md b/content/integrate/redis-data-integration/data-pipelines/secret-providers.md index e9d55ea130..1cedc29f82 100644 --- a/content/integrate/redis-data-integration/data-pipelines/secret-providers.md +++ b/content/integrate/redis-data-integration/data-pipelines/secret-providers.md @@ -25,11 +25,109 @@ See the sections below to learn how to do this. If you prefer to set the secrets [Set secrets]({{< relref "/integrate/redis-data-integration/data-pipelines/set-secrets" >}}) for more information. +{{< note >}} +This page is a work in progress. The sections marked **TODO** below need +confirmation from the RDI developers before the page is published. Please +review and comment. +{{< /note >}} + ## Configure an external provider -### Vault +You define a secret provider in the `secret-providers` section of your +[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/pipeline-config" >}}) +file. Each provider has: + +- a unique **id** (the key under `secret-providers`) that you use to refer to it +- a **`type`** (`vault` or `aws`) +- a **`parameters`** object that configures the connection to the provider and + lists the secret **`objects`** to fetch from it. + +Each entry in `objects` maps a secret stored in the provider to a name that you +can reference elsewhere in the configuration: + +```yaml +secret-providers: + my-provider: + type: vault # or "aws" + parameters: + # connection parameters (see the sections below) + objects: + - objectName: credentials # the name you reference + secretPath: secret/data/db-pass + secretKey: password +``` + +You then reference a fetched secret using the syntax +`${secret:::}`. For example: + +```yaml +connection: + user: ${secret:my-provider:credentials:user} + password: ${secret:my-provider:credentials:password} +``` + +Note that this differs from the `${SECRET_NAME}` syntax used for +[secrets that you set manually]({{< relref "/integrate/redis-data-integration/data-pipelines/set-secrets" >}}). + +> **TODO (needs dev input):** Confirm where a provider can be defined. The +> `config.yaml` reference lists `secret-providers` as a config section, but the +> API also exposes `PUT /api/v1/pipelines/secret-providers/{name}`. Can +> providers be configured in `config.yaml`, through the API, and/or in Redis +> Insight? + +## Vault + +Set `type: vault` and provide the following connection parameters: -### AWS Secret Manager +| Parameter | Description | +| :-- | :-- | +| `vaultAddress` | URL of the Vault server, for example `http://vault.default:8200`. | +| `roleName` | The Vault role that RDI uses to authenticate. | + +Each entry in `objects` has the following fields: + +| Field | Description | +| :-- | :-- | +| `objectName` | The name you use to reference the secret in the `${secret:...}` syntax. | +| `secretPath` | The path to the secret in Vault, for example `secret/data/db-pass`. | +| `secretKey` | The key within the secret whose value you want to read. | + +For example: + +```yaml +secret-providers: + my-vault: + type: vault + parameters: + vaultAddress: http://vault.default:8200 + roleName: database + objects: + - objectName: credentials + secretPath: secret/data/db-pass + secretKey: password +``` + +> **TODO (needs dev input):** +> +> - Which Vault **authentication method** does RDI use (token, Kubernetes auth, +> AppRole)? How are the credentials or token supplied? The `roleName` +> parameter suggests a role-based method. +> - The API example includes an extra `someField` parameter. What is the +> complete, real list of `parameters` for the Vault provider? +> - Are there any TLS options for the connection to Vault? + +## AWS Secrets Manager + +Set `type: aws`. + +> **TODO (needs dev input):** The `config.yaml` reference and the API schema +> currently document only `type: aws` with no parameters, so this section needs +> the full parameter set from the RDI developers, including: +> +> - **Region** and any endpoint configuration. +> - **Authentication** — IAM role, IRSA, or static access keys? +> - The **`objects`** mapping fields — the AWS equivalents of `objectName`, the +> secret name or ARN, and the key or JSON field to read. ## Secret rotation @@ -38,3 +136,7 @@ by the provider according to a schedule. RDI versions 1.10.0 and above let you configure the pipeline to restart the appropriate K8s pods automatically whenever a secret rotates in the external provider that you have configured. + +> **TODO (needs dev input):** Document how to enable the automatic pod restart — +> which `config.yaml` setting controls it, and whether it has any per-provider +> requirements.