DOC-6252 sections about failover behaviour when all endpoints are unhealthy

content/develop/clients/failover.md

@@ -32,6 +32,7 @@ your client library to learn how to configure it for failover and failback:
 
 - [Jedis]({{< relref "/develop/clients/jedis/failover" >}})
 - [redis-py]({{< relref "/develop/clients/redis-py/failover" >}}) (preview)
+- [Lettuce]({{< relref "/develop/clients/lettuce/failover" >}}) (preview)
 
 ## Concepts
 
@@ -93,10 +94,20 @@ still better than the current failover target, so it might be worth
 failing back to that server even if it is not optimal.
 
 Clients periodically run a "health check" on each server to see if it has recovered.
-The health check can be as simple as sending a Redis
-[`PING`]({{< relref "/commands/ping" >}}) or
-[ECHO]({{< relref "/commands/echo" >}}) command and ensuring that it gives the
-expected response.
+Several health check strategies are implemented in all clients:
+
+-   **Ping**: This is the default strategy, which just sends a
+    [`PING`]({{< relref "/commands/ping" >}}) command and ensures that it gives the
+    expected response.
+-   **Lag aware** (Redis Software only): This strategy uses the
+    [REST API]({{< relref "/operate/rs/references/rest-api" >}}) to check the
+    synchronization lag between a specific database and the others in the Active-Active
+    setup. If the lag is within a specified tolerance, the server is considered healthy.
+-   **Custom**: You can implement your own health check strategy to use information
+    or take action that is specific to your application.
+
+See the documentation for your client library for more information on how to
+configure health checks.
 
 You can also configure the client to run health checks on the current target
 server during periods of inactivity, even if no failover has occurred. This can

content/develop/clients/jedis/failover.md

@@ -172,6 +172,18 @@ The `MultiDbConfig.RetryConfig` builder has the following options to configure r
 | `includedExceptionList()` | See description | `List` of `Throwable` classes that should be considered as failures to be retried. By default, it includes just `JedisConnectionException`. |
 | `ignoreExceptionList()` | `null` | `List` of `Throwable` classes that should be ignored for retry. |
 
+### General failover configuration
+
+The `MultiDbConfig` builder also has the following options to configure general failover behavior:
+
+| Builder method | Default value | Description|
+| --- | --- | --- |
+| `maxNumFailoverAttempts()` | `10` | Number of attempts to fail over to a new endpoint before giving up. |
+| `delayInBetweenFailoverAttempts()` | `12000` | Time interval in milliseconds between successive failover attempts. |
+| `gracePeriod()` | `60000` | Time interval in milliseconds to keep the unhealthy endpoint disabled even if it recovers (this prevents rapid oscillation between endpoints when intermittent faults occur). |
+| `fastFailover()` | `false` | If true, existing connections to an unhealthy endpoint are forced to close immediately when a failover occurs, otherwise the connections are closed gracefully. Forcefully closing connections can make failover faster, but it might also cause in-flight operations to fail. |
+| `retryOnFailover()` | `false` | If true, commands that fail during a failover are automatically retried on the replacement endpoint. |
+
 ### Failover callbacks
 
 You may want to take some custom action when a failover occurs.
@@ -394,6 +406,24 @@ systems to trigger this method in your application. For example, if your applica
 exposes a REST API, you might consider creating a REST endpoint to call
 `setActiveDatabase()`.
 
+## Behavior when all endpoints are unhealthy
+
+In the extreme case where no endpoint is healthy, a command will throw a
+`JedisTemporarilyNotAvailableException`. This indicates that the client is periodically
+checking to see if any endpoint becomes healthy again. The number of
+times it will keep checking is configured by the `maxNumFailoverAttempts()` option in
+the `MultiDbConfig` builder and
+the delay between attempts is configured by the `delayInBetweenFailoverAttempts()` option (see [General failover configuration](#general-failover-configuration)). With the default settings,
+`maxNumFailoverAttempts` * `delayInBetweenFailoverAttempts`  gives a period of 120 seconds to find
+a healthy endpoint.
+
+You can still keep retrying commands after a `JedisTemporarilyNotAvailableException` is thrown (for example,
+you could add this exception to the `includedExceptionList`, as described
+in the [Retry configuration]({{< relref "#retry-configuration" >}}) section). However, if the number of
+failover attempts exceeds the value set by `maxNumFailoverAttempts()`, commands will throw a `JedisPermanentlyNotAvailableException`. Note that this is intended to notify your app
+that the problem is likely to be persistent, but it *doesn't* mean that Jedis will stop trying
+to connect to a healthy endpoint if one becomes available.
+
 ## Troubleshooting
 
 This section lists some common problems and their solutions.

content/develop/clients/redis-py/failover.md

@@ -215,21 +215,19 @@ a database that is already unhealthy.
 | `health_check` | Custom list of `HealthCheck` objects to specify how to perform each probe during a health check. This defaults to just the simple [`PingHealthCheck`](#pinghealthcheck-default). |
 | `initial_health_check_policy` | `InitialHealthCheck` enum value to specify the policy to use during the initial health check. The options are `InitialHealthCheck.ALL_HEALTHY` (all probes must succeed), `InitialHealthCheck.ANY_HEALTHY` (at least one probe must succeed), and `InitialHealthCheck.MAJORITY_HEALTHY` (more than half the probes must succeed). The default policy is `InitialHealthCheck.ALL_HEALTHY`. |
 
-### Health check strategies
-
 There are several strategies available for health checks that you can configure using the
 `MultiClusterClientConfig` builder. The sections below explain these strategies
 in more detail.
 
-#### `PingHealthCheck` (default)
+### `PingHealthCheck` (default)
 
 The default strategy, `PingHealthCheck`, periodically sends a Redis
 [`PING`]({{< relref "/commands/ping" >}}) command
 and checks that it gives the expected response. Any unexpected response
 or exception indicates an unhealthy server. Although `PingHealthCheck` is
 very simple, it is a good basic approach for most Redis deployments.
 
-#### `LagAwareHealthCheck` (Redis Software only) {#lag-aware-health-check}
+### `LagAwareHealthCheck` (Redis Software only) {#lag-aware-health-check}
 
 `LagAwareHealthCheck` is designed specifically for
 Redis Software [Active-Active]({{< relref "/operate/rs/databases/active-active" >}})
@@ -294,7 +292,7 @@ The `LagAwareHealthCheck` constructor accepts the following options:
 | `client_key_file` | Path to client private key file for mutual TLS. |
 | `client_key_password` | Password for encrypted client private key |
 
-#### Custom health check strategy
+### Custom health check strategy
 
 You can supply your own custom health check strategy by
 deriving a new class from the `AbstractHealthCheck` class.
@@ -392,6 +390,21 @@ Note that `set_active_database()` is thread-safe.
 
 If you decide to implement manual failback, you will need a way for external systems to trigger this method in your application. For example, if your application exposes a REST API, you might consider creating a REST endpoint to call `set_active_database()`.
 
+## Behavior when all endpoints are unhealthy
+
+In the extreme case where no endpoint is healthy, a command will throw a `TemporaryUnavailableException`.
+This indicates that the client is periodically checking to see if any endpoint becomes healthy again. The number of
+times it will keep checking is configured by the `failover_attempts` option in `MultiDbConfig` and
+the delay between attempts is configured by the `failover_delay` option (see [General failover configuration](#general-failover-configuration)). With the default settings, `failover_attempts` * `failover_delay` 
+gives a period of 120 seconds to find a healthy endpoint.
+
+You can still keep retrying commands after a `TemporaryUnavailableException` is thrown (for example,
+you could add this exception to the `supported_errors` list in your `Retry` configuration, as described
+in [Retries]({{< relref "/develop/clients/redis-py/produsage#retries" >}})). However, if the client exhausts
+all the available failover attempts before any endpoint becomes healthy again, commands will throw a `NoValidDatabaseException`. The client won't recover automatically from this situation, so you
+should handle it by reconnecting with the `MultiDBClient` constructor after a suitable delay (see
+[Failover configuration](#failover-configuration) for a connection example).
+
 ## Troubleshooting
 
 This section lists some common problems and their solutions.
@@ -400,12 +413,19 @@ This section lists some common problems and their solutions.
 
 If all health checks fail, you should first rule out authentication
 problems with the Redis server and also make sure there are no persistent
-network connectivity problems. If you are using
+network connectivity problems.
+
+If you are using [`PingHealthCheck`](#pinghealthcheck-default) or a
+[custom health check strategy](#custom-health-check-strategy),
+check that the `socket_timeout` is not too low for your network conditions
+(see [Timeouts]({{< relref "/develop/clients/redis-py/produsage#timeouts" >}}) for more information).
+
+For
 [`LagAwareHealthCheck`](#lag-aware-health-check), check that the `health_check_url`
-is set correctly for each endpoint. You can also try increasing the timeout
-for health checks and the interval between them. See
-[Health check configuration](#health-check-configuration) and
-[Endpoint configuration](#endpoint-configuration) for more information about these options.
+is set correctly for each endpoint. Note that health checks might be taking longer to
+execute than you anticipated, so make sure your `timeout` setting is not too low.
+
+
 
 ### Slow failback after recovery