Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/en/latest/plugins/ai-proxy-multi.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ When an instance's `provider` is set to `bedrock`, the Plugin expects requests i
| max_req_body_size | integer | False | 67108864 | greater than or equal to 1 | Maximum request body size in bytes that the plugin reads into memory. Requests whose body exceeds this limit are rejected with `413`. Prevents unbounded memory buffering of large request bodies. |
| max_stream_duration_ms | integer | False | | greater than or equal to 1 | Maximum wall-clock duration (in milliseconds) for a streaming AI response. If the upstream keeps sending data past this deadline, the gateway closes the connection. Unset means no cap. Use this to protect the gateway from upstream bugs that produce tokens indefinitely. When the limit is hit mid-stream, the downstream SSE stream is truncated (no protocol-specific terminator such as `[DONE]`, `message_stop`, or `response.completed`); well-behaved clients should treat a missing terminator as an incomplete response. |
| max_response_bytes | integer | False | | greater than or equal to 1 | Maximum total bytes read from the upstream for a single AI response (streaming or non-streaming). If exceeded, the gateway closes the connection. For non-streaming responses with `Content-Length`, the check is performed before reading the body; for chunked (no-`Content-Length`) non-streaming responses and for streaming responses, the cap is enforced incrementally as bytes are received. Unset means no cap. |
| streaming_flush_interval_ms | integer | False | 10 | greater than or equal to 0 | Interval in milliseconds for the background flush thread. When greater than 0 (default: 10), a background timer calls `ngx.flush(false)` every N milliseconds, batching output for bursty upstreams. When 0, the background thread is disabled and each chunk is flushed synchronously through `ngx.flush(true)`, guaranteeing immediate client delivery. |
| keepalive | boolean | False | true | | If true, keep the connection alive when requesting the LLM service. |
| keepalive_timeout | integer | False | 60000 | greater than or equal to 1000 | Request timeout in milliseconds when requesting the LLM service. |
| keepalive_pool | integer | False | 30 | | Keepalive pool size for when connecting with the LLM service. |
Expand Down
Loading