feat(stream): support set_real_ip_from to trust an inbound PROXY protocol header

apisix/cli/ngx_tpl.lua

@@ -173,6 +173,12 @@ stream {
     lua_ssl_trusted_certificate {* ssl.ssl_trusted_certificate *};
     {% end %}
 
+    {% if stream.real_ip_from then %}
+    {% for _, real_ip in ipairs(stream.real_ip_from) do %}
+    set_real_ip_from {*real_ip*};
+    {% end %}
+    {% end %}
+
     # for stream logs, off by default
     {% if stream.enable_access_log == true then %}
     log_format main escape={* stream.access_log_format_escape *} '{* stream.access_log_format *}';

conf/config.yaml.example

@@ -218,6 +218,13 @@ nginx_config:                     # Config for render the template to generate n
       upstream-healthcheck: 10m
 
   stream:
+    # real_ip_from:                          # https://nginx.org/en/docs/stream/ngx_stream_realip_module.html#set_real_ip_from
+    #   - 192.168.1.0/24                     # Trusted addresses that are known to send correct PROXY protocol
+    #                                        # headers. On a connection from a trusted address that carries an
+    #                                        # inbound PROXY protocol header, the client address is replaced by
+    #                                        # the one in the header. Only takes effect on ports that accept the
+    #                                        # PROXY protocol. Empty by default, so the client address is always
+    #                                        # the directly connected peer.
     enable_access_log: false                 # Enable stream proxy access logging.
     access_log: logs/access_stream.log       # Location of the stream access log.
     access_log_format: |

docs/en/latest/stream-proxy.md

@@ -277,3 +277,44 @@ apisix:
 ```
 
 The accept side (`proxy_protocol`) is a per-listen directive, so ports with different settings can share one listener. The upstream side (`proxy_protocol_to_upstream`) is a server-level directive, so APISIX renders ports that send the PROXY protocol upstream into a separate `server` block. UDP listens never send the PROXY protocol upstream, so they always stay in the plain `server` block.
+
+:::warning
+
+Only enable `proxy_protocol_to_upstream` for upstreams that expect the PROXY protocol. An upstream that does not will read the plaintext `PROXY` line as application data and typically close the connection immediately — a TLS upstream, for example, cannot parse it as a TLS record.
+
+:::
+
+### Preserving the client address behind a load balancer
+
+`proxy_protocol_to_upstream` builds the outbound header from the address APISIX is connected to. When APISIX sits behind a load balancer that speaks the PROXY protocol, that is the load balancer's address, so accepting a header and sending one is not by itself enough to carry the client address through to the upstream.
+
+Set `nginx_config.stream.real_ip_from` to the addresses of the load balancers you trust. On a connection from a trusted address that carries an inbound PROXY protocol header, APISIX replaces the client address with the one from the header:
+
+```yaml
+apisix:
+  proxy_mode: http&stream
+  stream_proxy:
+    tcp:
+      - addr: 9100
+        proxy_protocol: true              # accept the header from the load balancer
+        proxy_protocol_to_upstream: true  # rebuild it toward the upstream
+nginx_config:
+  stream:
+    real_ip_from:
+      - 192.168.1.0/24                    # the load balancer's network
+```
+
+The header APISIX then sends upstream carries the client address, and so do the stream `$remote_addr`, the access log, and address-based matching such as the `ip-restriction` plugin. The directly connected address stays available as `$realip_remote_addr`.
+
+`real_ip_from` is empty by default and only takes effect on ports that accept the PROXY protocol, and only for peers that match it. Trust only load balancers you control: a peer you trust can claim to be any client.
+
+### Choosing a configuration
+
+Which combination you want depends on who needs to see the client address:
+
+| Configuration | `proxy_protocol` | `proxy_protocol_to_upstream` | `real_ip_from` | Result |
+|---|---|---|---|---|
+| Pass through | off | off | — | APISIX never looks at the header and proxies it to the upstream as ordinary stream bytes. The upstream sees the client, APISIX does not. Not usable on ports where APISIX has to read the stream itself, such as TLS ports or routes that match on preread data. |
+| Terminate | on | off | — | APISIX consumes the header and connects to the upstream without one. Use this for upstreams that do not speak the PROXY protocol. |
+| Terminate and rebuild | on | on | load balancer network | APISIX consumes the header and sends a new one carrying the client address. Both APISIX and the upstream see the client. |
+| Terminate and rebuild, nothing trusted | on | on | — | The upstream still gets a header, but it carries the address APISIX is connected to — the load balancer — so the client address is lost. |

docs/zh/latest/stream-proxy.md

@@ -268,3 +268,44 @@ apisix:
 ```
 
 接收侧（`proxy_protocol`）是 listen 级别的指令，因此设置不同的端口可以共用一个监听块。上游侧（`proxy_protocol_to_upstream`）是 server 级别的指令，因此 APISIX 会把向上游发送 PROXY 协议的端口渲染到单独的 `server` 块中。UDP 监听永远不会向上游发送 PROXY 协议，因此始终保留在普通的 `server` 块中。
+
+:::warning
+
+只应对期待 PROXY 协议的上游启用 `proxy_protocol_to_upstream`。不支持该协议的上游会把明文的 `PROXY` 行当作应用数据读取，通常会立即关闭连接——例如 TLS 上游无法将其解析为 TLS record。
+
+:::
+
+### 在负载均衡器之后保留客户端地址
+
+`proxy_protocol_to_upstream` 使用 APISIX 实际连接到的对端地址来构造发往上游的头部。当 APISIX 位于一台使用 PROXY 协议的负载均衡器之后时，该地址是负载均衡器的地址，因此仅仅"接收头部并发送头部"并不足以把客户端地址传递到上游。
+
+将 `nginx_config.stream.real_ip_from` 设置为你信任的负载均衡器地址。当连接来自受信任的地址且携带入站 PROXY 协议头部时，APISIX 会用头部中的地址替换客户端地址：
+
+```yaml
+apisix:
+  proxy_mode: http&stream
+  stream_proxy:
+    tcp:
+      - addr: 9100
+        proxy_protocol: true              # 接收来自负载均衡器的头部
+        proxy_protocol_to_upstream: true  # 向上游重建该头部
+nginx_config:
+  stream:
+    real_ip_from:
+      - 192.168.1.0/24                    # 负载均衡器所在网段
+```
+
+此时 APISIX 发往上游的头部携带的是客户端地址，stream 的 `$remote_addr`、访问日志以及基于地址的匹配（如 `ip-restriction` 插件）同样如此。直连对端的地址仍可通过 `$realip_remote_addr` 获取。
+
+`real_ip_from` 默认为空，且仅在接收 PROXY 协议的端口上、且仅对与之匹配的对端生效。请只信任你自己掌控的负载均衡器：受信任的对端可以声称自己是任意客户端。
+
+### 如何选择配置
+
+具体选择哪种组合，取决于谁需要看到客户端地址：
+
+| 配置 | `proxy_protocol` | `proxy_protocol_to_upstream` | `real_ip_from` | 效果 |
+|---|---|---|---|---|
+| 透传 | 关闭 | 关闭 | — | APISIX 完全不解析头部，将其作为普通 stream 字节代理到上游。上游能看到客户端，APISIX 看不到。在 APISIX 自身需要读取流内容的端口上不可用，例如 TLS 端口或依赖 preread 数据匹配的路由。 |
+| 终结 | 开启 | 关闭 | — | APISIX 消费头部，并在不携带头部的情况下连接上游。适用于不支持 PROXY 协议的上游。 |
+| 终结并重建 | 开启 | 开启 | 负载均衡器网段 | APISIX 消费头部，并发送携带客户端地址的新头部。APISIX 和上游都能看到客户端。 |
+| 终结并重建，但未配置信任网段 | 开启 | 开启 | — | 上游仍会收到头部，但其中携带的是 APISIX 的直连对端地址（即负载均衡器），客户端地址丢失。 |

t/cli/test_stream_proxy_protocol.sh

@@ -36,6 +36,33 @@ block_with_listen() {
     ' conf/nginx.conf
 }
 
+# Print the top-level stream{} block. `set_real_ip_from` is rendered into the
+# http block too, so assertions about the stream side have to be scoped.
+stream_block() {
+    awk '
+    /^stream[ \t]*\{/ { depth = 0; inblock = 1 }
+    inblock {
+        print
+        depth += gsub(/\{/, "{")
+        depth -= gsub(/\}/, "}")
+        if (depth == 0) { inblock = 0 }
+    }
+    ' conf/nginx.conf
+}
+
+# Open a TCP connection to <port>, announce <claimed client ip> in a PROXY
+# protocol v1 header, and print whatever comes back. The tests below set
+# `admin_key` explicitly rather than relying on the shipped default, so that
+# the `yq` lookup finds a key whatever the default happens to be.
+pp_request() {
+    local port="$1"
+    local claimed_ip="$2"
+    exec 3<>/dev/tcp/127.0.0.1/"$port"
+    printf 'PROXY TCP4 %s 127.0.0.1 56324 %s\r\n' "$claimed_ip" "$port" >&3
+    timeout 3 cat <&3
+    exec 3<&-
+}
+
 # === Default: no PROXY protocol anywhere ===
 echo '
 apisix:
@@ -228,4 +255,154 @@ if block_with_listen 9100 | grep -E "ssl_certificate " > /dev/null; then
 fi
 echo "passed: per-group ssl follows the TLS port into its server block"
 
+# === No trusted addresses by default ===
+echo '
+apisix:
+  proxy_mode: "http&stream"
+  stream_proxy:
+    tcp:
+      - addr: 9100
+        proxy_protocol: true
+' > conf/config.yaml
+make init
+
+if stream_block | grep -E "set_real_ip_from" > /dev/null; then
+    echo "failed: no address should be trusted by default"
+    exit 1
+fi
+echo "passed: no set_real_ip_from by default"
+
+# === stream.real_ip_from renders one set_real_ip_from per entry ===
+echo '
+apisix:
+  proxy_mode: "http&stream"
+  stream_proxy:
+    tcp:
+      - addr: 9100
+        proxy_protocol: true
+nginx_config:
+  stream:
+    real_ip_from:
+      - 192.168.1.0/24
+      - 10.0.0.0/8
+' > conf/config.yaml
+make init
+
+if ! stream_block | grep -E "^\s*set_real_ip_from 192\.168\.1\.0/24;" > /dev/null; then
+    echo "failed: stream.real_ip_from should render the trusted network"
+    exit 1
+fi
+if ! stream_block | grep -E "^\s*set_real_ip_from 10\.0\.0\.0/8;" > /dev/null; then
+    echo "failed: stream.real_ip_from should render every entry"
+    exit 1
+fi
+echo "passed: stream.real_ip_from renders set_real_ip_from"
+
+# === End to end: a trusted peer's PROXY header reaches the upstream ===
+# The upstream is a PROXY-protocol-aware stream server that echoes the address
+# it was told about, so this asserts on the header APISIX actually rebuilds
+# rather than on what nginx.conf says.
+echo '
+apisix:
+  proxy_mode: "http&stream"
+  stream_proxy:
+    tcp:
+      - addr: 9100
+        proxy_protocol: true
+        proxy_protocol_to_upstream: true
+deployment:
+  admin:
+    admin_key:
+      - name: admin
+        key: edd1c9f034335f136f87ad84b625c8f1
+        role: admin
+nginx_config:
+  stream:
+    real_ip_from:
+      - 127.0.0.1
+  stream_configuration_snippet: |
+    server {
+        listen 9101 proxy_protocol;
+        return "upstream saw $proxy_protocol_addr";
+    }
+' > conf/config.yaml
+make run
+wait_for_tcp 127.0.0.1 9180
+wait_for_tcp 127.0.0.1 9100
+
+admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
+curl -i http://127.0.0.1:9180/apisix/admin/stream_routes/1 \
+    -H "X-API-KEY: $admin_key" -X PUT -d \
+    '{"upstream":{"nodes":{"127.0.0.1:9101":1},"type":"roundrobin"}}'
+
+# Retry under a 10s deadline to cover etcd->stream-worker propagation.
+ok=0
+deadline=$(( $(date +%s) + 10 ))
+{ set +x; } 2>/dev/null
+while [ "$(date +%s)" -lt "$deadline" ]; do
+    if pp_request 9100 198.51.100.7 | grep -q "upstream saw 198.51.100.7"; then
+        ok=1
+        break
+    fi
+    sleep 0.3
+done
+set -x
+if [ "$ok" -ne 1 ]; then
+    echo "failed: upstream should see the client address, not the direct peer"
+    exit 1
+fi
+
+make stop
+echo "passed: trusted peer's client address reaches the upstream"
+
+# === End to end: without a trusted peer the upstream sees the direct peer ===
+echo '
+apisix:
+  proxy_mode: "http&stream"
+  stream_proxy:
+    tcp:
+      - addr: 9100
+        proxy_protocol: true
+        proxy_protocol_to_upstream: true
+deployment:
+  admin:
+    admin_key:
+      - name: admin
+        key: edd1c9f034335f136f87ad84b625c8f1
+        role: admin
+nginx_config:
+  stream_configuration_snippet: |
+    server {
+        listen 9101 proxy_protocol;
+        return "upstream saw $proxy_protocol_addr";
+    }
+' > conf/config.yaml
+make run
+wait_for_tcp 127.0.0.1 9180
+wait_for_tcp 127.0.0.1 9100
+
+admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
+curl -i http://127.0.0.1:9180/apisix/admin/stream_routes/1 \
+    -H "X-API-KEY: $admin_key" -X PUT -d \
+    '{"upstream":{"nodes":{"127.0.0.1:9101":1},"type":"roundrobin"}}'
+
+ok=0
+deadline=$(( $(date +%s) + 10 ))
+{ set +x; } 2>/dev/null
+while [ "$(date +%s)" -lt "$deadline" ]; do
+    if pp_request 9100 198.51.100.7 | grep -q "upstream saw 127.0.0.1"; then
+        ok=1
+        break
+    fi
+    sleep 0.3
+done
+set -x
+if [ "$ok" -ne 1 ]; then
+    echo "failed: an untrusted peer's claim must not be adopted"
+    exit 1
+fi
+
+make stop
+echo "passed: untrusted peer's claim is ignored"
+
 echo "All stream PROXY protocol tests passed."