Add Envoy as an alternative network proxy backend

docs/arch/15-envoy-network-proxy.md

@@ -0,0 +1,274 @@
+# Envoy Network Proxy
+
+## Status
+
+Experimental — selected with `TOOLHIVE_NETWORK_PROXY=envoy`. Squid remains the
+default.
+
+## Problem
+
+When network isolation is enabled (`--isolate-network`), ToolHive currently
+starts **three** auxiliary containers per workload:
+
+| Container | Role |
+|-----------|------|
+| `<name>-egress` | Squid forward proxy — routes outbound traffic through an allowlist |
+| `<name>-ingress` | Squid reverse proxy — receives traffic from the proxy runner |
+| `<name>-dns` | dnsmasq — provides DNS to the internal network |
+
+Three containers means three image pulls, three startup sequences, three sets of
+resources, and three things that can fail or restart. The Squid egress and ingress
+containers are logically a single gateway — splitting them into two processes is
+an implementation artifact rather than a deliberate design.
+
+## Solution
+
+Replace the two Squid containers with a **single Envoy container** that handles
+both egress and ingress as separate listeners inside the same process. The DNS
+container (dnsmasq) is unchanged.
+
+```
+Before:  <name>-egress (Squid)  +  <name>-ingress (Squid)  +  <name>-dns
+After:   <name>-egress (Envoy, two listeners)               +  <name>-dns
+```
+
+This reduces auxiliary container count from 3 → 2, simplifies the startup
+sequence, and uses a single bootstrap configuration file to describe the entire
+proxy behaviour.
+
+## Why Envoy
+
+### Consolidation
+
+Envoy's `HttpConnectionManager` supports multiple listeners in a single process.
+The egress forward proxy (`:3128`) and ingress reverse proxy share the same Envoy
+instance, the same access logs, and the same lifecycle.
+
+### L7 authority-based enforcement
+
+Squid matches destination hostnames via `dstdomain` ACLs and destination IPs via
+`dst` ACLs. Envoy, running as a forward proxy, enforces the equivalent controls
+at L7 by matching the HTTP `:authority` header.
+
+Envoy's RBAC filter is used for both allow and deny decisions:
+- **Header match on `:authority`** — L7 match on the forward-proxy target for
+  both plain HTTP requests and HTTPS CONNECT tunnels. This is the authority the
+  client asks the proxy to reach, so a single match covers hostnames *and* IP
+  literals presented as the CONNECT target.
+
+Note that RBAC `destination_ip` is **not** usable for target filtering in a
+forward proxy: `destination_ip` resolves to Envoy's own listener socket (the
+address the client connected to), not the proxied target, so a CIDR match on it
+is inert. All target enforcement therefore happens through `:authority` regexes,
+which mirror Squid's `dstdomain`/`dst` denies. The `Internal: true` Docker
+network remains the fail-closed backstop for non-cooperative traffic that ignores
+the proxy entirely.
+
+### Proper dynamic forward proxy
+
+Envoy's `dynamic_forward_proxy` cluster performs per-request DNS resolution and
+handles HTTP CONNECT tunnelling natively. HTTPS flows through a CONNECT tunnel
+exactly as a client would expect, with Envoy acting as a transparent TCP relay
+after the CONNECT handshake — no TLS inspection, no certificate pinning, no CA
+changes.
+
+### Access logging
+
+Both listeners write structured access logs to stdout, visible via `docker logs`.
+Squid logged differently for egress and ingress with no unified view.
+
+### Configuration as code
+
+Envoy reads a protobuf-JSON bootstrap file generated by ToolHive at workload
+start. The configuration is typed Go structs serialised to JSON — unit-testable,
+diffable, and reproducible. Squid required template-rendered text files.
+
+### Future extensibility
+
+Envoy's xDS API makes it possible to update listeners, clusters, and RBAC
+policies at runtime without a container restart. This is not used today, but the
+groundwork is there for dynamic policy updates. The transparent L3/L4
+interception path (Phase 2, not yet implemented) requires an `original_dst`
+listener and iptables rules that Envoy handles cleanly.
+
+## What Envoy Does Not Do
+
+- **Decrypt TLS.** Like Squid, Envoy filters HTTPS on the CONNECT target hostname
+  and then relays the encrypted stream as-is. No certificate inspection, no
+  man-in-the-middle.
+- **Block non-cooperative traffic.** A workload that opens a raw TCP connection
+  ignoring `HTTP_PROXY` is contained by the `Internal: true` Docker network
+  blackhole, not by Envoy. Envoy only sees traffic that goes through the proxy.
+- **Replace dnsmasq.** DNS for the workload's internal network is still served by
+  the dnsmasq container.
+- **Run in Kubernetes.** Network isolation is a local-Docker feature only; the
+  Kubernetes operator has a separate egress gateway path.
+
+## Architecture
+
+### Egress listener (`:3128` — forward proxy)
+
+```
+HTTP_PROXY / HTTPS_PROXY → Envoy :3128
+  └── HCM (upgrade: CONNECT)
+        ├── [optional] RBAC DENY  — docker gateway (:authority: IP literal + hostnames)
+        ├── RBAC ALLOW            — outbound allowlist (or allow-all)
+        ├── dynamic_forward_proxy — per-request DNS + CONNECT tunnel
+        └── router
+```
+
+The RBAC filters are evaluated top-to-bottom. The gateway DENY filter is present
+unless `--allow-docker-gateway` is set. It blocks the Docker gateway purely at
+L7, by matching the `:authority` header (the forward-proxy target for both plain
+HTTP and HTTPS CONNECT) against anchored, case-insensitive, port-tolerant
+regexes for:
+- The resolved Docker bridge gateway IP literal (e.g. `172.17.0.1`, with an
+  optional `:port`)
+- The Docker-internal hostnames `host.docker.internal` and
+  `gateway.docker.internal` (with an optional `:port`, so HTTPS CONNECT
+  authorities such as `host.docker.internal:443` match too)
+
+There is deliberately no `destination_ip`/L3 CIDR component: as noted above, a
+forward proxy's `destination_ip` is Envoy's own listener, so it would never match
+the gateway. Matching the IP literal in `:authority` is what actually catches a
+client that connects directly to the gateway address, giving parity with Squid's
+combined `dst`/`dstdomain` denies.
+
+The ALLOW filter implements the permission profile's `Outbound` rules:
+- `InsecureAllowAll: true` → single wildcard policy (`any: true`)
+- `AllowHost: [...]` → per-host `:authority` regex match that mirrors Squid's
+  `dstdomain` semantics: `example.com` matches that host exactly, while
+  `.example.com` (leading dot, also accepted as `*.example.com`) matches the apex
+  **and** all subdomains. The generated pattern is anchored, case-insensitive,
+  and tolerates an optional `:port` so HTTPS CONNECT authorities
+  (`example.com:443`) match too.
+- No outbound permissions configured → empty policy map → Envoy deny-all
+
+This preserves parity with the existing Squid backend so that permission
+profiles written for Squid's `dstdomain` syntax behave identically under Envoy.
+
+When `--allow-docker-gateway` is set, ToolHive not only omits the gateway DENY
+filter but also injects explicit ALLOW policies for the gateway (the IP literal
+and both Docker-internal hostnames, matched the same way as the deny). This is
+required because the ALLOW filter is default-deny: under a non-permissive profile
+(one without `InsecureAllowAll`), simply dropping the deny would leave the
+gateway unreachable, since nothing in the allowlist would match it. The explicit
+ALLOW entries ensure the gateway is actually permitted rather than merely
+un-denied.
+
+### Ingress listener (`0.0.0.0:<port>` — reverse proxy)
+
+```
+Proxy runner → host:127.0.0.1:<port> → Docker port binding → Envoy :port
+  └── HCM
+        ├── router
+        └── route → STRICT_DNS cluster → <name>:<mcp-port>
+```
+
+The ingress listener binds to `0.0.0.0` inside the container so Docker's port
+forwarding (which targets the container's bridge IP, not its loopback) can reach
+it. The host-side port binding restricts to `127.0.0.1`, so the ingress is only
+reachable from the local machine.
+
+The upstream STRICT_DNS cluster resolves the MCP container's hostname inside the
+internal Docker network and forwards HTTP traffic to the MCP server port.
+
+The admin interface binds to `127.0.0.1` inside the Envoy container (container
+loopback, not reachable via Docker port forwarding) as a precaution against the
+admin API being accessible from other containers.
+
+### Bootstrap lifecycle
+
+1. ToolHive generates a protobuf-JSON bootstrap file in `os.TempDir()` at mode
+   `0600`.
+2. The file is bind-mounted read-only into the Envoy container at
+   `/etc/envoy/envoy.json`.
+3. Envoy reads it once at startup.
+
+The bootstrap file is **not** removed when the workload is torn down — there is
+no cleanup hook for it today. The only removal path is a best-effort cleanup if
+generation fails partway through writing. As a result, a bootstrap file persists
+in the OS temp directory (at mode `0600`) for the lifetime of the host until the
+OS reclaims the temp directory. This is a known limitation (see below).
+
+## Selection
+
+```bash
+TOOLHIVE_NETWORK_PROXY=envoy thv run --isolate-network <server>
+```
+
+`TOOLHIVE_NETWORK_PROXY` accepts:
+- `""` or `"squid"` — Squid backend (default)
+- `"envoy"` — Envoy backend
+
+An unknown value causes `NewClient` to fail at startup with a descriptive error.
+The env var is intentionally not exposed as a CLI flag or CRD field while the
+backend is experimental; chart surface and `RunConfig` wiring come later once the
+backend is stable.
+
+## Comparison with Squid
+
+| Aspect | Squid (current default) | Envoy |
+|--------|------------------------|-------|
+| Containers per workload | 3 (egress + ingress + dns) | 2 (combined + dns) |
+| Forward proxy | ✓ | ✓ (dynamic_forward_proxy) |
+| Reverse proxy (ingress) | ✓ (separate container) | ✓ (second listener, same container) |
+| HTTPS CONNECT tunnelling | ✓ | ✓ |
+| TLS inspection | ✗ | ✗ |
+| L7 hostname deny | ✓ (`dstdomain`) | ✓ (`:authority` header match) |
+| Destination IP deny | ✓ (`dst` ACL, resolved IP) | ~ (IP literal in `:authority` only; resolved IP not enforced — see Known Limitations) |
+| Wildcard host allowlist | ✓ (`.example.com` dot-prefix) | ✓ (same syntax, regex match incl. apex + port) |
+| Per-request DNS resolution | Via Squid resolver | Via DFP cluster |
+| Access logs | Per-container, text format | Unified stdout, structured |
+| Config format | Text template | Typed Go structs → protobuf-JSON |
+| Runtime config update | Restart required | xDS-capable (not yet used) |
+| Upstream image | Stacklok-built | Upstream distroless (pinned tag) |
+
+## Known Limitations
+
+- **Tag-pinned image.** The Envoy image is pinned by tag (`v1.32.3`), not by
+  digest. A future PR should pin by digest and add a `TOOLHIVE_ENVOY_IMAGE`
+  override for supply-chain policy requirements (the env var already exists).
+  Tracked in #5903.
+- **Admin interface port.** The admin API on `:9901` (loopback-only inside the
+  container) is always enabled. A follow-up can disable it entirely or make it
+  conditional.
+- **CONNECT access log timing.** Envoy logs CONNECT tunnel entries when the
+  tunnel closes, not when it opens. With keep-alive HTTP clients the log entry
+  may be delayed by minutes. Egress access logs are visible in `docker logs` but
+  appear after the connection closes.
+- **No transparent L3/L4.** Non-cooperative traffic (workloads that ignore
+  `HTTP_PROXY`) is contained by the `Internal: true` network, not Envoy. True
+  non-bypassable enforcement requires iptables TPROXY + an init container with
+  `CAP_NET_ADMIN` — this is Phase 2 and requires its own architecture doc.
+- **No port-based allowlist.** `AllowPort` from the permission profile is not
+  yet translated into Envoy egress policy, so an allowlisted host is reachable on
+  any port. Squid honours `AllowPort`; the Envoy backend currently does not,
+  which is a parity gap. Tracked in #5915.
+- **Gateway deny matches the requested name, not the resolved IP.** The deny
+  filter matches the `:authority` string (the gateway IP literal and the two
+  Docker-internal hostnames). Because an HTTP RBAC filter cannot see the address
+  the `dynamic_forward_proxy` cluster ultimately resolves and dials, a hostname
+  that resolves to the gateway IP — via DNS rebinding, an attacker-controlled
+  record, or an allowlisted host repointed at the gateway — is not caught, whereas
+  Squid's `dst` ACL denies on the resolved IP. Enforcing on the resolved address
+  requires L3/L4 interception (Phase 2, #5905); until then this is a known
+  weakening of the gateway block versus the Squid backend.
+- **IPv4-only DFP DNS.** The `dynamic_forward_proxy` cluster's DNS lookup is
+  hardcoded to `V4_ONLY`. A host that resolves only to IPv6 addresses is
+  unreachable through the Envoy egress path even if it is allowlisted.
+- **Wildcard ingress virtual host.** The ingress listener uses a wildcard virtual
+  host (`domains: ["*"]`) rather than a specific host match. This is safe because
+  it is bounded by the host-side port binding to `127.0.0.1`: only the local
+  proxy runner can reach the ingress listener, so the wildcard is not exposed
+  beyond the local machine.
+- **Bootstrap file not cleaned up.** The generated bootstrap file (mode `0600` in
+  the OS temp directory) is not removed on workload teardown; it is only removed
+  on a mid-write generation failure. Files accumulate until the OS reclaims the
+  temp directory.
+- **Deny-all on empty profile (deliberate divergence).** A nil or empty
+  permission profile produces an empty ALLOW policy map, which under Envoy's
+  default-deny RBAC means deny-all. Squid, by contrast, defaults to allow-all
+  when no rules are configured. The Envoy behaviour is fail-closed (safer) but is
+  an intentional divergence from the default backend, so a profile that is
+  permissive-by-omission under Squid will block all egress under Envoy.

docs/arch/README.md

@@ -120,6 +120,13 @@ Welcome to the ToolHive architecture documentation. This directory contains comp
     - Component inventory and per-client dropped-component warnings
     - Name/repo consistency check, extraction safety, TOML mutation under file lock
 
+15. **[Envoy Network Proxy](15-envoy-network-proxy.md)**
+    - Experimental Envoy backend (`TOOLHIVE_NETWORK_PROXY=envoy`) replacing two Squid containers with one
+    - Egress forward proxy and ingress reverse proxy as separate listeners in a single process
+    - L7 `:authority`-based RBAC allow/deny and `dstdomain` parity (incl. Docker-gateway blocking)
+    - Dynamic forward proxy with per-request DNS and native HTTPS CONNECT tunnelling
+    - Squid-vs-Envoy comparison and known limitations (`AllowPort` gap, V4_ONLY DNS, deny-all on empty profile)
+
 ### Existing Documentation
 
 For middleware architecture, see: **[docs/middleware.md](../middleware.md)**

pkg/container/docker/client_stop_test.go

@@ -138,3 +138,69 @@ func TestStopWorkload_NotFound_ReturnsNil(t *testing.T) {
 	// StopWorkload should treat a not-found workload as success
 	require.NoError(t, err)
 }
+
+// TestStopProxyContainer_HandlesEnvoyAndSquidTopologies exercises the building
+// block of the shared cleanup path. StopWorkload iterates the fixed proxy
+// suffixes (-egress, -ingress, -dns) and calls stopProxyContainer for each; this
+// test verifies stopProxyContainer's per-container contract directly: stop a
+// container that exists, and silently tolerate one that does not. That tolerance
+// is what lets the same suffix iteration clean up both the 3-container Squid
+// workload and the 2-container Envoy workload (which has no -ingress container)
+// without backend-specific logic. See #5902.
+func TestStopProxyContainer_HandlesEnvoyAndSquidTopologies(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name          string
+		containerName string
+		present       bool // whether the named proxy container exists
+		wantStopped   bool
+	}{
+		{
+			name:          "present container is stopped (Squid egress/ingress/dns, Envoy egress/dns)",
+			containerName: "app-egress",
+			present:       true,
+			wantStopped:   true,
+		},
+		{
+			name:          "absent ingress is tolerated (Envoy has no -ingress container)",
+			containerName: "app-ingress",
+			present:       false,
+			wantStopped:   false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			var stoppedID string
+			api := &fakeDockerAPI{
+				listFunc: func(_ context.Context, _ mobyclient.ContainerListOptions) ([]container.Summary, error) {
+					if !tt.present {
+						return []container.Summary{}, nil
+					}
+					return []container.Summary{
+						{ID: "cid-" + tt.containerName, Names: []string{"/" + tt.containerName}},
+					}, nil
+				},
+				stopFunc: func(_ context.Context, id string, _ mobyclient.ContainerStopOptions) error {
+					stoppedID = id
+					return nil
+				},
+			}
+			c := &Client{api: api}
+
+			// Must not panic or error regardless of whether the container exists.
+			c.stopProxyContainer(t.Context(), tt.containerName, 30)
+
+			if tt.wantStopped {
+				assert.Equal(t, "cid-"+tt.containerName, stoppedID,
+					"expected the existing proxy container to be stopped")
+			} else {
+				assert.Empty(t, stoppedID,
+					"expected no ContainerStop call for an absent proxy container")
+			}
+		})
+	}
+}