Fix websocket test flakes and complete TCP half-closed support - #2012
Open
jrajahalme wants to merge 9 commits into
Open
Fix websocket test flakes and complete TCP half-closed support#2012jrajahalme wants to merge 9 commits into
jrajahalme wants to merge 9 commits into
Conversation
jrajahalme
force-pushed
the
websocket-half-close-support
branch
from
August 25, 2026 09:00
f1720b9 to
ab62d42
Compare
The tests currently send a payload equal to the configured buffer limit. WebSocket framing makes the encoded data exceed that limit, potentially disabling reads before the FIN is observed and making the test timing dependent. Double the buffer limits while retaining the 50 MiB payload. The payload remains larger than the kernel socket buffer, so the tests still exercise upstream flushing without crossing the Envoy buffer limit solely because of WebSocket framing overhead. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Update the WebSocket encapsulation tests to perform an orderly CLOSE exchange instead of relying on the outer TCP transport FIN to delimit the tunneled stream. Keep the TCP client open after sending its write-side FIN, wait for the resulting WebSocket CLOSE, and have the peer send its remaining data followed by its own CLOSE. This more accurately exercises TCP half-close semantics and verifies that reverse-direction data can still be delivered after the first direction has ended. Also fix the downstream flush test's 64-bit WebSocket payload header. The old header provided only four of the required eight extended-length bytes. The decoder consequently consumed the beginning of the payload as part of the frame header, but the test happened to pass because the outer transport FIN terminated processing. Encode the full 10-byte frame header so that the following CLOSE is parsed at the correct frame boundary. These changes make the tests use valid WebSocket framing and explicit CLOSE messages for logical stream termination, without requiring any codec behavior changes. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
… end A WebSocket CLOSE represents a directional FIN for the tunneled TCP connection. The codec previously forwarded CLOSE with end_stream=true, coupling the tunneled FIN to a FIN on the underlying WebSocket transport. When both tunnel directions ended close together, final data, CLOSE, and transport FIN could be delivered together. Transport teardown could then race flushing the final frames. Sanitizer instrumentation made this timing race visible in CI as intermittent loss of the last tunneled data. Keep the WebSocket transport open after the first CLOSE. Only the server codec explicitly sends transport end-stream, and only after both the encoder and decoder have ended. This follows the closing asymmetry recommended by RFC 6455 section 7.1.1 and allows client-generated final frames to enter the transport without an attached FIN, so normal transport flushing can complete before shutdown. Handle either CLOSE ordering and guard against sending transport end-stream twice when propagating a tunneled FIN synchronously invokes the reverse codec path. Buffered client data released after the handshake likewise does not end the transport. Extend the integration tests to verify that the first CLOSE leaves the reverse direction usable and that final reverse-direction data is delivered before the WebSocket transport closes. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
A protocol filter may expose an end stream for its decoded data while the underlying protocol connection remains active for control traffic. Envoy currently rejects writes after ConnectionImpl::write_end_stream_ is set before invoking the write filter chain. This prevents the WebSocket codec from receiving upstream WebSocket control frames after it has translated a CLOSE into a FIN on the decoded downstream TCP connection. Carry an Envoy patch that gives write filters an opportunity to consume non-empty data after end stream. Any data left unconsumed is still rejected and cannot be written to the socket after FIN, preserving the underlying TCP invariant. This is the write-side counterpart to the receive_before_connect support we previously contributed upstream in envoyproxy/envoy#38189. That change allows protocol filters to receive downstream data before TCP proxy has established its upstream connection. This patch allows a protocol filter to continue receiving upstream protocol traffic after its decoded downstream stream has ended, and consequently to send control responses such as PONG back upstream. Together they decouple the lifetime of a protocol filter's control plane from the lifetime of the proxied TCP data stream. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
jrajahalme
force-pushed
the
websocket-half-close-support
branch
from
August 25, 2026 12:13
ab62d42 to
79376c4
Compare
jrajahalme
changed the title
jrajahalme
marked this pull request as draft
August 25, 2026 13:45
jrajahalme
force-pushed
the
websocket-half-close-support
branch
from
August 25, 2026 18:18
8de66bd to
a2acd73
Compare
jrajahalme
marked this pull request as ready for review
August 25, 2026 18:21
Normally TCPProxy upstream is closed without delay when the downstream connection gets closed. This can lead to droppoing data already received from the downstream TCP connection, but not yet sent to the upstream connection (e.g., being buffered at the filter chain). Similarly, an abrupt closure without flushing the data on the upstream socket receive buffer, or any receive direction data that happens to be in-flight, can cause a TCP RST. Add an Envoy patch to exposes the existing delayed connection close functionality to TCPProxy filter chains. This allows urderly delivery of any buffered data to be sent to the upstream socket, and allows protocols implemented in network filters to continue processing after downstream closes, for example, to perform a graceful shutdown. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Configure TcpProxy::UpstreamFlushWaitTimeoutMs filter state to enable delayed close on the upstream TCP connection used for WebSocket tunnel, so that WebSocket codec can cleanly encode any pending data and finish WebSocket CLOSE handshake after the downstream connection closes. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
RFC 6455 requires every client-to-server WebSocket frame to be masked, but the decapsulation integration tests sent most of their client frames unmasked. The decoder accepted these frames, causing otherwise successful tests to depend on invalid protocol traffic. Add a helper for constructing masked client binary frames and use it throughout the existing positive test coverage. Preserve coverage for non-minimal 16-bit and 64-bit payload-length encodings, frames received back-to-back, and frames split across multiple writes. This prepares the tests for enforcing client masking without changing production behavior. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Envoy TCP Proxy supports TCP half-close semantics, allowing one direction to end while data continues flowing in the reverse direction. Complete the corresponding behavior for TCP connections tunneled over WebSocket. Treat WebSocket CLOSE as a directional FIN. Track sent and received CLOSE independently, and emit end stream toward the corresponding tunneled TCP connection exactly once while leaving the reverse direction available. Delay a CLOSE response until the reverse TCP direction reaches end stream, allowing all pending reverse-direction data to be encoded first. Preserve the received CLOSE payload in the response, including its status-code bytes and optional reason. If the local TCP direction ends first, send CLOSE and continue processing peer control traffic until the peer's CLOSE arrives. Keep PING and PONG processing active while only one tunnel direction has closed. After receiving CLOSE, discard data-frame payloads while retaining frame synchronization so that subsequent control frames can still be processed. Refactor control-frame encoding so that CLOSE, PING, and PONG, including empty control frames, use the same encoder path. Client-generated control frames remain masked as required by the protocol. Extend integration coverage for client-first and server-first half-close, reverse-direction data after CLOSE, early data with FIN, buffer flushing, TCP resets, control frames during the half-closed interval, policy-filtered connections, and CLOSE payload propagation. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
Reject WebSocket frames that violate RFC 6455 instead of accepting them or treating malformed input as an orderly tunneled TCP shutdown. Require every client-to-server frame to be masked and every server-to-client frame to be unmasked. Require control frames to be final and limit their payloads to 125 bytes. Validate CLOSE payload structure by rejecting one-byte payloads and requiring the optional reason following the status-code bytes to be valid UTF-8. Keep the status-code bytes opaque so the codec remains compatible with future protocol revisions. Distinguish a valid WebSocket CLOSE from termination of the underlying transport. Treat a transport FIN received without CLOSE, as well as other frame protocol errors, as an abort that closes both tunnel directions immediately. Add integration coverage showing that unmasked client frames, malformed CLOSE payloads, and transport termination without CLOSE are rejected. Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
jrajahalme
force-pushed
the
websocket-half-close-support
branch
from
August 25, 2026 18:35
a2acd73 to
01d7864
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes two WebSocket tunnel shutdown issues exposed by sanitizer CI, completes tunneled TCP half-close support, and tightens frame decoding to RFC 6455.
The first three commits form a self-contained backport set. They give flush tests enough buffer headroom for WebSocket framing, make test shutdowns use an orderly CLOSE exchange, and fix the production race between final frames and transport shutdown. CLOSE remains a directional FIN for the tunneled TCP stream, but the client codec no longer attaches transport end-stream to its final frames. Only the server codec explicitly ends the transport after both tunnel directions have ended, allowing queued frames to be flushed before shutdown.
The remaining commits complete bidirectional half-close semantics. Receiving CLOSE ends the corresponding tunneled TCP direction exactly once while reverse-direction data and control traffic remain active. A CLOSE response is delayed until reverse data has drained, and it preserves the received status and reason payload. PING and PONG continue to work during the half-closed interval.
The PR carries two Envoy patches that lets write filters consume data after decoded end stream before Envoy rejects any unconsumed write, and delays the close of an upstream socket on a TCPProxy connection when the downstream closes. The first patch lets the WebSocket filter handle control traffic such as PING and generate PONG after the tunneled TCP direction has ended without permitting application data to be written after FIN. It is the write-side counterpart to envoyproxy/envoy#38189. The second patch allows any buffered final data to be encoded to the WebSocket tunnel, and WebSocket CLOSE handshake to finish gracefully after downstream TCP connection closes.
The tests now use correctly masked client frames and cover both half-close orderings, final data, flushing, control traffic, CLOSE payload propagation, and aborts. Decoder hardening rejects incorrect masking, fragmented or oversized control frames, malformed CLOSE payloads, invalid UTF-8 reasons, and transport FIN without CLOSE.