[fix](arrow-flight) Bound idle Arrow Flight SQL sessions separately from wait_timeout - #67504
Open
raghav-reglobe wants to merge 1 commit into
Open
Conversation
…rom wait_timeout A Flight SQL session idles under the same wait_timeout as a MySQL connection (28800s by default). Since the coordinator of a BE-served Flight query is kept alive across GetFlightInfo -> DoGet until the session's next query or its close, an abandoned session - a client that opens a session per query and never sends CloseSession - keeps that query's workload-group queue slot for the whole wait_timeout. Eight such sessions fill a max_concurrency=8 group and every later query in it fails with "query queue timeout". Add a mutable FE config, arrow_flight_session_idle_timeout_second (default 3600), applied by the existing connection timeout checker to ARROW_FLIGHT_SQL contexts only as min(wait_timeout, max(config, exec timeout)). The exec-timeout floor matters: a Flight session is COM_SLEEP while the client drains the result via DoGet and its idle clock runs from the query's start, so a bound below query_timeout would kill a long result stream before the query's own timeout could. 0 disables the bound. MySQL-protocol connections are unchanged. The kill log line now reports the effective idle timeout. Signed-off-by: Raghvendra Singh <raghav@cashify.in> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #67503
Related PR: #64799 (kept the coordinator alive across GetFlightInfo → DoGet so the BE can fetch external-table splits; this PR bounds how long an abandoned session can hold it)
Problem Summary:
A Flight SQL session idles under the same
wait_timeoutas a MySQL connection (28800s by default). Since #64799 the coordinator of a BE-served Flight query is kept alive until the session's next query or its close, and that coordinator holds the query's workload-group queue slot andactive_queriesregistration. Most Flight clients open a session per query and never sendCloseSession(the ADBC/JDBC drivers don't do it on connection close, as the comment inDorisFlightSqlProducer.closeSessionalready notes), so each abandoned session pins one slot for up to eight hours.Observed on a 2-FE cluster with a
max_concurrency=8workload group: eight finished Flight queries sat ininformation_schema.active_queriesas RUNNING for over an hour, and every later query in the group failed withquery queue timeout, timeout: 60000 ms. Killing the idle sessions (SHOW PROCESSLISTrows withHost = 0.0.0.0:0) released the slots immediately. Full write-up in #67503.This PR adds a mutable FE config,
arrow_flight_session_idle_timeout_second(default 3600), and applies it in the existing connection timeout checker toARROW_FLIGHT_SQLcontexts only:The exec-timeout floor is deliberate: a Flight session is
COM_SLEEPwhile the client drains the result via DoGet, and its idle clock runs from the query's start, so a bound belowquery_timeoutwould kill a long result stream before the query's own timeout could.0disables the bound. MySQL-protocol connections are unchanged; the kill log line now reports the effective idle timeout.Verified on a test cluster with the bound set to 8s: an abandoned Flight session and its
active_queriesrow were gone within 14s and the kill was logged asconnection type: ARROW_FLIGHT_SQL, ... idle timeout: 8; an idle MySQL connection survived the same window;0left the Flight session alive.Release note
Arrow Flight SQL sessions get their own idle timeout (
arrow_flight_session_idle_timeout_second, default 1h), so an abandoned Flight session no longer holds its query's workload-group queue slot untilwait_timeout.Check List (For Author)
Test
FlightSqlSessionIdleTimeoutTest: the bound tightens but never widenswait_timeout, the exec-timeout floor,0disables, MySQL context untouched)Behavior changed:
wait_timeout(8h); setarrow_flight_session_idle_timeout_second=0for the previous behavior.Does this need documentation?
arrow_flight_session_idle_timeout_second— doris-website PR to follow once this is reviewed.Check List (For Reviewer who merge this PR)