[GLUTEN-12924][VL] Defer literal node construction for InSet to reduce memory - #12925
Open
WangGuangxin wants to merge 1 commit into
Open
[GLUTEN-12924][VL] Defer literal node construction for InSet to reduce memory#12925WangGuangxin wants to merge 1 commit into
WangGuangxin wants to merge 1 commit into
Conversation
…mory footprint Keep raw IN-set values and the value data type on SingularOrListNode and build the substrait LiteralNode options lazily in toProtobuf(), instead of eagerly materializing one LiteralNode per value. This avoids holding many LiteralNode objects in memory for large IN lists. - Add ExpressionBuilder.makeSingularOrListNode(value, rawValues, dataType) and a matching SingularOrListNode constructor; build options from raw values lazily. - InSetTransformer passes sorted raw hset values (deterministic) and defers LiteralNode creation to protobuf serialization.
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
This PR reduces driver-side memory usage when converting large
InSetexpressions to Substrait.Before this change,
InSetTransformereagerly materialized one SubstraitLiteralNodefor each element in the IN set and stored all of them inSingularOrListNode. For large IN lists, this creates a largenumber of intermediate objects and increases retained heap usage before protobuf serialization.
This PR changes the flow as follows:
InSetTransformerkeeps the raw IN-set values together with the child data type, instead of eagerly building oneLiteralNodeper value.SingularOrListNodeconstructs the corresponding Substrait literal protobuf entries lazily intoProtobuf().List<Expression>.Literal(_, valueType).toString()logic as before.In short, this patch reduces the number of long-lived intermediate objects created for large
InSetexpressions and also avoids an extra temporary protobuf list during serialization.Why are the changes needed?
Queries with very large
IN (...)predicates may be optimized by Spark intoInSet. In Gluten's Substrait conversion path, eagerly creating and retaining oneLiteralNodeper IN-set value can cause high memorypressure on the driver.
The issue is especially visible for large IN lists, where the conversion step itself may consume much more memory than necessary even though the final serialized representation is only needed at protobuf
generation time.
By deferring literal node construction until
toProtobuf(), we reduce retained heap usage in the expression tree and lower peak allocation pressure in the conversion path.How was this patch tested?
Manually.
Was this patch authored or co-authored using generative AI tooling?
No.
Related issue: #12924