[GLUTEN-12909][VL] Support HiveGenericUDTF in HiveUDFTransformer - #12910
[GLUTEN-12909][VL] Support HiveGenericUDTF in HiveUDFTransformer#12910WangGuangxin wants to merge 2 commits into
Conversation
062c437 to
c937d84
Compare
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
c937d84 to
d6eaf32
Compare
|
Run Gluten Clickhouse CI on x86 |
|
@philo-he @jinchengchenghh please review this when you are convenient |
philo-he
left a comment
philo-he
left a comment
There was a problem hiding this comment.
Looks good overall. Just one minor suggestion on the test. Please check if it makes sense. Thanks.
| @@ -53,6 +57,47 @@ class GlutenHiveUDFSuite | |||
| } | |||
| } | |||
|
|
|||
| testGluten("HiveGenericUDTF is recognized and mapped by HiveUDFTransformer") { | |||
There was a problem hiding this comment.
I assume such test is independent of the supported Spark versions, and we don't need to extend HiveUDFSuite for enabling this test. Then, can we move the new tests to a new test suite under gluten-ut/test to avoid duplicate?
Generally, gluten-ut/sparkxxx should hold the corresponding Spark UTs and their variants.
|
Run Gluten Clickhouse CI on x86 |
|
|
||
| @SlowHiveTest | ||
| class GlutenHiveUDFTransformerSuite | ||
| extends HiveUDFSuite |
There was a problem hiding this comment.
Can we remove extending HiveUDFSuite if this test don't really depend on it?
2 participants
What changes are proposed in this pull request?
This PR extends
HiveUDFTransformerto recognizeHiveGenericUDTFin addition toHiveSimpleUDFandHiveGenericUDF.Previously,
HiveUDFTransformeronly handled Hive simple UDFs and generic UDFswhen checking whether an expression is a Hive UDF and when extracting the Hive
function name and implementation class. As a result,
HiveGenericUDTFcould notbe recognized or mapped through the same transformer path.
This patch updates
HiveUDFTransformerso that:isHiveUDFreturns true forHiveGenericUDTFgetHiveUDFNameAndClassNamecan extract the function name and class name fromHiveGenericUDTFIn addition, this PR adds unit tests to verify that
HiveGenericUDTFis properlyrecognized, mapped, and converted through
HiveUDFTransformeracross supportedSpark versions.
How was this patch tested?
Added unit tests in
gluten-utfor Spark 3.3, 3.4, 3.5, 4.0, and 4.1 to verify:HiveGenericUDTFcan be found from the analyzed planHiveUDFTransformer.isHiveUDFrecognizes it correctlyHiveUDFTransformer.getHiveUDFNameAndClassNamereturns the expected functionname and implementation class
HiveUDFTransformer.replaceWithExpressionTransformercan map it to aGenericExpressionTransformerWas this patch authored or co-authored using generative AI tooling?
Generated-by: TraeCode GPT-5
Related issue: #12909