[GLUTEN-12897][VL][TEST] Run seven UDF and two ANSI SQL query test files on Spark 3.5/4.0/4.1 - #12890
Merged
jackylee-ch merged 3 commits intoAug 31, 2026
Conversation
… two dead entries
|
Run Gluten Clickhouse CI on x86 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Velox SQL query test allowlists for Spark 3.4, 3.5, 4.0, and 4.1 so that nine previously Spark-3.3-only .sql query test files continue to run after gluten-ut/spark33 is removed.
Changes:
- Add nine previously Spark-3.3-only SQL query tests to
SUPPORTED_SQL_QUERY_LISTin the Spark 3.4/3.5/4.0/4.1 Velox settings. - Remove two Spark 3.4 entries (
udf-union.sql,udf-window.sql) from the supported list.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala | Adds the nine missing SQL tests to the Spark 3.4 Velox supported list; also removes two UDF entries. |
| gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala | Adds the nine missing SQL tests to the Spark 3.5 Velox supported list. |
| gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala | Adds the nine missing SQL tests to the Spark 4.0 Velox supported list. |
| gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxSQLQueryTestSettings.scala | Adds the nine missing SQL tests to the Spark 4.1 Velox supported list. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
203
to
206
| "union.sql", | ||
| "using-join.sql", | ||
| "window.sql", | ||
| "udf-union.sql", | ||
| "udf-window.sql", | ||
| "ansi/cast.sql", |
|
Run Gluten Clickhouse CI on x86 |
LuciferYang
changed the title
LuciferYang
marked this pull request as draft
August 25, 2026 12:53
Comment on lines
+212
to
+213
| "ansi/datetime-special.sql", | ||
| "ansi/parse-schema-string.sql", |
LuciferYang
marked this pull request as ready for review
August 25, 2026 14:14
LuciferYang
marked this pull request as draft
August 25, 2026 14:14
|
Run Gluten Clickhouse CI on x86 |
LuciferYang
changed the title
LuciferYang
marked this pull request as ready for review
August 26, 2026 04:58
Contributor
Author
|
cc @jackylee-ch this one is ready to go |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nine
.sqlfiles run ongluten-ut/spark34but on none ofspark35,spark40,spark41. This adds them where they are missing and where the file still exists.udf/filesThe two
ansi/files are spark35 only. Spark 4.0 removed theansi/input directory inb6265286eca([SPARK-50313][SQL][TESTS] Enable ANSI in SQL *SQLQueryTestSuite by default): with ANSI on by default those files moved to the top level and anonansi/directory appeared for the other variants. Soansi/datetime-special.sqlandansi/parse-schema-string.sqldo not exist on 4.0 and 4.1, and both modules' lists already cover the same content under the 4.0 names,datetime-special.sqlandparse-schema-string.sqlplus the twononansi/entries.Evidence
GlutenSQLQueryTestSuitecarries@ExtendedSQLTest, so it runs in thespark-test-sparkNN-slow (extended)jobs, not in the numbered groups, which exclude that tag. From those jobs' surefire reports on this PR:udf/files ran, Scala UDF passedudf/files ran, Scala UDF passedThe Regular Python UDF and Scalar Pandas UDF variants report as skipped because the CI image has no pyspark, pandas or pyarrow, and the
_analyzer_testvariants on 4.0/4.1 are gated separately. Both predate this change.An earlier revision of this PR also added the two
ansi/entries to spark40 and spark41; the run above is what showed they produce no test there, which is how the 4.0 directory change came to light.How this PR changed shape
It opened claiming these nine would stop running when
gluten-ut/spark33is removed under #12807, and it also deleted two entries from spark34's list as unreachable. Both were wrong, and the reason is #12897: spark34 matches the supported list withcontainswhile 3.5+ match with==. So spark34's short entries (count.sql,cross-join.sql,having.sql,inner-join.sql,join-empty-relation.sql,natural-join.sql,outer-join.sql,datetime-special.sql,parse-schema-string.sql) already enable these files there, and"udf-union.sql"was not a dead entry either.I ran the suite against Spark 3.4.4 with Velox twice, once with the nine entries added to spark34 and once without: both give 222 tests, 222 succeeded, 0 failed, 84 ignored, with the nine present either way. The spark34 half was a no-op, so it is gone and the two deletions are reverted. What remains is an extension rather than a migration, and removing
gluten-ut/spark33does not depend on it.Related: #12897, #12863, #12807.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5