Tests for parquet filter pushdown project.

[rhou1]

Hi Rahul,

Can you review these?

Thanks.

--Robert

[rchallapalli]

Review still in progress....

At a glance, the tests look good. I am curious to know how you generated the baselines. If you manually verified the baselines, then I would express some concern and would request you to use mysql or postgres to generate them.

[rhou1]

I manually verified the baselines. The tests were designed to generate certain answers so that I could verify them. So for example, I created a test that would return 3020 rows, and it returned 3020 rows.

[rchallapalli]

The tests look good Robert. Once we generate the baselines using hive/mysql, we should be ready to commit (after the running the regression test again after re-basing onto latest framework master). Below are a few observations while looking at the tests

1. Did we test the flag"planner.store.parquet.rowgroup.filter.pushdown.threshold" ? This is present in our documentation
2. Metadata Caching : If we have a table with 4 rowgroups and we added one new parquet file to the table folder.....do we update the metadata cache and then consider the row groups from the newly added file? Any tests surrounding this usecase?
3. CTAS auto-partitioned datasets : Its hard to say whether any of your data sets are auto-partitioned. Do we have tests around this?
4. The documentation says we support '<>' for parquet filter pushdown. However I couldn't find any tests. Did I miss something obvious?
5. Do we have files with multiple row groups and tests around that?