Adds support for zstd compressed parquet files#29
Conversation
|
Thanks for your PR. Could you add some external sources for verifying interoperability? (At least reading) |
Two follow-ups to the initial zstd support: 1. ParquetWriter could not actually emit zstd files. DataFactory's CompressionMethodToCodec table was missing the Zstd entry, so GetThriftCompression() threw "codec '6' is not supported" the moment any caller set CompressionMethod::Zstd. The read path worked because DataStreamFactory has its own codec → method map. 2. ZstdStreamWrapper had three different "default level" values (const = 6, property = 9, stream_open fallback = 6) and a docblock that described gzip's 1-9 range instead of zstd's 1-22. Everything now flows from DEFAULT_COMPRESSION_LEVEL, raised to 9 (small write-time cost, slightly better ratio on the per-page chunks parquet produces; higher levels show diminishing returns until ~15 because each page is compressed independently).
|
There are some issues in this PR. I'm very interested in having this integrated so took the liberty to fix them. @gms8994 I've created a PR to your branch with the fixes, namely write to ZSTD didn't work and the compression level details were all out of whack: I was able to dump some data from DuckDB in ztd compression and read it in successfully. And in addition write some data out that was read by duckdb. Here's the simple sample from DuckDB if its relevant (had to zip it as github rejected .parquet): I can open a separate PR directly here if don't hear back from the original PR author. |
Fix zstd write path and clean up compression-level defaults
|
@serpentblade that's been merged on my side and the PR is up to date here - @Katalystical any chance we can get another review? |
3 participants
This replicates much of the functionality that exists via the snappy compression to use zstd compression.