Skip to content

Performance: fast-path top-level flat scalar fields in ArrayToDataColumnsConverter#32

Open
serpentblade wants to merge 2 commits into
codename-hub:masterfrom
One-Learning-Community:feature/flat-path-performance
Open

Performance: fast-path top-level flat scalar fields in ArrayToDataColumnsConverter#32
serpentblade wants to merge 2 commits into
codename-hub:masterfrom
One-Learning-Community:feature/flat-path-performance

Conversation

@serpentblade

Copy link
Copy Markdown

Summary

ArrayToDataColumnsConverter::processData dispatches every field through
processDataRecursively — a static method with 8 parameters and several
conditionals, designed to handle the full range of parquet's nesting,
repetition, and definition-level semantics. For nested types (lists,
maps, structs) this is necessary. For top-level flat scalar fields —
which dominate most real-world tabular schemas (DB dumps,
CSV→parquet, analytics events) — the recursive machinery is
~10× more expensive than a tight foreach over the row buffer.

This PR adds a fast path at the top of processData that handles flat
scalar fields directly and falls through to the existing recursive code
for anything more complex. Output is byte-identical; the change is
purely additive.

Motivation

Measured against three production tabular exports of varying column-count
and fast-path fit

Table Rows Cols Fast-path % Before After Speedup
~34 cols, mixed 686K 34 62% 77 s 69 s 1.1×
~44 cols, mostly scalar 401K 44 84% 67 s 57 s 1.2×
~430 cols, mostly scalar+bool 249K 430 97% 180 s 109 s 1.65×

The wide-column case (typical for any export that pre-computes
boolean features per row) sees the largest win. Narrower tables
benefit proportionally to their fast-path fit. No table regressed
under matched cache conditions.

At the converter level alone the savings are even sharper:

Benchmark: ArrayToDataColumnsConverter::toDataColumns
           5000 rows × 410 boolean columns

  Before:  1.046 s
  After:   0.133 s     (7.9× faster)

  Projected for 50 row groups (250K rows):
    Before:  52.3 s
    After:    6.7 s    (saves ~46 s on this dimension alone)

The remaining wall-clock savings come from downstream effects (smaller
buffer sizes, fewer GC cycles, faster per-row-group setup).

The patch

Adds a fast path gated on a single check:

$isFlatScalar =
  count($path) === 1
  && !$df->isArray
  && empty($options['nullable_levels'])
  && empty($options['repeated_levels']);

These four conditions together mean: top-level field (not inside a
struct/map/list), not declared as an array, and no list/map ancestors.
Under those conditions the inner work reduces to:

foreach ($this->data as $row) {
    $v = $row[$colname] ?? null;
    if ($v === null) {
        if (!$hasNulls) throw new \Exception('Malformed data: null value in not-nullable field');
        $data[] = null;
        $definitionLevels[] = 0;
    } else {
        $data[] = $v;
        if ($hasNulls) $definitionLevels[] = 1;
    }
}

— a direct equivalent of what the recursive code computes for this
case, without the per-call overhead.

Correctness

The fast path produces output provably equivalent to the recursive
code for the gated case:

Original behavior Fast path
$data[] populated with $row[$colname] Same
For hasNulls=true: definition level 1 for value, 0 for null Same
For hasNulls=false: definition levels omitted (since maxDefinitionLevel === 0) Same
Repetition levels omitted Same
Exception text 'Malformed data: null value in not-nullable field' for null in non-nullable Same exact text

Tested via round-trip cases:

  • Non-nullable flat scalars (int, string, boolean): byte-identical write→read
  • Nullable scalars with null/value mix: nulls preserved correctly
  • Non-nullable field receiving null: same exception thrown
  • Mixed schema with a ListField: scalar field hits fast path, list still
    goes through existing recursive code unchanged

Reproducible benchmark

A self-contained benchmark script that runs against a 20K-row × 100-column
synthetic schema intermixing integer / bigint / float / string / boolean /
timestamp / list types is included in this PR (or as a gist if preferred):
benchmark-wide-schema.php.

Run it once on master, apply the patch, and run it again. On the
maintainer's machine the converter time should drop ~3× and end-to-end
write time ~1.5×; file output is byte-identical pre/post patch:

Unpatched (master):
  ArrayToDataColumnsConverter::toDataColumns       1.450 s
  End-to-end ParquetDataWriter → php://memory      2.647 s
  file size                                        1.8 MiB

Patched:
  ArrayToDataColumnsConverter::toDataColumns       493.6 ms
  End-to-end ParquetDataWriter → php://memory      1.763 s
  file size                                        1.8 MiB

Compatibility

  • No public API changes.
  • No new dependencies.
  • Same minimum PHP version.
  • Works against the existing test fixtures

…verter

For top-level flat scalar fields (no nesting, no repetition) the existing
recursive processDataRecursively() pays O(rows) PHP function-call overhead
per field. On wide tabular schemas — a common shape for DB exports,
CSV-to-parquet, and analytics events — this dominates the converter cost.

This change adds an early return at the top of processData() that handles
the gated case directly with a tight foreach:

  count($path) === 1
  && !$df->isArray
  && empty($options['nullable_levels'])
  && empty($options['repeated_levels'])

Anything more complex (lists, maps, structs, nested fields) falls through
to the existing recursive path with zero behavior change. Output bytes
are identical pre/post for the gated case; the same
"Malformed data: null value in not-nullable field" exception is raised
for the null-in-non-nullable case.

Measured against a 20K rows x 100 columns intermixed-type synthetic
schema (see benchmark-wide-schema.php in the follow-up commit):

  ArrayToDataColumnsConverter::toDataColumns: 1.450s -> 493ms  (~2.9x)
  End-to-end ParquetDataWriter:               2.647s -> 1.763s (~1.5x)
  File size:                                  1.8 MiB (byte-identical)

Real-world payoff on a 250K-row x 430-column boolean-heavy export: 180s -> 109s.
Self-contained script: 20,000 rows x 100 columns intermixing integer,
bigint, float, string, boolean, timestamp, and list types. Times the
converter alone and end-to-end ParquetDataWriter. Includes one list
column so the existing recursive path stays exercised — useful as a
no-regression check.

Output is deterministic so parquet bytes are byte-stable across runs;
the file size in the benchmark output can be compared pre/post patch
to confirm output is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant