Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
*
!build/flow.phar
36 changes: 26 additions & 10 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,10 +1,21 @@
# Stage 1: Build stage
ARG FLOW_PHP_VERSION=8.3.28
ARG FLOW_BASE_IMAGE_TAG_SUFFIX=cli-bookworm
ARG FLOW_BASE_IMAGE_TAG=${FLOW_PHP_VERSION}-${FLOW_BASE_IMAGE_TAG_SUFFIX}
ARG FLOW_BASE_IMAGE=php:${FLOW_BASE_IMAGE_TAG}
ARG FLOW_PHP_VERSION=8.5.8
ARG FLOW_DEBIAN_SUITE=trixie
ARG FLOW_BASE_IMAGE=php:${FLOW_PHP_VERSION}-cli-${FLOW_DEBIAN_SUITE}

FROM ${FLOW_BASE_IMAGE} AS builder
# Stage 1: Shared base so builder and final stage resolve libpq from the same apt source.
FROM ${FLOW_BASE_IMAGE} AS base

ARG FLOW_DEBIAN_SUITE
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates gnupg curl \
&& curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \
| gpg --dearmor -o /usr/share/keyrings/pgdg.gpg \
&& echo "deb [signed-by=/usr/share/keyrings/pgdg.gpg] http://apt.postgresql.org/pub/repos/apt ${FLOW_DEBIAN_SUITE}-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
&& apt-get purge -y --auto-remove gnupg \
&& rm -rf /var/lib/apt/lists/*

# Stage 2: Build stage
FROM base AS builder

# Install dependencies and PHP extensions
RUN apt-get update && apt-get install -y --no-install-recommends \
Expand All @@ -18,6 +29,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libgmp-dev \
libpq-dev \
libsqlite3-dev \
libzip-dev \
default-libmysqlclient-dev \
libprotobuf-dev \
libprotobuf-c-dev \
Expand All @@ -26,7 +38,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
clang \
libclang-dev \
&& rm -rf /var/lib/apt/lists/* \
&& docker-php-ext-install bcmath gmp pdo_mysql pdo_pgsql pdo_sqlite \
&& docker-php-ext-install bcmath gmp pdo_mysql pdo_pgsql pdo_sqlite pgsql zip \
&& pecl install protobuf \
&& docker-php-ext-enable protobuf \
&& curl -L https://github.com/php/pie/releases/latest/download/pie.phar -o /usr/local/bin/pie \
&& chmod +x /usr/local/bin/pie \
&& php /usr/local/bin/pie install kjdev/brotli \
Expand All @@ -38,15 +52,17 @@ RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"

RUN php /usr/local/bin/pie install flow-php/arrow-ext:1.x-dev \
&& php /usr/local/bin/pie install flow-php/pg-query-ext:1.x-dev
&& php /usr/local/bin/pie install flow-php/pg-query-ext:1.x-dev \
&& php /usr/local/bin/pie install flow-php/flow-php-ext:1.x-dev

# Stage 2: Final Image
FROM ${FLOW_BASE_IMAGE} AS flow
# Stage 3: Final Image
FROM base AS flow

# Install runtime libraries needed by extensions
RUN apt-get update && apt-get install -y --no-install-recommends \
libpq5 \
libgmp10 \
libzip5 \
&& rm -rf /var/lib/apt/lists/*

# Copy the built extensions from the builder stage
Expand Down
6 changes: 3 additions & 3 deletions documentation/components/extensions/arrow-ext.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The Arrow ecosystem provides high-performance implementations for common data op
Parquet, CSV, JSON, and Arrow IPC — in C++, Rust, Java, Python, and other languages.

This extension brings the [Arrow Rust ecosystem](https://github.com/apache/arrow-rs) into PHP via
[ext-php-rs](https://github.com/davidcole1340/ext-php-rs). It exposes Arrow's native readers and writers through
[ext-php-rs](https://github.com/extphprs/ext-php-rs). It exposes Arrow's native readers and writers through
PHP streaming interfaces, letting PHP applications benefit from Rust-level performance without leaving the PHP runtime.

> [!TIP]
Expand Down Expand Up @@ -316,7 +316,7 @@ make test

## Architecture

- Built with [ext-php-rs](https://github.com/davidcole1340/ext-php-rs), which generates PHP bindings from Rust code
- Built with [ext-php-rs](https://github.com/extphprs/ext-php-rs), which generates PHP bindings from Rust code
- Uses Apache Arrow and Parquet Rust crates from the [Arrow ecosystem](https://github.com/apache/arrow-rs)
- All compression codecs compiled into the extension — no external PHP compression extensions needed
- PHP streaming interfaces (`RandomAccessFile`, `OutputStream`) called from Rust via ext-php-rs callbacks
Expand All @@ -326,6 +326,6 @@ make test
## See Also

- [parquet library](/documentation/components/libs/parquet.md)
- [ext-php-rs](https://github.com/davidcole1340/ext-php-rs)
- [ext-php-rs](https://github.com/extphprs/ext-php-rs)
- [Apache Arrow Rust](https://github.com/apache/arrow-rs)
- [Nix Development Environment](/documentation/contributing/nix.md)
381 changes: 365 additions & 16 deletions documentation/contributing/wasm.md

Large diffs are not rendered by default.

76 changes: 72 additions & 4 deletions documentation/installation/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

[DOC_LINK:../installation.md]

Since some of the Flow adapters require additional PHP extensions, we have prepared a Docker image with all the necessary dependencies.
Since some of the Flow adapters require additional PHP extensions, we have prepared a Docker image with all the
necessary dependencies.

```shell
$ docker pull ghcr.io/flow-php/flow:latest
Expand Down Expand Up @@ -42,11 +43,78 @@ Now you can use Flow CLI as follows:
flow --help
```

If you would like to try Flow, fork this repository, navigate to it through command line interface and execute following command:
## Running a pipeline

Write a pipeline file that *returns* a `DataFrame` — `run` executes it for you, so do not call
`->run()` yourself:

```php
<?php

use function Flow\ETL\DSL\{data_frame, from_array, to_output};

return data_frame()
->read(from_array([
['id' => 1, 'name' => 'User 01', 'active' => true],
['id' => 2, 'name' => 'User 02', 'active' => false],
]))
->write(to_output(truncate: false));
```

Save it as `pipeline.php` and run it through the image:

```shell
$ docker run -v $(pwd):/flow-workspace --rm -it flow-php/flow:latest run /flow-workspace/examples/topics/aggregations/daily_revenue.php
$ docker run -v $(pwd):/flow-workspace --rm -it ghcr.io/flow-php/flow:latest run /flow-workspace/pipeline.php
```

Flow CLI will grab the pipeline definition from `examples/topics/aggregations/daily_revenue.php` file and execute it.
## Bundled extensions

Alongside PHP 8.5 and the extensions Flow's adapters need — `bcmath`, `gmp`, `pdo_mysql`,
`pdo_pgsql`, `pdo_sqlite`, `pgsql`, and the `brotli`, `lz4`, `snappy`, `zstd` codecs — the image ships four extensions
that Flow detects and uses automatically:

| Extension | Package | Effect when loaded |
|------------|-------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
| `flow_php` | [flow-php/flow-php-ext](/documentation/components/extensions/flow-php-ext.md) | `AdaptiveRowHydrator` and `AdaptiveFloeEncoder` run native, fusing every Floe read/write and every raw-scalar hydration into one native call per batch |
| `arrow` | [flow-php/arrow-ext](/documentation/components/extensions/arrow-ext.md) | `AdaptiveParquetEngine` selects `ArrowParquetEngine`, so Parquet reads and writes run native |
| `pg_query` | [flow-php/pg-query-ext](/documentation/components/extensions/pg-query-ext.md) | `Flow\PostgreSql\Parser` becomes usable at all — SQL parsing, normalization and AST manipulation |
| `protobuf` | `pecl/protobuf` | `Flow\PostgreSql\Parser` decodes the parse tree in C instead of pure PHP — measured ~69x faster end to end |

`pdo_pgsql` and `pgsql` link libpq 18 from the PGDG repository, matching the PostgreSQL 18 grammar
`pg_query` is built against. PHP 8.5 additionally compiles in `lexbor`, `uri` and Zend OPcache unconditionally.

> [!NOTE]
> `Parser::parse()` decodes a protobuf AST, and protobuf caps message nesting at 100 levels — roughly 23 levels of
> nested subqueries. Deeper SQL fails to decode regardless of whether `protobuf` is loaded; the extension changes
> speed, not that ceiling.

### Opting out of the native path

Engine selection happens per read and per write, so a single step can be pinned to the PHP implementation:

```php
<?php

use Flow\ETL\Row\PhpRowHydrator;
use Flow\Floe\FloeEngine;
use Flow\Parquet\Engine\PhpParquetEngine;

use function Flow\ETL\Adapter\Parquet\from_parquet;
use function Flow\ETL\DSL\{config_builder, data_frame};
use function Flow\Floe\DSL\to_floe;

return data_frame(config_builder()->hydrator(new PhpRowHydrator()))
->read(from_parquet(__DIR__ . '/input.parquet', engine: new PhpParquetEngine()))
->write(to_floe(__DIR__ . '/output.floe', engine: FloeEngine::php));
```

To take the whole container off one native path, mount an empty file over that extension's ini:

```shell
$ docker run --rm -v /dev/null:/usr/local/etc/php/conf.d/docker-php-ext-flow_php.ini \
-v $(pwd):/flow-workspace ghcr.io/flow-php/flow:latest run /flow-workspace/pipeline.php
```

Once a native engine is selected it does **not** silently degrade. Extension failures surface as
`Flow\Floe\Exception\ExtensionException`, which `FloeReader` and `FloeWriter` wrap as
`Flow\Floe\Exception\FloeException`.
2 changes: 1 addition & 1 deletion src/extension/arrow-ext/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Extension: Arrow

A Rust-powered PHP extension for reading and writing Apache Parquet files using [ext-php-rs](https://github.com/davidcole1340/ext-php-rs)
A Rust-powered PHP extension for reading and writing Apache Parquet files using [ext-php-rs](https://github.com/extphprs/ext-php-rs)
and the official Apache Arrow/Parquet Rust crates. This extension provides streaming read/write interfaces for Parquet files
with support for all flat Arrow types, nested types (LIST, STRUCT, MAP), and multiple compression algorithms
(SNAPPY, GZIP, BROTLI, ZSTD, LZ4_RAW).
Expand Down
6 changes: 5 additions & 1 deletion src/extension/pg-query-ext/ext/pg_query.c
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,11 @@ PHP_MINIT_FUNCTION(pg_query)

PHP_MSHUTDOWN_FUNCTION(pg_query)
{
pg_query_exit();
/* Deliberately no pg_query_exit(). It frees libpg_query's TopMemoryContext, but pg_query_init()
* guards on a flag it never resets, so the contexts are gone and cannot be validly rebuilt.
* SAPIs that start the engine more than once per process -- embed, which the WASM playground
* uses -- then crash on the second startup. MSHUTDOWN runs at process teardown, so freeing
* here bought nothing anyway. */
return SUCCESS;
}

Expand Down
2 changes: 1 addition & 1 deletion wasm/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
*.phar.br
manifest.json
/libxml2-*
/libpg_query
/libpg_query-*
/php-ext-snappy
/libzip-*
# Build logs
Expand Down
Loading
Loading