Skip to content

Remote py apis - #2675

Draft
shivamka1 wants to merge 128 commits into
db_v4from
remote-py-apis
Draft

Remote py apis#2675
shivamka1 wants to merge 128 commits into
db_v4from
remote-py-apis

Conversation

@shivamka1

@shivamka1 shivamka1 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Remote Python APIs

Adds a Python RemoteGraph client that is a drop-in replacement for the local
Graph API, backed by a typed transport over GraphQL. Local code migrates by
swapping the constructor — method names, signatures, and semantics match the
local API; server-only extras are additive.

Architecture

  • Transport trait — the client core is execute(&Op) -> Result<Option<Prop>>
    over a typed operation tree (Op = Read(ReadExpr) | Write(WriteOp)). The
    GraphQL rendering lives behind the trait; nothing above it builds query strings.
  • Lazy views, one RPC per terminal — view chains (window, layers,
    at, filters, …) accumulate in a ReadExpr tree and fire a single request
    when a terminal (degree(), count(), collect(), …) is reached.
  • Static queries + JSON variables — all GraphQL is static text with
    serde-serialized variables (no string interpolation or templates). The serde
    impls on the wire types are the single source of truth for the format.
  • View-faithful collection materializationcollect() returns handles
    that replay the parent's op log (views + filters), so members evaluate under
    the same view chain as the collection they came from.

Unified filtering

  • One filter(expr: GqlFilter) field on Graph/Node/Edge and all collections.
    GqlFilter is a oneOf input: nodes, edges, graph, and, or, not,
    plus flat graph-view spellings (window, at, layers, snapshotAt, …).
  • and is intersection, matching local g.filter(a & b); cross-kind or
    produces a proper graph union.
  • Python filter expressions (including graph-view chains and mixed node/edge
    compositions like filter.Graph.window(1, 4) & (filter.Node.property("x") > 1))
    are exported from core as a kind-tagged FilterTree and carried to the wire
    without loss.
  • Empty and/or lists are rejected — previously an empty or matched
    everything, which mattered for persisted access filters.

Wire fidelity

  • Exact dtypes — the server exposes a round-trippable PropertyType
    scalar and dtype fields; the client decodes stored values type-directed,
    so numeric widths (U8 stays U8), datetimes, and decimals survive the
    wire exactly. Older servers without dtype fall back to shape-based decoding.
  • Non-finite floats — NaN/±Infinity round-trip via tagged
    f64Special/f32Special input variants and string sentinels on output
    (previously: error on write, silent null on read).
  • Map orderingProp::Map is now insertion-ordered (IndexMap) and
    the wire preserves key order end to end, matching local behavior.

Performance

  • Response payloads are moved, not deep-copied, on every read RPC.
  • Edge.event(...) pins events via indexed layer/time views instead of
    scanning the full exploded history.
  • Client handles share state via Arc; nested columnar reads batch per
    source; history slicing fetches pages rather than full histories.

Breaking changes

  • filterNodes / filterEdges are removed from the GraphQL schema.
    Migration is mechanical and can preserve response shapes with an alias:
    filterNodes(expr: X)filterNodes: filter(expr: { nodes: X }).
  • Persisted filters using the legacy node/edge keys still load (serde
    aliases); hand-written GraphQL must use nodes/edges.
  • PropertyOutput emits "NaN"/"Infinity"/"-Infinity" strings for
    non-finite floats where it previously emitted null.

Build fix

pyo3/extension-module is now pinned under [tool.maturin] — without it the
wheel links libpython and import raphtory segfaults on statically-linked
interpreters (e.g. conda).

Testing

  • Local-vs-remote parity e2e suite covering the full client surface
    (views, filters, collections, properties, write path), including key-order,
    NaN, and dtype round-trips.
  • GraphQL filter suites migrated to the unified field.
  • Rust unit tests for wire round-trips (filters, values, special floats).

Not in this PR (local APIs the remote client does not yet cover)

  • Rolling / expanding windowsrolling() and expanding() (and the
    WindowSet they return) are not available on any remote object. All other
    time views (window, at, before, after, snapshot_*, shrink_*)
    are supported.
  • Collection-level history and node states.history on collections
    (and .deletions on edge collections) are not exposed remotely; per-entity
    node.history / edge.deletions are. Same root gap: there is no remote
    NodeState, so collection accessors like nodes.degree() return plain lists
    rather than the sortable/groupable NodeState views the local API returns.
  • Algorithms — the raphtory.algorithms module (and
    Graph.largest_connected_component) operate on local graphs only.
  • Vectors / embeddingsGraph.vectorise and the raphtory.vectors
    module (server-side vector endpoints remain reachable via raw
    client.query(...) on vectors-enabled servers).
  • DataFrame / Parquet bulk IOload_nodes / load_edges /
    load_*_metadata, from_parquet / to_parquet. Batch remote writes exist
    (RemoteGraph.add_nodes / add_edges) but take update lists, not
    DataFrames.
  • Exportto_networkx, to_pyvis, and collection to_df.
  • Cross-graph import — the import_node(s) / import_edge(s) (_as)
    family, which copies entities from another in-process graph.
  • Materialization and flavour conversionmaterialize,
    materialize_at, event_graph, persistent_graph.
  • Local persistence (not applicable remotely)save_to_file /
    load_from_file / serialise / deserialise and the disk cache
    (cache_view, flush) are in-process concepts; whole-graph transfer is
    covered by RaphtoryClient.send_graph / receive_graph / upload_graph.
  • Minor surface gaps — columnar collection property views support
    get/keys/items/values remotely but not in/[]/iteration and lack
    the collection-level .temporal view; History.merge /
    compose_histories are unavailable on remote history handles.
  • Known limitationexplode_layers().collect() on remote edges raises
    Unsupported.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.

Benchmark suite Current: 2b624f9 Previous: 9823ef7 Ratio
lotr_graph/num_edges 4 ns/iter (± 0) 0 ns/iter (± 0) +∞
lotr_graph/num_nodes 5 ns/iter (± 0) 1 ns/iter (± 0) 5
lotr_graph/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph/graph_latest 3 ns/iter (± 0) 0 ns/iter (± 0) +∞
lotr_graph_materialise/materialize 8050631 ns/iter (± 40172) 1564816 ns/iter (± 35303) 5.14
lotr_graph_window_100/num_nodes 15 ns/iter (± 0) 5 ns/iter (± 0) 3
lotr_graph_window_100_materialise/materialize 8089774 ns/iter (± 64519) 1669150 ns/iter (± 10700) 4.85
lotr_graph_window_10/has_node_existing 143 ns/iter (± 9) 62 ns/iter (± 11) 2.31
lotr_graph_window_10_materialise/materialize 3345959 ns/iter (± 47014) 971980 ns/iter (± 4278) 3.44
lotr_graph_subgraph_10pc/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph_subgraph_10pc_materialise/materialize 2009904 ns/iter (± 22468) 334634 ns/iter (± 1287) 6.01
lotr_graph_subgraph_10pc_windowed/has_node_existing 149 ns/iter (± 9) 62 ns/iter (± 14) 2.40
lotr_graph_subgraph_10pc_windowed_materialise/materialize 1217703 ns/iter (± 11569) 230399 ns/iter (± 2617) 5.29
lotr_graph_window_50_layered/num_edges_temporal 155717 ns/iter (± 2232) 70121 ns/iter (± 7586) 2.22
lotr_graph_window_50_layered/has_node_existing 390 ns/iter (± 24) 129 ns/iter (± 12) 3.02
lotr_graph_window_50_layered/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph_window_50_layered/graph_latest 85415 ns/iter (± 2064) 36649 ns/iter (± 916) 2.33
lotr_graph_window_50_layered_materialise/materialize 30515575 ns/iter (± 95819) 3488825 ns/iter (± 24948) 8.75
lotr_graph_persistent_window_50_layered/num_edges_temporal 646047 ns/iter (± 24446) 192686 ns/iter (± 1569) 3.35
lotr_graph_persistent_window_50_layered/has_node_existing 442 ns/iter (± 435) 174 ns/iter (± 83) 2.54
lotr_graph_persistent_window_50_layered/has_node_nonexisting 5 ns/iter (± 0) 2 ns/iter (± 0) 2.50
lotr_graph_persistent_window_50_layered/iterate_exploded_edges 3480647 ns/iter (± 10244) 1659940 ns/iter (± 19402) 2.10
lotr_graph_persistent_window_50_layered/graph_latest 137704 ns/iter (± 4154) 57549 ns/iter (± 4809) 2.39
lotr_graph_persistent_window_50_layered_materialise/materialize 53184087 ns/iter (± 282815) 5298035 ns/iter (± 147912) 10.04

This comment was automatically generated by workflow using github-action-benchmark.

github-actions Bot and others added 20 commits July 3, 2026 22:37
…Degree/OutDegree/Name) with Python bindings + tests
…ypes, ExcludeNodes) + Valid, DefaultLayer, and graph Path/Namespace/Name terminals

@ljeub-pometry ljeub-pometry left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness:

  • Make sure property types match between remote/local apis (need to pass along the data type to cast the output correctly)
  • Filter apis in graphql need to be aligned with rust/python such that all filters pass through the remote client correctly
  • get_dtype_of needs to return PropType, not String
  • Change the inner type of Prop::Map to IndexMap, so we don't have to worry about scrambling the order everywhere

Performance:

  • A lot of String allocations when building the queries can be avoided by passing in a mutable String as a buffer
  • event and event_layer on edges need to be implemented efficiently as part of EdgeViewOps instead of doing a linear search over the exploded edges

Tidy:

  • the client modules can be tidied up a bit
  • a lot of unnecessary manual conversion to Py<PyAny>> in the python apis

Comment thread python/tests/test_base_install/test_graphql/test_remote_graph_transport.py Outdated
Comment thread python/tests/test_base_install/test_graphql/test_remote_graph_transport.py Outdated
Comment thread python/tests/test_base_install/test_graphql/test_remote_graph_transport.py Outdated
Comment thread raphtory-graphql/src/client/graphql_transport.rs Outdated
// The two view types share an identical method surface and pivot logic; the
// only difference is which client handle (and thus which container) they wrap.
// A macro keeps the two `#[pymethods]` blocks in lockstep without duplication.
macro_rules! columnar_view_methods {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems to be missing get_dtypes_of

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no get_dtypes_of on the local collection views either, only per-entity Properties.get_dtype_of, which the remote now mirrors (returning a real PropType).

Comment thread raphtory-graphql/src/python/client/remote_metadata.rs Outdated
Comment thread raphtory-graphql/src/python/client/remote_metadata.rs Outdated
Comment thread raphtory-graphql/src/python/client/remote_metadata.rs
Comment thread raphtory-graphql/src/python/client/remote_nested_edges.rs Outdated
shivamka1 and others added 29 commits August 4, 2026 11:11
…add items() for pairs; drop the RemoteProperty wrapper
…p: ViewOp }: one vocabulary, data-driven render/parse, ctx inspectable
Co-authored-by: Shivam <4599890+shivamka1@users.noreply.github.com>
… everything — a fail-open for stored access filters
…accept string GIDs in id ordering, reject degree op-chains early, normalize Layer::None/All
…rsion; legacy node/edge keys still load via serde aliases
…e/Edge/collections; remove filterNodes/filterEdges; collapse the client's six filter ops into one
…GraphQL shape produced only at the transport boundary
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: arienandalibi <arienandalibi2@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Louis Chan <louisch@fastmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ben Steer <ben.steer@pometry.com>
…eir declared dtype so numeric widths, datetimes, and decimals survive the wire exactly
…stances via a layer view instead of scanning
…ple returns, engine filter types on select, new_graph returns the RemoteGraph handle, indexed edge event lookup
…stead of allocating a fresh string per recursion level
# Conflicts:
#	Cargo.lock
#	python/Cargo.toml
#	raphtory-graphql/src/auth.rs
#	raphtory-graphql/src/config/app_config.rs
#	raphtory-graphql/src/data.rs
#	raphtory-graphql/src/model/graph/graph.rs
#	raphtory-graphql/src/model/graph/mod.rs
#	raphtory-graphql/src/model/mod.rs
#	raphtory-graphql/src/server.rs
#	raphtory/src/errors.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants