Remote py apis - #2675
Draft
shivamka1 wants to merge 128 commits into
Draft
Conversation
…ree read wired end-to-end
…ata, update_metadata, delete_edge) through Transport
…date_metadata) through Transport
…tadata) through Transport
…aph.node/edge to Rust methods (fixes view-chain preservation in Python)
Contributor
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rust Benchmark'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 2.
| Benchmark suite | Current: 2b624f9 | Previous: 9823ef7 | Ratio |
|---|---|---|---|
lotr_graph/num_edges |
4 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph/num_nodes |
5 ns/iter (± 0) |
1 ns/iter (± 0) |
5 |
lotr_graph/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph/graph_latest |
3 ns/iter (± 0) |
0 ns/iter (± 0) |
+∞ |
lotr_graph_materialise/materialize |
8050631 ns/iter (± 40172) |
1564816 ns/iter (± 35303) |
5.14 |
lotr_graph_window_100/num_nodes |
15 ns/iter (± 0) |
5 ns/iter (± 0) |
3 |
lotr_graph_window_100_materialise/materialize |
8089774 ns/iter (± 64519) |
1669150 ns/iter (± 10700) |
4.85 |
lotr_graph_window_10/has_node_existing |
143 ns/iter (± 9) |
62 ns/iter (± 11) |
2.31 |
lotr_graph_window_10_materialise/materialize |
3345959 ns/iter (± 47014) |
971980 ns/iter (± 4278) |
3.44 |
lotr_graph_subgraph_10pc/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_subgraph_10pc_materialise/materialize |
2009904 ns/iter (± 22468) |
334634 ns/iter (± 1287) |
6.01 |
lotr_graph_subgraph_10pc_windowed/has_node_existing |
149 ns/iter (± 9) |
62 ns/iter (± 14) |
2.40 |
lotr_graph_subgraph_10pc_windowed_materialise/materialize |
1217703 ns/iter (± 11569) |
230399 ns/iter (± 2617) |
5.29 |
lotr_graph_window_50_layered/num_edges_temporal |
155717 ns/iter (± 2232) |
70121 ns/iter (± 7586) |
2.22 |
lotr_graph_window_50_layered/has_node_existing |
390 ns/iter (± 24) |
129 ns/iter (± 12) |
3.02 |
lotr_graph_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_window_50_layered/graph_latest |
85415 ns/iter (± 2064) |
36649 ns/iter (± 916) |
2.33 |
lotr_graph_window_50_layered_materialise/materialize |
30515575 ns/iter (± 95819) |
3488825 ns/iter (± 24948) |
8.75 |
lotr_graph_persistent_window_50_layered/num_edges_temporal |
646047 ns/iter (± 24446) |
192686 ns/iter (± 1569) |
3.35 |
lotr_graph_persistent_window_50_layered/has_node_existing |
442 ns/iter (± 435) |
174 ns/iter (± 83) |
2.54 |
lotr_graph_persistent_window_50_layered/has_node_nonexisting |
5 ns/iter (± 0) |
2 ns/iter (± 0) |
2.50 |
lotr_graph_persistent_window_50_layered/iterate_exploded_edges |
3480647 ns/iter (± 10244) |
1659940 ns/iter (± 19402) |
2.10 |
lotr_graph_persistent_window_50_layered/graph_latest |
137704 ns/iter (± 4154) |
57549 ns/iter (± 4809) |
2.39 |
lotr_graph_persistent_window_50_layered_materialise/materialize |
53184087 ns/iter (± 282815) |
5298035 ns/iter (± 147912) |
10.04 |
This comment was automatically generated by workflow using github-action-benchmark.
…eld from RemoteGraph
…e impls from Py* types to op types
…Degree/OutDegree/Name) with Python bindings + tests
…)/.dst() navigation
…optional-string machinery
…ypes, ExcludeNodes) + Valid, DefaultLayer, and graph Path/Namespace/Name terminals
…ighbour accessors
… wrong answers under view chains
…est, layer, shrink_*, etc.)
…est, layer, shrink_*, etc.)
…p through base_graph on all Remote types
ljeub-pometry
requested changes
Aug 3, 2026
ljeub-pometry
left a comment
Collaborator
There was a problem hiding this comment.
Correctness:
- Make sure property types match between remote/local apis (need to pass along the data type to cast the output correctly)
- Filter apis in graphql need to be aligned with rust/python such that all filters pass through the remote client correctly
- get_dtype_of needs to return PropType, not String
- Change the inner type of
Prop::MaptoIndexMap, so we don't have to worry about scrambling the order everywhere
Performance:
- A lot of String allocations when building the queries can be avoided by passing in a mutable String as a buffer
eventandevent_layeron edges need to be implemented efficiently as part ofEdgeViewOpsinstead of doing a linear search over the exploded edges
Tidy:
- the client modules can be tidied up a bit
- a lot of unnecessary manual conversion to
Py<PyAny>>in the python apis
| // The two view types share an identical method surface and pivot logic; the | ||
| // only difference is which client handle (and thus which container) they wrap. | ||
| // A macro keeps the two `#[pymethods]` blocks in lockstep without duplication. | ||
| macro_rules! columnar_view_methods { |
Collaborator
There was a problem hiding this comment.
this seems to be missing get_dtypes_of
Collaborator
Author
There was a problem hiding this comment.
There's no get_dtypes_of on the local collection views either, only per-entity Properties.get_dtype_of, which the remote now mirrors (returning a real PropType).
…add items() for pairs; drop the RemoteProperty wrapper
…wer: remote_nested_edges.rs:39)
…p: ViewOp }: one vocabulary, data-driven render/parse, ctx inspectable
… Transport contract it implements
…ilding HashMaps per request
…anup; yield RemoteGraph
Co-authored-by: Shivam <4599890+shivamka1@users.noreply.github.com>
… everything — a fail-open for stored access filters
… combinator; regenerate schema
…both conversion directions
…accept string GIDs in id ordering, reject degree op-chains early, normalize Layer::None/All
…rsion; legacy node/edge keys still load via serde aliases
…e/Edge/collections; remove filterNodes/filterEdges; collapse the client's six filter ops into one
…GraphQL shape produced only at the transport boundary
…ient via a core FilterTree export
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: arienandalibi <arienandalibi2@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Louis Chan <louisch@fastmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ben Steer <ben.steer@pometry.com>
…eir declared dtype so numeric widths, datetimes, and decimals survive the wire exactly
…the full exploded history
…stances via a layer view instead of scanning
…ple returns, engine filter types on select, new_graph returns the RemoteGraph handle, indexed edge event lookup
…cting and joining per-field allocations
…stead of allocating a fresh string per recursion level
# Conflicts: # Cargo.lock # python/Cargo.toml # raphtory-graphql/src/auth.rs # raphtory-graphql/src/config/app_config.rs # raphtory-graphql/src/data.rs # raphtory-graphql/src/model/graph/graph.rs # raphtory-graphql/src/model/graph/mod.rs # raphtory-graphql/src/model/mod.rs # raphtory-graphql/src/server.rs # raphtory/src/errors.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remote Python APIs
Adds a Python
RemoteGraphclient that is a drop-in replacement for the localGraphAPI, backed by a typed transport over GraphQL. Local code migrates byswapping the constructor — method names, signatures, and semantics match the
local API; server-only extras are additive.
Architecture
Transporttrait — the client core isexecute(&Op) -> Result<Option<Prop>>over a typed operation tree (
Op = Read(ReadExpr) | Write(WriteOp)). TheGraphQL rendering lives behind the trait; nothing above it builds query strings.
window,layers,at, filters, …) accumulate in aReadExprtree and fire a single requestwhen a terminal (
degree(),count(),collect(), …) is reached.serde-serialized variables (no string interpolation or templates). The serde
impls on the wire types are the single source of truth for the format.
collect()returns handlesthat replay the parent's op log (views + filters), so members evaluate under
the same view chain as the collection they came from.
Unified filtering
filter(expr: GqlFilter)field on Graph/Node/Edge and all collections.GqlFilteris a oneOf input:nodes,edges,graph,and,or,not,plus flat graph-view spellings (
window,at,layers,snapshotAt, …).andis intersection, matching localg.filter(a & b); cross-kindorproduces a proper graph union.
compositions like
filter.Graph.window(1, 4) & (filter.Node.property("x") > 1))are exported from core as a kind-tagged
FilterTreeand carried to the wirewithout loss.
and/orlists are rejected — previously an emptyormatchedeverything, which mattered for persisted access filters.
Wire fidelity
PropertyTypescalar and
dtypefields; the client decodes stored values type-directed,so numeric widths (
U8staysU8), datetimes, and decimals survive thewire exactly. Older servers without
dtypefall back to shape-based decoding.f64Special/f32Specialinput variants and string sentinels on output(previously: error on write, silent null on read).
Prop::Mapis now insertion-ordered (IndexMap) andthe wire preserves key order end to end, matching local behavior.
Performance
Edge.event(...)pins events via indexed layer/time views instead ofscanning the full exploded history.
Arc; nested columnar reads batch persource; history slicing fetches pages rather than full histories.
Breaking changes
filterNodes/filterEdgesare removed from the GraphQL schema.Migration is mechanical and can preserve response shapes with an alias:
filterNodes(expr: X)→filterNodes: filter(expr: { nodes: X }).node/edgekeys still load (serdealiases); hand-written GraphQL must use
nodes/edges.PropertyOutputemits"NaN"/"Infinity"/"-Infinity"strings fornon-finite floats where it previously emitted
null.Build fix
pyo3/extension-moduleis now pinned under[tool.maturin]— without it thewheel links libpython and
import raphtorysegfaults on statically-linkedinterpreters (e.g. conda).
Testing
(views, filters, collections, properties, write path), including key-order,
NaN, and dtype round-trips.
Not in this PR (local APIs the remote client does not yet cover)
rolling()andexpanding()(and theWindowSetthey return) are not available on any remote object. All othertime views (
window,at,before,after,snapshot_*,shrink_*)are supported.
.historyon collections(and
.deletionson edge collections) are not exposed remotely; per-entitynode.history/edge.deletionsare. Same root gap: there is no remoteNodeState, so collection accessors like
nodes.degree()return plain listsrather than the sortable/groupable NodeState views the local API returns.
raphtory.algorithmsmodule (andGraph.largest_connected_component) operate on local graphs only.Graph.vectoriseand theraphtory.vectorsmodule (server-side vector endpoints remain reachable via raw
client.query(...)on vectors-enabled servers).load_nodes/load_edges/load_*_metadata,from_parquet/to_parquet. Batch remote writes exist(
RemoteGraph.add_nodes/add_edges) but take update lists, notDataFrames.
to_networkx,to_pyvis, and collectionto_df.import_node(s)/import_edge(s)(_as)family, which copies entities from another in-process graph.
materialize,materialize_at,event_graph,persistent_graph.save_to_file/load_from_file/serialise/deserialiseand the disk cache(
cache_view,flush) are in-process concepts; whole-graph transfer iscovered by
RaphtoryClient.send_graph/receive_graph/upload_graph.get/keys/items/valuesremotely but notin/[]/iteration and lackthe collection-level
.temporalview;History.merge/compose_historiesare unavailable on remote history handles.explode_layers().collect()on remote edges raisesUnsupported.