warehouse: docs by atravitz · Pull Request #1988 · OpenFreeEnergy/openfe

atravitz · 2026-06-05T20:05:09Z

resolves #2071

Checklist

All new code is appropriately documented (user-facing code must have complete docstrings).
Added a news entry, or the changes are not user-facing.
Ran pre-commit: you can run pre-commit locally or comment on this PR with pre-commit.ci autofix.

Manual Tests: these are slow so don't need to be run every commit, only before merging and when relevant changes are made (generally at reviewer-discretion).

GPU integration tests
example notebook testing
packaging tests: run this for any large feature PRs or PRs that add test data.

Developers certificate of origin

I certify that this contribution is covered by the MIT License here and the Developer Certificate of Origin at https://developercertificate.org/.

github-actions · 2026-06-05T20:06:34Z

No API break detected ✅

codecov · 2026-06-05T20:41:15Z

Codecov Report

❌ Patch coverage is 98.49246% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.60%. Comparing base (5e6fb2e) to head (12a82ec).
⚠️ Report is 8 commits behind head on epic/execution_improvements.

Files with missing lines	Patch %	Lines
src/openfe/storage/warehouse.py	95.77%	3 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                       @@
##           epic/execution_improvements    #1988      +/-   ##
===============================================================
- Coverage                        95.10%   90.60%   -4.50%     
===============================================================
  Files                              212      208       -4     
  Lines                            20703    20314     -389     
===============================================================
- Hits                             19689    18405    -1284     
- Misses                            1014     1909     +895

Flag	Coverage Δ
fast-tests	`90.60% <98.49%> (?)`
slow-tests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

atravitz · 2026-07-13T21:51:23Z

+
+Naively, we could require that all three of these storage types be directories that can be accessed locally by the Protocol, but this significantly limits the ways in which the Protocol can be executed, e.g. a Protocol could not store **result** data on a remote machine or cloud storage.
+
+Where a Warehouse stores its data is defined by a ``WarehouseStores`` object, which is a small TypedDict that containing ``'setup'`` and ``'result'`` keys that correspond to gufe ``ExternalStorage`` objects. 


we should probably pull the gufe docs in ExternalStorage and the like
https://github.com/OpenFreeEnergy/gufe/pull/753/files

rather, just link to the ExternalStorage api docs

atravitz · 2026-07-13T21:52:10Z

+.. code-block ::
+    from openfe.storage import FileSystemWarehouse
+    my_warehouse = FileSystemWarehouse(root_dir="my_warehouse")
+
+    ...
+
+    my_warehouse.store_result_tokenizable(result)
+.. TODO: add example of dropping in non-filesystem storage once gufe supports it


not sure if adding an example like this is useful or not. maybe just an example showing how you get deduplication built-in is better?

ethanholz

Minor comments but this is fantastic!

ethanholz · 2026-07-21T19:46:00Z

+        """
+
+        def __init__(self, root_dir: str = "warehouse"):
+            setup_store = FileStorage(f"{root_dir}/setup")


Is it worth linking out to the FileStorage implementation?

yes I think that's a good idea

IAlibay · 2026-07-28T13:23:08Z

+
+**openfe**'s ``Warehouse`` defines the interface for an execution engine to store and access data during execution.
+
+A Warehouse is any instance of a derived class of the abstract :class:`.WarehouseBaseClass`. In other words, *where* the data is stored is decided by the derived class, but *how* the data is accessed is defined by ``WarehouseBaseClass``.


This jumps in directly into the class but doesn't actually give users any info about what warehouse does - some examples and why it's needed before this line would be useful.

IAlibay · 2026-07-28T13:37:54Z

+
+
+The below example implementation, :class:`.FileSystemWarehouse`, is a derived class that inherits from ``WarehouseBaseClass``.
+This is a simple example of how to construct a Warehouse given a root directory, which is uses to create ``WarehouseStores``.


Suggested change

This is a simple example of how to construct a Warehouse given a root directory, which is uses to create ``WarehouseStores``.

This is a simple example of how to construct a Warehouse given a root directory, which is used to create ``WarehouseStores``.

IAlibay · 2026-07-28T13:42:28Z

+
+You can think of the ``WarehouseBaseClass`` as a set of specifications that must be met by a Warehouse implementation (subclass), such that any openfe Protocol can then interact appropriately with its data.
+
+For example, **openfe** Protocols require several types of storage - scratch, setup, and result.


Distinction between setup and result here is a bit confusing. I assume by "result" you mean "anything that could be useful for a user to access in the future", and "setup" is "everything that is generated as an input to a further simulation in the dag"?

IAlibay · 2026-07-28T13:46:01Z

I'm a bit lost with this documentation, it looks like mostly dev orientated docs, but it doesn't actually give me any insight as to how things will behave within a Protocol.

* refactor: remove result_server references in favor of gufe ExternalStorage (#1632) * feat: remove result_server references in favor of gufe external_storage Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * fix: remove extra external_store * test: change from result_server to external_storage * test: remove external_store reference * test: change extension to item per class changes * refactor: remove result server import * chore: remove metadata store from resultclient Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * refactor: remove result server * refactor: remove metadata store --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * refactor: rename to WarehouseBaseClass (#1660) * refactor(warehouse): rename to warehouse * refactor: rename _ResultContainerTest to _ContainerTest * refactor: rename _Container instances to _DataContainer * feat/refactor: add setup store (#1671) * refactor: remove extra implementations We may end up adding these back later but for the time being, we are going to remove them to keep life easy. * refactor: remove load_* and store_* functions Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * refactor: keys are being stored really strangely * refactor(warehouse): huge refactor to simplify working with Warehouse * refactor(warehouse): remove dead code * fix: delete failed due to incorrect object * test: add a test for the FileSystemWarehouse * refactor: remove dead code * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * docs: update warehouse docstrings * feat(warehouse): leverage keyed_chain for object dedup * Update openfe/storage/warehouse.py * import Literal * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * docs(warehouse): change from return, to raises * refactor(warehouse): move _key_exists to exists * chore(warehouse): remove extra todo * chore(warehouse): remove _load_stream * fix(warehouse): deduplicate objects on the filesystem * chore(warehouse): add typehint to exists --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io> * Temporarily build pooch from main w/ hotfix (#1806) * build with pooch@main to see if hotfix works * add link * Add support for result tokenizables to warehouse (#1763) * feat(warehouse): add result tokenizable store * docs(warehouse): add docs on the result store * chore: add property for result store * fix: use the correct function signature for handling setup --------- Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io> * test: cleanup warehouse test to be more modular (#1809) * fix dict type * add news item * run mypy CI on PRs into all branches (not just main) (#2112) * ignore a couple type check violations for now --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Ethan Holz <ethan.holz@omsf.io>

* refactor: remove result_server references in favor of gufe ExternalStorage (#1632) * feat: remove result_server references in favor of gufe external_storage Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * fix: remove extra external_store * test: change from result_server to external_storage * test: remove external_store reference * test: change extension to item per class changes * refactor: remove result server import * chore: remove metadata store from resultclient Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * refactor: remove result server * refactor: remove metadata store --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * refactor: rename to WarehouseBaseClass (#1660) * refactor(warehouse): rename to warehouse * refactor: rename _ResultContainerTest to _ContainerTest * refactor: rename _Container instances to _DataContainer * feat/refactor: add setup store (#1671) * refactor: remove extra implementations We may end up adding these back later but for the time being, we are going to remove them to keep life easy. * refactor: remove load_* and store_* functions Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * refactor: keys are being stored really strangely * refactor(warehouse): huge refactor to simplify working with Warehouse * refactor(warehouse): remove dead code * fix: delete failed due to incorrect object * test: add a test for the FileSystemWarehouse * refactor: remove dead code * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * docs: update warehouse docstrings * feat(warehouse): leverage keyed_chain for object dedup * Update openfe/storage/warehouse.py * import Literal * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * Update openfe/storage/warehouse.py Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> * docs(warehouse): change from return, to raises * refactor(warehouse): move _key_exists to exists * chore(warehouse): remove extra todo * chore(warehouse): remove _load_stream * fix(warehouse): deduplicate objects on the filesystem * chore(warehouse): add typehint to exists --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io> * Temporarily build pooch from main w/ hotfix (#1806) * build with pooch@main to see if hotfix works * add link * Add support for result tokenizables to warehouse (#1763) * feat(warehouse): add result tokenizable store * docs(warehouse): add docs on the result store * chore: add property for result store * fix: use the correct function signature for handling setup --------- Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io> * test: cleanup warehouse test to be more modular (#1809) * feat: add warehouse primitives for handling protocol units * feat: inital worker for exorcist * test: add tests for warehouse * fix: can now return protocol unit * refactor: make things more consistent * test: initial test setup for orchestration subpackage * test: initial exorcist utility testing * refactor: provide a root path to the exorcist DB * test: inital worker testing * feat: add shared_store * feat: add better handling for CLI application Signed-off-by: Ethan Holz <ethan.holz@omsf.io> * test: add new worker tests * feat: add exorcist worker to CLI * test: add for worker CLI command * docs: add numpy docstrings * feat: add support for planning an RBFE to a Warehouse * fix: correct edge direction for task graph * refactor: remove extra debugging from warehouse * refactor: cleanup handling of tasks for worker * refactor: fix issues in the CLI for running the worker * Update src/openfe/orchestration/__init__.py * fix dict type * add news item * ignore a couple type check violations for now * fix type checking * add exorcist to docs build --------- Signed-off-by: Ethan Holz <ethan.holz@omsf.io> Co-authored-by: Alyssa Travitz <31974495+atravitz@users.noreply.github.com> Co-authored-by: Alyssa Travitz <alyssa.travitz@omsf.io>

* only use protocolunit as key * add specific protocol dag store * add get_protocol_dags * fix protocol dag store deduplication order * type checking and require a name * default task db to use warehouse name * update docstrings * update type hint for python 3.12

atravitz changed the base branch from main to feat/warehouse June 5, 2026 21:46

atravitz force-pushed the warehouse/docs branch from 35bbd57 to 6ebdebb Compare June 5, 2026 23:30

atravitz commented Jun 5, 2026

View reviewed changes

Comment thread src/openfe/tests/storage/test_warehouse.py

atravitz mentioned this pull request Jun 9, 2026

Exorcist & Warehouse Integration Plan #1990

Open

9 tasks

atravitz force-pushed the warehouse/docs branch from 6ebdebb to 271900b Compare July 13, 2026 18:35

atravitz commented Jul 13, 2026

View reviewed changes

Comment thread src/openfe/storage/warehouse.py

atravitz force-pushed the warehouse/docs branch from 8b25014 to cb06195 Compare July 13, 2026 21:50

atravitz commented Jul 13, 2026

View reviewed changes

atravitz force-pushed the warehouse/docs branch 2 times, most recently from 217d499 to 6e19c62 Compare July 21, 2026 18:26

atravitz requested a review from ethanholz July 21, 2026 18:30

ethanholz reviewed Jul 21, 2026

View reviewed changes

atravitz self-assigned this Jul 27, 2026

atravitz changed the title ~~[WIP] warehouse: docs~~ warehouse: docs Jul 27, 2026

atravitz linked an issue Jul 27, 2026 that may be closed by this pull request

warehouse: add user-facing docs #2071

Open

IAlibay reviewed Jul 28, 2026

View reviewed changes

atravitz mentioned this pull request Jul 30, 2026

epic: add warehouse #1864

Merged

7 tasks

Base automatically changed from feat/warehouse to epic/execution_improvements July 30, 2026 21:58

atravitz force-pushed the epic/execution_improvements branch 2 times, most recently from 6fc30c6 to a07f1c8 Compare August 7, 2026 19:51

atravitz force-pushed the epic/execution_improvements branch from a07f1c8 to a85734f Compare August 12, 2026 19:39

atravitz and others added 3 commits August 14, 2026 09:01

atravitz force-pushed the epic/execution_improvements branch from a6fd672 to cc8edd5 Compare August 14, 2026 16:01

atravitz added 2 commits August 17, 2026 13:37

add warehouse to api docs

38ac623

fixing types

f32a9c7

atravitz added 2 commits August 17, 2026 13:39

more type hints for me

295e7dc

brain dump warehouse docs

935e804

atravitz force-pushed the warehouse/docs branch from 12a82ec to f9700c3 Compare August 17, 2026 20:41

atravitz added 5 commits August 17, 2026 13:43

add news item

5bd9f6d

formatting and clarity

5a01b01

add more examples

e3ce60e

fix docs formatting

6dd1e16

update text for clarity

8c6998e

atravitz force-pushed the warehouse/docs branch from f9700c3 to 8c6998e Compare August 17, 2026 20:44

atravitz force-pushed the epic/execution_improvements branch 3 times, most recently from 4bf5974 to 15029e7 Compare August 24, 2026 15:25


		Naively, we could require that all three of these storage types be directories that can be accessed locally by the Protocol, but this significantly limits the ways in which the Protocol can be executed, e.g. a Protocol could not store result data on a remote machine or cloud storage.

		Where a Warehouse stores its data is defined by a ``WarehouseStores`` object, which is a small TypedDict that containing ``'setup'`` and ``'result'`` keys that correspond to gufe ``ExternalStorage`` objects.


		openfe's ``Warehouse`` defines the interface for an execution engine to store and access data during execution.

		A Warehouse is any instance of a derived class of the abstract :class:`.WarehouseBaseClass`. In other words, where the data is stored is decided by the derived class, but how the data is accessed is defined by ``WarehouseBaseClass``.



		The below example implementation, :class:`.FileSystemWarehouse`, is a derived class that inherits from ``WarehouseBaseClass``.
		This is a simple example of how to construct a Warehouse given a root directory, which is uses to create ``WarehouseStores``.


		You can think of the ``WarehouseBaseClass`` as a set of specifications that must be met by a Warehouse implementation (subclass), such that any openfe Protocol can then interact appropriately with its data.

		For example, openfe Protocols require several types of storage - scratch, setup, and result.

Uh oh!

Conversation

atravitz commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Developers certificate of origin

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

codecov Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethanholz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

atravitz commented Jun 5, 2026 •

edited

Loading

codecov Bot commented Jun 5, 2026 •

edited

Loading