Skip to content

@W-23751657: Add external module resolution to Python binding - #167

Merged
mlischetti merged 20 commits into
masterfrom
python-module-resolver-design
Aug 25, 2026
Merged

@W-23751657: Add external module resolution to Python binding#167
mlischetti merged 20 commits into
masterfrom
python-module-resolver-design

Conversation

@mlischetti

Copy link
Copy Markdown
Contributor

Summary

  • add synchronous external DataWeave module resolution to explicit Python DataWeave instances
  • add map, directory, JAR, and composed resolver factories without new runtime dependencies
  • bridge Python resolvers through the existing run_script_with_resolver ABI with safe callback, buffer, concurrency, re-entry, and isolate lifecycle handling
  • recover six Python TCK module scenarios using the existing shared read-only fixture

API

from dataweave import DataWeave, modules_from_map

resolver = modules_from_map({
    "org/company/lib.dwl": "%dw 2.0\nfun answer() = 42",
})

with DataWeave(resolve_module=resolver) as dw:
    result = dw.run("import org::company::lib\n---\nlib::answer()")

Custom module resolution is intentionally limited to synchronous DataWeave.run(). Module-level convenience functions and streaming APIs remain resolver-less.

TCK impact

  • selected: 729
  • passed/executed: 679
  • active exclusions: 31, down from 37
  • strict xfails: 19
  • structural module cases: 17
  • failed: 0
  • unaccounted: 0

runtime/module-singleton-out.json remains excluded because the approved shared fixture lacks its three singleton modules.

Validation

  • ./gradlew native-lib:pythonTest (147 passed)
  • ./gradlew native-lib:stageTckSuites native-lib:pythonTck (719 pytest passes, 31 skips, 19 xfails; TCK accounting 729/729)
  • focused resolver/runtime/lifecycle suites (105 passed)
  • ./gradlew native-lib:test -PskipNodeTests=true -PskipPythonTests=true --rerun-tasks
  • ./gradlew native-lib:buildPythonWheel plus isolated virtualenv import smoke test
  • git diff --check

Scope

  • no changes under native-lib/node
  • no Java/native source or ABI changes
  • no new runtime dependencies

@mlischetti
mlischetti requested a review from a team as a code owner August 25, 2026 12:24
@mlischetti mlischetti changed the title Add external module resolution to Python binding @W-23751657: Add external module resolution to Python binding Aug 25, 2026

@svacas svacas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one issue that should be addressed before merging.

P2: Directory resolver fails when base_dir contains ..

modules_from_directory() (

lexical_root = Path(base_dir).absolute()
canonical_root = lexical_root.resolve(strict=True)
if not canonical_root.is_dir():
raise NotADirectoryError(f"Module root is not a directory: {base_dir}")
def resolve(module_path: str) -> Optional[str]:
requested_path = Path(module_path)
if requested_path.is_absolute():
return None
candidate = Path(os.path.abspath(lexical_root / requested_path))
try:
candidate.relative_to(lexical_root)
) creates lexical_root with Path.absolute(),
which preserves .. path segments. The candidate is later normalized with os.path.abspath(), so this containment check rejects valid files:

candidate.relative_to(lexical_root)

For example, an existing module resolved through a base path such as /tmp/modules/../modules incorrectly returns None.

Normalize the lexical root without resolving symlinks:

lexical_root = Path(os.path.abspath(base_dir))

Please also add a regression test using a valid base directory containing ...

All current CI checks pass, and I found no other blocking issues in callback lifetime, thread attachment, execution serialization, or cleanup handling.

@mlischetti
mlischetti merged commit 91ebf6f into master Aug 25, 2026
5 checks passed
@mlischetti
mlischetti deleted the python-module-resolver-design branch August 25, 2026 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants