Skip to content

[Bug]: Orphaned map/parallel branch can checkpoint after parent completion #640

Description

@zhongkechen

Expected Behavior

When a map or parallel operation completes early because its completion
criteria are met, an in-flight branch marked as orphaned must not checkpoint
its terminal SUCCEED or FAIL result after the parent context completes.

Checkpoint admission should have a deterministic order:

  • If the branch checkpoint is admitted first, it may be sent before the parent
    completion.
  • If the parent completion marks the branch orphaned first, the branch
    checkpoint must be rejected and must not reach the checkpoint service.

Actual Behavior

ExecutionState.create_checkpoint() validates _parent_done while holding
_parent_done_lock, releases that lock, invokes the plugin hook, and only later
enqueues the update while holding _completion_lock.

This creates a check-to-enqueue race:

  1. An in-flight branch calls create_checkpoint() with its terminal context
    SUCCEED update and passes the _parent_done check.
  2. Before the branch update is enqueued, the map/parallel parent checkpoints
    SUCCEED and _mark_orphans() marks the branch operation ID as orphaned.
  3. The branch resumes and enqueues its terminal SUCCEED update because orphan
    status is not checked again.
  4. checkpoint_batches_forever() sends every queued update without
    revalidating orphan status.

A deterministic reproduction recorded two successful checkpoint-service calls:

service call 1: parent SUCCEED with BatchResult branch status STARTED
service call 2: branch SUCCEED with payload "branch-result"

Both synchronous checkpoint callers were released after the mock service
returned a new checkpoint token.

The customer-visible BatchResult remains replay-consistent in the current
implementation because normal payloads replay the parent's serialized result
and ReplayChildren summaries record startedIndexes. However, the durable
operation hierarchy can contain a terminal branch result checkpointed after
its parent context completed.

Relevant code:

  • packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/state.py,
    ExecutionState.create_checkpoint() around the _parent_done_lock
    validation and later _checkpoint_queue.put()
  • packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/state.py,
    checkpoint_batches_forever() where queued updates are sent without orphan
    revalidation

Steps to Reproduce

  1. Create an ExecutionState with a branch operation registered as a child of
    a map/parallel parent.
  2. Start a branch terminal CONTEXT/SUCCEED checkpoint on a worker thread.
  3. Pause that thread after its initial orphan validation but before queue
    insertion, for example using a blocking synchronous plugin hook.
  4. Checkpoint the parent CONTEXT/SUCCEED, causing _mark_orphans() to mark
    the branch operation ID.
  5. Allow the parent update to be sent successfully.
  6. Release the branch thread.
  7. Observe that the branch terminal update is sent in a subsequent checkpoint
    API call and the synchronous branch caller completes successfully.

SDK Version

1.8.0, repository HEAD 7ac7acc6a7dae231f2abbb8e37f9780cc9b89af0

Python Version

3.14

Is this a regression?

No known regression.

Possible Fix

Make orphan validation and queue insertion atomic relative to parent
completion:

  1. Inside the final _completion_lock block, acquire _parent_done_lock using
    a consistent global lock order.
  2. Recheck whether operation_update.operation_id is in _parent_done
    immediately before queue insertion.
  3. Optionally also reject when operation_update.parent_id is in
    _parent_done, preventing new durable descendants from being created under
    an already orphaned branch.
  4. Insert the queued operation while still holding _parent_done_lock.

This produces deterministic ordering: either the child is queued first, or
parent completion wins and the child is rejected.

Add deterministic unit tests for:

  • A terminal branch checkpoint paused between initial validation and enqueue.
  • A new operation created under an already orphaned branch.
  • The valid child-first ordering.
  • Replay consistency for normal and ReplayChildren map/parallel results.

Additional Context

The existing core test suite passes (1504 passed, plus 5 subtests), but it
does not currently cover this interleaving.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions