Expected Behavior
When a map or parallel operation completes early because its completion
criteria are met, an in-flight branch marked as orphaned must not checkpoint
its terminal SUCCEED or FAIL result after the parent context completes.
Checkpoint admission should have a deterministic order:
- If the branch checkpoint is admitted first, it may be sent before the parent
completion.
- If the parent completion marks the branch orphaned first, the branch
checkpoint must be rejected and must not reach the checkpoint service.
Actual Behavior
ExecutionState.create_checkpoint() validates _parent_done while holding
_parent_done_lock, releases that lock, invokes the plugin hook, and only later
enqueues the update while holding _completion_lock.
This creates a check-to-enqueue race:
- An in-flight branch calls
create_checkpoint() with its terminal context
SUCCEED update and passes the _parent_done check.
- Before the branch update is enqueued, the map/parallel parent checkpoints
SUCCEED and _mark_orphans() marks the branch operation ID as orphaned.
- The branch resumes and enqueues its terminal
SUCCEED update because orphan
status is not checked again.
checkpoint_batches_forever() sends every queued update without
revalidating orphan status.
A deterministic reproduction recorded two successful checkpoint-service calls:
service call 1: parent SUCCEED with BatchResult branch status STARTED
service call 2: branch SUCCEED with payload "branch-result"
Both synchronous checkpoint callers were released after the mock service
returned a new checkpoint token.
The customer-visible BatchResult remains replay-consistent in the current
implementation because normal payloads replay the parent's serialized result
and ReplayChildren summaries record startedIndexes. However, the durable
operation hierarchy can contain a terminal branch result checkpointed after
its parent context completed.
Relevant code:
packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/state.py,
ExecutionState.create_checkpoint() around the _parent_done_lock
validation and later _checkpoint_queue.put()
packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/state.py,
checkpoint_batches_forever() where queued updates are sent without orphan
revalidation
Steps to Reproduce
- Create an
ExecutionState with a branch operation registered as a child of
a map/parallel parent.
- Start a branch terminal
CONTEXT/SUCCEED checkpoint on a worker thread.
- Pause that thread after its initial orphan validation but before queue
insertion, for example using a blocking synchronous plugin hook.
- Checkpoint the parent
CONTEXT/SUCCEED, causing _mark_orphans() to mark
the branch operation ID.
- Allow the parent update to be sent successfully.
- Release the branch thread.
- Observe that the branch terminal update is sent in a subsequent checkpoint
API call and the synchronous branch caller completes successfully.
SDK Version
1.8.0, repository HEAD 7ac7acc6a7dae231f2abbb8e37f9780cc9b89af0
Python Version
3.14
Is this a regression?
No known regression.
Possible Fix
Make orphan validation and queue insertion atomic relative to parent
completion:
- Inside the final
_completion_lock block, acquire _parent_done_lock using
a consistent global lock order.
- Recheck whether
operation_update.operation_id is in _parent_done
immediately before queue insertion.
- Optionally also reject when
operation_update.parent_id is in
_parent_done, preventing new durable descendants from being created under
an already orphaned branch.
- Insert the queued operation while still holding
_parent_done_lock.
This produces deterministic ordering: either the child is queued first, or
parent completion wins and the child is rejected.
Add deterministic unit tests for:
- A terminal branch checkpoint paused between initial validation and enqueue.
- A new operation created under an already orphaned branch.
- The valid child-first ordering.
- Replay consistency for normal and
ReplayChildren map/parallel results.
Additional Context
The existing core test suite passes (1504 passed, plus 5 subtests), but it
does not currently cover this interleaving.
Expected Behavior
When a
maporparalleloperation completes early because its completioncriteria are met, an in-flight branch marked as orphaned must not checkpoint
its terminal
SUCCEEDorFAILresult after the parent context completes.Checkpoint admission should have a deterministic order:
completion.
checkpoint must be rejected and must not reach the checkpoint service.
Actual Behavior
ExecutionState.create_checkpoint()validates_parent_donewhile holding_parent_done_lock, releases that lock, invokes the plugin hook, and only laterenqueues the update while holding
_completion_lock.This creates a check-to-enqueue race:
create_checkpoint()with its terminal contextSUCCEEDupdate and passes the_parent_donecheck.SUCCEEDand_mark_orphans()marks the branch operation ID as orphaned.SUCCEEDupdate because orphanstatus is not checked again.
checkpoint_batches_forever()sends every queued update withoutrevalidating orphan status.
A deterministic reproduction recorded two successful checkpoint-service calls:
Both synchronous checkpoint callers were released after the mock service
returned a new checkpoint token.
The customer-visible
BatchResultremains replay-consistent in the currentimplementation because normal payloads replay the parent's serialized result
and
ReplayChildrensummaries recordstartedIndexes. However, the durableoperation hierarchy can contain a terminal branch result checkpointed after
its parent context completed.
Relevant code:
packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/state.py,ExecutionState.create_checkpoint()around the_parent_done_lockvalidation and later
_checkpoint_queue.put()packages/aws-durable-execution-sdk-python/src/aws_durable_execution_sdk_python/state.py,checkpoint_batches_forever()where queued updates are sent without orphanrevalidation
Steps to Reproduce
ExecutionStatewith a branch operation registered as a child ofa map/parallel parent.
CONTEXT/SUCCEEDcheckpoint on a worker thread.insertion, for example using a blocking synchronous plugin hook.
CONTEXT/SUCCEED, causing_mark_orphans()to markthe branch operation ID.
API call and the synchronous branch caller completes successfully.
SDK Version
1.8.0, repository HEAD7ac7acc6a7dae231f2abbb8e37f9780cc9b89af0Python Version
3.14
Is this a regression?
No known regression.
Possible Fix
Make orphan validation and queue insertion atomic relative to parent
completion:
_completion_lockblock, acquire_parent_done_lockusinga consistent global lock order.
operation_update.operation_idis in_parent_doneimmediately before queue insertion.
operation_update.parent_idis in_parent_done, preventing new durable descendants from being created underan already orphaned branch.
_parent_done_lock.This produces deterministic ordering: either the child is queued first, or
parent completion wins and the child is rejected.
Add deterministic unit tests for:
ReplayChildrenmap/parallel results.Additional Context
The existing core test suite passes (
1504 passed, plus 5 subtests), but itdoes not currently cover this interleaving.