fix(01-harness): stop 07-oauth reporting success for a failed invoke, and leaking on failure

[rmncardoso]

Summary

07-oauth printed a full success narrative even when the agent invocation failed, and leaked every resource it created if any step raised. Every fix below was reproduced against upstream main and re-verified live in a real AWS account (7 deploy/teardown cycles).

Correctness

1. False success on a failed invoke. InvokeHarness over HTTPS returns HTTP 200 and reports agent-side failures as {"message": ...} frames inside the binary event stream. The parser read only obj["delta"], discarded them, printed (No text deltas found in stream) plus raw bytes, and then printed the success narrative claiming all three auth hops had worked. Now: raise on non-200, collect and raise on error frames, and raise when there are neither deltas nor an error frame.
2. Missing memory permissions made the harness fail at runtime with AccessDeniedException on ListEvents. Added the 8 memory/event actions. CreateMemory and ListMemories take no memory ID, so they are implicitDeny under memory/* and must be scoped to "*" (confirmed with iam:SimulateCustomPolicy); the other 6 stay on memory/*.
3. Three fall-through polls treated terminal statuses as "not ready yet" and, once the loop expired, carried straight on against a resource still CREATING — surfacing as a confusing HTTP error rather than the real cause. Replaced with deadline loops that raise on terminal status and on timeout.
4. No try/finally. A failure between Step 1a and Step 4 left two Cognito pools, a credential provider, a Lambda, a gateway + target, a harness and three IAM roles alive — all billable, and all blocking the next run with ConflictException.
5. _wait_gone returned success on timeout; it now returns a boolean the caller honours.
6. ✅ Cleanup complete! printed even when resources had been skipped.
7. _ensure_policy swallowed EntityAlreadyExists, so a re-run silently kept a stale policy document. It now publishes a new default version, pruning the oldest non-default at the 5-version service limit.
8. The Cognito pool lookup read only the first 60-pool page.
9. _del_role could not delete a policy that had non-default versions.
10. Six unpaginated AgentCore list calls. ListHarnesses, ListGateways and ListGatewayTargets all return a nextToken. Routed through a single iter_paginated helper. This has two distinct consequences, neither of which mentions pagination: a lookup after ConflictException fails with "conflict but not found" for a resource that demonstrably exists, and cleanup reports "not found" and walks past a resource it created — leaking it, and in the harness case leaking its managed memory with it.
11. The target conflict fallback took the first target with no name check, so a gateway carrying any other target hands back the wrong targetId — and the poll then waits on a resource this script never created.
12. Removed a needless get_gateway per gateway in the gateway conflict fallback; ListGateways already reports name, so only the match needs describing. The original described every gateway in the account, which on a busy account risks throttling during what is meant to be a recovery path.

Also converted 6 except Exception: pass handlers into reported skips, so cleanup failures are visible instead of silent.

Docs and lint

13. The README documented no IAM permissions at all, and told readers to pip install requests — already line 9 of ../../requirements.txt. Added a permission table (enumerated by walking the AST of every boto3 call in the sample) and a helper-function table.
14. utils/lambda_function_code.py carried a latent import-order violation and a docstring naming the wrong tools. It had never failed CI because CI lints only changed files and this file had never been modified.

Testing

7 live deploy/teardown cycles in us-west-2, on a fresh-account baseline:

Run	Scenario	Result
1	pristine upstream (BEFORE)	reproduces items 1 and 2
2	fixed bytes (AFTER)	exit 0, full auth chain
3	fresh + cleanup	exit 0; Deleted 13 resources, skipped 1
4	fresh + --skip-cleanup	exit 0; prints Harness ID + Gateway ARN
5	re-run over existing resources	exit 0; Deleted 14 resources, skipped 0; policies v1 → v2 default
6	fresh + --skip-cleanup, final bytes	exit 0; HTTP 200
7	re-run over existing resources, final bytes	exit 0; all three rewritten conflict fallbacks exercised; Deleted 13 resources, skipped 1

Every run exited 0, and after each one the account returned to its exact pre-flight baseline with zero leaked resources. The harness's managed harness_* memory self-reaps a couple of minutes after the harness delete completes — confirmed, so cleanup does not need to (and cannot) delete it directly.

Also: 63 unit checks covering the branches a healthy account cannot reach (terminal statuses, timeouts, policy-version pruning, pagination across a page boundary, and static assertions that the success narrative sits after every guard that protects it). ruff check and ruff format --check are clean on all four files at both ruff 0.15.0 and 0.16.1.

Deliberately unchanged

• _wait_gone's broad except — a missing gateway raises AccessDeniedException, not ResourceNotFoundException (verified live), so narrowing it would break the "gone" detection.
• The IAM list calls are bounded by service limits (10 managed policies per role, 5 versions per policy), both well under one page, so they need no pagination.

[github-actions]

Latest scan for commit: 4c9498f | Updated: 2026-08-01 22:22:05 UTC

Security Scan Results

Scan Metadata

• Project: ASH
• Scan executed: 2026-08-01T22:21:51+00:00
• ASH version: 3.0.0

Summary

Scanner Results

The table below shows findings by scanner, with status based on severity thresholds and dependencies:

Column Explanations:

Severity Levels (S/C/H/M/L/I):

• Suppressed (S): Security findings that have been explicitly suppressed/ignored and don't affect the scanner's pass/fail status
• Critical (C): The most severe security vulnerabilities requiring immediate remediation (e.g., SQL injection, remote code execution)
• High (H): Serious security vulnerabilities that should be addressed promptly (e.g., authentication bypasses, privilege escalation)
• Medium (M): Moderate security risks that should be addressed in normal development cycles (e.g., weak encryption, input validation issues)
• Low (L): Minor security concerns with limited impact (e.g., information disclosure, weak recommendations)
• Info (I): Informational findings for awareness with minimal security risk (e.g., code quality suggestions, best practice recommendations)

Other Columns:

• Time: Duration taken by each scanner to complete its analysis
• Action: Total number of actionable findings at or above the configured severity threshold that require attention

Scanner Results:

• PASSED: Scanner found no security issues at or above the configured severity threshold - code is clean for this scanner
• FAILED: Scanner found security vulnerabilities at or above the threshold that require attention and remediation
• MISSING: Scanner could not run because required dependencies/tools are not installed or available
• SKIPPED: Scanner was intentionally disabled or excluded from this scan
• ERROR: Scanner encountered an execution error and could not complete successfully

Severity Thresholds (Thresh Column):

• CRITICAL: Only Critical severity findings cause scanner to fail
• HIGH: High and Critical severity findings cause scanner to fail
• MEDIUM (MED): Medium, High, and Critical severity findings cause scanner to fail
• LOW: Low, Medium, High, and Critical severity findings cause scanner to fail
• ALL: Any finding of any severity level causes scanner to fail

Threshold Source: Values in parentheses indicate where the threshold is configured:

• (g) = global: Set in the global_settings section of ASH configuration
• (c) = config: Set in the individual scanner configuration section
• (s) = scanner: Default threshold built into the scanner itself

Statistics calculation:

• All statistics are calculated from the final aggregated SARIF report
• Suppressed findings are counted separately and do not contribute to actionable findings
• Scanner status is determined by comparing actionable findings to the threshold

Scanner	S	C	H	M	L	I	Time	Action	Result	Thresh
bandit	0	0	0	0	0	0	717ms	0	PASSED	MED (g)
cdk-nag	0	0	0	0	0	0	6.8s	0	PASSED	MED (g)
cfn-nag	0	0	0	0	0	0	9ms	0	PASSED	MED (g)
checkov	0	0	0	0	0	0	5.7s	0	PASSED	MED (g)
detect-secrets	0	0	0	0	0	0	1.3s	0	PASSED	MED (g)
grype	0	0	0	0	0	0	57.1s	0	PASSED	MED (g)
npm-audit	0	0	0	0	0	0	203ms	0	PASSED	MED (g)
opengrep	0	0	0	0	0	0	<1ms	0	SKIPPED	MED (g)
semgrep	0	0	0	0	0	0	<1ms	0	MISSING	MED (g)
syft	0	0	0	0	0	0	2.2s	0	PASSED	MED (g)