Description
The adk conformance test CLI command supports several options and input combinations, but these CLI behaviors should have explicit automated coverage to prevent regressions.
The command currently supports:
- Running conformance tests with no path, defaulting to the
tests directory.
- Running tests against one or more explicitly provided directories.
- Selecting
replay or live mode.
- Generating a Markdown report with
--generate_report.
- Specifying a custom report directory with
--report_dir.
- Selecting a streaming mode with
--streaming-mode.
This issue proposes adding focused unit tests for the CLI argument handling and dispatch behavior without requiring real LLM execution.
Proposed Changes
Add pytest coverage for the adk conformance test CLI command covering:
- No path provided → defaults to the
tests directory.
- A single test directory is correctly passed to the conformance runner.
- Multiple test directories are accepted.
--mode replay is correctly propagated.--mode live is correctly propagated.--generate_report is correctly propagated.--report_dir <path> is correctly propagated.--streaming-mode values are parsed and propagated correctly.- Invalid
--mode values are rejected by Click.
- Invalid
--streaming-mode values are rejected by Click.
Where appropriate, the conformance runner should be mocked so these tests remain fast and deterministic.
Expected Outcome
The CLI's argument parsing and dispatch behavior should have regression coverage, making future changes to the conformance command less likely to introduce unnoticed CLI regressions.
Testing
Run the relevant CLI unit tests with pytest and ensure the existing ADK test suite continues to pass.
Scope
This should be limited to automated test coverage for the existing CLI behavior. No changes to the underlying conformance execution logic are required.
Description
The
adk conformance testCLI command supports several options and input combinations, but these CLI behaviors should have explicit automated coverage to prevent regressions.The command currently supports:
testsdirectory.replayorlivemode.--generate_report.--report_dir.--streaming-mode.This issue proposes adding focused unit tests for the CLI argument handling and dispatch behavior without requiring real LLM execution.
Proposed Changes
Add pytest coverage for the
adk conformance testCLI command covering:testsdirectory.--mode replayis correctly propagated.--mode liveis correctly propagated.--generate_reportis correctly propagated.--report_dir <path>is correctly propagated.--streaming-modevalues are parsed and propagated correctly.--modevalues are rejected by Click.--streaming-modevalues are rejected by Click.Where appropriate, the conformance runner should be mocked so these tests remain fast and deterministic.
Expected Outcome
The CLI's argument parsing and dispatch behavior should have regression coverage, making future changes to the conformance command less likely to introduce unnoticed CLI regressions.
Testing
Run the relevant CLI unit tests with pytest and ensure the existing ADK test suite continues to pass.
Scope
This should be limited to automated test coverage for the existing CLI behavior. No changes to the underlying conformance execution logic are required.