From a1526d962d62357b6b50cd22e03dfdff7aaf4961 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 14 Aug 2026 23:37:47 +0200 Subject: [PATCH 01/31] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20refactor=20skill=20d?= =?UTF-8?q?ocumentation=20and=20evaluation=20framework?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reorganized dotnet-remote-testing skill documentation and evaluation definitions for better clarity and maintainability. Updated SKILL.md, FORMS.md, and reference materials to align with current implementation capabilities. Defined comprehensive evaluation suite with test fixtures to validate skill behavior across diverse environments. --- skills/dotnet-remote-testing/FORMS.md | 37 ++++-- skills/dotnet-remote-testing/SKILL.md | 116 +++++++++++++++--- skills/dotnet-remote-testing/evals/evals.json | 86 +++++++++++++ .../test/Catalog.Tests/Catalog.Tests.csproj | 12 ++ .../test/Catalog.Tests/CatalogTests.cs | 9 ++ .../multi-configured/testenvironments.json | 15 +++ .../test/Matrix.Tests/Matrix.Tests.csproj | 12 ++ .../test/Matrix.Tests/MatrixTests.cs | 9 ++ .../references/release-discovery.md | 38 +++++- .../references/testenvironments-json.md | 2 +- 10 files changed, 305 insertions(+), 31 deletions(-) create mode 100644 skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/Catalog.Tests.csproj create mode 100644 skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/CatalogTests.cs create mode 100644 skills/dotnet-remote-testing/evals/files/multi-configured/testenvironments.json create mode 100644 skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj create mode 100644 skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs diff --git a/skills/dotnet-remote-testing/FORMS.md b/skills/dotnet-remote-testing/FORMS.md index 8fea89d..72b3f3b 100644 --- a/skills/dotnet-remote-testing/FORMS.md +++ b/skills/dotnet-remote-testing/FORMS.md @@ -1,6 +1,18 @@ # .NET Remote Testing Input Form -Collect only the fields that are still unresolved after inspecting the request and the repository. Most remote-test requests are fully determined and need **no** questions — for example, "remote test this solution" against a repository with a single applicable environment. Prefer native structured controls when the host provides them; otherwise use the plain-text fallback below without changing field order, defaults, or the final confirmation. +This form is a **fallback for genuine ambiguity, not an intake checklist**. The default path collects nothing: a request to remote test is executed, not surveyed. + +## Autonomy gate — evaluate before presenting any field + +Present a field only when one of these is true: + +1. The runner exited `SelectionRequired` (`16`) — present `environment`, restricted to the `candidates` it returned. +2. The developer explicitly asked to choose something ("let me pick the environment", "which options do I have?"). +3. The developer supplied a value that is genuinely unusable (for example a project path that does not exist). + +If none apply, run with the defaults — auto-resolved target, `Debug`, no coverage — and present **no** fields and **no** confirmation. A single applicable Docker environment in `testenvironments.json` is a resolved answer, not a question. Never walk the field list top-to-bottom to "gather requirements", and never ask `test_scope`, `configuration`, or `coverage` unprompted; those are defaults the developer overrides by saying so. + +Prefer native structured controls when the host provides them; otherwise use the plain-text fallback below without changing field order, defaults, or the final confirmation. ## Fields @@ -8,9 +20,9 @@ Collect only the fields that are still unresolved after inspecting the request a - **type:** single-choice - **prompt:** Which environment should run the tests? -- **choices:** Dynamically list the environments from `remote-test.cs list` — the configured Docker environments from `testenvironments.json`, or the Microsoft-derived environments (for example `dotnet-10-lts`, `dotnet-9-sts`, `dotnet-11-preview`) when no `testenvironments.json` exists -- **default:** The only applicable Docker environment, or the environment explicitly named by the user (Recommended) -- **required:** true +- **choices:** The `candidates` returned by the runner's `SelectionRequired` result — the configured Docker environments from `testenvironments.json`, or the Microsoft-derived environments (for example `dotnet-10-lts`, `dotnet-9-sts`, `dotnet-11-preview`) when no `testenvironments.json` exists +- **default:** The environment explicitly named by the user +- **required:** Only when the runner exits `SelectionRequired`. A single applicable Docker environment resolves automatically and is never asked. ### test_scope @@ -21,7 +33,7 @@ Collect only the fields that are still unresolved after inspecting the request a - A specific project - A class or test filter - **default:** Entire solution / auto-resolved target (Recommended) -- **required:** true +- **required:** false — the default applies silently; ask only when the developer asks to narrow the run but does not say how ### project @@ -50,7 +62,7 @@ Collect only the fields that are still unresolved after inspecting the request a - Debug (Recommended) - Release - **default:** Debug (Recommended) -- **required:** true +- **required:** false — `Debug` applies silently unless the developer names a configuration ### coverage @@ -60,7 +72,7 @@ Collect only the fields that are still unresolved after inspecting the request a - No (Recommended) - Yes - **default:** No (Recommended) -- **required:** true +- **required:** false — never ask; coverage is collected only when the developer requests it ### confirmation @@ -70,15 +82,16 @@ Collect only the fields that are still unresolved after inspecting the request a - Yes (Recommended) - No - **default:** Yes (Recommended) -- **required:** true +- **required:** Only when at least one other field was presented. On the autonomous path there is nothing to confirm — the run is the answer. ## Presentation rules -- Infer explicit answers from the request and from `remote-test.cs list`/`plan`; do not ask them again. -- Ask one unresolved field at a time. Never bundle multiple questions. +- Clear the autonomy gate above before presenting anything. In practice most invocations present no fields at all. +- Infer explicit answers from the request and from the runner's own output; do not ask them again. +- Ask one unresolved field at a time. Never bundle multiple questions, and never turn a single blocking choice into a broader intake. - Present the recommended/default choice first and suffix it with `(Recommended)`. -- For the `environment` field, offer the discovered environment names as selectable choices rather than free text. When exactly one environment applies, select it without asking. +- For the `environment` field, offer the runner's `candidates` as selectable choices rather than free text. When exactly one environment applies, select it without asking. - For `project`, offer the auto-resolved target as a selectable choice alongside a free-text path. - In plain-text fallback mode, start immediately with `Field: ` and show numbered choices. Do not add a conversational preamble. - If the user leaves a shown computed/default choice blank, accept it and continue. -- After all fields are resolved, summarize the exact environment, target, configuration, and coverage, then ask `confirmation`. +- When fields were presented, summarize the exact environment, target, configuration, and coverage after they are resolved, then ask `confirmation`. When no field was presented, skip the summary and the confirmation and run. diff --git a/skills/dotnet-remote-testing/SKILL.md b/skills/dotnet-remote-testing/SKILL.md index 349dd67..2b43ba5 100644 --- a/skills/dotnet-remote-testing/SKILL.md +++ b/skills/dotnet-remote-testing/SKILL.md @@ -1,28 +1,61 @@ --- name: dotnet-remote-testing description: > - Run .NET tests inside a resolved remote Docker environment and return structured results — Visual Studio's Remote Testing experience (choose an environment, run tests, see results) without hand-writing container plumbing. Use when asked to remote test, run tests in Docker or a container, run tests against a specific .NET SDK, list or select test environments, or honor an existing testenvironments.json. Honors testenvironments.json Docker environments or derives zero-config environments from Microsoft's live .NET release index (LTS, STS, preview) using official mcr.microsoft.com/dotnet/sdk images via the bundled runner scripts/remote-test.cs. Docker only; WSL and SSH are reported unsupported. Do NOT use to author or refactor test code, choose a testing framework, generate Dockerfiles, or run tests on the host. + Run .NET tests inside a resolved remote Docker environment — Visual Studio's Remote Testing without hand-writing container plumbing. Invoking this skill IS the request: run the tests immediately. Never reply with a menu of options or a questionnaire. Use when asked to remote test, run tests in Docker or a container, target a specific .NET SDK, list or select test environments, or honor an existing testenvironments.json. Honors configured Docker environments, or derives them from Microsoft's live .NET release index using mcr.microsoft.com/dotnet/sdk images, plus codebeltnet/ubuntu-testrunner for multi-targeted repos, via the runner scripts/remote-test.cs. Docker only; WSL and SSH are unsupported. Do NOT use to author or refactor test code, choose a testing framework, generate Dockerfiles, or run tests on the host. compatibility: > - Requires the .NET SDK, Docker, and PowerShell 7+. Zero-config discovery needs network access to Microsoft's release index and mcr.microsoft.com; a cache enables offline reuse. + Requires the .NET 10 SDK or later (`dotnet run --file`), a running Docker daemon, and PowerShell 7+. Zero-config discovery needs network access; a cache enables offline reuse. --- # .NET Remote Testing -Give developers the experience Visual Studio's experimental [Remote Testing](https://learn.microsoft.com/en-us/visualstudio/test/remote-testing?view=visualstudio) was meant to provide: +## Do this now + +**You were invoked. That is the request. Run the tests.** + +Your first action is this command — not a question, not a menu, not a summary of what you could do: + +``` +dotnet run --file "/scripts/remote-test.cs" -- run --repo-root "" +``` + +Run it immediately, then report the result. This holds for a bare `/dotnet-remote-testing` with no other words, and for "remote test this repo", "run my tests in Docker", or any equivalent. There is nothing to clarify first: the runner resolves the environment, the target, and the configuration by itself, and it is the *only* thing that decides whether a question is needed. + +**Forbidden as a first response:** listing your capabilities; "here are the typical workflows"; offering `list` / `plan` / `run` as choices; asking which project, environment, .NET version, or filter to use; asking for confirmation. If you are about to write "What would you like me to do?", you have already failed — run the command instead. + +You may ask a question in exactly one situation: the command exited `16` (`SelectionRequired`), which means a genuine choice remains. Then ask one question listing only the `candidates` it returned, and rerun with `-e `. Every other exit code is an outcome to report, never a question. The full mapping is in [Failure handling](#failure-handling). + +Resolve the two placeholders exactly once and reuse them verbatim: + +- `` is the directory containing this `SKILL.md`. Build the path from it; do not guess a relative path from the working directory and do not copy `` through literally. +- `` is the workspace/solution root — the current directory unless the developer named another. Always pass it explicitly rather than relying on the process default, and always quote both paths (Windows paths contain backslashes and often spaces). + +Prerequisites, and the exact way each one fails: + +| Requirement | Why | If missing | +|---|---|---| +| .NET 10 SDK or later | `dotnet run --file` (file-based apps) | The CLI rejects `--file`; report the SDK requirement — do not rewrite the runner into a project | +| Docker, running | Test execution | The runner exits `DockerUnavailable` (`5`); report it, never fall back to the host | +| Network (first run) | Release metadata, image pull | Use `--offline` with `--cache-root` when a cache exists; otherwise exit `15` explains it | + +## Why this skill exists + +Visual Studio's experimental [Remote Testing](https://learn.microsoft.com/en-us/visualstudio/test/remote-testing?view=visualstudio) promised: > **Choose a test environment → run tests → see results.** -Everything between those two actions — configuration discovery, .NET release discovery, image resolution, source staging, NuGet caching, restore, build, test execution, result collection, cancellation, and cleanup — is infrastructure that belongs *behind* the abstraction. Your job is orchestration: understand what the developer means, then hand the work to the deterministic runner. Do not turn a routine "run my tests in .NET 10" into a Docker tutorial, and never compose ad-hoc `docker run` command lines yourself. +Everything between those actions — configuration discovery, .NET release discovery, image resolution, source staging, NuGet caching, restore, build, test execution, result collection, cancellation, and cleanup — is infrastructure that belongs *behind* the abstraction. A developer who reached for this skill has already chosen remote testing; handing that choice back as a questionnaire is the friction this skill exists to remove. Do not turn a routine "run my tests in .NET 10" into a Docker tutorial, and never compose ad-hoc `docker run` command lines yourself. ## Architecture: you orchestrate, the runner executes -The bundled .NET file-based program `scripts/remote-test.cs` is the **execution layer**. It is deterministic and self-tested. You are the **orchestration layer**. Always route execution through it instead of driving Docker directly: +The bundled .NET file-based program `scripts/remote-test.cs` is the **execution layer**. It is deterministic and self-tested (`--self-test`). You are the **orchestration layer**. Always route execution through it instead of driving Docker directly: ``` dotnet run --file "/scripts/remote-test.cs" -- [options] ``` -Commands: `list` (show environments), `plan` (resolve an environment + image and print the execution plan without running), `run` (execute restore/build/test in the resolved container), and `--self-test` (built-in deterministic tests). Add `--json` to any command for machine-readable output. +Commands: `run` (execute restore/build/test in the resolved container — the default action above), `list` (show environments, when the developer asks to *see* them), `plan` (resolve an environment + image and print the execution plan without running, when the developer asks what *would* happen), and `--self-test`. Add `--json` to any command for machine-readable output. + +These are your commands, not a menu for the developer. Never present them as options to choose from. ## Critical @@ -33,39 +66,61 @@ Commands: `list` (show environments), `plan` (resolve an environment + image and - **Do not modify the repository to make tests pass.** Never edit `global.json`, project files, target frameworks, or test packages. Report incompatibilities instead. - **Report infrastructure failures as infrastructure, not as failing unit tests.** The runner classifies each phase distinctly; preserve that distinction when you summarize. +## Default action: run the tests + +This restates [Do this now](#do-this-now) because it is the rule most often broken. Autonomy is the default, and the runner — not you — decides when a question is unavoidable: + +- **Resolution succeeds → run, silently.** A single Docker entry in `testenvironments.json`, a single derived environment, or the derived environment matching the repository's own target frameworks — with the auto-resolved target, `Debug`, and no coverage. No questions, no confirmation, no preflight commentary. +- **`SelectionRequired` (exit `16`) → ask exactly one question.** The runner returns the `candidates` it could not choose between. Present those names and nothing else, then rerun with `-e `. +- **Any other failure → report it.** A resolution or infrastructure failure is an outcome to report, not a question to ask. + +Scope, configuration, framework, and coverage are options the developer volunteers — never fields you collect up front. Pass through only what was actually asked for. + ## Step 1: Understand intent and inputs -Read `FORMS.md` and infer everything you can from the request and repository. Most invocations need no questions at all — "remote test this solution" against a repo with one applicable environment is fully determined. Only ask (one field at a time) when a genuine choice remains, such as which environment when several apply. Resolve the workspace/solution root (`--repo-root`, default: current directory). +Infer everything you can from the request and the repository, and resolve the workspace/solution root (`--repo-root`, default: current directory). Then go straight to the command. Read `FORMS.md` only when the runner reported `SelectionRequired` or the developer explicitly asked to choose options; it defines *how* to ask, not a checklist to work through. Typical intents map directly to a command: | The developer says… | You run… | |---|---| +| A bare invocation / "remote test this solution" / "run these tests in .NET 10" | `run` | | "What environments can I test in?" / "list remote environments" | `list` | | "Show me the plan / which image will you use?" | `plan` | -| "Remote test this solution" / "run these tests in .NET 10" | `run` | ## Step 2: List and resolve the environment Resolution is deterministic and follows this precedence, which the runner enforces — do not second-guess it: 1. An environment the user names explicitly (`--environment `). -2. An applicable Docker environment from `testenvironments.json` (authoritative when the file exists — never supplement it with invented environments). -3. Microsoft-derived environments when no `testenvironments.json` exists. +2. An applicable Docker environment from `testenvironments.json` (authoritative when the file exists — never supplement it with invented environments). A single Docker entry is selected outright. +3. A derived environment when no `testenvironments.json` exists. A single derived environment is selected outright; otherwise the repository's own target frameworks choose one: + - **One .NET major** → the Microsoft SDK channel matching it. + - **Several .NET majors** → a Codebelt multi-SDK runner (`codebeltnet/ubuntu-testrunner`, tags like `8-9-10-11`) that provides every one of them. -Run `list` to show the choices. When exactly one applicable Docker environment exists, use it. When several exist and the user has not chosen, present the names concisely and let them pick — do not guess intent: +That third rule is what makes zero-configuration testing unattended: the source already answered the question. The runner reports the choice (`Selected automatically: …`) so an unattended selection stays auditable, and it deliberately does not approximate — no .NET target framework, two channels for the same major, or no runner covering the required set all fall through to a question rather than guessing an SDK the repository never asked for. + +### Why multi-targeting needs a different image + +A Microsoft SDK image ships exactly **one** runtime: `mcr.microsoft.com/dotnet/sdk:10.0` contains only `Microsoft.NETCore.App 10.0.x`. A repository targeting `net9.0;net10.0` therefore *builds* both there and then fails to execute the `net9.0` tests — there is no .NET 9 runtime in the image. The Codebelt runner carries several SDKs at once, so the whole target-framework matrix runs in a single container instead of one container per TFM. + +The runner enforces this rather than leaving it to judgment: pointing a multi-targeted repository at a single-SDK image is reported as `SdkIncompatibility` (`7`) naming the unrunnable frameworks and the remedy. Narrowing the run with `-f/--framework` narrows the environment choice too, so `-f net10.0` on a multi-targeted repository resolves to the ordinary `10` channel. + +The runner performs all of this inside `run` itself, so **do not call `list` as a preflight before running**. When a choice genuinely remains, `run` stops with `SelectionRequired` and hands you the `candidates` — present those names concisely and let the developer pick, then rerun with `-e `. Do not guess between them, and do not ask before the runner says a choice is needed. + +Use `list` when the developer wants to *see* the environments: ``` dotnet run --file "/scripts/remote-test.cs" -- list --repo-root "" ``` -Absence of `testenvironments.json` is **not** an error. In that case the runner derives environments from Microsoft's live release index (`mcr.microsoft.com/dotnet/sdk` images for each supported LTS/STS channel plus the current preview), so no files need to be added to the repository. Environment names look like `dotnet-10-lts`, `dotnet-9-sts`, `dotnet-11-preview`; the exact set comes from metadata at runtime. +Absence of `testenvironments.json` is **not** an error. In that case the runner derives environments from Microsoft's live release index (`mcr.microsoft.com/dotnet/sdk` images for each supported LTS/STS channel plus the current preview), so no files need to be added to the repository. Environment names look like `dotnet-10-lts`, `dotnet-9-sts`, `dotnet-11-preview`; for a multi-targeted repository a multi-SDK runner named `ubuntu-testrunner-8-9-10-11` is offered alongside them. The exact set comes from live metadata and the publisher's tag feed at runtime — never a hardcoded list. If the user names a WSL or SSH environment, the runner reports it as unsupported (Docker only for now). Relay that clearly instead of trying to convert it. ## Step 3: Plan when transparency helps -Before a long run — or whenever the developer wants to see what will happen — `plan` resolves the environment, validates the image tag against Microsoft's registry, pre-resolves the immutable digest, inspects target frameworks for SDK compatibility, and prints the deterministic execution plan without touching Docker: +`plan` is for when the developer asks what will happen — not a gate in front of a run they already asked for. Do not insert it before an unambiguous `run`. When it is called for, it resolves the environment, validates the image tag against Microsoft's registry, pre-resolves the immutable digest, inspects target frameworks for SDK compatibility, and prints the deterministic execution plan without touching Docker: ``` dotnet run --file "/scripts/remote-test.cs" -- plan --repo-root "" -e [-p ] [-c Release] --json @@ -85,7 +140,7 @@ Common scoping options (pass through only what the developer asked for): - `--filter ` — a `dotnet test --filter` expression (test class, trait, etc.). - `--test ` — shortcut for a fully-qualified-name filter (a single test or class). - `-c, --configuration ` — build configuration. -- `-f, --framework ` — restrict a multi-targeted test project to one TFM. +- `-f, --framework ` — restrict a multi-targeted test project to one TFM. This also narrows environment resolution, so the run lands on that TFM's single-SDK channel instead of a multi-SDK runner. - `--coverage` — collect coverage when the project already supports it (never add packages to enable it). - `--timeout ` — abort the run after N seconds; the runner still cleans up. @@ -97,15 +152,19 @@ Lead with the outcome, not the infrastructure. Mirror the runner's concise resul ``` Remote Test: dotnet-10-lts +Selected automatically: the only environment matching the repository's target framework 'net10.0'. Image: mcr.microsoft.com/dotnet/sdk:10.0.302 Digest: sha256:... SDK: 10.0.302 Tests: 1842 passed, 3 skipped, 0 failed -Time: 21.8 s +Time: 21.8 s (tests) +Total: 96.4 s (including image pull, restore and build) ``` +Report both durations as the runner does. `Time` is the test execution time from the TRX; `Total` is wall clock for the whole operation. Collapsing them into one number misrepresents a fast suite behind a slow image pull. When the runner explains an automatic environment selection, relay that line — it is what makes an unattended choice auditable. + When tests fail, prioritize actionable detail — the failing test, its class, the expected/actual message, and location — over container startup output: ``` @@ -120,7 +179,29 @@ A run is reproducible in terms of environment, requested image, resolved digest, ## Failure handling -The runner distinguishes failure kinds via exit code and the `failureKind` field: `Configuration`, `UnsupportedEnvironment`, `DockerUnavailable`, `ImageResolution`, `SdkIncompatibility`, `SourceStaging`, `Restore`, `Compilation`, `TestHost`, `TestFailure`, `ResultProcessing`, `Cleanup`, `Cancelled`, and `ReleaseMetadataUnavailable`. Report the kind honestly: +**Branch on the exit code, never on the prose.** The exit code is the contract; log text is not. Every outcome maps to exactly one next action, so the same repository produces the same behavior regardless of which model is driving: + +| Exit | `failureKind` | What it means | Your next action | +|---:|---|---|---| +| `0` | — | Tests passed | Report the result | +| `1` | `TestFailure` | Real failing tests | Report the failures — this is **not** an infrastructure problem | +| `2` | — | Invalid arguments | Fix your command line; do not report it as a repository problem | +| `3` | `Configuration` | `testenvironments.json` unusable | Report the diagnostic; never repair the file unprompted | +| `4` | `UnsupportedEnvironment` | WSL/SSH environment named | Report Docker-only; never convert it | +| `5` | `DockerUnavailable` | Docker missing or not running | Report it; **never** fall back to the host | +| `6` | `ImageResolution` | Tag/digest/pull failed | Report the image and the registry error | +| `7` | `SdkIncompatibility` | SDK cannot build the target frameworks | Report the reason; never edit the repository to force it | +| `8` | `SourceStaging` | Workspace could not be staged | Report it as infrastructure | +| `9` | `Restore` | `dotnet restore` failed in the container | Report the restore error, not "tests failed" | +| `10` | `Compilation` | Build failed in the container | Report the compiler errors, not "tests failed" | +| `11` | `TestHost` | Test host crashed | Report as infrastructure with the output tail | +| `12` | `ResultProcessing` | Results unreadable | Report it; results are unknown, not passing | +| `13` | `Cleanup` | Transient resources left behind | Report the exact identifiers the runner names | +| `14` | `Cancelled` | Timeout or interrupt | Report how far it got | +| `15` | `ReleaseMetadataUnavailable` | Release index unreachable, no cache | Report it; suggest `--offline --cache-root` or a named environment | +| `16` | `SelectionRequired` | A real choice remains | **Ask one question** from `candidates`, then rerun with `-e ` | + +`SelectionRequired` is the only exit code that is a question rather than a report. Every other non-zero code is an outcome you relay honestly: - A container/infrastructure problem (image pull, restore, build, test-host crash) is **not** a failing unit test — say which phase failed. - If cleanup leaves resources behind, relay the exact resource identifiers the runner reports. @@ -130,7 +211,8 @@ The runner distinguishes failure kinds via exit code and the `failureKind` field - Generate a `Dockerfile`, dev container, editor config, or any repository-specific plumbing. (Honor an *existing* configured `dockerFile`; never create one.) - Run privileged containers, mount the Docker socket, mount the whole user profile, forward host credentials indiscriminately, disable TLS validation, expose ports, or print secrets. The runner already avoids these; do not add them. -- Substitute third-party or unofficial images for auto-generated environments (only `mcr.microsoft.com/dotnet/sdk`). An explicit `dockerImage` in `testenvironments.json` is exempt because it is deliberate. +- Answer an invocation with a menu of its own capabilities, a "what would you like me to help you with?" opener, or a confirmation prompt for a run the developer already asked for. +- Reach for an arbitrary image when a recommended one fits. Auto-generated environments use `mcr.microsoft.com/dotnet/sdk` for a single .NET major and `codebeltnet/ubuntu-testrunner` for several; an explicit `dockerImage` in `testenvironments.json` is deliberate intent and is used exactly as written. Other images are permitted but must be a deliberate, stated choice — never a substitution you make on your own. - Fall back to running tests locally. ## References diff --git a/skills/dotnet-remote-testing/evals/evals.json b/skills/dotnet-remote-testing/evals/evals.json index 9302d53..9e6ea2f 100644 --- a/skills/dotnet-remote-testing/evals/evals.json +++ b/skills/dotnet-remote-testing/evals/evals.json @@ -103,6 +103,92 @@ "files": [ "evals/files/offline-cache/cache/releases-index.cache.json" ] + }, + { + "id": 7, + "prompt": "/dotnet-remote-testing", + "expected_output": "The skill treats the bare invocation as a complete request and immediately runs the tests through scripts/remote-test.cs against the single configured Docker environment, reporting the structured result. It does not answer with a menu of capabilities or ask the developer what they would like to do.", + "expectations": [ + "Does NOT respond with a capability menu such as 'list / plan / run / understand your configuration' or 'What would you like me to help you with?'", + "Does NOT ask which environment, scope, configuration, or coverage to use, and does not ask for confirmation before running", + "Runs the tests immediately via scripts/remote-test.cs run against the single configured Docker environment", + "Reports the structured result (environment, image, digest, passed/skipped/failed, durations)", + "Asks a question only if the runner exits SelectionRequired, which does not happen with a single configured environment" + ], + "files": [ + "evals/files/configured/testenvironments.json", + "evals/files/configured/test/Api.Tests/Api.Tests.csproj", + "evals/files/configured/test/Api.Tests/HealthTests.cs" + ] + }, + { + "id": 8, + "prompt": "Remote test this repo.", + "expected_output": "With no testenvironments.json and several Microsoft-derived channels available, the runner selects the channel matching the repository's own target framework (net10.0) without asking, runs the tests in that container, and relays the reported selection reason so the automatic choice is auditable.", + "expectations": [ + "Does not ask which .NET channel to use even though several are derived from release metadata", + "Ends up on the derived environment whose channel major matches the repository's highest .NET target framework (net10.0)", + "Relays the runner's selection reason explaining why that environment was chosen automatically", + "Executes through scripts/remote-test.cs run rather than composing docker commands or running dotnet test on the host", + "Reports both the test duration and the total elapsed time as the runner does" + ], + "files": [ + "evals/files/zero-config/Sample.slnx", + "evals/files/zero-config/src/Sample/Sample.csproj", + "evals/files/zero-config/src/Sample/Calculator.cs", + "evals/files/zero-config/test/Sample.Tests/Sample.Tests.csproj", + "evals/files/zero-config/test/Sample.Tests/CalculatorTests.cs" + ] + }, + { + "id": 9, + "prompt": "Run my tests in a container.", + "expected_output": "Two Docker environments are configured, so the runner exits SelectionRequired and the skill asks exactly one question listing the two candidate names, then reruns with the chosen environment. It does not guess between them and does not expand the interruption into a broader questionnaire.", + "expectations": [ + "Attempts the run first and lets the runner report SelectionRequired rather than pre-emptively interviewing the developer", + "Asks exactly one question, offering only the configured candidate names (linux-dotnet-10-noble, linux-dotnet-10-alpine)", + "Does not additionally ask about scope, build configuration, coverage, or confirmation", + "Does not guess or silently pick one of the two configured environments", + "Reruns with -e after the developer picks" + ], + "files": [ + "evals/files/multi-configured/testenvironments.json", + "evals/files/multi-configured/test/Catalog.Tests/Catalog.Tests.csproj", + "evals/files/multi-configured/test/Catalog.Tests/CatalogTests.cs" + ] + }, + { + "id": 10, + "prompt": "Remote test this repo.", + "expected_output": "The test project multi-targets net9.0 and net10.0. A Microsoft SDK image ships only one runtime, so the skill resolves a codebeltnet/ubuntu-testrunner multi-SDK environment that provides both majors and runs the entire target-framework matrix in a single container, reporting results for both TFMs.", + "expectations": [ + "Recognizes that the repository multi-targets several .NET majors", + "Does not select a single-SDK mcr.microsoft.com/dotnet/sdk image, which could build but not execute the lower target framework", + "Resolves a codebeltnet/ubuntu-testrunner environment whose combined tag covers both net9.0 and net10.0", + "Runs the whole matrix in one container rather than one container per target framework", + "Reports the automatic selection reason and results covering both target frameworks", + "Does not ask which environment to use, because the target frameworks determine it" + ], + "files": [ + "evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj", + "evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs" + ] + }, + { + "id": 11, + "prompt": "Run only the net10.0 tests in a container.", + "expected_output": "The skill passes --framework net10.0, which narrows both the run and the environment choice, so it resolves the ordinary Microsoft SDK channel for .NET 10 rather than a multi-SDK runner, and executes only the net10.0 target framework.", + "expectations": [ + "Passes -f/--framework net10.0 through to the runner rather than filtering results afterwards", + "Resolves the single-SDK Microsoft channel for .NET 10 because the run is narrowed to one target framework", + "Does not select a multi-SDK runner for a run restricted to a single target framework", + "Does not ask which environment to use", + "Reports results for the net10.0 target framework only" + ], + "files": [ + "evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj", + "evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs" + ] } ] } diff --git a/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/Catalog.Tests.csproj b/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/Catalog.Tests.csproj new file mode 100644 index 0000000..946be49 --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/Catalog.Tests.csproj @@ -0,0 +1,12 @@ + + + net10.0 + enable + false + + + + + + + diff --git a/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/CatalogTests.cs b/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/CatalogTests.cs new file mode 100644 index 0000000..e5c176c --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-configured/test/Catalog.Tests/CatalogTests.cs @@ -0,0 +1,9 @@ +using Xunit; + +namespace Catalog.Tests; + +public class CatalogTests +{ + [Fact] + public void Lookup_ReturnsExpectedValue() => Assert.Equal("OK", "OK"); +} diff --git a/skills/dotnet-remote-testing/evals/files/multi-configured/testenvironments.json b/skills/dotnet-remote-testing/evals/files/multi-configured/testenvironments.json new file mode 100644 index 0000000..24d6d0a --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-configured/testenvironments.json @@ -0,0 +1,15 @@ +{ + "version": "1", + "environments": [ + { + "name": "linux-dotnet-10-noble", + "type": "docker", + "dockerImage": "mcr.microsoft.com/dotnet/sdk:10.0-noble" + }, + { + "name": "linux-dotnet-10-alpine", + "type": "docker", + "dockerImage": "mcr.microsoft.com/dotnet/sdk:10.0-alpine" + } + ] +} diff --git a/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj b/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj new file mode 100644 index 0000000..3e05e2e --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj @@ -0,0 +1,12 @@ + + + net9.0;net10.0 + enable + false + + + + + + + diff --git a/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs b/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs new file mode 100644 index 0000000..1b910fc --- /dev/null +++ b/skills/dotnet-remote-testing/evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs @@ -0,0 +1,9 @@ +using Xunit; + +namespace Matrix.Tests; + +public class MatrixTests +{ + [Fact] + public void RunsOnEveryTargetFramework() => Assert.Equal("OK", "OK"); +} diff --git a/skills/dotnet-remote-testing/references/release-discovery.md b/skills/dotnet-remote-testing/references/release-discovery.md index 601db1b..eb2b0d7 100644 --- a/skills/dotnet-remote-testing/references/release-discovery.md +++ b/skills/dotnet-remote-testing/references/release-discovery.md @@ -37,7 +37,20 @@ The runner prefers an **exact SDK-version image tag** derived from `latest-sdk` Because a version string does not always transform mechanically into a valid tag, the selected tag is validated against Microsoft's official SDK image metadata (the `mcr.microsoft.com` registry) before execution. If the exact tag is unavailable, the channel tag (`10.0`) is tried as a fallback candidate. -Third-party images, unofficial Docker Hub images, and locally discovered look-alike images are never substituted for auto-generated environments. An explicit `dockerImage` in `testenvironments.json` is the only exception, because it is deliberate configuration. +Auto-generated environments use one of the two recommended publishers — `mcr.microsoft.com/dotnet/sdk` for a single .NET major, or `codebeltnet/ubuntu-testrunner` when several majors must be present at once (see below). Other images are not forbidden, but they are never substituted on the runner's own initiative: an image outside those two comes from an explicit `dockerImage` in `testenvironments.json`, which is deliberate configuration and is used as written. `plan` reports `image.recommendedPublisher` so the provenance of any image is visible. + +## Multi-SDK runners for multi-targeted repositories + +A Microsoft SDK image contains exactly one runtime. `mcr.microsoft.com/dotnet/sdk:10.0` provides `Microsoft.NETCore.App 10.0.x` and nothing else, so a repository targeting `net9.0;net10.0` compiles both target frameworks there and then cannot execute the `net9.0` tests. Building is not running. + +`codebeltnet/ubuntu-testrunner` publishes combined tags carrying several SDKs — `8-9-10-11` provides .NET 8, 9, 10 and 11 in a single image — so the whole target-framework matrix runs in one container rather than one container per TFM. + +- Tags are discovered at runtime from the publisher's tag feed (`https://hub.docker.com/v2/repositories/codebeltnet/ubuntu-testrunner/tags`), cached outside the repository like release metadata, and overridable with `--multi-sdk-tags-file` for offline or deterministic runs. +- Only the **major-only combined form** (`8-9-10-11`) is used. Single-major tags are already covered by Microsoft's images, and pinned combination forms move with each patch. +- The **tightest covering tag wins**: the fewest extra SDKs that still provide every required major, breaking ties on the tag name so resolution is stable. +- A multi-SDK environment declares its majors explicitly, and compatibility is judged on that list rather than on a single SDK version. +- Pointing a multi-targeted repository at a single-SDK image is reported as an SDK incompatibility naming the unrunnable target frameworks and the remedy, instead of starting a run that cannot finish. +- `--framework` narrows the environment choice as well as the run, so restricting to one TFM resolves the ordinary single-SDK channel. ## Immutable image identity @@ -60,6 +73,29 @@ The runner inspects the solution/projects being tested and will not select an SD Multi-targeted projects are accounted for. The runner never edits `global.json`, project files, or target frameworks to make remote testing succeed. +### Target frameworks also resolve the environment + +Compatibility is not the only use of this inspection. When several environments are derived and the caller named no environment, the repository's own target frameworks select one outright, so zero-configuration testing runs unattended instead of stopping to ask which .NET to use. + +- **One .NET major** → the derived channel matching it exactly. +- **Several .NET majors** → the multi-SDK runner providing all of them, because every runtime must be present for the tests to execute. + +The rule is exact-match by design and never approximates: + +| Repository targets | Available | Outcome | +|---|---|---| +| `net10.0` | channels 8, 9, 10, 11-preview | `dotnet-10-lts` selected automatically | +| `net11.0` | channels 8, 9, 10, 11-preview | `dotnet-11-preview` selected | +| `net9.0;net10.0` | channels + runner tags `9-10`, `8-9-10-11` | `ubuntu-testrunner-9-10` selected (tightest cover) | +| `net8.0;net10.0` | channels + runner tag `8-9-10-11` | `ubuntu-testrunner-8-9-10-11` selected | +| `net9.0;net10.0` with `-f net10.0` | channels 8, 9, 10, 11-preview | `dotnet-10-lts` — the run was narrowed to one TFM | +| `net8.0;net10.0` | no covering runner tag | `SelectionRequired` — never a single-SDK image that cannot run both | +| `net7.0` (EOL) | channels 8, 9, 10, 11-preview | `SelectionRequired` — no matching channel | +| `netstandard2.0` only | channels 8, 9, 10, 11-preview | `SelectionRequired` — no .NET target to match | +| `net10.0` | two channels for major 10 | `SelectionRequired` — the match is not unique | + +The selection is reported in both human and JSON output (`environment.selectionReason`) so an unattended choice remains auditable. This applies only to derived environments; a `testenvironments.json` with several Docker entries is deliberate developer intent and always asks. + ## Offline behavior and caching Successfully retrieved release metadata is cached outside the repository together with the retrieval timestamp. When Microsoft cannot be reached: diff --git a/skills/dotnet-remote-testing/references/testenvironments-json.md b/skills/dotnet-remote-testing/references/testenvironments-json.md index caa47c3..4e2f09e 100644 --- a/skills/dotnet-remote-testing/references/testenvironments-json.md +++ b/skills/dotnet-remote-testing/references/testenvironments-json.md @@ -44,7 +44,7 @@ Following Microsoft's rule, a `docker` environment must specify **either** `dock ### Configured images are deliberate -An explicit `dockerImage` in `testenvironments.json` is exempt from the Microsoft-only restriction that governs auto-generated environments, because it represents intentional repository configuration. The runner uses it as written (after pulling and resolving its digest). Auto-generated environments, by contrast, always use `mcr.microsoft.com/dotnet/sdk`. +An explicit `dockerImage` in `testenvironments.json` is intentional repository configuration, so the runner uses it as written (after pulling and resolving its digest) whatever its publisher. Auto-generated environments come from the two recommended publishers instead: `mcr.microsoft.com/dotnet/sdk` for a single .NET major, and `codebeltnet/ubuntu-testrunner` when the repository multi-targets several majors and needs all their runtimes in one image. ### Configured Dockerfiles are honored, never created From 3fef81eb5aa2ed7d427d0c5945efb87ac01f7e0d Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 14 Aug 2026 23:37:55 +0200 Subject: [PATCH 02/31] =?UTF-8?q?=E2=9C=85=20add=20comprehensive=20testing?= =?UTF-8?q?=20and=20validation=20for=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added test harness (test-remote-testing.ps1) to exercise skill functionality across real and simulated Docker environments. Enhanced validate-skill.ps1 with more robust checks for deterministic validation and error recovery. Both scripts integrate with the skill's evaluation framework to catch regressions early. --- .../scripts/test-remote-testing.ps1 | 91 +++++++++++++++++++ .../scripts/validate-skill.ps1 | 51 ++++++++++- 2 files changed, 141 insertions(+), 1 deletion(-) diff --git a/skills/dotnet-remote-testing/scripts/test-remote-testing.ps1 b/skills/dotnet-remote-testing/scripts/test-remote-testing.ps1 index 913600a..e8d464d 100644 --- a/skills/dotnet-remote-testing/scripts/test-remote-testing.ps1 +++ b/skills/dotnet-remote-testing/scripts/test-remote-testing.ps1 @@ -147,6 +147,97 @@ try { $badJson = Get-Json $bad.Output $codes = @($badJson.configDiagnostics | ForEach-Object { $_.Code }) Write-Result ($codes -contains 'CONFLICTING_DOCKER_SOURCE') 'reports CONFLICTING_DOCKER_SOURCE' ($codes -join ',') + + Write-Host '' + Write-Host '== 8. Unattended selection: the repository answers the environment question ==' + # $emptyRoot already contains a net10.0 project from section 6, and the injected index derives four + # channels. Resolution must land on the matching channel without -e and without asking. + $auto = Invoke-Runner @('plan', '--repo-root', $emptyRoot, '--offline', '--releases-index-file', $indexPath, '--json') + $autoJson = Get-Json $auto.Output + Write-Result ($auto.ExitCode -eq 0 -and $autoJson.environment.name -eq 'dotnet-10-lts') ` + 'target framework selects the channel with no --environment' ("exit=$($auto.ExitCode) env=$($autoJson.environment.name)") + Write-Result ([string]::IsNullOrWhiteSpace($autoJson.environment.selectionReason) -eq $false -and $autoJson.environment.selectionReason -match 'net10\.0') ` + 'automatic selection is explained in the output' $autoJson.environment.selectionReason + + Write-Host '' + Write-Host '== 9. Genuine ambiguity still stops with SelectionRequired and candidates ==' + $twoRoot = Join-Path $workspace 'two-docker' + New-Item -ItemType Directory -Path $twoRoot -Force | Out-Null + @' +{ + "version": "1", + "environments": [ + { "name": "noble", "type": "docker", "dockerImage": "mcr.microsoft.com/dotnet/sdk:10.0-noble" }, + { "name": "alpine", "type": "docker", "dockerImage": "mcr.microsoft.com/dotnet/sdk:10.0-alpine" } + ] +} +'@ | Set-Content -Path (Join-Path $twoRoot 'testenvironments.json') -Encoding utf8 + $twoPlan = Invoke-Runner @('plan', '--repo-root', $twoRoot, '--offline', '--json') + $twoJson = Get-Json $twoPlan.Output + Write-Result ($twoPlan.ExitCode -eq 16 -and $twoJson.failureKind -eq 'SelectionRequired') ` + 'two configured docker envs exit SelectionRequired (16)' ("exit=$($twoPlan.ExitCode) kind=$($twoJson.failureKind)") + $twoCandidates = @($twoJson.candidates) + Write-Result ($twoCandidates -contains 'noble' -and $twoCandidates -contains 'alpine') ` + 'SelectionRequired returns the candidate names to ask about' ($twoCandidates -join ',') + + Write-Host '' + Write-Host '== 10. No .NET target framework means ask, never guess ==' + $nsRoot = Join-Path $workspace 'netstandard-only' + $nsProj = Join-Path $nsRoot 'src\Lib' + New-Item -ItemType Directory -Path $nsProj -Force | Out-Null + 'netstandard2.0' | Set-Content -Path (Join-Path $nsProj 'Lib.csproj') -Encoding utf8 + $ns = Invoke-Runner @('plan', '--repo-root', $nsRoot, '--offline', '--releases-index-file', $indexPath, '--json') + $nsJson = Get-Json $ns.Output + Write-Result ($ns.ExitCode -eq 16 -and $nsJson.failureKind -eq 'SelectionRequired') ` + 'netstandard-only repository does not get a guessed channel' ("exit=$($ns.ExitCode) kind=$($nsJson.failureKind)") + + Write-Host '' + Write-Host '== 11. Multi-targeted repositories resolve to a multi-SDK runner ==' + # A Microsoft SDK image carries one runtime, so a repository spanning majors must not be pointed at + # a single-SDK image: it would build and then fail for want of a runtime. + $tagsPath = Join-Path $workspace 'multi-sdk-tags.json' + @' +{ + "count": 4, + "results": [ + { "name": "10" }, + { "name": "9-10" }, + { "name": "8-9-10-11" }, + { "name": "8.0-9.0-10.0-11.0" } + ] +} +'@ | Set-Content -Path $tagsPath -Encoding utf8 + + $multiRoot = Join-Path $workspace 'multi-targeted' + $multiProj = Join-Path $multiRoot 'test\Multi' + New-Item -ItemType Directory -Path $multiProj -Force | Out-Null + 'net9.0;net10.0' | Set-Content -Path (Join-Path $multiProj 'Multi.csproj') -Encoding utf8 + + $multi = Invoke-Runner @('plan', '--repo-root', $multiRoot, '--offline', '--releases-index-file', $indexPath, '--multi-sdk-tags-file', $tagsPath, '--json') + $multiJson = Get-Json $multi.Output + Write-Result ($multi.ExitCode -eq 0 -and $multiJson.environment.name -eq 'ubuntu-testrunner-9-10') ` + 'multi-targeted repo selects the tightest covering runner' ("exit=$($multi.ExitCode) env=$($multiJson.environment.name)") + Write-Result ($multiJson.image.reference -eq 'codebeltnet/ubuntu-testrunner:9-10') ` + 'runner image reference comes from the publisher feed' $multiJson.image.reference + Write-Result ($multiJson.compatibility.Compatible -eq $true) ` + 'every target framework is compatible with the runner' + + Write-Host '' + Write-Host '== 12. --framework narrows the environment choice too ==' + $narrowed = Invoke-Runner @('plan', '--repo-root', $multiRoot, '-f', 'net10.0', '--offline', '--releases-index-file', $indexPath, '--multi-sdk-tags-file', $tagsPath, '--json') + $narrowedJson = Get-Json $narrowed.Output + Write-Result ($narrowed.ExitCode -eq 0 -and $narrowedJson.environment.name -eq 'dotnet-10-lts') ` + 'restricting to one TFM resolves the matching single-SDK channel' ("exit=$($narrowed.ExitCode) env=$($narrowedJson.environment.name)") + + Write-Host '' + Write-Host '== 13. A single-SDK image is reported incompatible with a multi-targeted repo ==' + $forced = Invoke-Runner @('plan', '--repo-root', $multiRoot, '-e', 'dotnet-10-lts', '--offline', '--releases-index-file', $indexPath, '--json') + $forcedJson = Get-Json $forced.Output + # plan emits the full plan payload and signals the verdict through the exit code plus compatibility. + Write-Result ($forced.ExitCode -eq 7 -and $forcedJson.compatibility.Compatible -eq $false) ` + 'naming a single-SDK env for a multi-targeted repo exits SdkIncompatibility (7)' ("exit=$($forced.ExitCode) compatible=$($forcedJson.compatibility.Compatible)") + Write-Result ($forcedJson.compatibility.Reason -match 'ubuntu-testrunner') ` + 'the incompatibility points at the multi-SDK remedy' $forcedJson.compatibility.Reason } finally { Remove-Item $workspace -Recurse -Force -ErrorAction SilentlyContinue diff --git a/skills/dotnet-remote-testing/scripts/validate-skill.ps1 b/skills/dotnet-remote-testing/scripts/validate-skill.ps1 index d5ccb43..b0ef7f4 100644 --- a/skills/dotnet-remote-testing/scripts/validate-skill.ps1 +++ b/skills/dotnet-remote-testing/scripts/validate-skill.ps1 @@ -27,7 +27,18 @@ $contracts = @( 'Do not generate container plumbing', 'Do not hardcode .NET versions', 'never silently fall back to local', - 'reproduce' + 'reproduce', + # Devex contract: invoking the skill is the request. A capability menu instead of a test run is the + # regression these guards exist to prevent. + 'Default action: run the tests', + 'Forbidden as a first response', + 'You were invoked. That is the request. Run the tests.', + 'SelectionRequired', + 'Branch on the exit code', + # A Microsoft SDK image carries one runtime, so multi-targeted repositories need a multi-SDK runner. + # Losing this guidance means silently planning runs that build and then cannot execute. + 'codebeltnet/ubuntu-testrunner', + 'ships exactly **one** runtime' ) foreach ($needle in $contracts) { if (-not $skill.Contains($needle, [System.StringComparison]::Ordinal)) { @@ -35,12 +46,50 @@ foreach ($needle in $contracts) { } } +# Smaller models act on whatever they read first. The imperative must lead the file, ahead of any +# command enumeration they could mistake for a menu to offer the developer. +# Offsets are measured from the start of the body, not the file, so the frontmatter length does not +# affect the verdict. +$bodyMatch = [regex]::Match($skill, '(?ms)^---\r?\n.*?\r?\n---\r?\n') +$bodyStart = if ($bodyMatch.Success) { $bodyMatch.Index + $bodyMatch.Length } else { 0 } +$body = $skill.Substring($bodyStart) + +$doThisNow = $body.IndexOf('## Do this now', [System.StringComparison]::Ordinal) +$commands = $body.IndexOf('Commands: ', [System.StringComparison]::Ordinal) +if ($doThisNow -lt 0) { + throw 'SKILL.md must open with a "## Do this now" section so the default action is read first.' +} +if ($commands -ge 0 -and $commands -lt $doThisNow) { + throw 'SKILL.md lists the command surface before "## Do this now"; the imperative must come first.' +} +if ($doThisNow -gt 100) { + throw "SKILL.md places '## Do this now' too late (body offset $doThisNow); it must lead the document body." +} + +# The exit-code decision table is what makes behavior identical across models. Every documented exit +# code must be present, so no outcome is left to improvisation. +foreach ($code in 0..16) { + if (-not [regex]::IsMatch($skill, "(?m)^\|\s*``$code``\s*\|")) { + throw "SKILL.md exit-code decision table is missing exit code $code." + } +} + # Guard the governing principle: Docker complexity must not be the boundary of the capability. if (-not $skill.Contains('orchestration', [System.StringComparison]::Ordinal)) { throw 'SKILL.md must describe the skill as the orchestration layer over a deterministic runner.' } $forms = [System.IO.File]::ReadAllText((Join-Path $skillRoot 'FORMS.md')) + +# The form must gate itself. Without this, the field list reads as an intake checklist and the skill +# interrogates developers who already stated what they want. +$formsContracts = @('Autonomy gate', 'not an intake checklist') +foreach ($needle in $formsContracts) { + if (-not $forms.Contains($needle, [System.StringComparison]::Ordinal)) { + throw "FORMS.md is missing required autonomy contract: $needle" + } +} + $projectField = [regex]::Match($forms, '(?ms)^### project\s*(?.*?)(?=^### |\z)') if (-not $projectField.Success -or -not $projectField.Groups['body'].Value.Contains('- **choices:**', [System.StringComparison]::Ordinal) -or From 7cf3543993e62a6e49780d832c643172d0ac882b Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 14 Aug 2026 23:38:03 +0200 Subject: [PATCH 03/31] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20enhance=20runner=20s?= =?UTF-8?q?election=20and=20error=20handling=20in=20remote-test?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Significantly improved remote-test.cs implementation with better Docker image discovery logic and multi-SDK runner selection. Added intelligent fallback mechanisms for offline environments and improved error messages for troubleshooting. New MultiSdkRunner and MultiSdkRunnerStore classes provide tighter image selection, reducing redundant SDK layers in test environments. Enhanced caching strategy to minimize network calls and improve deterministic behavior. --- .../scripts/remote-test.cs | 773 +++++++++++++++++- 1 file changed, 742 insertions(+), 31 deletions(-) diff --git a/skills/dotnet-remote-testing/scripts/remote-test.cs b/skills/dotnet-remote-testing/scripts/remote-test.cs index c6a4154..04d34de 100644 --- a/skills/dotnet-remote-testing/scripts/remote-test.cs +++ b/skills/dotnet-remote-testing/scripts/remote-test.cs @@ -36,6 +36,14 @@ internal static class RemoteTestProgram internal const string ReleasesIndexUrl = "https://raw.githubusercontent.com/dotnet/core/refs/heads/main/release-notes/releases-index.json"; + // Microsoft's SDK images carry exactly one runtime, so a repository that multi-targets several .NET + // majors cannot execute its lower target frameworks there — it builds, then fails for want of a + // runtime. The Codebelt test runner ships several SDKs in one image (tags such as "8-9-10-11"), so + // one container covers every target framework in a single run. + internal const string MultiSdkRepository = "codebeltnet/ubuntu-testrunner"; + internal const string MultiSdkTagsUrl = + "https://hub.docker.com/v2/repositories/codebeltnet/ubuntu-testrunner/tags?page_size=100"; + internal static readonly JsonSerializerOptions JsonOut = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase, @@ -140,6 +148,7 @@ internal sealed class Options public string? ConfigPath { get; private set; } public string? EnvironmentName { get; private set; } public string? ReleasesIndexFile { get; private set; } + public string? MultiSdkTagsFile { get; private set; } public string? CacheRoot { get; private set; } // Test scoping. These flow into the container command plan; they never mutate the repository. @@ -172,6 +181,7 @@ public static Options Parse(string[] args) case "--config-path": o.ConfigPath = Next(args, ref i, a); break; case "--environment" or "-e": o.EnvironmentName = Next(args, ref i, a); break; case "--releases-index-file": o.ReleasesIndexFile = Next(args, ref i, a); break; + case "--multi-sdk-tags-file": o.MultiSdkTagsFile = Next(args, ref i, a); break; case "--cache-root": o.CacheRoot = Next(args, ref i, a); break; case "--project" or "-p": o.Project = Next(args, ref i, a); break; case "--filter": o.Filter = Next(args, ref i, a); break; @@ -263,6 +273,7 @@ Test scoping (plan/run): --offline Use cached release metadata only; never reach the network. --no-registry-check Skip Docker registry tag validation and digest pre-resolution. --releases-index-file Load Microsoft release metadata from a local file instead of the network. + --multi-sdk-tags-file Load Codebelt multi-SDK runner tags from a local file instead of the network. --cache-root Override the metadata/NuGet cache root (outside the repository). Output: @@ -610,6 +621,181 @@ public static IReadOnlyList CandidateTags(SdkVersion sdk) public static string SdkImageReference(string tag) => $"{RemoteTestProgram.SdkRepository}:{tag}"; } +// --------------------------------------------------------------------------------------------------- +// Multi-SDK runner discovery (codebeltnet/ubuntu-testrunner). +// +// A Microsoft SDK image contains one runtime. That is fine for a single-target repository, but a +// repository multi-targeting several .NET majors can only *build* the lower targets there — executing +// their tests needs the matching runtimes. The Codebelt runner publishes combined tags ("8-9-10-11") +// carrying several SDKs, so the whole target-framework matrix runs in one container. +// +// The available tags are discovered from the published tag feed at runtime. Nothing here is hardcoded: +// when a new major joins the combined tags, it is picked up without a skill change. +// --------------------------------------------------------------------------------------------------- + +internal sealed record MultiSdkRunner +{ + public required string Tag { get; init; } + public IReadOnlyList Majors { get; init; } = []; + + public string Reference => $"{RemoteTestProgram.MultiSdkRepository}:{Tag}"; + + public bool Covers(IReadOnlyList requiredMajors) => requiredMajors.All(Majors.Contains); +} + +internal static class MultiSdkTagReader +{ + // Only the major-only combined form ("8-9-10-11") is used. It is a moving tag that tracks the + // current patch of each major, and it is the form the publisher documents for consumers. Single + // majors ("10"), channel forms ("10.0"), and fully pinned combinations are deliberately ignored + // here — single majors are already covered by Microsoft's images. + private static readonly Regex CombinedMajorTag = new(@"^\d+(?:-\d+)+$", RegexOptions.Compiled); + + public static IReadOnlyList Parse(string json) + { + var runners = new List(); + JsonDocument document; + try + { + document = JsonDocument.Parse(json); + } + catch (JsonException) + { + return runners; + } + + using (document) + { + if (!document.RootElement.TryGetProperty("results", out var results) || + results.ValueKind != JsonValueKind.Array) + { + return runners; + } + + foreach (var entry in results.EnumerateArray()) + { + if (!entry.TryGetProperty("name", out var nameElement) || + nameElement.GetString() is not { } name || + !CombinedMajorTag.IsMatch(name)) + { + continue; + } + + var majors = new List(); + var usable = true; + foreach (var part in name.Split('-')) + { + if (int.TryParse(part, NumberStyles.Integer, CultureInfo.InvariantCulture, out var major) && major > 0) + { + majors.Add(major); + } + else + { + usable = false; + break; + } + } + + if (usable && majors.Count > 1) + { + runners.Add(new MultiSdkRunner { Tag = name, Majors = [.. majors.Distinct().OrderBy(m => m)] }); + } + } + } + + return runners; + } + + // Tightest fit wins: the fewest extra SDKs that still cover every required major. Ties break on the + // tag name so the same repository always resolves to the same image. + public static MultiSdkRunner? Select(IReadOnlyList runners, IReadOnlyList requiredMajors) + { + if (requiredMajors.Count < 2) + { + return null; + } + + return runners + .Where(r => r.Covers(requiredMajors)) + .OrderBy(r => r.Majors.Count) + .ThenBy(r => r.Tag, StringComparer.Ordinal) + .FirstOrDefault(); + } +} + +internal sealed record MultiSdkResult(IReadOnlyList Runners, string? Error); + +internal static class MultiSdkRunnerStore +{ + private static readonly HttpClient Http = new() { Timeout = TimeSpan.FromSeconds(15) }; + + private static string CacheFile(string cacheRoot) => Path.Combine(cacheRoot, "multi-sdk-tags.cache.json"); + + public static async Task LoadAsync(Options options, CancellationToken ct) + { + // An explicit local file is a deliberate input (used by the deterministic test harness). + if (!string.IsNullOrWhiteSpace(options.MultiSdkTagsFile)) + { + return File.Exists(options.MultiSdkTagsFile) + ? new MultiSdkResult(MultiSdkTagReader.Parse(await File.ReadAllTextAsync(options.MultiSdkTagsFile, ct)), null) + : new MultiSdkResult([], $"multi-SDK tags file not found: {options.MultiSdkTagsFile}"); + } + + var cacheFile = CacheFile(options.CacheDirectory); + if (options.Offline) + { + return LoadFromCache(cacheFile, "Offline mode: "); + } + + try + { + var json = await Http.GetStringAsync(RemoteTestProgram.MultiSdkTagsUrl, ct); + var runners = MultiSdkTagReader.Parse(json); + if (runners.Count == 0) + { + return LoadFromCache(cacheFile, "Multi-SDK tag feed returned no combined tags: "); + } + + TryWriteCache(cacheFile, json); + return new MultiSdkResult(runners, null); + } + catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException or IOException) + { + return LoadFromCache(cacheFile, $"Could not reach the multi-SDK tag feed ({ex.Message}); "); + } + } + + private static MultiSdkResult LoadFromCache(string cacheFile, string prefix) + { + if (!File.Exists(cacheFile)) + { + return new MultiSdkResult([], prefix + "no cached multi-SDK runner tags are available."); + } + + try + { + return new MultiSdkResult(MultiSdkTagReader.Parse(File.ReadAllText(cacheFile)), null); + } + catch (IOException ex) + { + return new MultiSdkResult([], prefix + $"the cached multi-SDK runner tags could not be read ({ex.Message})."); + } + } + + private static void TryWriteCache(string cacheFile, string json) + { + try + { + Directory.CreateDirectory(Path.GetDirectoryName(cacheFile)!); + File.WriteAllText(cacheFile, json); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + // Caching is an optimization; never fail discovery because the cache could not be written. + } + } +} + // --------------------------------------------------------------------------------------------------- // Environment resolution. // A resolved environment is what the runner actually executes against, whether it came from @@ -632,9 +818,34 @@ internal sealed record ResolvedEnvironment public string? LocalRoot { get; init; } - // Whether this environment's image is exempt from the Microsoft-only restriction (configured images - // are deliberate; generated images must come from mcr.microsoft.com/dotnet/sdk). + // The .NET majors this environment can both build and *run*. Empty means "single SDK, inferred from + // Channel/Sdk". A multi-SDK runner states them explicitly, which is what makes it usable for a + // repository whose target frameworks span several majors. + public IReadOnlyList SupportedMajors { get; init; } = []; + + public bool IsMultiSdk => SupportedMajors.Count > 1; + + // Whether this environment's image came from the repository's own configuration rather than being + // generated. Configured images are deliberate intent and are always used exactly as written. public bool ImageIsConfigured => Origin == EnvironmentOrigin.Configured; + + // The .NET major version this environment's channel represents ("10.0" -> 10), or 0 when the channel + // is unknown (configured environments carry no channel). + public int ChannelMajor + { + get + { + if (string.IsNullOrWhiteSpace(Channel)) + { + return 0; + } + + var head = Channel.Split('.')[0]; + return int.TryParse(head, NumberStyles.Integer, CultureInfo.InvariantCulture, out var major) && major > 0 + ? major + : 0; + } + } } internal static class GeneratedEnvironments @@ -677,6 +888,17 @@ public static IReadOnlyList FromMetadata(ReleaseMetadata me return result; } + // A multi-SDK runner environment. It carries no single channel: its value is that every listed major + // is present, so a multi-targeted repository runs its whole matrix in one container. + public static ResolvedEnvironment FromMultiSdkRunner(MultiSdkRunner runner) => new() + { + Name = $"ubuntu-testrunner-{runner.Tag}", + Origin = EnvironmentOrigin.Generated, + ReleaseType = "Multi-SDK", + DockerImage = runner.Reference, + SupportedMajors = runner.Majors, + }; + public static ResolvedEnvironment FromConfigured(EnvironmentDefinition def) => new() { Name = def.Name, @@ -695,6 +917,10 @@ internal sealed record EnvironmentResolution public ResolvedEnvironment? Environment { get; init; } public IReadOnlyList Candidates { get; init; } = []; public string? Message { get; init; } + + // Why this environment was chosen when the caller did not name one. Surfaced so an automatic + // selection is always explainable rather than looking arbitrary. + public string? SelectionReason { get; init; } } internal static class EnvironmentResolver @@ -702,11 +928,14 @@ internal static class EnvironmentResolver // Deterministic precedence: // 1. An environment explicitly named by the user (configured first, then generated). // 2. An applicable Docker environment from testenvironments.json (authoritative when present). - // 3. Microsoft-derived environments when no testenvironments.json exists. + // 3. Microsoft-derived environments when no testenvironments.json exists. When several are + // derived, the repository's own highest .NET target framework picks exactly one of them + // (see SelectByTargetFramework) so "run my tests" does not need a question to answer. public static EnvironmentResolution Resolve( TestEnvironmentsConfig? config, IReadOnlyList generated, - string? requestedName) + string? requestedName, + TargetFrameworkInfo? repoTargets = null) { var configured = config?.SupportedDockerEnvironments ?? []; @@ -789,6 +1018,14 @@ public static EnvironmentResolution Resolve( return Resolved(generated[0]); } + // Several channels are available. The repository already states which .NET it targets, so use + // that instead of asking a question the source code has already answered. + var byTargetFramework = SelectByTargetFramework(generated, repoTargets); + if (byTargetFramework is not null) + { + return byTargetFramework; + } + return new EnvironmentResolution { Status = ResolutionStatus.Ambiguous, @@ -797,6 +1034,68 @@ public static EnvironmentResolution Resolve( }; } + // Deterministic tie-break: the repository's highest .NET target framework major must match exactly + // one derived channel. Highest wins because an SDK builds its own major and every lower one, so the + // newest target is the only channel guaranteed to build the whole repository. + // + // This deliberately does not "pick something close". No target frameworks, no .NET target (only + // netstandard/net48), or more than one channel for the same major all fall through to a question — + // guessing an SDK the repository never asked for is worse than asking once. + private static EnvironmentResolution? SelectByTargetFramework( + IReadOnlyList generated, + TargetFrameworkInfo? repoTargets) + { + if (repoTargets is null) + { + return null; + } + + var majors = repoTargets.NetCoreMajors; + if (majors.Count == 0) + { + return null; + } + + // Multi-targeted repositories need every runtime present, not just the newest SDK, so a runner + // covering the whole matrix wins outright when one is available. + if (majors.Count > 1) + { + var covering = generated.Where(e => e.IsMultiSdk && majors.All(e.SupportedMajors.Contains)).ToList(); + if (covering.Count == 0) + { + return null; + } + + var runner = covering + .OrderBy(e => e.SupportedMajors.Count) + .ThenBy(e => e.Name, StringComparer.Ordinal) + .First(); + + var targeted = string.Join(", ", majors.Select(m => $"net{m}.0")); + return Resolved(runner) with + { + SelectionReason = + $"Selected automatically: the repository targets {targeted}, and this runner provides every one of them, " + + "so the whole target-framework matrix runs in a single container.", + }; + } + + var targetMajor = majors.Max(); + var matches = generated.Where(e => !e.IsMultiSdk && e.ChannelMajor == targetMajor).ToList(); + if (matches.Count != 1) + { + return null; + } + + var tfm = repoTargets.TargetFrameworks + .FirstOrDefault(t => TargetFrameworkInspector.NetMajor(t) == targetMajor) ?? $"net{targetMajor}.0"; + + return Resolved(matches[0]) with + { + SelectionReason = $"Selected automatically: the only environment matching the repository's target framework '{tfm}'.", + }; + } + private static EnvironmentResolution Resolved(ResolvedEnvironment env) => new() { Status = ResolutionStatus.Resolved, Environment = env }; } @@ -925,19 +1224,33 @@ public static TargetFrameworkInfo Inspect(string sourceRoot, string? project) // Can a channel (identified by its SDK version) build these target frameworks? An SDK builds its own // major and every lower one; it cannot build a newer runtime major, and the Linux SDK cannot build // .NET Framework (net4x) targets. - public static SdkCompatibility CanBuild(SdkVersion? channelSdk, TargetFrameworkInfo tfms) + public static SdkCompatibility CanBuild( + SdkVersion? channelSdk, + TargetFrameworkInfo tfms, + IReadOnlyList? supportedMajors = null) { - if (channelSdk is null) - { - return new SdkCompatibility(false, "The environment SDK version could not be determined."); - } - if (tfms.HasNetFramework) { return new SdkCompatibility(false, "The project targets .NET Framework (net4x), which cannot be built by a Linux .NET SDK container."); } + // An environment that states its majors explicitly (a multi-SDK runner) is judged on that list: + // every target framework must be present, because presence is what allows the tests to run. + if (supportedMajors is { Count: > 0 }) + { + var missing = tfms.NetCoreMajors.Where(m => !supportedMajors.Contains(m)).ToList(); + return missing.Count == 0 + ? new SdkCompatibility(true, null) + : new SdkCompatibility(false, + $"The project targets {string.Join(", ", missing.Select(m => $"net{m}.0"))}, which the selected image does not provide."); + } + + if (channelSdk is null) + { + return new SdkCompatibility(false, "The environment SDK version could not be determined."); + } + foreach (var major in tfms.NetCoreMajors) { if (major > channelSdk.Major) @@ -947,6 +1260,18 @@ public static SdkCompatibility CanBuild(SdkVersion? channelSdk, TargetFrameworkI } } + // A single-SDK image ships exactly one runtime. Lower target frameworks compile there but have + // no runtime to execute on, so a multi-targeted repository needs a multi-SDK runner instead of + // a silently doomed run. + var unrunnable = tfms.NetCoreMajors.Where(m => m != channelSdk.Major).ToList(); + if (unrunnable.Count > 0) + { + return new SdkCompatibility(false, + $"The project targets {string.Join(", ", unrunnable.Select(m => $"net{m}.0"))} in addition to net{channelSdk.Major}.0, " + + $"but the selected image ships only the {channelSdk.Major}.x runtime. Use a multi-SDK runner image " + + $"({RemoteTestProgram.MultiSdkRepository}) or restrict the run with --framework."); + } + // Honor an explicit global.json pin: the container SDK major must satisfy it. var pinned = SdkVersion.TryParse(tfms.GlobalJsonSdkVersion); if (pinned is not null @@ -1773,6 +2098,117 @@ public static async Task RemoveContainerAsync(string name, CancellationTok } } +// --------------------------------------------------------------------------------------------------- +// Image preparation — a .NET build routinely shells out to host tooling: MinVer, Nerdbank.GitVersioning, +// GitInfo and SourceLink all invoke `git` while building. An image without it fails the build (MinVer +// reports MINVER1007) even though the code compiles fine, which reads as a repository problem when it is +// really a missing tool in the image. Microsoft's SDK images ship git; a minimal runner image may not. +// When it is missing, one thin layer is derived from the resolved base image and cached under a +// digest-addressed tag, so the cost is paid once per image and never inside the repository. +// --------------------------------------------------------------------------------------------------- + +internal sealed record PreparedImage(string Reference, bool Provisioned, string? Note = null); + +internal static class ImageProvisioner +{ + // Tooling the container must expose on PATH for a build to behave the way it does on the host. + public static readonly string[] RequiredTools = ["git"]; + + private const string TagPrefix = "dotnet-remote-testing/prepared"; + + public static string ToolList => string.Join(", ", RequiredTools); + + // Content-addressed tag: the same base image and tool set always produce the same prepared image, + // so a later run reuses the cached layer instead of rebuilding it. + public static string DerivedTag(string baseReference, string? digest) + { + var key = digest is not null && digest.Contains(':', StringComparison.Ordinal) + ? digest[(digest.IndexOf(':', StringComparison.Ordinal) + 1)..] + : StableHash(baseReference); + var shortKey = key.Length > 16 ? key[..16] : key; + return $"{TagPrefix}:{string.Join('-', RequiredTools)}-{shortKey}"; + } + + // Verifies the tools are on PATH inside the image, without assuming a specific shell or entrypoint. + public static string ProbeCommand() => + string.Join(" && ", RequiredTools.Select(t => $"command -v {t} >/dev/null 2>&1")); + + // A single RUN that adapts to whichever package manager the base image ships. The Dockerfile is + // written to the run's own temporary directory — never into the repository being tested. + public static string Dockerfile(string baseReference) + { + var tools = string.Join(' ', RequiredTools); + return $""" + FROM {baseReference} + USER root + RUN set -e; \ + if command -v apt-get >/dev/null 2>&1; then \ + apt-get update && apt-get install -y --no-install-recommends {tools} && rm -rf /var/lib/apt/lists/*; \ + elif command -v apk >/dev/null 2>&1; then \ + apk add --no-cache {tools}; \ + elif command -v microdnf >/dev/null 2>&1; then \ + microdnf install -y {tools} && microdnf clean all; \ + elif command -v dnf >/dev/null 2>&1; then \ + dnf install -y {tools} && dnf clean all; \ + elif command -v yum >/dev/null 2>&1; then \ + yum install -y {tools} && yum clean all; \ + else \ + echo 'No supported package manager in the base image.' >&2; exit 1; \ + fi + + """; + } + + // Best effort by design: when the tooling cannot be added, the base image is used anyway and the + // reason is reported, because a repository that never invokes git still runs perfectly well there. + public static async Task EnsureAsync( + string baseReference, string? digest, string workRoot, bool offline, CancellationToken ct) + { + var tag = DerivedTag(baseReference, digest); + + if (await DockerClient.ResolveImageIdAsync(tag, ct) is not null) + { + return new PreparedImage(tag, true, $"Reused prepared image providing {ToolList}."); + } + + var probe = await DockerClient.RunAsync( + ["run", "--rm", "--entrypoint", "sh", baseReference, "-c", ProbeCommand()], ct); + if (probe.ExitCode == 0) + { + return new PreparedImage(baseReference, false); + } + + if (offline) + { + return new PreparedImage(baseReference, false, + $"The image does not provide {ToolList} and adding it needs network access (--offline). " + + "A build that invokes it will fail."); + } + + var contextDir = Path.Combine(workRoot, "image-prep"); + Directory.CreateDirectory(contextDir); + var dockerfile = Path.Combine(contextDir, "Dockerfile"); + await File.WriteAllTextAsync(dockerfile, Dockerfile(baseReference), ct); + + var build = await DockerClient.BuildAsync(dockerfile, contextDir, tag, ct); + if (build.ExitCode != 0) + { + var reason = build.StdErr.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries) + .LastOrDefault() ?? "docker build failed."; + return new PreparedImage(baseReference, false, + $"The image does not provide {ToolList} and it could not be added: {reason}"); + } + + return new PreparedImage(tag, true, $"Added {ToolList} to the image (cached for later runs)."); + } + + private static string StableHash(string value) + { + var bytes = System.Security.Cryptography.SHA256.HashData(Encoding.UTF8.GetBytes(value)); + return Convert.ToHexStringLower(bytes); + } +} + // --------------------------------------------------------------------------------------------------- // Source staging — copy the source into an isolated, disposable workspace so container builds never // pollute the developer's working tree with Linux bin/obj artifacts. @@ -1895,7 +2331,8 @@ internal sealed record ResolveContext( ReleaseMetadata? Metadata, string? MetadataError, IReadOnlyList Generated, - EnvironmentResolution Resolution); + EnvironmentResolution Resolution, + string? MultiSdkError = null); internal static class Commands { @@ -1928,8 +2365,47 @@ bool Match(EnvironmentDefinition e) => } } - var resolution = EnvironmentResolver.Resolve(config, generated, options.EnvironmentName); - return new ResolveContext(config, metadata, metadataError, generated, resolution); + // Generated environments are resolved against the repository's own target frameworks. Configured + // environments never need this (they are deliberate intent), and a named environment short-circuits + // before it is used, so the inspection only runs when it can actually decide something. + TargetFrameworkInfo? repoTargets = null; + string? multiSdkError = null; + if (options.EnvironmentName is null && config?.SourcePath is null && generated.Count > 1) + { + repoTargets = NarrowToRequestedFramework( + TargetFrameworkInspector.Inspect(options.RepoRoot, options.Project), options.Framework); + + // Only a genuinely multi-targeted repository needs the multi-SDK runner feed; do not spend a + // network call to answer a question a single target framework already answers. + if (repoTargets.NetCoreMajors.Count > 1) + { + var multi = await MultiSdkRunnerStore.LoadAsync(options, ct); + multiSdkError = multi.Error; + var runner = MultiSdkTagReader.Select(multi.Runners, repoTargets.NetCoreMajors); + if (runner is not null) + { + generated = [.. generated, GeneratedEnvironments.FromMultiSdkRunner(runner)]; + } + } + } + + var resolution = EnvironmentResolver.Resolve(config, generated, options.EnvironmentName, repoTargets); + return new ResolveContext(config, metadata, metadataError, generated, resolution, multiSdkError); + } + + // --framework narrows what actually runs, so it must narrow what the environment is chosen for too. + // Without this, "-f net10.0" on a multi-targeted repository would still be resolved as multi-targeted. + private static TargetFrameworkInfo NarrowToRequestedFramework(TargetFrameworkInfo info, string? framework) + { + if (string.IsNullOrWhiteSpace(framework)) + { + return info; + } + + var match = info.TargetFrameworks.FirstOrDefault( + t => string.Equals(t, framework, StringComparison.OrdinalIgnoreCase)); + + return info with { TargetFrameworks = match is null ? [framework] : [match] }; } public static async Task ListAsync(Options options) @@ -1956,6 +2432,20 @@ public static async Task ListAsync(Options options) metadata = result.Metadata; metadataError = result.Error; environments = metadata is not null ? GeneratedEnvironments.FromMetadata(metadata) : []; + + // A multi-targeted repository cannot execute its lower target frameworks on a single-SDK + // image, so offer the runner that can — listing only what cannot work would be misleading. + var repoMajors = NarrowToRequestedFramework( + TargetFrameworkInspector.Inspect(options.RepoRoot, options.Project), options.Framework).NetCoreMajors; + if (repoMajors.Count > 1) + { + var multi = await MultiSdkRunnerStore.LoadAsync(options, cts.Token); + var runner = MultiSdkTagReader.Select(multi.Runners, repoMajors); + if (runner is not null) + { + environments = [GeneratedEnvironments.FromMultiSdkRunner(runner), .. environments]; + } + } } if (options.Json) @@ -1970,6 +2460,7 @@ public static async Task ListAsync(Options options) { e.Name, origin = e.Origin.ToString(), e.Channel, e.ReleaseType, e.Sdk, image = e.DockerImage, dockerFile = e.DockerFile, + supportedMajors = e.SupportedMajors.Count > 0 ? e.SupportedMajors : null, }), unsupported = unsupported.Select(e => new { e.Name, type = e.RawType }), configDiagnostics = config?.Diagnostics.Select(d => new { d.Code, d.Message, environment = d.EnvironmentName }), @@ -1999,6 +2490,11 @@ public static async Task ListAsync(Options options) Console.WriteLine($" {e.ReleaseType}"); } + if (e.SupportedMajors.Count > 0) + { + Console.WriteLine($" Provides .NET {string.Join(", ", e.SupportedMajors)} in one image"); + } + if (e.Sdk is not null) { Console.WriteLine($" SDK {e.Sdk}"); @@ -2046,7 +2542,8 @@ public static async Task PlanAsync(Options options) var env = ctx.Resolution.Environment!; var sourceRoot = ResolveSourceRoot(options, env); - var tfmInfo = TargetFrameworkInspector.Inspect(sourceRoot, options.Project); + var tfmInfo = NarrowToRequestedFramework( + TargetFrameworkInspector.Inspect(sourceRoot, options.Project), options.Framework); // Image identity: for generated environments, validate candidate tags against MCR and pre-resolve // the digest without pulling. Offline / --no-registry-check skips the network probe. @@ -2060,6 +2557,14 @@ public static async Task PlanAsync(Options options) { imageNote = $"Configured Dockerfile '{env.DockerFile}' will be built into a local image."; } + else if (env.IsMultiSdk) + { + // The tag came from the publisher's own tag feed, so it exists by construction. Its digest is + // resolved at pull time in `run`; there is no Microsoft registry probe to make here. + requestedTag = env.DockerImage?.Split(':').Last(); + imageNote = $"Multi-SDK runner providing .NET {string.Join(", ", env.SupportedMajors)}; " + + "the whole target-framework matrix runs in one container."; + } else if (env.Origin == EnvironmentOrigin.Generated && channelSdk is not null) { var candidates = ImageTagResolver.CandidateTags(channelSdk); @@ -2087,7 +2592,7 @@ public static async Task PlanAsync(Options options) } } - var compatibility = TargetFrameworkInspector.CanBuild(channelSdk, tfmInfo); + var compatibility = TargetFrameworkInspector.CanBuild(channelSdk, tfmInfo, env.SupportedMajors); var compatibilityBlocking = SdkCompatibilityPolicy.IsBlocking(env.Origin, compatibility.Compatible); // Configured environments trust their image SDK (validated at run time); do not present or fail // them as incompatible just because the SDK could not be determined statically. @@ -2110,7 +2615,15 @@ public static async Task PlanAsync(Options options) Console.WriteLine(JsonSerializer.Serialize(new { tool = RemoteTestProgram.ToolName, - environment = new { env.Name, origin = env.Origin.ToString(), env.Channel, env.ReleaseType, env.Sdk }, + environment = new + { + env.Name, + origin = env.Origin.ToString(), + env.Channel, + env.ReleaseType, + env.Sdk, + selectionReason = ctx.Resolution.SelectionReason, + }, image = new { requested = env.DockerImage ?? reference, @@ -2119,7 +2632,7 @@ public static async Task PlanAsync(Options options) dockerFile = env.DockerFile, digest, digestResolved = digest is not null, - microsoftOnlyEnforced = env.Origin == EnvironmentOrigin.Generated, + recommendedPublisher = IsRecommendedPublisher(env.DockerImage ?? reference), note = imageNote, }, targetFrameworks = new @@ -2138,6 +2651,11 @@ public static async Task PlanAsync(Options options) else { Console.WriteLine($"Plan: {env.Name}"); + if (ctx.Resolution.SelectionReason is not null) + { + Console.WriteLine($" Selection: {ctx.Resolution.SelectionReason}"); + } + Console.WriteLine($" Image: {env.DockerImage ?? reference ?? "(from Dockerfile)"}"); if (requestedTag is not null) { @@ -2171,6 +2689,14 @@ public static async Task PlanAsync(Options options) Coverage = options.Coverage, }; + // Recommended publishers for auto-generated environments: Microsoft's official SDK images and the + // Codebelt multi-SDK test runner. This is reported, not enforced — an image from anywhere else is + // allowed (a configured dockerImage is deliberate intent), it simply is not one we vouch for. + private static bool IsRecommendedPublisher(string? reference) => + reference is not null + && (reference.StartsWith(RemoteTestProgram.SdkRepository + ":", StringComparison.Ordinal) + || reference.StartsWith(RemoteTestProgram.MultiSdkRepository + ":", StringComparison.Ordinal)); + private static string ResolveSourceRoot(Options options, ResolvedEnvironment env) { if (string.IsNullOrWhiteSpace(env.LocalRoot)) @@ -2206,6 +2732,7 @@ private static int ReportResolutionProblem(Options options, ResolveContext ctx) message, candidates = res.Candidates, metadataError = ctx.MetadataError, + multiSdkError = ctx.MultiSdkError, }, RemoteTestProgram.JsonOut)); } else @@ -2215,6 +2742,11 @@ private static int ReportResolutionProblem(Options options, ResolveContext ctx) { Console.Error.WriteLine("Available: " + string.Join(", ", res.Candidates)); } + + if (ctx.MultiSdkError is not null) + { + Console.Error.WriteLine($"Multi-SDK runner discovery: {ctx.MultiSdkError}"); + } } return (int)kind; @@ -2226,7 +2758,9 @@ private sealed record RunImage( string? RequestedTag = null, string? Sdk = null, FailureKind Kind = FailureKind.None, - string? Error = null); + string? Error = null, + string? PreparedReference = null, + string? ProvisionNote = null); private sealed record CleanupReport(bool ContainerRemoved, bool WorkspaceRemoved, IReadOnlyList Leftovers); @@ -2242,6 +2776,10 @@ public static async Task RunAsync(Options options) void OnCancel(object? _, ConsoleCancelEventArgs e) { e.Cancel = true; cancelled = true; cts.Cancel(); } Console.CancelKeyPress += OnCancel; + // Wall clock for the whole operation. Reported alongside the test duration from the TRX so a + // fast test suite behind a slow image pull never looks like the run itself took no time. + var wallClock = Stopwatch.StartNew(); + var containerName = ContainerPlanner.ContainerName(Guid.NewGuid().ToString("N")[..8]); var runRoot = Path.Combine(Path.GetTempPath(), "dotnet-remote-testing", containerName); string? stagingRoot = null; @@ -2263,9 +2801,10 @@ public static async Task RunAsync(Options options) } var sourceRoot = ResolveSourceRoot(options, env); - var tfmInfo = TargetFrameworkInspector.Inspect(sourceRoot, options.Project); + var tfmInfo = NarrowToRequestedFramework( + TargetFrameworkInspector.Inspect(sourceRoot, options.Project), options.Framework); var channelSdk = SdkVersion.TryParse(env.Sdk); - var compat = TargetFrameworkInspector.CanBuild(channelSdk, tfmInfo); + var compat = TargetFrameworkInspector.CanBuild(channelSdk, tfmInfo, env.SupportedMajors); if (SdkCompatibilityPolicy.IsBlocking(env.Origin, compat.Compatible)) { return Error(options, FailureKind.SdkIncompatibility, compat.Reason ?? "The selected SDK cannot build the requested target framework."); @@ -2277,6 +2816,11 @@ public static async Task RunAsync(Options options) return Error(options, image.Kind, image.Error); } + // The image runs the build, not just the tests, so it must carry what the build shells out to. + var prepared = await ImageProvisioner.EnsureAsync( + image.Reference!, image.Digest, runRoot, options.Offline, cts.Token); + image = image with { PreparedReference = prepared.Provisioned ? prepared.Reference : null, ProvisionNote = prepared.Note }; + var resultsRoot = Path.Combine(runRoot, "results"); stagingRoot = Path.Combine(runRoot, "workspace"); Directory.CreateDirectory(resultsRoot); @@ -2298,7 +2842,7 @@ public static async Task RunAsync(Options options) new ContainerMount(nugetCache, "/nuget", ReadOnly: false), new ContainerMount(resultsRoot, "/results", ReadOnly: false), }; - var plan = ContainerPlanner.Build(image.Reference!, containerName, mounts, testOptions); + var plan = ContainerPlanner.Build(prepared.Reference, containerName, mounts, testOptions); ProcessResult proc; try @@ -2319,7 +2863,9 @@ public static async Task RunAsync(Options options) var cleanup = await CleanupAsync(containerName, runRoot, cts.Token); - return EmitRunResult(options, env, image, tfmInfo, results, outcome, proc, cleanup); + return EmitRunResult( + options, env, image, tfmInfo, results, outcome, proc, cleanup, + ctx.Resolution.SelectionReason, wallClock.Elapsed.TotalSeconds); } catch (OperationCanceledException) { @@ -2377,7 +2923,9 @@ private static async Task ResolveImageForRunAsync( } else { - // Configured dockerImage is deliberate repository intent — exempt from the Microsoft-only rule. + // A configured dockerImage (deliberate repository intent) or a multi-SDK runner tag taken + // from the publisher's tag feed. Both are used exactly as written; the digest is resolved + // from the pulled image below. reference = env.DockerImage!; } @@ -2426,7 +2974,9 @@ private static int EmitRunResult( TestRunResult results, ExecutionOutcome outcome, ProcessResult proc, - CleanupReport cleanup) + CleanupReport cleanup, + string? selectionReason = null, + double? elapsedSeconds = null) { var exit = FailureClassifier.ToExitCode(outcome.Kind); if (outcome.Kind == FailureKind.None && cleanup.Leftovers.Count > 0) @@ -2450,10 +3000,19 @@ private static int EmitRunResult( failureKind = outcome.Kind == FailureKind.None ? null : outcome.Kind.ToString(), phase = outcome.Phase, message = outcome.Message, - environment = new { env.Name, origin = env.Origin.ToString(), env.Channel, env.ReleaseType }, - image = new { requested = image.RequestedTag, reference = image.Reference, digest = image.Digest, sdk = image.Sdk }, + environment = new { env.Name, origin = env.Origin.ToString(), env.Channel, env.ReleaseType, selectionReason }, + image = new + { + requested = image.RequestedTag, + reference = image.Reference, + digest = image.Digest, + sdk = image.Sdk, + prepared = image.PreparedReference, + provisioning = image.ProvisionNote, + }, targetFrameworks = tfmInfo.TargetFrameworks, tests = new { results.Total, results.Passed, results.Skipped, results.Failed, results.DurationSeconds, trxFiles = results.TrxFilesParsed }, + elapsedSeconds = elapsedSeconds is null ? (double?)null : Math.Round(elapsedSeconds.Value, 1), failures = results.Failures.Select(f => new { f.TestName, f.ClassName, f.Message, f.StackTrace }), cleanup = new { cleanup.ContainerRemoved, cleanup.WorkspaceRemoved, cleanup.Leftovers }, diagnostics = status == "error" ? new { containerExitCode = proc.ExitCode, stdoutTail = LastLines(proc.StdOut, 30), stderrTail = LastLines(proc.StdErr, 20) } : null, @@ -2463,6 +3022,11 @@ private static int EmitRunResult( // Concise human result: environment → image/digest/sdk → tests → duration, actionable on failure. Console.WriteLine($"Remote Test: {env.Name}"); + if (selectionReason is not null) + { + Console.WriteLine(selectionReason); + } + Console.WriteLine(); Console.WriteLine($"Image: {image.Reference}"); if (image.Digest is not null) @@ -2475,12 +3039,21 @@ private static int EmitRunResult( Console.WriteLine($"SDK: {image.Sdk}"); } + if (image.ProvisionNote is not null) + { + Console.WriteLine($"Tools: {image.ProvisionNote}"); + } + Console.WriteLine(); if (outcome.Kind is FailureKind.None or FailureKind.TestFailure) { Console.WriteLine($"Tests: {results.Passed} passed, {results.Skipped} skipped, {results.Failed} failed"); - Console.WriteLine($"Time: {results.DurationSeconds.ToString("0.0", CultureInfo.InvariantCulture)} s"); + Console.WriteLine($"Time: {results.DurationSeconds.ToString("0.0", CultureInfo.InvariantCulture)} s (tests)"); + if (elapsedSeconds is not null) + { + Console.WriteLine($"Total: {elapsedSeconds.Value.ToString("0.0", CultureInfo.InvariantCulture)} s (including image pull, restore and build)"); + } if (results.Failed > 0) { Console.WriteLine(); @@ -2500,10 +3073,14 @@ private static int EmitRunResult( else { Console.Error.WriteLine($"{outcome.Kind}: {outcome.Message}"); - var tail = LastLines(proc.StdErr, 20); - if (!string.IsNullOrWhiteSpace(tail)) + // The container writes compiler/restore diagnostics to stdout, so a stderr-only tail would + // leave the developer with a verdict and no cause. Prefer the actual error lines. + var detail = FirstNonEmpty( + ErrorLines(proc.StdOut, 20), LastLines(proc.StdErr, 20), LastLines(proc.StdOut, 20)); + if (!string.IsNullOrWhiteSpace(detail)) { - Console.Error.WriteLine(tail); + Console.Error.WriteLine(); + Console.Error.WriteLine(detail); } } @@ -2576,6 +3153,7 @@ public static int Run() ReleaseMetadataTests(); SdkAndImageTagTests(); EnvironmentSelectionTests(); + MultiSdkRunnerTests(); UnsupportedEnvironmentTests(); TargetFrameworkTests(); CommandPlanningTests(); @@ -2740,6 +3318,135 @@ private static void EnvironmentSelectionTests() var notFound = EnvironmentResolver.Resolve(config, generated, "does-not-exist"); Check("unknown name reported as not found", notFound.Status == ResolutionStatus.NotFound); + + // Deterministic tie-break: the repository's own target framework answers the question. + var net10 = new TargetFrameworkInfo { TargetFrameworks = ["net10.0"] }; + var byTfm = EnvironmentResolver.Resolve(null, generated, null, net10); + Check("target framework selects the matching channel without asking", + byTfm.Status == ResolutionStatus.Resolved && byTfm.Environment!.Name == "dotnet-10-lts"); + Check("automatic selection explains itself", byTfm.SelectionReason is not null && byTfm.SelectionReason.Contains("net10.0")); + + // A single-SDK image ships one runtime, so a multi-targeted repository must not be silently + // pointed at the newest channel — those lower target frameworks would build and then fail to run. + var multiTargeted = new TargetFrameworkInfo { TargetFrameworks = ["net8.0", "net9.0", "net10.0"] }; + Check("multi-targeted repository is not sent to a single-SDK image", + EnvironmentResolver.Resolve(null, generated, null, multiTargeted).Status == ResolutionStatus.Ambiguous); + + var runner = GeneratedEnvironments.FromMultiSdkRunner( + new MultiSdkRunner { Tag = "8-9-10-11", Majors = [8, 9, 10, 11] }); + var withRunner = EnvironmentResolver.Resolve(null, [.. generated, runner], null, multiTargeted); + Check("multi-targeted repository selects the covering multi-SDK runner", + withRunner.Status == ResolutionStatus.Resolved && withRunner.Environment!.Name == "ubuntu-testrunner-8-9-10-11"); + Check("multi-SDK selection explains the whole-matrix benefit", + withRunner.SelectionReason is not null && withRunner.SelectionReason.Contains("single container")); + + Check("single-target repository still prefers the matching single-SDK channel", + EnvironmentResolver.Resolve(null, [.. generated, runner], null, net10).Environment!.Name == "dotnet-10-lts"); + + var uncovered = new TargetFrameworkInfo { TargetFrameworks = ["net7.0", "net10.0"] }; + Check("a runner that does not cover every target framework is not selected", + EnvironmentResolver.Resolve(null, [.. generated, runner], null, uncovered).Status == ResolutionStatus.Ambiguous); + + var previewOnly = new TargetFrameworkInfo { TargetFrameworks = ["net11.0"] }; + Check("preview channel is selectable by target framework", + EnvironmentResolver.Resolve(null, generated, null, previewOnly).Environment!.Name == "dotnet-11-preview"); + + var unsupportedMajor = new TargetFrameworkInfo { TargetFrameworks = ["net7.0"] }; + Check("target framework with no supported channel still asks", + EnvironmentResolver.Resolve(null, generated, null, unsupportedMajor).Status == ResolutionStatus.Ambiguous); + + var noNetTarget = new TargetFrameworkInfo { TargetFrameworks = ["netstandard2.0"] }; + Check("non-.NET target framework does not guess a channel", + EnvironmentResolver.Resolve(null, generated, null, noNetTarget).Status == ResolutionStatus.Ambiguous); + + Check("empty target framework info does not guess a channel", + EnvironmentResolver.Resolve(null, generated, null, new TargetFrameworkInfo()).Status == ResolutionStatus.Ambiguous); + + var duplicateMajor = new List(generated) + { + generated.First(e => e.ChannelMajor == 10) with { Name = "dotnet-10-alt" }, + }; + Check("two channels for the same major stay ambiguous", + EnvironmentResolver.Resolve(null, duplicateMajor, null, net10).Status == ResolutionStatus.Ambiguous); + + Check("configured environments are never auto-selected by target framework", + EnvironmentResolver.Resolve(twoConfig, generated, null, net10).Status == ResolutionStatus.Ambiguous); + + Check("channel major parsed from channel version", + generated.First(e => e.Name == "dotnet-10-lts").ChannelMajor == 10); + Check("configured environment has no channel major", + GeneratedEnvironments.FromConfigured(config.SupportedDockerEnvironments[0]).ChannelMajor == 0); + } + + private const string SampleMultiSdkTags = """ + { + "count": 8, + "results": [ + { "name": "11.0.100-preview.7" }, + { "name": "10" }, + { "name": "10.0" }, + { "name": "8-9-10-11" }, + { "name": "8.0-9.0-10.0-11.0" }, + { "name": "9-10" }, + { "name": "8.0.421-9.0.314-10.0.300-11.0.100-preview.4" }, + { "name": "mono-net8.0.418-9.0.311-10.0.103" } + ] + } + """; + + private static void MultiSdkRunnerTests() + { + Section("Multi-SDK runner discovery"); + + var runners = MultiSdkTagReader.Parse(SampleMultiSdkTags); + var tags = runners.Select(r => r.Tag).ToList(); + Check("combined major tags are discovered", tags.Contains("8-9-10-11") && tags.Contains("9-10")); + Check("single-major tags are ignored", !tags.Contains("10") && !tags.Contains("10.0")); + Check("channel and pinned combination forms are ignored", + !tags.Contains("8.0-9.0-10.0-11.0") && !tags.Contains("8.0.421-9.0.314-10.0.300-11.0.100-preview.4")); + Check("prefixed tags are ignored", !tags.Any(t => t.StartsWith("mono", StringComparison.Ordinal))); + Check("majors parsed from the tag", + runners.Single(r => r.Tag == "8-9-10-11").Majors.SequenceEqual([8, 9, 10, 11])); + Check("image reference built from the publisher repository", + runners.Single(r => r.Tag == "9-10").Reference == "codebeltnet/ubuntu-testrunner:9-10"); + + Check("malformed feed yields no runners", MultiSdkTagReader.Parse("not json").Count == 0); + Check("feed without results yields no runners", MultiSdkTagReader.Parse("""{ "count": 0 }""").Count == 0); + + // Tightest fit: cover every required major without dragging in SDKs the repository never asked for. + Check("tightest covering tag wins", + MultiSdkTagReader.Select(runners, [9, 10])!.Tag == "9-10"); + Check("wider tag used when the tight one does not cover", + MultiSdkTagReader.Select(runners, [8, 10])!.Tag == "8-9-10-11"); + Check("no covering tag returns null", + MultiSdkTagReader.Select(runners, [7, 10]) is null); + Check("single major never selects a multi-SDK runner", + MultiSdkTagReader.Select(runners, [10]) is null); + + var env = GeneratedEnvironments.FromMultiSdkRunner(runners.Single(r => r.Tag == "8-9-10-11")); + Check("runner environment is named after its tag", env.Name == "ubuntu-testrunner-8-9-10-11"); + Check("runner environment is multi-SDK", env.IsMultiSdk && env.SupportedMajors.SequenceEqual([8, 9, 10, 11])); + Check("runner environment carries no single channel", env.Channel is null && env.ChannelMajor == 0); + + // Compatibility is judged on declared majors, because presence is what lets the tests run. + var spread = new TargetFrameworkInfo { TargetFrameworks = ["net8.0", "net10.0"] }; + Check("multi-SDK image is compatible with every provided target", + TargetFrameworkInspector.CanBuild(null, spread, env.SupportedMajors).Compatible); + Check("multi-SDK image rejects a target it does not provide", + !TargetFrameworkInspector.CanBuild(null, new TargetFrameworkInfo { TargetFrameworks = ["net7.0"] }, env.SupportedMajors).Compatible); + Check("multi-SDK image still cannot build .NET Framework", + !TargetFrameworkInspector.CanBuild(null, new TargetFrameworkInfo { TargetFrameworks = ["net48"] }, env.SupportedMajors).Compatible); + + // The runtime gap that motivates the multi-SDK runner in the first place. + var sdk10 = SdkVersion.TryParse("10.0.302"); + var singleSdkSpread = TargetFrameworkInspector.CanBuild(sdk10, spread); + Check("single-SDK image is incompatible with a multi-targeted repository", !singleSdkSpread.Compatible); + Check("the incompatibility names the missing runtime and the remedy", + singleSdkSpread.Reason is not null + && singleSdkSpread.Reason.Contains("net8.0") + && singleSdkSpread.Reason.Contains(RemoteTestProgram.MultiSdkRepository)); + Check("single-SDK image remains compatible with its own single target", + TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net10.0"] }).Compatible); } private static void UnsupportedEnvironmentTests() @@ -2783,7 +3490,11 @@ private static void TargetFrameworkTests() Check("global.json parsed", sdk == "10.0.302" && roll == "latestFeature"); var sdk10 = SdkVersion.TryParse("10.0.302"); - Check("sdk builds equal/lower target", TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net8.0", "net10.0"] }).Compatible); + Check("sdk builds and runs its own target", TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net10.0"] }).Compatible); + // A lower target compiles on a newer SDK but has no runtime in that image, so it is not runnable + // there. This is why a multi-targeted repository needs a multi-SDK runner. + Check("sdk alone cannot run a lower target it can build", + !TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net8.0", "net10.0"] }).Compatible); Check("sdk cannot build newer target", !TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net11.0"] }).Compatible); Check("linux sdk cannot build net framework", !TargetFrameworkInspector.CanBuild(sdk10, new TargetFrameworkInfo { TargetFrameworks = ["net48"] }).Compatible); Check("global.json disable pin mismatch is incompatible", !TargetFrameworkInspector.CanBuild(sdk10, From 47f4a5604f1d9b128acbdd7fd14a430e30df98b7 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 14 Aug 2026 23:38:11 +0200 Subject: [PATCH 04/31] =?UTF-8?q?=F0=9F=92=AC=20update=20repository=20docu?= =?UTF-8?q?mentation=20for=20skill=20changes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Synced README.md with latest dotnet-remote-testing skill enhancements, including improved feature descriptions and updated capability documentation to reflect the expanded testing infrastructure and enhanced Docker environment discovery. --- README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 92d517b..3d7b88f 100644 --- a/README.md +++ b/README.md @@ -130,7 +130,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | | [dotnet-test](skills/dotnet-test/SKILL.md) | Bootstraps and refactors xUnit projects to Codebelt conventions. It deterministically inspects project roles, target frameworks, xUnit generation, package ownership, inheritance, application entry points—including Bootstrapper `MinimalConsoleProgram`, `MinimalWorkerProgram`, and `MinimalWebProgram` hosts—and every selected `WebApplicationFactory` usage; classifies ordinary unit, ASP.NET Core functional, and console/worker functional tests; modernizes xUnit v2 projects to xUnit v3 plus Microsoft Testing Platform without moving package ownership or changing frameworks; and resolves current stable compatible packages through NuGet-backed isolated compatibility-project restores, including the selected combined package set. Focused web tests use `WebApplicationTestFactory` with an explicit entrypoint-owned `ManagedWebApplicationFixture`, directly or through a narrow `Test`-derived harness; shared web fixtures use `WebApplicationTest` with `ManagedWebApplicationFixture`; focused console/worker tests use `ApplicationTestFactory` with `ManagedApplicationFixture`; and shared non-web fixtures use `ApplicationTest` with `ManagedApplicationFixture`. Deprecated blocking fixtures are migration inputs only and are never emitted because they are scheduled for removal. Functional migrations fail closed unless the chosen Codebelt pattern and managed fixture are present, the legacy or blocking fixture is absent, and test code does not reconstruct the production composition root with its own `WebApplication`, `TestServer`, or `HostBuilder`. Migrations preserve entrypoint-owned startup, host configuration, lazy start, clients, services, configuration, sync/async disposal, isolation, and existing test names, while fresh bootstraps add source-grounded behavior tests. Non-web tests stay in-process and require a resolvable Generic Host; test-only scope reports the exact production adaptation instead of silently rewriting startup or launching a process. | | [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, requires exact per-case correctness oracles plus a semantic preflight for truthful workload labels, hard-gates interpretation on a complete valid BenchmarkDotNet summary, preserves workload invariants such as selectivity and hit/miss ratios as sizes scale, distinguishes deferred pipeline creation from terminal/materialization work, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash; after the first valid full result it stops unless deeper diagnostics could change a real engineering decision. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | -| [dotnet-remote-testing](skills/dotnet-remote-testing/SKILL.md) | Run .NET tests inside a resolved remote Docker environment and return concise, structured results — Visual Studio's Remote Testing experience (choose an environment → run tests → see results) with the container plumbing hidden behind a deterministic runner (`scripts/remote-test.cs`) the skill orchestrates instead of composing ad-hoc `docker run` commands. It honors Microsoft's existing `testenvironments.json` version-1 contract (`name`, `localRoot`, `dockerImage`, `dockerFile`, with the either/or Docker-source rule), treats that file as authoritative when present, and reports WSL/SSH/unknown types as unsupported rather than converting or silently ignoring them. When no `testenvironments.json` exists it provides a zero-configuration experience built exclusively on official `mcr.microsoft.com/dotnet/sdk` images, discovering the currently supported LTS and STS channels plus the current preview from Microsoft's live `releases-index.json` using `support-phase`/`release-type` (never hardcoded version numbers or even/odd assumptions) and caching that metadata outside the repository for offline reuse. It prefers an exact `latest-sdk` image tag (stripping preview build metadata), validates the tag against Microsoft's registry, and pins each execution to the resolved immutable digest so results are reproducible across environment, image, digest, SDK, and architecture. Execution stages the source into an isolated workspace so container builds never leave Linux `bin`/`obj` in the working tree, mounts a persistent NuGet cache outside the repo, runs restore → build → test with structured TRX collection, classifies failures into distinct kinds (configuration, unsupported environment, Docker unavailable, image resolution, SDK incompatibility, staging, restore, compilation, test-host, test failure, result-processing, cleanup, cancellation, release-metadata) so infrastructure problems are never reported as failing unit tests, and always cleans up transient Docker resources. It never generates a `Dockerfile`, dev container, compose file, or editor configuration (an existing configured `dockerFile` is honored, never created), never runs privileged containers or mounts the Docker socket, and never silently falls back to running tests on the host. Docker is the only transport for now, designed so WSL/SSH can be added later without disturbing the deterministic Docker path, which is covered by a comprehensive built-in `--self-test` plus a PowerShell harness. | +| [dotnet-remote-testing](skills/dotnet-remote-testing/SKILL.md) | Run .NET tests inside a resolved remote Docker environment and return concise, structured results — Visual Studio's Remote Testing experience (choose an environment → run tests → see results) with the container plumbing hidden behind a deterministic runner (`scripts/remote-test.cs`) the skill orchestrates instead of composing ad-hoc `docker run` commands. Invoking it is the request: with one applicable Docker environment it runs immediately — no capability menu, no parameter questionnaire, no confirmation — and when several channels are derived, the repository's own highest target framework selects the matching one and the choice is reported. The runner decides when a question is unavoidable, exiting `SelectionRequired` (16) with the exact candidates, and every exit code maps to exactly one next action so behavior is identical across models. It honors Microsoft's existing `testenvironments.json` version-1 contract (`name`, `localRoot`, `dockerImage`, `dockerFile`, with the either/or Docker-source rule), treats that file as authoritative when present, and reports WSL/SSH/unknown types as unsupported rather than converting or silently ignoring them. When no `testenvironments.json` exists it provides a zero-configuration experience built exclusively on official `mcr.microsoft.com/dotnet/sdk` images, discovering the currently supported LTS and STS channels plus the current preview from Microsoft's live `releases-index.json` using `support-phase`/`release-type` (never hardcoded version numbers or even/odd assumptions) and caching that metadata outside the repository for offline reuse. It prefers an exact `latest-sdk` image tag (stripping preview build metadata), validates the tag against Microsoft's registry, and pins each execution to the resolved immutable digest so results are reproducible across environment, image, digest, SDK, and architecture. Execution stages the source into an isolated workspace so container builds never leave Linux `bin`/`obj` in the working tree, mounts a persistent NuGet cache outside the repo, runs restore → build → test with structured TRX collection, classifies failures into distinct kinds (configuration, unsupported environment, Docker unavailable, image resolution, SDK incompatibility, staging, restore, compilation, test-host, test failure, result-processing, cleanup, cancellation, release-metadata) so infrastructure problems are never reported as failing unit tests, and always cleans up transient Docker resources. It never generates a `Dockerfile`, dev container, compose file, or editor configuration (an existing configured `dockerFile` is honored, never created), never runs privileged containers or mounts the Docker socket, and never silently falls back to running tests on the host. Docker is the only transport for now, designed so WSL/SSH can be added later without disturbing the deterministic Docker path, which is covered by a comprehensive built-in `--self-test` plus a PowerShell harness. | | [dotnet-segregated-assets](skills/dotnet-segregated-assets/SKILL.md) | Migrate or configure ASP.NET Core static delivery with `codebeltnet/web-cdn-origin:2.0.0` while keeping `wwwroot` as the authoring root. The deterministic runner inspects and verifies topology, publish exclusion, Static Web Assets risks, Cuemon signals, competing `AppAssetOptions`-style abstractions, actual `app-*`/`cdn-*` markup, and scheme-safe local origins; the agent performs semantic edits. For an existing Cuemon package reference, its plan resolves the highest stable version from NuGet.org at execution time, preserves Central Package Management versus inline ownership, excludes prereleases, and fails rather than copying an old fixture or example version. It reuses Cuemon `AppTagHelperOptions`/`CdnTagHelperOptions`, `BaseUrlMode`, and the public `app-link`, `app-script`, `app-img`, `cdn-link`, `cdn-script`, and `cdn-img` helpers when already available, otherwise reuses a suitable project abstraction without adding Cuemon. It keeps App and shared CDN ownership separate, preserves ordinary Project-based Development, adds opt-in segregated Development through a root Docker Compose profile, and makes `compose.assets.yml` directly build artifact-first `LocalDevelopment.Dockerfile` and `Assets.Dockerfile` images. Every generated file comes from a literal template in `assets/` and lands in one fixed location — the three Dockerfiles beside the web `.csproj`, orchestration at the repository root — and `verify --check-local` proves that placement along with the artifact-first contract: no SDK stage or `dotnet publish` inside an application image, a `.dockerignore` that still carries `artifacts/`, `LocalPublishDirectory` behind a guarded post-build target, Compose host ports derived from the ordinary Project profile, and a CI job that produces the artifact those images copy. Production CI publishes the same application artifact for the shell-less runtime `Dockerfile`. The skill excludes app-owned `wwwroot` with targeted MSBuild metadata, preserves `_content`/`_framework` and generated Static Web Assets, and proves publish/local invariants deterministically and idempotently. | | [agent-smith](skills/agent-smith/SKILL.md) | Apply a rigorous, consistent, evidence-driven software-craftsmanship standard across a whole engineering task. Invoke explicitly as `/agent-smith ` or let it auto-trigger for design, architecture, implementation, refactoring, code review, public API review, compatibility and Semantic Versioning analysis, testing, benchmarking, performance, skill authoring, documentation, security and DevSecOps, CI/CD, delivery, repository governance, and engineering assessment. Skill-authoring mode grounds instructions in real execution, requires an explicit bounded-concurrency assessment so independent data retrieval and eval work do not remain sequential by habit, favors reusable C#/.NET scripts and validators against the dynamically resolved latest supported LTS when local constraints do not decide, and follows the Agent Skills guidance for progressive disclosure, description optimization, candidate-versus-baseline evaluation, aggregation, and human review. Its optional .NET EditorConfig conformance mode handles targeted IDE/CA diagnostic remediation and full informational-or-higher `dotnet format` conformance without treating a clean build as proof of policy compliance: user-defined diagnostic IDs remain task-supplied data; target, path, and severity scope remains authoritative; informational workflows explicitly preserve `--severity info` because the formatter defaults to `warn`; targeted IDE and analyzer checks use category-specific formatter subcommands; every formatter invocation is read-only via `--verify-no-changes`; `--no-restore` is never treated as a conformance fallback; fixes are deliberate source edits; repeated multi-target findings are de-duplicated by physical file, diagnostic, and span; and the bundled `repair-roslyn-multiproject-artifacts.ps1` detects conflict artifacts independently of diagnostic ID, preflights directory repairs without partial writes, repairs only proven structural patterns, and refuses unrecognized shapes. Completion requires the same scoped formatter gate plus an artifact scan before affected builds and relevant tests. Technology-neutral work remains unaffected. Performs the requested work (not just a review), loads only relevant `references/`, respects repository conventions, scales process depth without lowering the standard, and reports evidence and risk honestly in concise feedback that may sacrifice grammar but never required evidence. Governing principle: consistency is key. | Invoke explicitly as `/agent-smith ` or let it auto-trigger for design, architecture, implementation, refactoring, code review, public API review, compatibility and Semantic Versioning analysis, testing, benchmarking, performance, skill authoring, documentation, security and DevSecOps, CI/CD, delivery, repository governance, and engineering assessment. Skill-authoring mode grounds instructions in real execution, requires an explicit bounded-concurrency assessment so independent data retrieval and eval work do not remain sequential by habit, favors reusable C#/.NET scripts and validators against the dynamically resolved latest supported LTS when local constraints do not decide, and follows the Agent Skills guidance for progressive disclosure, description optimization, candidate-versus-baseline evaluation, aggregation, and human review. Its optional .NET EditorConfig conformance mode handles targeted IDE/CA diagnostic remediation and full informational-or-higher `dotnet format` conformance without treating a clean build as proof of policy compliance: user-defined diagnostic IDs remain task-supplied data; target, path, and severity scope remains authoritative; informational workflows explicitly preserve `--severity info` because the formatter defaults to `warn`; targeted IDE and analyzer checks use category-specific formatter subcommands; every formatter invocation is read-only via `--verify-no-changes`; `--no-restore` is never treated as a conformance fallback; fixes are deliberate source edits; repeated multi-target findings are de-duplicated by physical file, diagnostic, and span; and the bundled `repair-roslyn-multiproject-artifacts.ps1` detects conflict artifacts independently of diagnostic ID, preflights directory repairs without partial writes, repairs only proven structural patterns, and refuses unrecognized shapes. Completion requires the same scoped formatter gate plus an artifact scan before affected builds and relevant tests. Technology-neutral work remains unaffected. Performs the requested work (not just a review), loads only relevant `references/`, respects repository conventions, scales process depth without lowering the standard, and reports evidence and risk honestly in concise feedback that may sacrifice grammar but never required evidence. Governing principle: consistency is key. | @@ -689,13 +689,17 @@ Cross-platform .NET developers usually get Linux test feedback the slow way: pus - **Microsoft's contract, not a new one** — honors the existing `testenvironments.json` version-1 schema (`name`, `localRoot`, `dockerImage`, `dockerFile`, either/or Docker source), treats it as authoritative when present, and never modifies it unless asked - **Zero-configuration by default** — with no `testenvironments.json`, it derives environments from Microsoft's live `releases-index.json` (supported LTS/STS channels plus the current preview) using `support-phase`/`release-type`, so no files are added to the repo and `.NET 10`/`.NET 11` are never hardcoded +- **Runs instead of asking** — invoking the skill *is* the request, so a repository with one applicable Docker environment goes straight to a test run with no menu, no questionnaire, and no confirmation; when several environments are derived, the repository's own target frameworks select one and the runner explains the choice +- **Whole matrix in one container** — a Microsoft SDK image ships a single runtime, so a repository targeting `net9.0;net10.0` builds there and then cannot execute the lower TFM; multi-targeted repositories resolve instead to a `codebeltnet/ubuntu-testrunner` combined tag (e.g. `8-9-10-11`) discovered from its live tag feed, running every target framework in one pass. Pointing a multi-targeted repo at a single-SDK image is reported as an SDK incompatibility naming the remedy, and `--framework` narrows the environment choice along with the run +- **One question, only when there is a real one** — the runner decides when a choice remains and exits `SelectionRequired` (16) with the exact candidates, so the single unavoidable question is precise and never expands into an intake form - **Offline-safe discovery and explicit scoping** — successful release metadata is cached outside the repo for offline reuse, and when a project choice is needed the form exposes the exact runner-computed target as the recommended option alongside a custom path -- **Official images, pinned to a digest** — auto-generated environments use only `mcr.microsoft.com/dotnet/sdk`, prefer the exact `latest-sdk` tag (preview build metadata stripped), validate the tag against Microsoft's registry, and resolve an immutable digest so a run is reproducible across environment, image, digest, SDK, and architecture +- **Recommended images, pinned to a digest** — auto-generated environments come from `mcr.microsoft.com/dotnet/sdk` for a single .NET major or `codebeltnet/ubuntu-testrunner` for several, prefer the exact `latest-sdk` tag (preview build metadata stripped), validate the tag against the registry, and resolve an immutable digest so a run is reproducible across environment, image, digest, SDK, and architecture; other images are never substituted on the runner's own initiative - **Tests run in Docker, the host stays clean** — source is staged into an isolated workspace so container builds never leave Linux `bin`/`obj` in the working tree, a persistent NuGet cache lives outside the repo, and it never silently falls back to running tests locally -- **Honest failure classification** — configuration, unsupported environment, Docker-unavailable, image-resolution, SDK-incompatibility, staging, restore, compilation, test-host, test-failure, result-processing, cleanup, cancellation, and release-metadata failures are distinct, so a container problem is never reported as a failing unit test +- **Honest failure classification** — configuration, unsupported environment, Docker-unavailable, image-resolution, SDK-incompatibility, staging, restore, compilation, test-host, test-failure, result-processing, cleanup, cancellation, release-metadata, and selection-required outcomes are distinct exit codes mapped to exactly one next action each, so behavior is the same whichever model is driving and a container problem is never reported as a failing unit test +- **Timing you can trust** — reports test duration and total wall clock separately, so a fast suite behind a slow image pull is never presented as an instant run - **No plumbing added, ever** — never generates a `Dockerfile`, dev container, compose file, or editor configuration (an existing configured `dockerFile` is honored, never created), never runs privileged containers or mounts the Docker socket, and always cleans up transient Docker resources — reporting exact identifiers if any remain - **Target-framework aware** — inspects the projects and `global.json`, refuses to pick an SDK that cannot build the requested target framework, and reports incompatibilities instead of editing the repository to force them -- **Deterministic and tested** — the runner ships a comprehensive built-in `--self-test` plus a PowerShell harness covering configuration discovery, release parsing, environment selection, unsupported handling, image resolution, command planning, result parsing, failure classification, cancellation, and cleanup +- **Deterministic and tested** — the runner ships a comprehensive built-in `--self-test` plus a PowerShell harness covering configuration discovery, release parsing, environment selection (including unattended target-framework selection and the cases that must still ask), unsupported handling, image resolution, command planning, result parsing, failure classification, cancellation, and cleanup ### Why dotnet-segregated-assets? `wwwroot` is where every ASP.NET Core developer expects to author static files — editors, hot reload, and the SDK all assume it. But shipping those files inside the deployed web application couples static delivery to business logic, bloats the app artifact, and puts asset caching on the wrong surface. The right shape is architectural: keep authoring in `wwwroot`, but let a separate, hardened static-content host serve the files in production. From c54b526b1417216b6fea014343e834f82cb650aa Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 00:09:09 +0200 Subject: [PATCH 05/31] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20document=20build-too?= =?UTF-8?q?ling=20preparation=20in=20dotnet-remote-testing=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The runner now probes images for required build tooling (git) and conditionally provisions it in a cached preparation layer. Update SKILL.md constraints and guidance to explain that the runner owns this behavior outside the repository, with docker-execution.md documenting the image preparation sequence, caching, and best-effort fallback when tooling cannot be added. --- skills/dotnet-remote-testing/SKILL.md | 12 +++++-- .../references/docker-execution.md | 36 +++++++++++++++---- 2 files changed, 39 insertions(+), 9 deletions(-) diff --git a/skills/dotnet-remote-testing/SKILL.md b/skills/dotnet-remote-testing/SKILL.md index 2b43ba5..bccf70d 100644 --- a/skills/dotnet-remote-testing/SKILL.md +++ b/skills/dotnet-remote-testing/SKILL.md @@ -61,7 +61,7 @@ These are your commands, not a menu for the developer. Never present them as opt - **Never run tests on the host and never silently fall back to local.** If remote testing was requested, tests must execute in the resolved Docker environment. If Docker is unavailable, report that (the runner exits `DockerUnavailable`) — do not run `dotnet test` locally instead. - **`testenvironments.json` is the configuration contract.** Honor Microsoft's existing version-1 schema. Do not invent a competing format, and do not modify `testenvironments.json` unless explicitly asked. -- **Do not generate container plumbing.** Never create a `Dockerfile`, `docker-compose.yml`/`compose.yml`, `.devcontainer/`, `.vscode/`, `Directory.Build.*`, or throwaway scripts to make remote testing work. Zero-configuration testing uses official Microsoft SDK images directly. An *existing* configured `dockerFile` is honored because it is deliberate repository intent. +- **Do not generate container plumbing.** Never create a `Dockerfile`, `docker-compose.yml`/`compose.yml`, `.devcontainer/`, `.vscode/`, `Directory.Build.*`, or throwaway scripts to make remote testing work. Zero-configuration testing uses official Microsoft SDK images directly. An *existing* configured `dockerFile` is honored because it is deliberate repository intent. The runner may derive a cached preparation layer for build tooling the image lacks (see [Build tooling in the image](#build-tooling-in-the-image)) — that happens inside the runner, outside the repository, and is never something you author. - **Do not hardcode .NET versions.** Supported LTS/STS channels and the current preview channel are discovered from Microsoft's release metadata at runtime. `.NET 10`/`.NET 11` are examples, never constants. - **Do not modify the repository to make tests pass.** Never edit `global.json`, project files, target frameworks, or test packages. Report incompatibilities instead. - **Report infrastructure failures as infrastructure, not as failing unit tests.** The runner classifies each phase distinctly; preserve that distinction when you summarize. @@ -146,6 +146,14 @@ Common scoping options (pass through only what the developer asked for): The runner establishes an isolated staged workspace (so container builds never leave Linux `bin`/`obj` in the working tree), mounts a persistent NuGet cache outside the repository, pins the image to its digest, runs restore → build → test, collects TRX results, and removes all transient Docker resources afterward. You do not manage any of that. +### Build tooling in the image + +The container runs the *build*, not just the tests, and a .NET build routinely shells out to `git` — MinVer, Nerdbank.GitVersioning, GitInfo and SourceLink all do. Microsoft's SDK images ship it; a minimal runner image may not, and the build then fails with `MINVER1007: "git" is not present in PATH` even though nothing is wrong with the code. + +The runner handles this: it probes the resolved image and, when the tooling is missing, layers it on in a cached image tagged `dotnet-remote-testing/prepared:git-` — built in the runner's own temp directory, never in the repository, and reused by every later run against the same base image. The reported `Image`/`Digest` stay the base image; preparation is reported on its own `Tools:` line. If it cannot be added (no package manager, `--offline`), the run proceeds on the base image and says so. + +Relay that line when present, but do not act on it: it is not a repository problem and never a reason to edit `testenvironments.json`, author a `Dockerfile`, or fall back to the host. A recurring `Added git to the image` for a repository's own image is worth mentioning once — the durable fix belongs in that image, not here. + ## Step 5: Report results concisely Lead with the outcome, not the infrastructure. Mirror the runner's concise result and suppress pull/restore/build log noise unless something failed: @@ -209,7 +217,7 @@ A run is reproducible in terms of environment, requested image, resolved digest, ## What this skill must never do -- Generate a `Dockerfile`, dev container, editor config, or any repository-specific plumbing. (Honor an *existing* configured `dockerFile`; never create one.) +- Generate a `Dockerfile`, dev container, editor config, or any repository-specific plumbing. (Honor an *existing* configured `dockerFile`; never create one. The runner's own cached preparation layer is not repository plumbing and is not yours to write.) - Run privileged containers, mount the Docker socket, mount the whole user profile, forward host credentials indiscriminately, disable TLS validation, expose ports, or print secrets. The runner already avoids these; do not add them. - Answer an invocation with a menu of its own capabilities, a "what would you like me to help you with?" opener, or a confirmation prompt for a run the developer already asked for. - Reach for an arbitrary image when a recommended one fits. Auto-generated environments use `mcr.microsoft.com/dotnet/sdk` for a single .NET major and `codebeltnet/ubuntu-testrunner` for several; an explicit `dockerImage` in `testenvironments.json` is deliberate intent and is used exactly as written. Other images are permitted but must be a deliberate, stated choice — never a substitution you make on your own. diff --git a/skills/dotnet-remote-testing/references/docker-execution.md b/skills/dotnet-remote-testing/references/docker-execution.md index b174f16..def3d59 100644 --- a/skills/dotnet-remote-testing/references/docker-execution.md +++ b/skills/dotnet-remote-testing/references/docker-execution.md @@ -9,11 +9,12 @@ Every `run` follows the same deterministic sequence: 1. Validate Docker availability. If unavailable, exit `DockerUnavailable` — never fall back to the host. 2. Resolve the environment (configured or Microsoft-derived). 3. Resolve the image and pin it to an immutable digest. -4. Establish an isolated, disposable staged workspace. -5. Prepare deterministic mounts and caches. -6. Execute restore → build → test. -7. Collect structured results (TRX). -8. Clean up transient resources. +4. Ensure the image carries the build tooling (see [Image preparation](#image-preparation)). +5. Establish an isolated, disposable staged workspace. +6. Prepare deterministic mounts and caches. +7. Execute restore → build → test. +8. Collect structured results (TRX). +9. Clean up transient resources. Cleanup is mandatory after success, test failure, build failure, restore failure, cancellation, and exceptions. If cleanup fails, the exact remaining Docker resource identifiers are reported. @@ -23,6 +24,27 @@ Container execution must not pollute or mutate the developer's working tree. The The runner never creates `Dockerfile`, `docker-compose.yml`/`compose.yml`, `.devcontainer/`, `.vscode/`, `Directory.Build.*`, temporary scripts, or generated test configuration in the repository. +## Image preparation + +The container runs the *build*, not only the tests, and a .NET build routinely shells out to host tooling: MinVer, Nerdbank.GitVersioning, GitInfo and SourceLink all invoke `git` during build. An image without it fails the build — MinVer reports `MINVER1007: "git" is not present in PATH` — even though the code is perfectly fine, which surfaces a missing tool as if it were a repository problem. + +Microsoft's SDK images ship `git`; a minimal runner image may not. So after the image is resolved and pinned, the runner probes it (`command -v git`) and, when the tooling is missing, derives a single layer from the resolved base: + +```dockerfile +FROM +USER root +RUN … install git with whichever package manager the base image ships (apt-get / apk / microdnf / dnf / yum) +``` + +This preparation layer is: + +- **outside the repository** — the Dockerfile is written into the run's own temporary directory, never into the working tree, so the "never generate container plumbing" rule is intact; +- **content-addressed and cached** — tagged `dotnet-remote-testing/prepared:git-`, so it is built once per base image and reused by every later run (`Reused prepared image providing git.`); +- **transparent** — the reported `Image`/`Digest` remain the resolved base image (reproducibility identity), with the preparation reported separately; +- **best effort** — if the tooling cannot be added (no package manager, `--offline`, no network), the base image is used anyway and the reason is reported, because a repository that never invokes `git` runs fine without it. + +`.git` itself is not staged into the workspace. Version-deriving tools therefore fall back to their default version (MinVer: `0.0.0-alpha.0`, a warning) instead of failing, which is the right trade for a test run. + ## Mounts Three bind mounts, nothing more: @@ -33,7 +55,7 @@ Three bind mounts, nothing more: | NuGet cache | `/nuget` | Persistent package cache owned outside the repository | Yes | | Results | `/results` | TRX output read back by the host | No (removed after the run) | -The dependency cache (`/nuget`, via `NUGET_PACKAGES`) is deliberately separated from the per-execution build/test workspace: the immutable, reusable dependency cache may persist for fast feedback, while the build/test workspace is isolated per execution so results never depend on stale source. +The dependency cache (`/nuget`, via `NUGET_PACKAGES`) is deliberately separated from the per-execution build/test workspace: the immutable, reusable dependency cache may persist for fast feedback, while the build/test workspace is isolated per execution so results never depend on stale source. `NUGET_PACKAGES` is exported with a trailing slash (`/nuget/`) because NuGet's package root becomes an MSBuild `SourceRoot`, and SourceLink fails the build on a `SourceRoot` that does not end in a separator. ## In-container phases @@ -59,7 +81,7 @@ Failures are classified into distinct kinds so a container/infrastructure proble ## Cancellation and cleanup -The container is given a deterministic, knowable name so it can always be targeted for cleanup — even after Ctrl+C or a `--timeout`. On cancellation the runner force-removes the container and deletes the staged workspace and results directory; the persistent NuGet cache is kept. `docker run --rm` also auto-removes the container on normal completion. +The container is given a deterministic, knowable name so it can always be targeted for cleanup — even after Ctrl+C or a `--timeout`. On cancellation the runner force-removes the container and deletes the staged workspace and results directory; the persistent NuGet cache and any cached preparation image are kept — both are reusable assets, not leftovers. `docker run --rm` also auto-removes the container on normal completion. ## Security posture From 6618d7f427ab2751d5d6d0f2db26d67026a036e3 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 00:09:18 +0200 Subject: [PATCH 06/31] =?UTF-8?q?=E2=9C=A8=20add=20image=20provisioning=20?= =?UTF-8?q?to=20dotnet-remote-testing=20runner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement ImageProvisioner to handle missing build tooling in container images. The runner probes the resolved image for git (required by MinVer, Nerdbank.GitVersioning, GitInfo, and SourceLink during dotnet build), and when absent, provisions a cached preparation layer using the appropriate package manager (apt-get, apk, microdnf). Add NuGetPackagesPath helper to ensure SourceLink receives SourceRoot paths with required trailing separator, ErrorLines helper for diagnostic output, FirstNonEmpty string utility, and ImagePreparationTests covering tag derivation, probing, dockerfile generation, and package-manager detection. --- .../scripts/remote-test.cs | 54 +++++++++++++++++-- 1 file changed, 51 insertions(+), 3 deletions(-) diff --git a/skills/dotnet-remote-testing/scripts/remote-test.cs b/skills/dotnet-remote-testing/scripts/remote-test.cs index 04d34de..ba698c7 100644 --- a/skills/dotnet-remote-testing/scripts/remote-test.cs +++ b/skills/dotnet-remote-testing/scripts/remote-test.cs @@ -1360,6 +1360,12 @@ internal static class ContainerPlanner return string.IsNullOrWhiteSpace(test) ? null : $"FullyQualifiedName~{test}"; } + // NuGet's package root becomes an MSBuild SourceRoot, and SourceLink rejects a SourceRoot that does + // not end in a separator ("SourceRoot paths are required to end with a slash or backslash"). The + // mount target stays clean; only the environment value carries the trailing slash. + public static string NuGetPackagesPath(string containerDir) => + containerDir.EndsWith('/') ? containerDir : containerDir + "/"; + // The in-container script. Phases run in order; each emits a machine-readable end marker with its // exit code so the host can classify restore vs build vs test outcomes precisely. restore/build stop // the run on failure; test always runs to completion so a TRX is produced even when tests fail. @@ -1373,7 +1379,7 @@ public static string BuildEntrypoint(TestCommandOptions o) var sb = new StringBuilder(); sb.Append("set -o pipefail\n"); - sb.Append($"export NUGET_PACKAGES={Shell.Quote(o.NuGetDir)}\n"); + sb.Append($"export NUGET_PACKAGES={Shell.Quote(NuGetPackagesPath(o.NuGetDir))}\n"); sb.Append("export DOTNET_CLI_TELEMETRY_OPTOUT=1\n"); sb.Append("export DOTNET_NOLOGO=1\n"); sb.Append("export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1\n"); @@ -1396,7 +1402,7 @@ public static ContainerPlan Build( { ["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1", ["DOTNET_NOLOGO"] = "1", - ["NUGET_PACKAGES"] = test.NuGetDir, + ["NUGET_PACKAGES"] = NuGetPackagesPath(test.NuGetDir), }; var entrypoint = BuildEntrypoint(test); @@ -3131,6 +3137,21 @@ private static string LastLines(string s, int count) var lines = s.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); return string.Join('\n', lines.TakeLast(count)); } + + // MSBuild/NuGet diagnostics, distilled from the build log so a failure names its cause. + private static string ErrorLines(string s, int count) + { + var lines = s.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries) + .Where(l => l.Contains(" error ", StringComparison.OrdinalIgnoreCase) + || l.Contains(": error", StringComparison.OrdinalIgnoreCase) + || l.StartsWith("error", StringComparison.OrdinalIgnoreCase)) + .Distinct(StringComparer.Ordinal) + .ToList(); + return string.Join('\n', lines.TakeLast(count)); + } + + private static string FirstNonEmpty(params string[] candidates) => + candidates.FirstOrDefault(c => !string.IsNullOrWhiteSpace(c)) ?? ""; } // --------------------------------------------------------------------------------------------------- @@ -3157,6 +3178,7 @@ public static int Run() UnsupportedEnvironmentTests(); TargetFrameworkTests(); CommandPlanningTests(); + ImagePreparationTests(); ResultParsingTests(); FailureClassificationTests(); CancellationAndCleanupTests(); @@ -3524,7 +3546,11 @@ private static void CommandPlanningTests() Check("entrypoint runs restore/build/test in order", entry.IndexOf("run_phase restore", StringComparison.Ordinal) < entry.IndexOf("run_phase build", StringComparison.Ordinal) && entry.IndexOf("run_phase build", StringComparison.Ordinal) < entry.IndexOf("run_phase test", StringComparison.Ordinal)); - Check("entrypoint sets NUGET_PACKAGES to the cache mount", entry.Contains("export NUGET_PACKAGES='/nuget'")); + // Trailing slash is required: NuGet's package root becomes an MSBuild SourceRoot and SourceLink + // fails the build without it. + Check("entrypoint sets NUGET_PACKAGES to the cache mount", entry.Contains("export NUGET_PACKAGES='/nuget/'")); + Check("nuget package root ends with a separator", + ContainerPlanner.NuGetPackagesPath("/nuget") == "/nuget/" && ContainerPlanner.NuGetPackagesPath("/nuget/") == "/nuget/"); Check("entrypoint uses --no-restore/--no-build to reuse phases", entry.Contains("--no-restore") && entry.Contains("--no-build")); Check("entrypoint honors configuration/framework/filter/coverage", entry.Contains("-c 'Release'") && entry.Contains("--framework 'net10.0'") && entry.Contains("--filter 'Category=Unit'") && entry.Contains("XPlat Code Coverage")); @@ -3568,6 +3594,28 @@ private static void CommandPlanningTests() } } + private static void ImagePreparationTests() + { + Section("Image preparation"); + + const string digest = "sha256:990d47a4f925dedf27c875271c8b592e201666536f955befef9147745652f29f"; + var tag = ImageProvisioner.DerivedTag("codebeltnet/ubuntu-testrunner:8-9-10-11", digest); + Check("prepared tag is derived from the base digest", tag == "dotnet-remote-testing/prepared:git-990d47a4f925dedf"); + Check("prepared tag is stable for the same image", ImageProvisioner.DerivedTag("other:tag", digest) == tag); + Check("prepared tag changes with the base image", + ImageProvisioner.DerivedTag("x:1", "sha256:abcdef0123456789abcdef") != tag); + Check("prepared tag needs no digest", ImageProvisioner.DerivedTag("x:1", null).StartsWith("dotnet-remote-testing/prepared:git-", StringComparison.Ordinal)); + + Check("probe asks the image for the tooling", ImageProvisioner.ProbeCommand() == "command -v git >/dev/null 2>&1"); + + var dockerfile = ImageProvisioner.Dockerfile("codebeltnet/ubuntu-testrunner:8-9-10-11"); + Check("provisioning layers onto the resolved base image", dockerfile.StartsWith("FROM codebeltnet/ubuntu-testrunner:8-9-10-11", StringComparison.Ordinal)); + Check("provisioning installs the required tooling", dockerfile.Contains("install -y --no-install-recommends git")); + Check("provisioning adapts to the image's package manager", + dockerfile.Contains("command -v apt-get") && dockerfile.Contains("command -v apk") && dockerfile.Contains("command -v microdnf")); + Check("provisioning fails loudly on an unknown package manager", dockerfile.Contains("No supported package manager")); + } + private static void ResultParsingTests() { Section("Result parsing"); From 5b4213e12351575addaebc77dba92f6fb41b8a3c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 00:09:25 +0200 Subject: [PATCH 07/31] =?UTF-8?q?=F0=9F=92=AC=20update=20changelog=20with?= =?UTF-8?q?=20build-tooling=20preparation=20details?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expand the v0.9.0 changelog entry for dotnet-remote-testing to document the build-tooling preparation feature: the runner probes images for git, provisions a cached preparation layer when missing, and reports its status transparently without treating it as a repository problem or reason to author container plumbing. --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f0c5931..08bda18 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-r - `dotnet-remote-testing` skill that runs .NET tests inside Docker using official `mcr.microsoft.com/dotnet/sdk` images, honoring an existing `testenvironments.json` as authoritative when present and otherwise deriving environments from Microsoft's live release index, while reporting WSL and SSH as unsupported instead of silently falling back to the host, - `dotnet-remote-testing` deterministic runner `remote-test.cs` owning configuration discovery, release parsing, digest-pinned image resolution, isolated source staging, NuGet caching, execution, result parsing, distinct failure classification, and cleanup behind a single entry point, with a built-in `--self-test` alongside a PowerShell harness, - Offline-safe release discovery for `dotnet-remote-testing`: successful release metadata is cached outside the repository so later runs work without network access, and the parameter form surfaces the exact runner-computed target as the recommended option, +- Build-tooling preparation in `dotnet-remote-testing`, probing the resolved image for the `git` that MinVer, Nerdbank.GitVersioning, GitInfo, and SourceLink invoke during `dotnet build`, and layering it on through a digest-addressed image cached outside the repository when the image lacks it, so a minimal runner image no longer fails a sound build with `MINVER1007` while the reported image and digest stay the resolved base, - `dotnet-segregated-assets` skill that migrates an ASP.NET Core application to serve deployed static content from Codebelt Static Content Provider (`codebeltnet/web-cdn-origin:2.0.0`) while `wwwroot` remains the authoring root, separating app-owned assets from shared CDN assets and preserving Razor Class Library, framework, and generated Static Web Assets, - `dotnet-segregated-assets` deterministic runner `segregate-assets.cs` that inspects static-asset topology, classifies existing segregation state, escalates Blazor, Razor Class Library, scoped-CSS, and frontend-build risk instead of blindly excluding it, resolves Cuemon TagHelper package versions from the NuGet V3 service index at plan time, reports cache-busting interfaces and registrations without rewriting Razor or C# source, and proves the publish invariant through `verify --run-publish` against an isolated temp directory, - Artifact-first container contract for `dotnet-segregated-assets` in which both application Dockerfiles package an already-published `artifacts/publish/` directory rather than compiling source, with the validator rejecting an SDK stage, a `dotnet build` or `dotnet publish` step, an `mcr.microsoft.com` runtime, or a missing artifact copy, From b3f9b306ace8f02bc09c78a75b135e79e5afbdab Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 00:17:16 +0200 Subject: [PATCH 08/31] =?UTF-8?q?=F0=9F=94=A8=20add=20sync-skill-install.p?= =?UTF-8?q?s1=20script=20for=20skill=20synchronization?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement deterministic skill synchronization script that copies whole skill trees from repository source to three local installs (Claude, global agents, Gemini Antigravity), compares SHA-256 hashes across all four locations, and exits non-zero on any drift. Script supports -Skill parameter to sync a single skill, -VerifyOnly to check without copying, and -Prune to delete install-only files. Excludes generated build output (bin/, obj/) which is regenerated per location. This is the mechanized equivalent of the previous manual sync guidance. --- scripts/sync-skill-install.ps1 | 154 +++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 scripts/sync-skill-install.ps1 diff --git a/scripts/sync-skill-install.ps1 b/scripts/sync-skill-install.ps1 new file mode 100644 index 0000000..38ef4ba --- /dev/null +++ b/scripts/sync-skill-install.ps1 @@ -0,0 +1,154 @@ +param( + [string[]]$Skill, + [switch]$VerifyOnly, + [switch]$Prune +) + +$ErrorActionPreference = 'Stop' + +Set-StrictMode -Version Latest + +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) +[Console]::InputEncoding = $utf8NoBom +[Console]::OutputEncoding = $utf8NoBom +$OutputEncoding = $utf8NoBom + +# Build output is regenerated per location, so its hashes never match across installs. Comparing it +# would bury real drift under noise and push agents back to fragile per-file copying. +$excludePattern = '(^|/)(bin|obj)/' + +function Get-RepoRoot { + return (Resolve-Path (Join-Path $PSScriptRoot '..')).Path +} + +function Get-InstallRoot { + param([string]$SkillName) + + $home_ = [Environment]::GetFolderPath('UserProfile') + return @( + (Join-Path $home_ ".claude/skills/$SkillName"), + (Join-Path $home_ ".agents/skills/$SkillName"), + (Join-Path $home_ ".gemini/antigravity-cli/skills/$SkillName") + ) +} + +function Get-RelativeFile { + param([string]$Root) + + if (-not (Test-Path $Root)) { + return @() + } + + $base = (Resolve-Path $Root).Path.TrimEnd('\', '/') + return @(Get-ChildItem -LiteralPath $Root -Recurse -File -Force | + ForEach-Object { $_.FullName.Substring($base.Length + 1).Replace('\', '/') } | + Where-Object { $_ -notmatch $excludePattern }) +} + +function Sync-SkillTree { + param( + [string]$SourceRoot, + [string]$InstallRoot, + [string[]]$RelativeFile + ) + + foreach ($rel in $RelativeFile) { + $destination = Join-Path $InstallRoot $rel + $destinationDir = Split-Path -Parent $destination + if (-not (Test-Path $destinationDir)) { + New-Item -ItemType Directory -Force -Path $destinationDir | Out-Null + } + + Copy-Item -LiteralPath (Join-Path $SourceRoot $rel) -Destination $destination -Force + } +} + +function Test-SkillTree { + param( + [string]$SourceRoot, + [string]$InstallRoot, + [string[]]$RelativeFile + ) + + $drift = @() + + foreach ($rel in $RelativeFile) { + $expected = (Get-FileHash -LiteralPath (Join-Path $SourceRoot $rel) -Algorithm SHA256).Hash + $destination = Join-Path $InstallRoot $rel + $actual = if (Test-Path -LiteralPath $destination) { + (Get-FileHash -LiteralPath $destination -Algorithm SHA256).Hash + } + else { + 'MISSING' + } + + if ($actual -ne $expected) { + $drift += " DRIFT $rel" + } + } + + # A rename or deletion in the repository leaves the old file behind in an install, where a stale + # skill keeps loading it. Extras are drift too, not cosmetic residue. + foreach ($rel in (Get-RelativeFile -Root $InstallRoot)) { + if ($RelativeFile -notcontains $rel) { + if ($Prune) { + Remove-Item -LiteralPath (Join-Path $InstallRoot $rel) -Force + } + else { + $drift += " EXTRA $rel" + } + } + } + + return $drift +} + +$repoRoot = Get-RepoRoot +$skillsRoot = Join-Path $repoRoot 'skills' + +if (-not $Skill -or $Skill.Count -eq 0) { + $Skill = @(Get-ChildItem -LiteralPath $skillsRoot -Directory | ForEach-Object { $_.Name }) +} + +$totalDrift = 0 + +foreach ($name in $Skill) { + $sourceRoot = Join-Path $skillsRoot $name + if (-not (Test-Path -LiteralPath $sourceRoot)) { + Write-Host "[SKIP] $name (not a repo-managed skill)" + continue + } + + $relativeFile = Get-RelativeFile -Root $sourceRoot + + foreach ($installRoot in (Get-InstallRoot -SkillName $name)) { + if (-not (Test-Path -LiteralPath $installRoot)) { + Write-Host "[SKIP] $name -> $installRoot (not installed)" + continue + } + + if (-not $VerifyOnly) { + Sync-SkillTree -SourceRoot $sourceRoot -InstallRoot $installRoot -RelativeFile $relativeFile + } + + $drift = @(Test-SkillTree -SourceRoot $sourceRoot -InstallRoot $installRoot -RelativeFile $relativeFile) + $totalDrift += $drift.Count + + if ($drift.Count -eq 0) { + Write-Host "[PASS] $name -> $installRoot ($($relativeFile.Count) files identical)" + } + else { + Write-Host "[FAIL] $name -> $installRoot ($($drift.Count) drifted)" + $drift | ForEach-Object { Write-Host $_ } + } + } +} + +Write-Host '' +if ($totalDrift -eq 0) { + Write-Host "Install sync: verified, 0 drift." + exit 0 +} + +Write-Host "Install sync: $totalDrift drifted entr$(if ($totalDrift -eq 1) { 'y' } else { 'ies' })." +exit 1 From ad660c1224ec30d78c98da9a2bcb893462b5302c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 00:17:30 +0200 Subject: [PATCH 09/31] =?UTF-8?q?=F0=9F=93=9A=20update=20repo=20governance?= =?UTF-8?q?=20for=20skill=20synchronization?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructure Local Install Sync section to reference the deterministic sync-skill-install.ps1 script, emphasizing that mechanical hash comparison is the required verification gate rather than manual file copying. Add new Blocking Completion Gates section that identifies required script executions and validators as hard requirements before completion; specifically mark sync-skill-install.ps1 as the final blocking gate since every prior step can still change files. This updates governance to enforce the mechanized sync process as repository policy. --- AGENTS.md | 726 +++++++++++++++++++++++++++--------------------------- 1 file changed, 359 insertions(+), 367 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index bfec2c1..92d099e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,24 +1,24 @@ -# Agent Guidelines - -Repository-level rules for AI agents working in this codebase. - -## Local Shell Execution - -Agents may use any appropriate local shell. When using PowerShell syntax or executing a `.ps1` script locally, use PowerShell 7+ through `pwsh`; never invoke `powershell` or `powershell.exe`. This does not prescribe GitHub Actions shell choices. - +# Agent Guidelines + +Repository-level rules for AI agents working in this codebase. + +## Local Shell Execution + +Agents may use any appropriate local shell. When using PowerShell syntax or executing a `.ps1` script locally, use PowerShell 7+ through `pwsh`; never invoke `powershell` or `powershell.exe`. This does not prescribe GitHub Actions shell choices. + ## Eval Isolation - -Eval workspaces and test repositories must **never** be created inside this repository. This includes: - -- `-workspace/` directories -- Temporary git repos for testing skills -- Test branches, throwaway commits, or config overrides (e.g. git aliases) - -When running evals or testing skills, create all workspaces in a temp location: - -- **Windows**: `$env:TEMP/-workspace/` -- **Unix**: `/tmp/-workspace/` - + +Eval workspaces and test repositories must **never** be created inside this repository. This includes: + +- `-workspace/` directories +- Temporary git repos for testing skills +- Test branches, throwaway commits, or config overrides (e.g. git aliases) + +When running evals or testing skills, create all workspaces in a temp location: + +- **Windows**: `$env:TEMP/-workspace/` +- **Unix**: `/tmp/-workspace/` + **Why:** Eval artifacts — branches, commits, local git config — leak into the real repo history and are painful to clean up. The skill source lives in a git repo; eval output does not belong here. ## AI/LLM Evaluation Automation Prohibition @@ -35,9 +35,9 @@ Repository scripts, CI jobs, skill runners, graders, optimizers, and custom exec This rule is Priority 1. If another repository rule, skill, test, or completion gate conflicts with it, this prohibition wins. ## Per-Skill Evals - -Every repo-managed skill must include its own `evals/evals.json` file at `skills//evals/evals.json`. - + +Every repo-managed skill must include its own `evals/evals.json` file at `skills//evals/evals.json`. + - Treat this as a required artifact for every first-party skill in this repo - Eval entries may include an optional `files` array of skill-relative fixture paths such as `evals/files/example.md` - When `files` is present, keep the paths relative to `skills//` and validate that every fixture exists @@ -47,20 +47,20 @@ Every repo-managed skill must include its own `evals/evals.json` file at `skills - Run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` once before completion for the repository gate - Follow the top-level **AI/LLM Evaluation Automation Prohibition** for every eval. No per-skill or third-party requirement overrides it. - Deterministic scaffold/template skills must keep local deterministic validators as well; evals supplement validators, they do not replace them - -If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` before considering the work complete. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. - -## Git Identity - -Never set or override `git user.name`, `git user.email`, or `alias.bot` in the **local** git config of this repository. Always use the global config. Local overrides silently shadow global settings and produce commits with the wrong author. - + +If you add a new skill or modify an existing repo-managed skill, update that skill's `evals/evals.json` before considering the work complete. Do not commit temp workspaces, benchmark outputs, or generated review files into this repository unless the user explicitly asks for checked-in artifacts. + +## Git Identity + +Never set or override `git user.name`, `git user.email`, or `alias.bot` in the **local** git config of this repository. Always use the global config. Local overrides silently shadow global settings and produce commits with the wrong author. + ## Git Operations Safeguards Agents must never automatically commit code changes or push to remote repositories. Both actions require explicit user approval: - -- **Commits**: Always request confirmation from the user before staging and committing code. Present a clear summary of changes and wait for user approval before executing the commit. -- **Remote Operations**: Do not push, pull, fetch, or interact with `origin` or any remote repository without explicit user instruction. These operations modify repository history and can cause data loss if performed unexpectedly. - + +- **Commits**: Always request confirmation from the user before staging and committing code. Present a clear summary of changes and wait for user approval before executing the commit. +- **Remote Operations**: Do not push, pull, fetch, or interact with `origin` or any remote repository without explicit user instruction. These operations modify repository history and can cause data loss if performed unexpectedly. + **Why:** Automatic commits can pollute history with incomplete work, debugging code, or unintended changes. Unexpected remote operations can overwrite or lose commits on shared branches. Always require the user to explicitly approve these operations. ### Commit Skill Routing @@ -70,336 +70,328 @@ When the user asks to commit or stage changes, write or review a commit message, Bare `yolo` or `auto` outside an explicit commit request does not invoke `git-visual-commits`. Likewise, those modifiers do not invoke `git-keep-a-changelog` unless the user explicitly requests a changelog or release-note output. Users can force deterministic CLI selection with `/git-visual-commits` when they do not want to rely on automatic skill selection. ## Skill Creation - -Always use the `skill-creator` skill (by Anthropic) when creating new skills, modifying existing skills, or running evals. It enforces best practices for structure, description quality, testing, and progressive disclosure. Do not create or edit skills manually without invoking it first. - -`skill-creator-agnostic` is deprecated, no longer maintained, and retained only for backward compatibility until 1.0.0. Agents must not use it for new skill creation, skill modification, or benchmarking; use Anthropic's `skill-creator` directly and apply the repository-specific requirements from this `AGENTS.md`. - -## Third-Party Skills - -Never modify skills maintained by others (e.g. `skill-creator` by Anthropic). If a third-party skill needs repo-specific behavior, add the rule here in `AGENTS.md` — not in the skill file itself, and not in a companion overlay around the third-party skill. Upstream updates will overwrite local edits without warning. - -## Local Install Sync - -Repo-managed skills live in four places that must stay in sync: - -- `skills//` — source control (and source of truth for edits) -- `~/.claude/skills//` — local Claude install -- `~/.agents/skills//` — local global agent install -- `~/.gemini/antigravity-cli/skills//` — local Gemini Antigravity install - -Changes often start in `~/.claude/skills//`, then get mirrored to the repo and the other local installs: - -- **Claude local → repo** (persist changes to source control): - ```powershell - Copy-Item "$HOME/.claude/skills//" "skills//" -Force - ``` -- **Claude local → agent installs** (keep `~/.agents` and Gemini current): - ```powershell - Copy-Item "$HOME/.claude/skills//" "$HOME/.agents/skills//" -Force - Copy-Item "$HOME/.claude/skills//" "$HOME/.gemini/antigravity-cli/skills//" -Force - ``` -- **Repo → local installs** (after pulling changes or cloning fresh): - ```powershell - Copy-Item "skills//" "$HOME/.claude/skills//" -Force - Copy-Item "skills//" "$HOME/.agents/skills//" -Force - Copy-Item "skills//" "$HOME/.gemini/antigravity-cli/skills//" -Force - ``` - -If you edit the `~/.agents/skills//` copy first, mirror it back to the repo and to `~/.claude/skills//` and `~/.gemini/antigravity-cli/skills//` using the same pattern. - -When renaming a skill, update **all four** locations — the repo folder, the local Claude install folder, the local global agent install folder, and the local Gemini Antigravity install folder. The folder name and the `name:` field in the SKILL.md frontmatter must match. A mismatch causes the skill to disappear from tooling or show stale instructions. - -A sync mismatch means one side runs a stale version, which leads to confusing eval results and wasted iterations. - -After the source copy passes its deterministic tests, SHA-256 identity across the repo and all three local installs is sufficient installation verification. Do not rerun the same deterministic suites from a hash-identical installed copy; that duplicates time, compute, and token use without adding evidence. Run an installed-copy test only when install-path resolution, loader behavior, permissions, or an actual hash mismatch is the subject of the test. - -After changing any repo-managed skill, sync the touched files across the repo copy, `~/.claude/skills//`, `~/.agents/skills//`, and `~/.gemini/antigravity-cli/skills//` before considering the task done. - -## Skill Directory Structure - -Every skill follows this layout: - -``` -skills// -├── SKILL.md # Required — the skill definition (loaded by Claude) -├── FORMS.md # Optional — structured form fields for parameter collection -├── assets/ # Optional — file templates, fonts, icons used in output -│ └── / # Group by variant when a skill supports multiple (e.g. library/, app/) -├── scripts/ # Optional — executable code (Python, Bash, etc.) -├── references/ # Optional — detailed reference docs the agent consults during generation -└── evals/ # Required for repo-managed skills — per-skill eval prompts and expectations - └── files/ # Optional — input fixtures referenced by evals/evals.json files[] -``` - -- `SKILL.md` is the entry point — it contains the workflow, conventions, and step-by-step instructions -- `assets/` holds file templates, fonts, icons, and other static content used in output (the agent reads and substitutes placeholders) -- `references/` holds detailed specs that `SKILL.md` references but are too long to inline -- `evals/` holds the per-skill `evals.json` definitions used to verify that the skill still works after changes -- `evals/files/` holds optional skill-local fixture inputs referenced by `evals/evals.json` when a benchmark needs attached source material - -## Template Files Are Literal - -Asset files in `assets/` are **not** processed by a templating engine. They contain real file content with placeholder values (e.g. `{ProjectName}`, `{TargetFramework}`) that the agent must read, understand, and substitute during generation. Agents should never copy asset files blindly — always read the content and adapt it to the user's specific parameters. - -## Prefer Dynamic Defaults - -When a skill needs time-sensitive or environment-sensitive values, prefer computing them from a reliable source instead of hardcoding them into prompts, defaults, or examples. - -- Prefer repo state, git metadata, official APIs, or vendor-maintained machine-readable feeds over date-stamped literals -- Use hardcoded fallback examples only when a dynamic source is unavailable or would add unreasonable complexity -- When a dynamic default exists, describe both the source and the fallback behavior in `FORMS.md` / `SKILL.md` -- If a value changes over time (supported frameworks, current versions, generated paths, repo-derived names), assume hardcoding will drift and design for refreshable computation - -## Scaffold Invariants - -For repo-managed .NET scaffolding skills, preserve semantic versioning infrastructure unless you are replacing it end-to-end in the same change. - -- App and library scaffolds rely on `MinVer` for versioning from git tags -- Do not remove `MinVer`, its package version, or its MSBuild hooks from scaffold templates unless a complete replacement workflow is implemented and validated in the same change -- Preserve the user-facing solution/product name in `PascalCase` for generated solution filenames such as `.slnx`; do not silently lowercase the solution filename -- Only derive lowercase values for fields that explicitly require them, such as repo slugs, package feeds, or Docker/image-style identifiers - -## Commit Discipline - -When committing changes to this repo, group by technology and logical purpose — don't mix unrelated changes. For example: - -- Skill instruction changes (`SKILL.md`) get their own commit -- Template files (`.csproj`, `.yml`, `.cs`) get their own commit(s) -- Documentation updates (`README.md`, `CONTRIBUTING.md`) get their own commit - -## Markdown Formatting - -All markdown files in this repository must use natural paragraph flow. Do not artificially break paragraphs at fixed column widths or insert hard line breaks within sentences. Paragraphs should flow as complete thoughts, allowing line wrapping to be determined by the reader's viewport or rendering engine, not by arbitrary character limits. - -**Why:** Natural paragraphs are more readable, easier to edit, and render correctly across all devices and markdown renderers. Artificially clipped paragraphs create maintenance friction and look awkward in source control diffs. - -## README Sync - -After modifying any skill (`SKILL.md`, `FORMS.md`) or repo-level config (`AGENTS.md`), **always update `README.md` before considering the task done**. This is a mandatory gate — not a nice-to-have. The README's "Available Skills" table, install examples, and "Why" sections must reflect the current state of all skills. A new skill without a README entry is incomplete work. - -## Blocking Completion Gates - -When repository guidance, an active skill, or a conversation summary identifies required follow-up work as pending, critical, blocking, or equivalent, treat those items as the active completion checklist for the current task rather than as background context. Do not call `task_complete`, describe the task as complete, or claim verification succeeded until every blocking item has either run successfully or been reported with the exact command, exit code, and remaining blocker. - -Before any completion message, reread the skill instructions and the current conversation summary's pending-task or blocker sections. If either one names a required script, validator, or maintenance step, that step is a hard gate, not optional polish. - -For script-backed workflows, creating or editing files is not enough on its own. If a skill requires deterministic maintenance or verification commands, run them before completion and report their concrete outcome. For `dotnet-docfx-digest`, `scripts/agents.cs` and `scripts/docfx.cs --build-api-model --validate-samples --verify-docfx-build` are blocking completion gates whenever the skill or task summary says they are required. - -## User Input UX - -When a skill collects parameters from the user, define the form in a dedicated `FORMS.md` file (Level 3 resource) rather than inlining field definitions in `SKILL.md`. This separates form structure from workflow logic and gives agents a parseable format to present fields correctly. - -Native input widgets are a **host/runtime feature**, not a guaranteed model capability. Treat them as an enhancement, not a dependency. - -- Skills must remain fully usable whether the host renders native fields or not -- When native fields are unavailable, the agent must follow a deterministic plain-text fallback defined in `FORMS.md` instead of improvising the interaction -- The fallback path must preserve the same field order, defaults, recommended choices, and final confirmation flow as the native-field path -- Do not switch interaction styles mid-collection unless the host explicitly upgrades from plain text to native controls -- Favor consistency and low-friction UX over conversational variety during parameter collection - -`FORMS.md` defines each field with: -- **type** — `text`, `single-choice`, or `multi-choice` -- **prompt** — the question to ask -- **choices** — options for choice types -- **default** — pre-filled value (mark as Recommended) -- **required** — whether the field is mandatory - -Presentation rules (enforced in every `FORMS.md`): -- Ask one field at a time — never bundle multiple questions -- Use selectable choices for `single-choice` and `multi-choice` fields — not free text -- When a default exists, present it first and append "(Recommended)" -- For `text` fields with a computed default, offer the computed value as a selectable choice alongside free text -- After all fields are collected, present a summary and ask for confirmation - -This applies to all skills that collect user input, not just scaffolding skills. - -## Status Update Hygiene - -Interim progress updates should describe user-relevant progress, evidence, blockers, and next steps. Do not narrate runner internals, sandbox mechanics, approved command paths, or retry plumbing unless that detail affects user approval, reproducibility, validation, or the final outcome. - -- Say what changed in the task state, not how the host executed the command -- Mention tool/runtime failures only when they block progress, require approval, or change the planned validation -- Prefer concise phrasing such as "The first read attempt failed before returning file content; I'm retrying and will report only if that changes the result" - -## Anthropic Skill Authoring Reference - -Essential conventions from [The Complete Guide to Building Skills for Claude](https://resources.anthropic.com/hubfs/The-Complete-Guide-to-Building-Skill-for-Claude.pdf) (Anthropic, Jan 2026). All skills in this repo must follow these rules. - -### File Structure - -``` -skill-name/ -├── SKILL.md # Required — exact spelling, case-sensitive -├── scripts/ # Optional — executable code (Python, Bash, etc.) -├── references/ # Optional — documentation loaded as needed -└── assets/ # Optional — templates, fonts, icons used in output -``` - -- **No `README.md`** inside the skill folder — all documentation goes in `SKILL.md` or `references/` -- Folder name must be **kebab-case** (no spaces, no underscores, no capitals) -- Folder name must match the `name:` field in YAML frontmatter - -### Progressive Disclosure (Three Levels) - -| Level | When loaded | Token cost | Content | -|-------|------------|------------|---------| -| **Level 1: Metadata** | Always (at startup) | ~100 tokens | `name` and `description` from YAML frontmatter | -| **Level 2: Instructions** | When skill is triggered | Under 5k tokens | SKILL.md body — workflows, steps, guidance | -| **Level 3: Resources** | As needed | Effectively unlimited | Linked files: scripts, references, assets, FORMS.md | - -Keep SKILL.md under **500 lines / 5,000 words**. Move detailed content to `references/`. Keep references **one level deep** from SKILL.md — nested references cause partial reads. - -### YAML Frontmatter - -Required fields: - -```yaml ---- -name: kebab-case-name # max 64 chars, lowercase + numbers + hyphens only -description: > # max 1024 chars, must include WHAT + WHEN + triggers - What it does. Use when user asks to [specific phrases]. ---- -``` - -Optional fields: - -```yaml -license: MIT # for open-source skills -compatibility: > # max 500 chars — environment requirements - Requires network access and Python 3.10+ -metadata: # custom key-value pairs - author: Company Name - version: 1.0.0 - mcp-server: server-name -``` - -**Forbidden**: XML angle brackets (`< >`), names containing "claude" or "anthropic" (reserved). - -### Description Field — The Most Important Part - -Structure: `[What it does] + [When to use it] + [Key capabilities]` - -```yaml -# ✅ Good — specific, actionable, includes triggers -description: > - Manages Linear project workflows including sprint planning, - task creation, and status tracking. Use when user mentions - "sprint", "Linear tasks", "project planning", or asks to - "create tickets". - -# ❌ Bad — too vague, no triggers -description: Helps with projects. -``` - -- Include trigger phrases users would actually say -- Mention file types if relevant -- Add negative triggers to prevent over-triggering: `Do NOT use for simple data exploration` - -### Writing Instructions - -- Be **specific and actionable** — `Run scripts/validate.py --input {filename}` not `Validate the data` -- Include **error handling** — common errors, causes, and solutions -- Use **feedback loops** — run validator → fix errors → repeat -- Put **critical instructions at the top** — use `## Critical` or `## Important` headers -- For critical validations, **use scripts over language instructions** — code is deterministic -- Prefer **dynamic defaults over hardcoded values** when the source data is available from the repo, environment, or an official machine-readable feed - -### Skill Categories - -| Category | Purpose | Example | -|----------|---------|---------| -| **Document & Asset Creation** | Consistent, high-quality output (docs, code, designs) | `frontend-design`, `docx`, `xlsx` | -| **Workflow Automation** | Multi-step processes with validation gates | `skill-creator`, scaffolding skills | -| **MCP Enhancement** | Workflow guidance layered on top of MCP tool access | `sentry-code-review` | - -### Common Patterns - -1. **Sequential workflow** — explicit step ordering with dependencies and rollback -2. **Multi-MCP coordination** — phase separation, data passing between services -3. **Iterative refinement** — draft → validate → fix → repeat until quality threshold -4. **Context-aware selection** — decision trees for choosing the right tool/approach -5. **Domain-specific intelligence** — compliance checks, governance, audit trails - -### Testing Checklist - -Before shipping a skill, verify: - -- [ ] Triggers on obvious tasks -- [ ] Triggers on paraphrased requests -- [ ] Does **not** trigger on unrelated topics -- [ ] Functional tests pass (correct outputs, error handling, edge cases) -- [ ] Performance improves over baseline (fewer messages, fewer errors, fewer tokens) - -Debug triggering: ask Claude `"When would you use the [skill name] skill?"` — it will quote the description back. - -### Troubleshooting Quick Reference - -| Symptom | Likely cause | Fix | -|---------|-------------|-----| -| Skill won't upload | `SKILL.md` misspelled or YAML invalid | Exact case `SKILL.md`, check `---` delimiters | -| Skill never triggers | Description too vague | Add trigger phrases, mention file types | -| Skill triggers too often | Description too broad | Add negative triggers, narrow scope | -| Instructions not followed | Too verbose or ambiguous | Shorten, use bullets, move detail to `references/` | -| Slow / degraded responses | Too much content loaded | Keep SKILL.md under 5k words, use progressive disclosure | - -## Karpathy Rules - -Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed. - -### 1. Think Before Coding - -**Don't assume. Don't hide confusion. Surface tradeoffs.** - -Before implementing: -- State your assumptions explicitly. If uncertain, ask. -- If multiple interpretations exist, present them - don't pick silently. -- If a simpler approach exists, say so. Push back when warranted. -- If something is unclear, stop. Name what's confusing. Ask. -- When the root cause is uncertain, do not present hypotheses as facts. State the uncertainty explicitly and ask whether to investigate before applying a fix. - -### 2. Simplicity First - -**Minimum code that solves the problem. Nothing speculative.** - -- No features beyond what was asked. -- No abstractions for single-use code. -- No "flexibility" or "configurability" that wasn't requested. -- No error handling for impossible scenarios. -- If you write 200 lines and it could be 50, rewrite it. - -Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. - -### 3. Surgical Changes - -**Touch only what you must. Clean up only your own mess.** - -When editing existing code: -- Don't "improve" adjacent code, comments, or formatting. -- Don't refactor things that aren't broken. -- Match existing style, even if you'd do it differently. -- If you notice unrelated dead code, mention it - don't delete it. - -When your changes create orphans: -- Remove imports/variables/functions that YOUR changes made unused. -- Don't remove pre-existing dead code unless asked. - -The test: Every changed line should trace directly to the user's request. - -### 4. Goal-Driven Execution - -**Define success criteria. Loop until verified.** - -Transform tasks into verifiable goals: -- "Add validation" → "Write tests for invalid inputs, then make them pass" -- "Fix the bug" → "Write a test that reproduces it, then make it pass" -- "Refactor X" → "Ensure tests pass before and after" - -For multi-step tasks, state a brief plan: -``` -1. [Step] → verify: [check] -2. [Step] → verify: [check] -3. [Step] → verify: [check] -``` - -Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification. + +Always use the `skill-creator` skill (by Anthropic) when creating new skills, modifying existing skills, or running evals. It enforces best practices for structure, description quality, testing, and progressive disclosure. Do not create or edit skills manually without invoking it first. + +`skill-creator-agnostic` is deprecated, no longer maintained, and retained only for backward compatibility until 1.0.0. Agents must not use it for new skill creation, skill modification, or benchmarking; use Anthropic's `skill-creator` directly and apply the repository-specific requirements from this `AGENTS.md`. + +## Third-Party Skills + +Never modify skills maintained by others (e.g. `skill-creator` by Anthropic). If a third-party skill needs repo-specific behavior, add the rule here in `AGENTS.md` — not in the skill file itself, and not in a companion overlay around the third-party skill. Upstream updates will overwrite local edits without warning. + +## Local Install Sync + +Repo-managed skills live in four places that must stay in sync: + +- `skills//` — source control (and source of truth for edits) +- `~/.claude/skills//` — local Claude install +- `~/.agents/skills//` — local global agent install +- `~/.gemini/antigravity-cli/skills//` — local Gemini Antigravity install + +Sync the **whole skill tree** with the repository as the source of truth, and prove it with hashes: + +``` +pwsh -NoProfile -File ./scripts/sync-skill-install.ps1 -Skill +``` + +The script copies every file under `skills//` into all three installs, then compares SHA-256 across all four locations and exits non-zero on any difference. `-VerifyOnly` checks without copying, `-Prune` deletes install files that no longer exist in the repository, and omitting `-Skill` sweeps every repo-managed skill. Generated build output (`bin/`, `obj/`) is excluded because it is regenerated per location and never matches; a file that exists only in an install is drift too, because a rename or deletion otherwise leaves the old one loading forever. + +**Run it as the last action before the completion message, after the final edit is in place.** Do not copy a remembered list of touched files: that list goes stale the moment you edit one more file, and a sync performed earlier in the session says nothing about what changed after it. Never report "synced" or "hash-identical" from memory, from an earlier turn, or from a partial per-file copy — the claim must be backed by this command's output in the same response that makes it. This is a [blocking completion gate](#blocking-completion-gates). + +If a change starts in `~/.claude/skills//` or another install, mirror the edited file back into `skills//` first, then run the script so the repository stays authoritative. + +When renaming a skill, update **all four** locations — the repo folder, the local Claude install folder, the local global agent install folder, and the local Gemini Antigravity install folder. The folder name and the `name:` field in the SKILL.md frontmatter must match. A mismatch causes the skill to disappear from tooling or show stale instructions. + +A sync mismatch means one side runs a stale version, which leads to confusing eval results and wasted iterations. + +After the source copy passes its deterministic tests, SHA-256 identity across the repo and all three local installs is sufficient installation verification. Do not rerun the same deterministic suites from a hash-identical installed copy; that duplicates time, compute, and token use without adding evidence. Run an installed-copy test only when install-path resolution, loader behavior, permissions, or an actual hash mismatch is the subject of the test. + +## Skill Directory Structure + +Every skill follows this layout: + +``` +skills// +├── SKILL.md # Required — the skill definition (loaded by Claude) +├── FORMS.md # Optional — structured form fields for parameter collection +├── assets/ # Optional — file templates, fonts, icons used in output +│ └── / # Group by variant when a skill supports multiple (e.g. library/, app/) +├── scripts/ # Optional — executable code (Python, Bash, etc.) +├── references/ # Optional — detailed reference docs the agent consults during generation +└── evals/ # Required for repo-managed skills — per-skill eval prompts and expectations + └── files/ # Optional — input fixtures referenced by evals/evals.json files[] +``` + +- `SKILL.md` is the entry point — it contains the workflow, conventions, and step-by-step instructions +- `assets/` holds file templates, fonts, icons, and other static content used in output (the agent reads and substitutes placeholders) +- `references/` holds detailed specs that `SKILL.md` references but are too long to inline +- `evals/` holds the per-skill `evals.json` definitions used to verify that the skill still works after changes +- `evals/files/` holds optional skill-local fixture inputs referenced by `evals/evals.json` when a benchmark needs attached source material + +## Template Files Are Literal + +Asset files in `assets/` are **not** processed by a templating engine. They contain real file content with placeholder values (e.g. `{ProjectName}`, `{TargetFramework}`) that the agent must read, understand, and substitute during generation. Agents should never copy asset files blindly — always read the content and adapt it to the user's specific parameters. + +## Prefer Dynamic Defaults + +When a skill needs time-sensitive or environment-sensitive values, prefer computing them from a reliable source instead of hardcoding them into prompts, defaults, or examples. + +- Prefer repo state, git metadata, official APIs, or vendor-maintained machine-readable feeds over date-stamped literals +- Use hardcoded fallback examples only when a dynamic source is unavailable or would add unreasonable complexity +- When a dynamic default exists, describe both the source and the fallback behavior in `FORMS.md` / `SKILL.md` +- If a value changes over time (supported frameworks, current versions, generated paths, repo-derived names), assume hardcoding will drift and design for refreshable computation + +## Scaffold Invariants + +For repo-managed .NET scaffolding skills, preserve semantic versioning infrastructure unless you are replacing it end-to-end in the same change. + +- App and library scaffolds rely on `MinVer` for versioning from git tags +- Do not remove `MinVer`, its package version, or its MSBuild hooks from scaffold templates unless a complete replacement workflow is implemented and validated in the same change +- Preserve the user-facing solution/product name in `PascalCase` for generated solution filenames such as `.slnx`; do not silently lowercase the solution filename +- Only derive lowercase values for fields that explicitly require them, such as repo slugs, package feeds, or Docker/image-style identifiers + +## Commit Discipline + +When committing changes to this repo, group by technology and logical purpose — don't mix unrelated changes. For example: + +- Skill instruction changes (`SKILL.md`) get their own commit +- Template files (`.csproj`, `.yml`, `.cs`) get their own commit(s) +- Documentation updates (`README.md`, `CONTRIBUTING.md`) get their own commit + +## Markdown Formatting + +All markdown files in this repository must use natural paragraph flow. Do not artificially break paragraphs at fixed column widths or insert hard line breaks within sentences. Paragraphs should flow as complete thoughts, allowing line wrapping to be determined by the reader's viewport or rendering engine, not by arbitrary character limits. + +**Why:** Natural paragraphs are more readable, easier to edit, and render correctly across all devices and markdown renderers. Artificially clipped paragraphs create maintenance friction and look awkward in source control diffs. + +## README Sync + +After modifying any skill (`SKILL.md`, `FORMS.md`) or repo-level config (`AGENTS.md`), **always update `README.md` before considering the task done**. This is a mandatory gate — not a nice-to-have. The README's "Available Skills" table, install examples, and "Why" sections must reflect the current state of all skills. A new skill without a README entry is incomplete work. + +## Blocking Completion Gates + +When repository guidance, an active skill, or a conversation summary identifies required follow-up work as pending, critical, blocking, or equivalent, treat those items as the active completion checklist for the current task rather than as background context. Do not call `task_complete`, describe the task as complete, or claim verification succeeded until every blocking item has either run successfully or been reported with the exact command, exit code, and remaining blocker. + +Before any completion message, reread the skill instructions and the current conversation summary's pending-task or blocker sections. If either one names a required script, validator, or maintenance step, that step is a hard gate, not optional polish. + +For script-backed workflows, creating or editing files is not enough on its own. If a skill requires deterministic maintenance or verification commands, run them before completion and report their concrete outcome. For `dotnet-docfx-digest`, `scripts/agents.cs` and `scripts/docfx.cs --build-api-model --validate-samples --verify-docfx-build` are blocking completion gates whenever the skill or task summary says they are required. + +Whenever a repo-managed skill was edited, `scripts/sync-skill-install.ps1` is a blocking completion gate in the same sense, and it is the last gate to run because every other step can still change a file. Report its actual output; an earlier run in the same session does not satisfy it. See [Local Install Sync](#local-install-sync). + +## User Input UX + +When a skill collects parameters from the user, define the form in a dedicated `FORMS.md` file (Level 3 resource) rather than inlining field definitions in `SKILL.md`. This separates form structure from workflow logic and gives agents a parseable format to present fields correctly. + +Native input widgets are a **host/runtime feature**, not a guaranteed model capability. Treat them as an enhancement, not a dependency. + +- Skills must remain fully usable whether the host renders native fields or not +- When native fields are unavailable, the agent must follow a deterministic plain-text fallback defined in `FORMS.md` instead of improvising the interaction +- The fallback path must preserve the same field order, defaults, recommended choices, and final confirmation flow as the native-field path +- Do not switch interaction styles mid-collection unless the host explicitly upgrades from plain text to native controls +- Favor consistency and low-friction UX over conversational variety during parameter collection + +`FORMS.md` defines each field with: +- **type** — `text`, `single-choice`, or `multi-choice` +- **prompt** — the question to ask +- **choices** — options for choice types +- **default** — pre-filled value (mark as Recommended) +- **required** — whether the field is mandatory + +Presentation rules (enforced in every `FORMS.md`): +- Ask one field at a time — never bundle multiple questions +- Use selectable choices for `single-choice` and `multi-choice` fields — not free text +- When a default exists, present it first and append "(Recommended)" +- For `text` fields with a computed default, offer the computed value as a selectable choice alongside free text +- After all fields are collected, present a summary and ask for confirmation + +This applies to all skills that collect user input, not just scaffolding skills. + +## Status Update Hygiene + +Interim progress updates should describe user-relevant progress, evidence, blockers, and next steps. Do not narrate runner internals, sandbox mechanics, approved command paths, or retry plumbing unless that detail affects user approval, reproducibility, validation, or the final outcome. + +- Say what changed in the task state, not how the host executed the command +- Mention tool/runtime failures only when they block progress, require approval, or change the planned validation +- Prefer concise phrasing such as "The first read attempt failed before returning file content; I'm retrying and will report only if that changes the result" + +## Anthropic Skill Authoring Reference + +Essential conventions from [The Complete Guide to Building Skills for Claude](https://resources.anthropic.com/hubfs/The-Complete-Guide-to-Building-Skill-for-Claude.pdf) (Anthropic, Jan 2026). All skills in this repo must follow these rules. + +### File Structure + +``` +skill-name/ +├── SKILL.md # Required — exact spelling, case-sensitive +├── scripts/ # Optional — executable code (Python, Bash, etc.) +├── references/ # Optional — documentation loaded as needed +└── assets/ # Optional — templates, fonts, icons used in output +``` + +- **No `README.md`** inside the skill folder — all documentation goes in `SKILL.md` or `references/` +- Folder name must be **kebab-case** (no spaces, no underscores, no capitals) +- Folder name must match the `name:` field in YAML frontmatter + +### Progressive Disclosure (Three Levels) + +| Level | When loaded | Token cost | Content | +|-------|------------|------------|---------| +| **Level 1: Metadata** | Always (at startup) | ~100 tokens | `name` and `description` from YAML frontmatter | +| **Level 2: Instructions** | When skill is triggered | Under 5k tokens | SKILL.md body — workflows, steps, guidance | +| **Level 3: Resources** | As needed | Effectively unlimited | Linked files: scripts, references, assets, FORMS.md | + +Keep SKILL.md under **500 lines / 5,000 words**. Move detailed content to `references/`. Keep references **one level deep** from SKILL.md — nested references cause partial reads. + +### YAML Frontmatter + +Required fields: + +```yaml +--- +name: kebab-case-name # max 64 chars, lowercase + numbers + hyphens only +description: > # max 1024 chars, must include WHAT + WHEN + triggers + What it does. Use when user asks to [specific phrases]. +--- +``` + +Optional fields: + +```yaml +license: MIT # for open-source skills +compatibility: > # max 500 chars — environment requirements + Requires network access and Python 3.10+ +metadata: # custom key-value pairs + author: Company Name + version: 1.0.0 + mcp-server: server-name +``` + +**Forbidden**: XML angle brackets (`< >`), names containing "claude" or "anthropic" (reserved). + +### Description Field — The Most Important Part + +Structure: `[What it does] + [When to use it] + [Key capabilities]` + +```yaml +# ✅ Good — specific, actionable, includes triggers +description: > + Manages Linear project workflows including sprint planning, + task creation, and status tracking. Use when user mentions + "sprint", "Linear tasks", "project planning", or asks to + "create tickets". + +# ❌ Bad — too vague, no triggers +description: Helps with projects. +``` + +- Include trigger phrases users would actually say +- Mention file types if relevant +- Add negative triggers to prevent over-triggering: `Do NOT use for simple data exploration` + +### Writing Instructions + +- Be **specific and actionable** — `Run scripts/validate.py --input {filename}` not `Validate the data` +- Include **error handling** — common errors, causes, and solutions +- Use **feedback loops** — run validator → fix errors → repeat +- Put **critical instructions at the top** — use `## Critical` or `## Important` headers +- For critical validations, **use scripts over language instructions** — code is deterministic +- Prefer **dynamic defaults over hardcoded values** when the source data is available from the repo, environment, or an official machine-readable feed + +### Skill Categories + +| Category | Purpose | Example | +|----------|---------|---------| +| **Document & Asset Creation** | Consistent, high-quality output (docs, code, designs) | `frontend-design`, `docx`, `xlsx` | +| **Workflow Automation** | Multi-step processes with validation gates | `skill-creator`, scaffolding skills | +| **MCP Enhancement** | Workflow guidance layered on top of MCP tool access | `sentry-code-review` | + +### Common Patterns + +1. **Sequential workflow** — explicit step ordering with dependencies and rollback +2. **Multi-MCP coordination** — phase separation, data passing between services +3. **Iterative refinement** — draft → validate → fix → repeat until quality threshold +4. **Context-aware selection** — decision trees for choosing the right tool/approach +5. **Domain-specific intelligence** — compliance checks, governance, audit trails + +### Testing Checklist + +Before shipping a skill, verify: + +- [ ] Triggers on obvious tasks +- [ ] Triggers on paraphrased requests +- [ ] Does **not** trigger on unrelated topics +- [ ] Functional tests pass (correct outputs, error handling, edge cases) +- [ ] Performance improves over baseline (fewer messages, fewer errors, fewer tokens) + +Debug triggering: ask Claude `"When would you use the [skill name] skill?"` — it will quote the description back. + +### Troubleshooting Quick Reference + +| Symptom | Likely cause | Fix | +|---------|-------------|-----| +| Skill won't upload | `SKILL.md` misspelled or YAML invalid | Exact case `SKILL.md`, check `---` delimiters | +| Skill never triggers | Description too vague | Add trigger phrases, mention file types | +| Skill triggers too often | Description too broad | Add negative triggers, narrow scope | +| Instructions not followed | Too verbose or ambiguous | Shorten, use bullets, move detail to `references/` | +| Slow / degraded responses | Too much content loaded | Keep SKILL.md under 5k words, use progressive disclosure | + +## Karpathy Rules + +Behavioral guidelines to reduce common LLM coding mistakes. Merge with project-specific instructions as needed. + +### 1. Think Before Coding + +**Don't assume. Don't hide confusion. Surface tradeoffs.** + +Before implementing: +- State your assumptions explicitly. If uncertain, ask. +- If multiple interpretations exist, present them - don't pick silently. +- If a simpler approach exists, say so. Push back when warranted. +- If something is unclear, stop. Name what's confusing. Ask. +- When the root cause is uncertain, do not present hypotheses as facts. State the uncertainty explicitly and ask whether to investigate before applying a fix. + +### 2. Simplicity First + +**Minimum code that solves the problem. Nothing speculative.** + +- No features beyond what was asked. +- No abstractions for single-use code. +- No "flexibility" or "configurability" that wasn't requested. +- No error handling for impossible scenarios. +- If you write 200 lines and it could be 50, rewrite it. + +Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. + +### 3. Surgical Changes + +**Touch only what you must. Clean up only your own mess.** + +When editing existing code: +- Don't "improve" adjacent code, comments, or formatting. +- Don't refactor things that aren't broken. +- Match existing style, even if you'd do it differently. +- If you notice unrelated dead code, mention it - don't delete it. + +When your changes create orphans: +- Remove imports/variables/functions that YOUR changes made unused. +- Don't remove pre-existing dead code unless asked. + +The test: Every changed line should trace directly to the user's request. + +### 4. Goal-Driven Execution + +**Define success criteria. Loop until verified.** + +Transform tasks into verifiable goals: +- "Add validation" → "Write tests for invalid inputs, then make them pass" +- "Fix the bug" → "Write a test that reproduces it, then make it pass" +- "Refactor X" → "Ensure tests pass before and after" + +For multi-step tasks, state a brief plan: +``` +1. [Step] → verify: [check] +2. [Step] → verify: [check] +3. [Step] → verify: [check] +``` + +Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification. From 8d8003f1f4d088cc7b8bb0118d39a5b72eccbcf5 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 00:17:43 +0200 Subject: [PATCH 10/31] =?UTF-8?q?=F0=9F=93=9D=20clarify=20skill=20synchron?= =?UTF-8?q?ization=20process=20in=20README?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expand the skill synchronization paragraph to explicitly document the sync-skill-install.ps1 script command and explain that mechanical hash-based verification is required, not just a remembered list of copied files. Clarify that a sync claim must be backed by actual command output in the same response that makes it, and that an earlier run does not satisfy the requirement. This aligns README user-facing documentation with the mechanized sync governance. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3d7b88f..154f582 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ DocFX prose, cleanup, unresolved ownership, and skip-marker diagnostics are not The DocFX validator also treats fallback `docfx.json` discovery conservatively: if a repo lacks a live root DocFX workspace, scaffold/template configs under skill assets are ignored unless their metadata globs resolve real projects. That keeps placeholder files like `skills/dotnet-new-lib-slnx/assets/library/.docfx/docfx.json` from masquerading as the active documentation workspace during repo-wide audits. -Local skill synchronization is verified efficiently: run deterministic tests against the repository source, copy touched files to the three local installs, and compare SHA-256 hashes across all four locations. Hash-identical copies are the same executable content, so agents do not repeat the same suites from an installed path unless install-path or loader behavior is specifically under test. +Local skill synchronization is verified efficiently and mechanically: run deterministic tests against the repository source, then run `pwsh -NoProfile -File ./scripts/sync-skill-install.ps1 -Skill ` as the final step, which copies the whole skill tree to the three local installs and compares SHA-256 hashes across all four locations, exiting non-zero on any difference. Syncing a remembered list of touched files is not enough — that list goes stale on the next edit — and a sync claim must be backed by that command's output rather than an earlier run. Hash-identical copies are the same executable content, so agents do not repeat the same suites from an installed path unless install-path or loader behavior is specifically under test. Resumed DocFX audits preserve every tracked and untracked documentation edit, regenerate the assessment work queue and example inventory, and process that queue in batches with a fast rerun after each batch. Encoding checks focus on actual damage: valid BOM-less UTF-8 is accepted, `ENCODING_BOM_MISSING` is not emitted, and audits do not create BOM-only or line-ending-only diffs. From c5dd446fd6b2a81f5b95e811f86b5e2bd317737a Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 01:22:22 +0200 Subject: [PATCH 11/31] =?UTF-8?q?=E2=9C=A8=20make=20dotnet-remote-testing?= =?UTF-8?q?=20results=20faithful=20to=20local=20environment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add git metadata staging to the workspace so version stamping, SourceLink, and repository-root detection behave as they do on the host. Introduce --no-git-metadata to opt out when .git dominates staging cost. Improve result reporting to break down per test assembly and target framework, with full failure detail (assertion message, stack trace, test output) so failures are actionable without a rerun. Add --show-log for diagnostics. --- skills/dotnet-remote-testing/SKILL.md | 39 +- skills/dotnet-remote-testing/evals/evals.json | 33 + .../references/docker-execution.md | 28 +- .../scripts/remote-test.cs | 578 ++++++++++++++++-- .../scripts/validate-skill.ps1 | 8 +- 5 files changed, 634 insertions(+), 52 deletions(-) diff --git a/skills/dotnet-remote-testing/SKILL.md b/skills/dotnet-remote-testing/SKILL.md index bccf70d..1d9a698 100644 --- a/skills/dotnet-remote-testing/SKILL.md +++ b/skills/dotnet-remote-testing/SKILL.md @@ -146,6 +146,23 @@ Common scoping options (pass through only what the developer asked for): The runner establishes an isolated staged workspace (so container builds never leave Linux `bin`/`obj` in the working tree), mounts a persistent NuGet cache outside the repository, pins the image to its digest, runs restore → build → test, collects TRX results, and removes all transient Docker resources afterward. You do not manage any of that. +Two diagnostic options exist for when a result needs explaining, not for routine runs: + +- `--show-log` — print the container's full restore/build/test log. Use it when the summarized failure detail is not enough, never by default. +- `--no-git-metadata` — skip staging `.git` (see [The staged workspace is still a repository](#the-staged-workspace-is-still-a-repository)). Only for a repository whose `.git` is large enough that copying it dominates the run, and only when the developer accepts the fidelity loss. + +### The staged workspace is still a repository + +The staged copy includes the repository's `.git` directory. This is not incidental: a .NET build and the code under test both read it. + +- MinVer, Nerdbank.GitVersioning and GitInfo derive the assembly version from git history. Without `.git` they silently fall back to `0.0.0`, so the container builds a differently-versioned assembly than the host. +- SourceLink stops embedding repository information. +- Application and test code commonly locates the repository root by walking up until a `.git` directory exists. Without it, that probe resolves somewhere else — and every path derived from it changes. + +That last one is why a suite can pass in Visual Studio's remote testing and fail here for reasons that have nothing to do with the container. If a run reports a failure that looks path-dependent, the staged workspace being a real repository is already accounted for; do not "fix" it by editing the repository. + +The runner installs `git` into the image when the image lacks it (see [Build tooling in the image](#build-tooling-in-the-image)) — the two halves belong together: the tooling and the metadata it reads. + ### Build tooling in the image The container runs the *build*, not just the tests, and a .NET build routinely shells out to `git` — MinVer, Nerdbank.GitVersioning, GitInfo and SourceLink all do. Microsoft's SDK images ship it; a minimal runner image may not, and the build then fails with `MINVER1007: "git" is not present in PATH` even though nothing is wrong with the code. @@ -156,7 +173,7 @@ Relay that line when present, but do not act on it: it is not a repository probl ## Step 5: Report results concisely -Lead with the outcome, not the infrastructure. Mirror the runner's concise result and suppress pull/restore/build log noise unless something failed: +Lead with the outcome, not the infrastructure. Mirror the runner's result and suppress pull/restore/build log noise unless something failed. The runner already reports at the granularity `dotnet test` does — one line per test assembly and target framework, then the totals: ``` Remote Test: dotnet-10-lts @@ -166,6 +183,8 @@ Image: mcr.microsoft.com/dotnet/sdk:10.0.302 Digest: sha256:... SDK: 10.0.302 +Passed! Cuemon.Core.Tests.dll (net10.0) — 1842 passed, 3 skipped, 0 failed, 21.8 s + Tests: 1842 passed, 3 skipped, 0 failed Time: 21.8 s (tests) Total: 96.4 s (including image pull, restore and build) @@ -173,16 +192,24 @@ Total: 96.4 s (including image pull, restore and build) Report both durations as the runner does. `Time` is the test execution time from the TRX; `Total` is wall clock for the whole operation. Collapsing them into one number misrepresents a fast suite behind a slow image pull. When the runner explains an automatic environment selection, relay that line — it is what makes an unattended choice auditable. -When tests fail, prioritize actionable detail — the failing test, its class, the expected/actual message, and location — over container startup output: +When tests fail, relay the runner's failure detail as it stands. It is deliberately shaped like `dotnet test` output — fully-qualified test name, the target framework it failed under, the assertion message, the stack trace and anything the test wrote itself — because that is what makes a red test fixable without a second run: ``` -3 tests failed +Failed! Cuemon.Text.Tests.dll (net10.0) — 110 passed, 0 skipped, 1 failed, 2.0 s +Passed! Cuemon.Text.Tests.dll (net9.0) — 111 passed, 0 skipped, 0 failed, 1.9 s -Cuemon.Text.Tests.StringUtilityTest - Sanitize_WithUnicode_ReturnsExpectedValue - Expected: ... Actual: ... +1 test failed: + + Failed Cuemon.Text.Tests.StringUtilityTest.Sanitize_WithUnicode_ReturnsExpectedValue [net10.0] (314 ms) + Assert.Equal() Failure: Values differ + Expected: 16 + Actual: 2 + Stack trace: + at Cuemon.Text.Tests.StringUtilityTest.Sanitize_WithUnicode_ReturnsExpectedValue() in /workspace/test/…/StringUtilityTest.cs:line 321 ``` +Do not compress this into a bare count. "1 test failed" without the name, the TFM and the message forces the developer to rerun the suite to learn what you already know. Note which target framework failed when a multi-targeted project fails under one TFM and passes under another — that asymmetry is usually the diagnosis. If the detail is still not enough, rerun with `--show-log` rather than guessing. + A run is reproducible in terms of environment, requested image, resolved digest, SDK, architecture, and runner version; include the image and digest so the result can be reproduced. ## Failure handling diff --git a/skills/dotnet-remote-testing/evals/evals.json b/skills/dotnet-remote-testing/evals/evals.json index 9e6ea2f..d476d09 100644 --- a/skills/dotnet-remote-testing/evals/evals.json +++ b/skills/dotnet-remote-testing/evals/evals.json @@ -189,6 +189,39 @@ "evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj", "evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs" ] + }, + { + "id": 12, + "prompt": "Remote test this repo.", + "expected_output": "The runner exits 1 with real failing tests. The skill reports them the way dotnet test would: which test assembly and target framework failed, the fully-qualified test name, the assertion message and the stack trace — and treats it as a test outcome to report, not an infrastructure problem and not a question.", + "expectations": [ + "Reports exit code 1 as a genuine test failure rather than an infrastructure or container problem", + "Names the failing test by its fully-qualified name and the target framework it failed under", + "Includes the assertion message and stack trace the runner reported instead of only a failure count", + "Distinguishes the failing test assembly/TFM from the ones that passed when a multi-targeted project fails under only one TFM", + "Does not rerun on the host, edit the repository, or modify the test to make it pass", + "Does not ask a question; a test failure is an outcome to report" + ], + "files": [ + "evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj", + "evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs" + ] + }, + { + "id": 13, + "prompt": "These tests pass under Visual Studio's remote testing against the same Docker image, but fail when I run them through you. Why?", + "expected_output": "The skill explains from the runner's own behavior rather than changing the repository: the staged workspace is a disposable copy that includes the repository's .git directory, so repository-root detection, MinVer/Nerdbank version stamping and SourceLink behave as they do on the host. It investigates the actual reported failure instead of assuming the container is at fault, and never edits the repository to force a pass.", + "expectations": [ + "Treats the difference as something to diagnose from the reported failure, not a reason to fall back to the host", + "Knows the staged workspace includes .git, so repository-root probes and version-deriving tools behave as on the host", + "Does not edit global.json, project files, target frameworks, test packages, or testenvironments.json to force a pass", + "Does not create a Dockerfile, dev container, or other repository plumbing", + "Offers --show-log for the full container log rather than guessing at the cause" + ], + "files": [ + "evals/files/multi-targeted/test/Matrix.Tests/Matrix.Tests.csproj", + "evals/files/multi-targeted/test/Matrix.Tests/MatrixTests.cs" + ] } ] } diff --git a/skills/dotnet-remote-testing/references/docker-execution.md b/skills/dotnet-remote-testing/references/docker-execution.md index def3d59..173db59 100644 --- a/skills/dotnet-remote-testing/references/docker-execution.md +++ b/skills/dotnet-remote-testing/references/docker-execution.md @@ -43,7 +43,23 @@ This preparation layer is: - **transparent** — the reported `Image`/`Digest` remain the resolved base image (reproducibility identity), with the preparation reported separately; - **best effort** — if the tooling cannot be added (no package manager, `--offline`, no network), the base image is used anyway and the reason is reported, because a repository that never invokes `git` runs fine without it. -`.git` itself is not staged into the workspace. Version-deriving tools therefore fall back to their default version (MinVer: `0.0.0-alpha.0`, a warning) instead of failing, which is the right trade for a test run. +## Git metadata in the workspace + +The repository's `.git` directory is copied into the staged workspace alongside the source. The copy is disposable, so the container may write to it freely; the developer's real repository is never mounted and never touched. + +This matters because staging the source without its git metadata is not a neutral omission — it changes observable build and test behavior: + +| Consumer | Without `.git` | +|---|---| +| MinVer / Nerdbank.GitVersioning / GitInfo | Silently fall back to `0.0.0`, so the container builds differently-versioned assemblies than the host | +| SourceLink | Stops embedding repository information | +| Repository-root probes (`walk up until a .git directory exists`) | Resolve to a different directory, changing every path derived from them — including paths the tests under it read | + +The third row is the subtle one: a suite that passes under Visual Studio's Remote Testing and fails here, for reasons unrelated to the container, is very often a repository-root probe landing somewhere else. Installing `git` into the image (above) and staging `.git` are two halves of one guarantee: the build tooling is present *and* the metadata it reads is present. + +A linked worktree or submodule stores `.git` as a `gitdir: ` pointer file; the runner resolves the pointer and stages the real directory, because the path it names does not exist inside the container. A source tree with no git metadata at all stages normally and silently — that is an ordinary case, not a degraded one. + +`--no-git-metadata` opts out for a repository whose `.git` is large enough that copying it dominates the run. The run then reports the fidelity loss rather than hiding it. ## Mounts @@ -75,10 +91,18 @@ Supported through options that pass straight into the phases: entire solution, a ## Result collection and classification -The runner parses every TRX in `/results` (multi-targeted projects emit one per TFM) into a single structured result: passed, skipped, failed counts, duration, and actionable failure detail (test name, class, message, location). It prioritizes machine-readable results and suppresses pull/restore/build noise on success. +The runner parses every TRX in `/results` into a single structured result, at the granularity `dotnet test` itself reports: + +- **Per test assembly and target framework** — a multi-targeted project emits one TRX per TFM, and the assembly path recorded in the TRX is the only thing that distinguishes them. Each becomes its own `Passed!`/`Failed!` line with its own counts and duration, so a failure is attributable to one test project under one TFM instead of a pooled total. +- **Aggregate counts and duration** across every assembly. +- **Per failing test**: fully-qualified name, owning class, target framework, elapsed time, assertion message, stack trace, and anything the test wrote to its own output helper. Detail is capped (15 failures, 10 stack frames, 15 output lines) so a wholesale failure stays readable; `--json` always carries the complete set. + +Pull/restore/build noise is suppressed on success. `--show-log` prints the container log in full when the summarized detail is not enough. Failures are classified into distinct kinds so a container/infrastructure problem is never misreported as a failing unit test: `Configuration`, `UnsupportedEnvironment`, `DockerUnavailable`, `ImageResolution`, `SdkIncompatibility`, `SourceStaging`, `Restore`, `Compilation`, `TestHost`, `TestFailure`, `ResultProcessing`, `Cleanup`, `Cancelled`, `ReleaseMetadataUnavailable`. A non-zero `dotnet test` exit with a TRX containing failures is a `TestFailure`; a non-zero exit with no failing results (crash, no discovered tests, missing adapter) is a `TestHost` failure. +Each phase emits a machine-readable end marker, so the log between two markers is exactly that phase's output. An infrastructure failure is reported with *its own* phase's log rather than a tail of everything — a build failure names the offending file and compiler error instead of trailing test-runner chatter. Any failing tests already recorded in a TRX are reported first even when the phase failed for another reason, so an assertion failure followed by a test-host crash does not disappear behind the crash. + ## Cancellation and cleanup The container is given a deterministic, knowable name so it can always be targeted for cleanup — even after Ctrl+C or a `--timeout`. On cancellation the runner force-removes the container and deletes the staged workspace and results directory; the persistent NuGet cache and any cached preparation image are kept — both are reusable assets, not leftovers. `docker run --rm` also auto-removes the container on normal completion. diff --git a/skills/dotnet-remote-testing/scripts/remote-test.cs b/skills/dotnet-remote-testing/scripts/remote-test.cs index ba698c7..4a22dba 100644 --- a/skills/dotnet-remote-testing/scripts/remote-test.cs +++ b/skills/dotnet-remote-testing/scripts/remote-test.cs @@ -160,6 +160,10 @@ internal sealed class Options public bool Coverage { get; private set; } public int TimeoutSeconds { get; private set; } + // Staging/reporting fidelity switches. Defaults reproduce what the developer sees locally. + public bool NoGitMetadata { get; private set; } + public bool ShowLog { get; private set; } + public static Options Parse(string[] args) { var o = new Options(); @@ -177,6 +181,8 @@ public static Options Parse(string[] args) case "--offline": o.Offline = true; break; case "--no-registry-check": o.NoRegistryCheck = true; break; case "--coverage": o.Coverage = true; break; + case "--no-git-metadata": o.NoGitMetadata = true; break; + case "--show-log": o.ShowLog = true; break; case "--repo-root": o.RepoRoot = Path.GetFullPath(Next(args, ref i, a)); break; case "--config-path": o.ConfigPath = Next(args, ref i, a); break; case "--environment" or "-e": o.EnvironmentName = Next(args, ref i, a); break; @@ -268,6 +274,9 @@ Test scoping (plan/run): -f, --framework Restrict multi-targeted test projects to one TFM. --coverage Collect code coverage (XPlat Code Coverage) when the project supports it. --timeout Abort the container run after N seconds (0 = no timeout). + --no-git-metadata Do not stage .git into the workspace (faster for a very large + repository, but repository-root detection, MinVer/Nerdbank + versioning and SourceLink will differ from the host). Release discovery / networking: --offline Use cached release metadata only; never reach the network. @@ -278,6 +287,7 @@ Test scoping (plan/run): Output: --json Emit machine-readable JSON. + --show-log Print the container's restore/build/test log in full. -h, --help Show this help. Exit codes: 0 success, 1 test failures, 2 invalid args, 3 configuration, 4 unsupported environment, @@ -1526,7 +1536,30 @@ private static string Relative(string root, string path) => // emit one per TFM) into a single structured result, prioritizing actionable failure detail. // --------------------------------------------------------------------------------------------------- -internal sealed record TestFailureDetail(string TestName, string? ClassName, string? Message, string? StackTrace); +// One failing test, with everything `dotnet test` would have printed about it: where it lives, what it +// asserted, where it threw, and whatever the test itself wrote to the output helper. +internal sealed record TestFailureDetail( + string TestName, + string? ClassName, + string? Message, + string? StackTrace, + string? Output = null, + string? Assembly = null, + string? Framework = null, + double DurationSeconds = 0); + +// One test assembly/TFM pair — the unit `dotnet test` reports a Passed!/Failed! line for. +internal sealed record TestAssemblyResult( + string Assembly, + string? Framework, + int Total, + int Passed, + int Failed, + int Skipped, + double DurationSeconds) +{ + public string Display => Framework is null ? Assembly : $"{Assembly} ({Framework})"; +} internal sealed record TestRunResult { @@ -1537,6 +1570,7 @@ internal sealed record TestRunResult public int NotExecuted { get; init; } public double DurationSeconds { get; init; } public IReadOnlyList Failures { get; init; } = []; + public IReadOnlyList Assemblies { get; init; } = []; public int TrxFilesParsed { get; init; } public static TestRunResult Empty => new(); @@ -1550,33 +1584,92 @@ internal sealed record TestRunResult NotExecuted = NotExecuted + other.NotExecuted, DurationSeconds = DurationSeconds + other.DurationSeconds, Failures = [.. Failures, .. other.Failures], + Assemblies = [.. Assemblies, .. other.Assemblies], TrxFilesParsed = TrxFilesParsed + other.TrxFilesParsed, }; } +// VSTest records the test assembly's path in lower case, so a TRX alone would report +// "acme.tests.dll" for an assembly the developer knows as "Acme.Tests.dll". The repository's own +// project files carry the authoritative casing. +internal static class AssemblyNameIndex +{ + private static readonly string[] ProjectPatterns = ["*.csproj", "*.fsproj", "*.vbproj"]; + + public static IReadOnlyDictionary Build(string sourceRoot) + { + var map = new Dictionary(StringComparer.OrdinalIgnoreCase); + if (!Directory.Exists(sourceRoot)) + { + return map; + } + + foreach (var pattern in ProjectPatterns) + { + IEnumerable files; + try + { + files = Directory.EnumerateFiles(sourceRoot, pattern, SearchOption.AllDirectories); + } + catch (Exception) + { + continue; + } + + foreach (var project in files) + { + var name = Path.GetFileNameWithoutExtension(project) + ".dll"; + map[name] = name; + } + } + + return map; + } +} + internal static class TrxParser { private static readonly XNamespace Ns = "http://microsoft.com/schemas/VisualStudio/TeamTest/2010"; - public static TestRunResult ParseFile(string path) => Parse(File.ReadAllText(path)); + public static TestRunResult ParseFile(string path, IReadOnlyDictionary? assemblyNames = null) => + Parse(File.ReadAllText(path), assemblyNames); - public static TestRunResult Parse(string trxXml) + public static TestRunResult Parse(string trxXml, IReadOnlyDictionary? assemblyNames = null) { var doc = XDocument.Parse(trxXml); var root = doc.Root ?? throw new InvalidOperationException("TRX has no root element."); - // Map testId -> class name via TestDefinitions so failures carry their owning class. + // Map testId -> class name / storage via TestDefinitions so failures carry their owning class and + // the assembly they came from. Multi-targeted projects emit one TRX per TFM, and only the storage + // path distinguishes them. var classById = new Dictionary(StringComparer.OrdinalIgnoreCase); + string? storage = null; + string? classAssembly = null; foreach (var ut in root.Descendants(Ns + "UnitTest")) { var id = ut.Attribute("id")?.Value; var className = ut.Element(Ns + "TestMethod")?.Attribute("className")?.Value; - if (id is not null && className is not null) + if (className is not null) { - classById[id] = className.Split(',')[0]; + var parts = className.Split(',', 2); + if (id is not null) + { + classById[id] = parts[0]; + } + + if (classAssembly is null && parts.Length == 2) + { + classAssembly = parts[1].Trim(); + } } + + storage ??= ut.Attribute("storage")?.Value + ?? ut.Element(Ns + "TestMethod")?.Attribute("codeBase")?.Value; } + var assemblyName = AssemblyDisplayName(storage, classAssembly, assemblyNames); + var framework = FrameworkFrom(storage); + int passed = 0, failed = 0, skipped = 0, notExecuted = 0, total = 0; var failures = new List(); double duration = 0; @@ -1593,18 +1686,24 @@ public static TestRunResult Parse(string trxXml) foreach (var r in root.Descendants(Ns + "UnitTestResult")) { var outcome = r.Attribute("outcome")?.Value ?? ""; - duration += ParseDuration(r.Attribute("duration")?.Value); + var testDuration = ParseDuration(r.Attribute("duration")?.Value); + duration += testDuration; if (string.Equals(outcome, "Failed", StringComparison.OrdinalIgnoreCase)) { var testId = r.Attribute("testId")?.Value; var testName = r.Attribute("testName")?.Value ?? "(unknown test)"; - var error = r.Element(Ns + "Output")?.Element(Ns + "ErrorInfo"); + var output = r.Element(Ns + "Output"); + var error = output?.Element(Ns + "ErrorInfo"); failures.Add(new TestFailureDetail( testName, testId is not null && classById.TryGetValue(testId, out var cls) ? cls : null, error?.Element(Ns + "Message")?.Value?.Trim(), - error?.Element(Ns + "StackTrace")?.Value?.Trim())); + error?.Element(Ns + "StackTrace")?.Value?.Trim(), + output?.Element(Ns + "StdOut")?.Value?.Trim(), + assemblyName, + framework, + Math.Round(testDuration, 3))); } else if (outcome is "NotExecuted" or "Skipped") { @@ -1632,11 +1731,14 @@ public static TestRunResult Parse(string trxXml) NotExecuted = notExecuted, DurationSeconds = Math.Round(duration, 3), Failures = failures, + Assemblies = assemblyName is null + ? [] + : [new TestAssemblyResult(assemblyName, framework, total, passed, failed, skipped, Math.Round(duration, 3))], TrxFilesParsed = 1, }; } - public static TestRunResult ParseDirectory(string directory) + public static TestRunResult ParseDirectory(string directory, IReadOnlyDictionary? assemblyNames = null) { var result = TestRunResult.Empty; if (!Directory.Exists(directory)) @@ -1648,7 +1750,7 @@ public static TestRunResult ParseDirectory(string directory) { try { - result = result.Merge(ParseFile(file)); + result = result.Merge(ParseFile(file, assemblyNames)); } catch (Exception) { @@ -1656,7 +1758,61 @@ public static TestRunResult ParseDirectory(string directory) } } - return result; + return result with + { + Assemblies = [.. result.Assemblies.OrderBy(a => a.Assembly, StringComparer.OrdinalIgnoreCase).ThenBy(a => a.Framework, StringComparer.OrdinalIgnoreCase)], + }; + } + + // "/workspace/test/Acme.Tests/bin/Debug/net10.0/Acme.Tests.dll" -> "Acme.Tests.dll". + internal static string? AssemblyNameFrom(string? storage) => + string.IsNullOrWhiteSpace(storage) ? null : Path.GetFileName(storage.Replace('\\', '/')); + + // VSTest lower-cases the storage path it records, which would report "acme.tests.dll" for an assembly + // the developer knows as "Acme.Tests.dll". Two sources restore the real casing, in order of + // authority: the repository's own project files, then the class name's assembly part + // ("Namespace.Type, Acme.Tests"), which some loggers include and which keeps its casing. + internal static string? AssemblyDisplayName( + string? storage, string? classAssembly, IReadOnlyDictionary? assemblyNames = null) + { + var fromStorage = AssemblyNameFrom(storage); + if (fromStorage is not null && assemblyNames is not null && assemblyNames.TryGetValue(fromStorage, out var known)) + { + return known; + } + + var simpleName = classAssembly?.Split(',')[0].Trim(); + if (string.IsNullOrWhiteSpace(simpleName)) + { + return fromStorage; + } + + var candidate = simpleName + ".dll"; + return fromStorage is null || string.Equals(candidate, fromStorage, StringComparison.OrdinalIgnoreCase) + ? candidate + : fromStorage; + } + + // The TFM is the output folder the test assembly was built into; it is the only place a TRX records + // which target framework produced it. + internal static string? FrameworkFrom(string? storage) + { + if (string.IsNullOrWhiteSpace(storage)) + { + return null; + } + + var segments = storage.Replace('\\', '/').Split('/', StringSplitOptions.RemoveEmptyEntries); + for (var i = segments.Length - 2; i >= 0; i--) + { + if (Regex.IsMatch(segments[i], @"^net(standard|coreapp|framework)?\d+(\.\d+)*(-[a-z0-9.]+)?$", RegexOptions.IgnoreCase) + || Regex.IsMatch(segments[i], @"^net\d{2,3}$", RegexOptions.IgnoreCase)) + { + return segments[i]; + } + } + + return null; } private static int IntAttr(XElement e, string name) => @@ -1776,14 +1932,42 @@ public static ExecutionOutcome Classify( public static IReadOnlyDictionary ParsePhaseMarkers(string output) { var map = new Dictionary(StringComparer.OrdinalIgnoreCase); - foreach (Match m in Regex.Matches(output, - Regex.Escape(ContainerPlanner.PhaseMarkerPrefix) + @"(?[a-zA-Z]+):(?-?\d+)##")) + foreach (Match m in MarkerPattern.Matches(output)) { map[m.Groups["name"].Value] = int.Parse(m.Groups["code"].Value, CultureInfo.InvariantCulture); } return map; } + + // The container emits one marker per phase, so the text between two markers is exactly that phase's + // log. Reporting the failing phase's own output — instead of a tail of everything — is what turns + // "the build failed" into "this file, this line, this compiler error". + public static string PhaseOutput(string output, string phase) + { + if (string.IsNullOrEmpty(output)) + { + return ""; + } + + var start = 0; + foreach (Match m in MarkerPattern.Matches(output)) + { + if (string.Equals(m.Groups["name"].Value, phase, StringComparison.OrdinalIgnoreCase)) + { + return output[start..m.Index].Trim('\r', '\n'); + } + + start = m.Index + m.Length; + } + + // The phase never completed (crash, cancellation): everything after the last marker is its log. + return output[start..].Trim('\r', '\n'); + } + + private static readonly Regex MarkerPattern = new( + Regex.Escape(ContainerPlanner.PhaseMarkerPrefix) + @"(?[a-zA-Z]+):(?-?\d+)##", + RegexOptions.Compiled); } // --------------------------------------------------------------------------------------------------- @@ -2218,15 +2402,28 @@ private static string StableHash(string value) // --------------------------------------------------------------------------------------------------- // Source staging — copy the source into an isolated, disposable workspace so container builds never // pollute the developer's working tree with Linux bin/obj artifacts. +// +// The staged copy must still *be* a repository. Dropping .git changes observable behavior: MinVer and +// Nerdbank.GitVersioning fall back to 0.0.0, SourceLink stops embedding, and any repository-root probe +// ("walk up until a .git directory exists") resolves somewhere else entirely — which silently changes +// what the tests under it see. That is the difference between "it passes in Visual Studio's remote +// testing and fails here", so .git is staged too, as a disposable copy the container may freely write. // --------------------------------------------------------------------------------------------------- -internal sealed record StagingResult(string? StagedPath, string? Error, int FileCount); +internal sealed record StagingResult( + string? StagedPath, + string? Error, + int FileCount, + bool GitMetadataStaged = false, + long GitMetadataBytes = 0, + string? GitMetadataNote = null); internal static class SourceStager { private static readonly string[] ExcludedDirs = ["bin", "obj", ".git", ".vs", ".vscode", "node_modules", "TestResults"]; - public static async Task StageAsync(string sourceRoot, string stagingRoot, CancellationToken ct) + public static async Task StageAsync( + string sourceRoot, string stagingRoot, CancellationToken ct, bool includeGitMetadata = true) { if (!Directory.Exists(sourceRoot)) { @@ -2244,7 +2441,11 @@ public static async Task StageAsync(string sourceRoot, string sta ? CopyEnumerated(sourceRoot, stagingRoot, files) : CopyRecursive(sourceRoot, stagingRoot); - return new StagingResult(stagingRoot, null, count); + var git = includeGitMetadata + ? StageGitMetadata(sourceRoot, stagingRoot) + : new GitStagingResult(false, 0, "Git metadata staging disabled; repository-root detection and version stamping will differ from the host."); + + return new StagingResult(stagingRoot, null, count, git.Staged, git.Bytes, git.Note); } catch (Exception ex) { @@ -2252,6 +2453,97 @@ public static async Task StageAsync(string sourceRoot, string sta } } + private sealed record GitStagingResult(bool Staged, long Bytes, string? Note); + + // Copy the repository's git directory verbatim into the staged workspace. Best-effort by design: a + // missing or unreadable .git is a fidelity note, never a reason to fail a run that can still execute. + private static GitStagingResult StageGitMetadata(string sourceRoot, string stagingRoot) + { + var gitDir = ResolveGitDirectory(sourceRoot); + if (gitDir is null) + { + return new GitStagingResult(false, 0, null); + } + + var destination = Path.Combine(stagingRoot, ".git"); + try + { + // A linked worktree stages its "gitdir:" pointer file as ordinary content; the real git + // directory has to replace it, because the path it points at does not exist in the container. + if (File.Exists(destination)) + { + File.Delete(destination); + } + + var bytes = CopyDirectory(gitDir, destination); + return new GitStagingResult(true, bytes, null); + } + catch (Exception ex) + { + return new GitStagingResult(false, 0, + $"Git metadata could not be staged ({ex.Message}); repository-root detection and version stamping may differ from the host."); + } + } + + // .git is a directory in an ordinary clone and a "gitdir: " pointer file in a linked worktree + // or submodule. Both resolve to a real directory that carries the repository state. + private static string? ResolveGitDirectory(string sourceRoot) + { + var candidate = Path.Combine(sourceRoot, ".git"); + if (Directory.Exists(candidate)) + { + return candidate; + } + + if (!File.Exists(candidate)) + { + return null; + } + + var pointer = File.ReadAllText(candidate).Trim(); + const string Prefix = "gitdir:"; + if (!pointer.StartsWith(Prefix, StringComparison.OrdinalIgnoreCase)) + { + return null; + } + + var target = pointer[Prefix.Length..].Trim(); + if (!Path.IsPathRooted(target)) + { + target = Path.GetFullPath(Path.Combine(sourceRoot, target)); + } + + return Directory.Exists(target) ? target : null; + } + + private static long CopyDirectory(string source, string destination) + { + long bytes = 0; + var stack = new Stack<(string Source, string Destination)>(); + stack.Push((source, destination)); + while (stack.Count > 0) + { + var (from, to) = stack.Pop(); + Directory.CreateDirectory(to); + foreach (var file in Directory.EnumerateFiles(from)) + { + var target = Path.Combine(to, Path.GetFileName(file)); + File.Copy(file, target, overwrite: true); + // The source .git may be read-only in places (packed objects); the staged copy is + // disposable and the container must be able to write to it. + new FileInfo(target).IsReadOnly = false; + bytes += new FileInfo(target).Length; + } + + foreach (var dir in Directory.EnumerateDirectories(from)) + { + stack.Push((dir, Path.Combine(to, Path.GetFileName(dir)))); + } + } + + return bytes; + } + private static async Task?> TryGitEnumerateAsync(string sourceRoot, CancellationToken ct) { if (!Directory.Exists(Path.Combine(sourceRoot, ".git"))) @@ -2831,7 +3123,7 @@ public static async Task RunAsync(Options options) stagingRoot = Path.Combine(runRoot, "workspace"); Directory.CreateDirectory(resultsRoot); - var staging = await SourceStager.StageAsync(sourceRoot, stagingRoot, cts.Token); + var staging = await SourceStager.StageAsync(sourceRoot, stagingRoot, cts.Token, includeGitMetadata: !options.NoGitMetadata); if (staging.Error is not null || staging.StagedPath is null) { return Error(options, FailureKind.SourceStaging, staging.Error ?? "Source staging produced no workspace."); @@ -2863,7 +3155,7 @@ public static async Task RunAsync(Options options) cancelled |= proc.TimedOut; - var results = TrxParser.ParseDirectory(resultsRoot); + var results = TrxParser.ParseDirectory(resultsRoot, AssemblyNameIndex.Build(sourceRoot)); var phaseMarkers = FailureClassifier.ParsePhaseMarkers(proc.StdOut); var outcome = FailureClassifier.Classify(phaseMarkers, proc.ExitCode, cancelled, results); @@ -2871,7 +3163,7 @@ public static async Task RunAsync(Options options) return EmitRunResult( options, env, image, tfmInfo, results, outcome, proc, cleanup, - ctx.Resolution.SelectionReason, wallClock.Elapsed.TotalSeconds); + ctx.Resolution.SelectionReason, wallClock.Elapsed.TotalSeconds, staging); } catch (OperationCanceledException) { @@ -2982,7 +3274,8 @@ private static int EmitRunResult( ProcessResult proc, CleanupReport cleanup, string? selectionReason = null, - double? elapsedSeconds = null) + double? elapsedSeconds = null, + StagingResult? staging = null) { var exit = FailureClassifier.ToExitCode(outcome.Kind); if (outcome.Kind == FailureKind.None && cleanup.Leftovers.Count > 0) @@ -3017,11 +3310,16 @@ private static int EmitRunResult( provisioning = image.ProvisionNote, }, targetFrameworks = tfmInfo.TargetFrameworks, + workspace = new { gitMetadataStaged = staging?.GitMetadataStaged, gitMetadataNote = staging?.GitMetadataNote }, tests = new { results.Total, results.Passed, results.Skipped, results.Failed, results.DurationSeconds, trxFiles = results.TrxFilesParsed }, + testAssemblies = results.Assemblies.Select(a => new { a.Assembly, a.Framework, a.Total, a.Passed, a.Failed, a.Skipped, a.DurationSeconds }), elapsedSeconds = elapsedSeconds is null ? (double?)null : Math.Round(elapsedSeconds.Value, 1), - failures = results.Failures.Select(f => new { f.TestName, f.ClassName, f.Message, f.StackTrace }), + failures = results.Failures.Select(f => new { f.TestName, f.ClassName, f.Assembly, f.Framework, f.DurationSeconds, f.Message, f.StackTrace, f.Output }), cleanup = new { cleanup.ContainerRemoved, cleanup.WorkspaceRemoved, cleanup.Leftovers }, - diagnostics = status == "error" ? new { containerExitCode = proc.ExitCode, stdoutTail = LastLines(proc.StdOut, 30), stderrTail = LastLines(proc.StdErr, 20) } : null, + diagnostics = status == "error" + ? new { containerExitCode = proc.ExitCode, phaseLogTail = LastLines(FailureClassifier.PhaseOutput(proc.StdOut, outcome.Phase), 40), stderrTail = LastLines(proc.StdErr, 20) } + : null, + containerLog = options.ShowLog ? proc.StdOut : null, }, RemoteTestProgram.JsonOut)); return (int)exit; } @@ -3050,46 +3348,67 @@ private static int EmitRunResult( Console.WriteLine($"Tools: {image.ProvisionNote}"); } + if (staging?.GitMetadataNote is not null) + { + Console.WriteLine($"Note: {staging.GitMetadataNote}"); + } + Console.WriteLine(); if (outcome.Kind is FailureKind.None or FailureKind.TestFailure) { + // Per assembly/TFM first — the same unit `dotnet test` reports on, so a failure is + // immediately attributable to one test project and one target framework. + var width = results.Assemblies.Count == 0 ? 0 : results.Assemblies.Max(a => a.Display.Length); + foreach (var a in results.Assemblies) + { + var verdict = a.Failed > 0 ? "Failed!" : "Passed!"; + Console.WriteLine( + $"{verdict,-8} {a.Display.PadRight(width)} — {a.Passed} passed, {a.Skipped} skipped, {a.Failed} failed, {a.DurationSeconds.ToString("0.0", CultureInfo.InvariantCulture)} s"); + } + + if (results.Assemblies.Count > 0) + { + Console.WriteLine(); + } + Console.WriteLine($"Tests: {results.Passed} passed, {results.Skipped} skipped, {results.Failed} failed"); Console.WriteLine($"Time: {results.DurationSeconds.ToString("0.0", CultureInfo.InvariantCulture)} s (tests)"); if (elapsedSeconds is not null) { Console.WriteLine($"Total: {elapsedSeconds.Value.ToString("0.0", CultureInfo.InvariantCulture)} s (including image pull, restore and build)"); } - if (results.Failed > 0) - { - Console.WriteLine(); - Console.WriteLine($"{results.Failed} test(s) failed:"); - foreach (var f in results.Failures) - { - Console.WriteLine(); - Console.WriteLine($" {f.ClassName}"); - Console.WriteLine($" {f.TestName}"); - if (f.Message is not null) - { - Console.WriteLine($" {f.Message}"); - } - } - } + + WriteFailureDetail(results); } else { Console.Error.WriteLine($"{outcome.Kind}: {outcome.Message}"); - // The container writes compiler/restore diagnostics to stdout, so a stderr-only tail would - // leave the developer with a verdict and no cause. Prefer the actual error lines. + + // Any TRX that was produced before the failure still names real failing tests; report those + // first so a test-host crash after a genuine assertion failure is not reduced to a log tail. + WriteFailureDetail(results); + + // The container writes compiler/restore/test diagnostics to stdout, so a stderr-only tail + // would leave the developer with a verdict and no cause. Prefer the failing phase's own log. + var phaseLog = FailureClassifier.PhaseOutput(proc.StdOut, outcome.Phase); var detail = FirstNonEmpty( - ErrorLines(proc.StdOut, 20), LastLines(proc.StdErr, 20), LastLines(proc.StdOut, 20)); + ErrorLines(phaseLog, 20), LastLines(phaseLog, 40), LastLines(proc.StdErr, 20), LastLines(proc.StdOut, 40)); if (!string.IsNullOrWhiteSpace(detail)) { Console.Error.WriteLine(); + Console.Error.WriteLine($"--- {outcome.Phase} output ---"); Console.Error.WriteLine(detail); } } + if (options.ShowLog && !string.IsNullOrWhiteSpace(proc.StdOut)) + { + Console.WriteLine(); + Console.WriteLine("--- container log ---"); + Console.WriteLine(proc.StdOut.TrimEnd()); + } + if (cleanup.Leftovers.Count > 0) { Console.Error.WriteLine("Cleanup left resources: " + string.Join(", ", cleanup.Leftovers)); @@ -3098,6 +3417,75 @@ private static int EmitRunResult( return (int)exit; } + // Caps so a suite that fails wholesale stays readable; the counts above remain authoritative and the + // full detail is always available in --json. + private const int MaxReportedFailures = 15; + private const int MaxStackFrames = 10; + private const int MaxOutputLines = 15; + + // The detail a developer actually needs to fix a red test: fully-qualified name, which TFM, the + // assertion message, the stack, and whatever the test wrote to its output helper. + private static void WriteFailureDetail(TestRunResult results) + { + if (results.Failures.Count == 0) + { + return; + } + + Console.WriteLine(); + Console.WriteLine(results.Failures.Count == 1 ? "1 test failed:" : $"{results.Failures.Count} tests failed:"); + + foreach (var f in results.Failures.Take(MaxReportedFailures)) + { + var name = f.ClassName is not null && !f.TestName.StartsWith(f.ClassName, StringComparison.Ordinal) + ? $"{f.ClassName}.{f.TestName}" + : f.TestName; + var where = f.Framework is null ? "" : $" [{f.Framework}]"; + var took = f.DurationSeconds > 0 ? $" ({(f.DurationSeconds * 1000).ToString("0", CultureInfo.InvariantCulture)} ms)" : ""; + + Console.WriteLine(); + Console.WriteLine($" Failed {name}{where}{took}"); + WriteIndented(f.Message, " ", int.MaxValue); + + if (!string.IsNullOrWhiteSpace(f.StackTrace)) + { + Console.WriteLine(" Stack trace:"); + WriteIndented(f.StackTrace, " ", MaxStackFrames); + } + + if (!string.IsNullOrWhiteSpace(f.Output)) + { + Console.WriteLine(" Output:"); + WriteIndented(f.Output, " ", MaxOutputLines); + } + } + + if (results.Failures.Count > MaxReportedFailures) + { + Console.WriteLine(); + Console.WriteLine($" … and {results.Failures.Count - MaxReportedFailures} more failing test(s); rerun with --json for the full list."); + } + } + + private static void WriteIndented(string? text, string indent, int maxLines) + { + if (string.IsNullOrWhiteSpace(text)) + { + return; + } + + var lines = text.Replace("\r\n", "\n", StringComparison.Ordinal).Split('\n'); + foreach (var line in lines.Take(maxLines)) + { + Console.WriteLine(indent + line.TrimEnd()); + } + + if (lines.Length > maxLines) + { + Console.WriteLine($"{indent}… {lines.Length - maxLines} more line(s)"); + } + } + private static int Error(Options options, FailureKind kind, string message) { if (options.Json) @@ -3179,6 +3567,7 @@ public static int Run() TargetFrameworkTests(); CommandPlanningTests(); ImagePreparationTests(); + SourceStagingTests(); ResultParsingTests(); FailureClassificationTests(); CancellationAndCleanupTests(); @@ -3616,6 +4005,68 @@ private static void ImagePreparationTests() Check("provisioning fails loudly on an unknown package manager", dockerfile.Contains("No supported package manager")); } + private static void SourceStagingTests() + { + Section("Source staging"); + + var root = Path.Combine(Path.GetTempPath(), "rt-stage-" + Guid.NewGuid().ToString("N")[..8]); + var source = Path.Combine(root, "repo"); + try + { + Directory.CreateDirectory(Path.Combine(source, "src")); + Directory.CreateDirectory(Path.Combine(source, "bin")); + Directory.CreateDirectory(Path.Combine(source, ".git", "refs")); + File.WriteAllText(Path.Combine(source, "src", "App.csproj"), ""); + File.WriteAllText(Path.Combine(source, "bin", "stale.dll"), "x"); + File.WriteAllText(Path.Combine(source, ".git", "HEAD"), "ref: refs/heads/main"); + + var staged = Path.Combine(root, "staged"); + var result = SourceStager.StageAsync(source, staged, CancellationToken.None).GetAwaiter().GetResult(); + + Check("staging succeeds", result.Error is null && result.StagedPath == staged); + Check("sources are staged", File.Exists(Path.Combine(staged, "src", "App.csproj"))); + + var index = AssemblyNameIndex.Build(source); + Check("project files provide the authoritative assembly casing", index["app.dll"] == "App.dll"); + Check("host build output is not staged", !Directory.Exists(Path.Combine(staged, "bin"))); + + // Without .git the staged workspace stops being a repository: MinVer/Nerdbank fall back to + // 0.0.0, SourceLink stops embedding, and any "walk up to the .git directory" repository-root + // probe resolves elsewhere — which silently changes what the tests under it observe. + Check("git metadata is staged", result.GitMetadataStaged && Directory.Exists(Path.Combine(staged, ".git"))); + Check("git metadata is staged verbatim", + File.ReadAllText(Path.Combine(staged, ".git", "HEAD")) == "ref: refs/heads/main" + && Directory.Exists(Path.Combine(staged, ".git", "refs"))); + Check("git metadata size is reported", result.GitMetadataBytes > 0); + + var without = Path.Combine(root, "staged-no-git"); + var opted = SourceStager.StageAsync(source, without, CancellationToken.None, includeGitMetadata: false).GetAwaiter().GetResult(); + Check("git metadata can be opted out", !opted.GitMetadataStaged && !Directory.Exists(Path.Combine(without, ".git"))); + Check("opting out is explained, not silent", opted.GitMetadataNote is not null); + + // A linked worktree or submodule stores .git as a "gitdir:" pointer file, not a directory. + var linked = Path.Combine(root, "linked"); + Directory.CreateDirectory(linked); + File.WriteAllText(Path.Combine(linked, "a.txt"), "a"); + File.WriteAllText(Path.Combine(linked, ".git"), $"gitdir: {Path.Combine(source, ".git")}"); + var linkedStaged = Path.Combine(root, "staged-linked"); + var linkedResult = SourceStager.StageAsync(linked, linkedStaged, CancellationToken.None).GetAwaiter().GetResult(); + Check("gitdir pointer file is resolved to the real git directory", + linkedResult.GitMetadataStaged && File.Exists(Path.Combine(linkedStaged, ".git", "HEAD"))); + + // A repository with no git metadata at all is ordinary, not an error. + var plain = Path.Combine(root, "plain"); + Directory.CreateDirectory(plain); + File.WriteAllText(Path.Combine(plain, "a.txt"), "a"); + var plainResult = SourceStager.StageAsync(plain, Path.Combine(root, "staged-plain"), CancellationToken.None).GetAwaiter().GetResult(); + Check("a non-git source stages without a note", plainResult.Error is null && !plainResult.GitMetadataStaged && plainResult.GitMetadataNote is null); + } + finally + { + try { Directory.Delete(root, recursive: true); } catch (Exception) { /* temp cleanup is best-effort */ } + } + } + private static void ResultParsingTests() { Section("Result parsing"); @@ -3626,12 +4077,12 @@ private static void ResultParsingTests() - Expected: foo Actual: barat StringUtilityTest.cs:line 142 + probing /workspace/tuningExpected: foo Actual: barat StringUtilityTest.cs:line 142 - + @@ -3643,9 +4094,41 @@ private static void ResultParsingTests() Check("failure detail captured", result.Failures.Count == 1 && result.Failures[0].TestName == "Sanitize_WithUnicode_ReturnsExpectedValue"); Check("failure class resolved from TestDefinitions", result.Failures[0].ClassName == "Cuemon.Text.Tests.StringUtilityTest"); Check("failure message captured", result.Failures[0].Message == "Expected: foo Actual: bar"); + Check("failure stack trace captured", result.Failures[0].StackTrace == "at StringUtilityTest.cs:line 142"); + Check("test-written output captured", result.Failures[0].Output == "probing /workspace/tuning"); + Check("failure carries its assembly and framework", + result.Failures[0].Assembly == "Cuemon.Text.Tests.dll" && result.Failures[0].Framework == "net10.0"); + + Check("per-assembly summary produced", + result.Assemblies.Count == 1 && result.Assemblies[0] is { Assembly: "Cuemon.Text.Tests.dll", Framework: "net10.0", Failed: 1 }); + Check("assembly summary displays framework", result.Assemblies[0].Display == "Cuemon.Text.Tests.dll (net10.0)"); + + Check("framework derived from the output path", + TrxParser.FrameworkFrom("/w/bin/Debug/net9.0/A.dll") == "net9.0" + && TrxParser.FrameworkFrom(@"C:\w\bin\Release\net10.0-windows\A.dll") == "net10.0-windows" + && TrxParser.FrameworkFrom("/w/bin/Debug/netstandard2.0/A.dll") == "netstandard2.0"); + Check("framework absent when the path has none", TrxParser.FrameworkFrom("/w/A.dll") is null); + Check("assembly name derived from the storage path", TrxParser.AssemblyNameFrom(@"C:\w\bin\Debug\net10.0\A.Tests.dll") == "A.Tests.dll"); + + // VSTest lower-cases the storage path; the developer knows the assembly by its real casing. + var known = new Dictionary(StringComparer.OrdinalIgnoreCase) { ["Acme.Tests.dll"] = "Acme.Tests.dll" }; + Check("assembly casing recovered from the repository's project files", + TrxParser.AssemblyDisplayName("/w/bin/debug/net10.0/acme.tests.dll", null, known) == "Acme.Tests.dll"); + Check("an unknown assembly keeps the name the TRX recorded", + TrxParser.AssemblyDisplayName("/w/bin/debug/net10.0/other.tests.dll", null, known) == "other.tests.dll"); + Check("assembly casing recovered from the class name", + TrxParser.AssemblyDisplayName("/w/bin/debug/net10.0/acme.tests.dll", "Acme.Tests") == "Acme.Tests.dll"); + Check("a qualified display name is reduced to its simple name", + TrxParser.AssemblyDisplayName("/w/bin/debug/net10.0/acme.tests.dll", "Acme.Tests, Version=1.0.0.0, Culture=neutral") == "Acme.Tests.dll"); + Check("a class name from another assembly never overrides storage", + TrxParser.AssemblyDisplayName("/w/bin/Debug/net10.0/Acme.Tests.dll", "Shared.Fixtures") == "Acme.Tests.dll"); + Check("class name alone still names the assembly", + TrxParser.AssemblyDisplayName(null, "Acme.Tests") == "Acme.Tests.dll"); + Check("neither source yields no assembly name", TrxParser.AssemblyDisplayName(null, null) is null); var merged = result.Merge(TrxParser.Parse(trx)); Check("multiple trx files aggregate", merged is { Total: 6, Failed: 2, TrxFilesParsed: 2 }); + Check("assembly summaries aggregate too", merged.Assemblies.Count == 2); } private static void FailureClassificationTests() @@ -3673,6 +4156,15 @@ private static void FailureClassificationTests() var markers = FailureClassifier.ParsePhaseMarkers("noise\n##RT_PHASE_END:restore:0##\nmore\n##RT_PHASE_END:build:2##\n"); Check("phase markers parsed from output", markers["restore"] == 0 && markers["build"] == 2); + + // Reporting the failing phase's own log — not a tail of everything — is what makes a build + // failure name the offending file instead of trailing test-runner chatter. + const string log = "restoring\n##RT_PHASE_END:restore:0##\nApp.cs(3,5): error CS1002: ; expected\n##RT_PHASE_END:build:1##\ntest chatter\n"; + Check("phase log isolates restore", FailureClassifier.PhaseOutput(log, "restore") == "restoring"); + Check("phase log isolates build", FailureClassifier.PhaseOutput(log, "build") == "App.cs(3,5): error CS1002: ; expected"); + Check("an incomplete phase yields everything after the last marker", + FailureClassifier.PhaseOutput(log, "test") == "test chatter"); + Check("phase log is empty when there is no output", FailureClassifier.PhaseOutput("", "build") == ""); } private static void CancellationAndCleanupTests() diff --git a/skills/dotnet-remote-testing/scripts/validate-skill.ps1 b/skills/dotnet-remote-testing/scripts/validate-skill.ps1 index b0ef7f4..e51fdda 100644 --- a/skills/dotnet-remote-testing/scripts/validate-skill.ps1 +++ b/skills/dotnet-remote-testing/scripts/validate-skill.ps1 @@ -38,7 +38,13 @@ $contracts = @( # A Microsoft SDK image carries one runtime, so multi-targeted repositories need a multi-SDK runner. # Losing this guidance means silently planning runs that build and then cannot execute. 'codebeltnet/ubuntu-testrunner', - 'ships exactly **one** runtime' + 'ships exactly **one** runtime', + # A staged workspace without .git stops being a repository: version stamping falls back to 0.0.0 and + # repository-root probes resolve elsewhere, which makes a suite fail here that passes in Visual + # Studio's remote testing. Losing this guidance sends the agent chasing the repository instead. + 'The staged workspace is still a repository', + # Reporting contract: a red test must be actionable from the report alone. + 'Do not compress this into a bare count' ) foreach ($needle in $contracts) { if (-not $skill.Contains($needle, [System.StringComparison]::Ordinal)) { From f133ba74ada7c7fa676b4ba231704a5206bda5de Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 01:22:35 +0200 Subject: [PATCH 12/31] =?UTF-8?q?=F0=9F=92=AC=20update=20changelog=20for?= =?UTF-8?q?=20v0.9.0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document git metadata staging and per-assembly result reporting enhancements for dotnet-remote-testing. --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 08bda18..c674038 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,8 @@ This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-r - `dotnet-remote-testing` deterministic runner `remote-test.cs` owning configuration discovery, release parsing, digest-pinned image resolution, isolated source staging, NuGet caching, execution, result parsing, distinct failure classification, and cleanup behind a single entry point, with a built-in `--self-test` alongside a PowerShell harness, - Offline-safe release discovery for `dotnet-remote-testing`: successful release metadata is cached outside the repository so later runs work without network access, and the parameter form surfaces the exact runner-computed target as the recommended option, - Build-tooling preparation in `dotnet-remote-testing`, probing the resolved image for the `git` that MinVer, Nerdbank.GitVersioning, GitInfo, and SourceLink invoke during `dotnet build`, and layering it on through a digest-addressed image cached outside the repository when the image lacks it, so a minimal runner image no longer fails a sound build with `MINVER1007` while the reported image and digest stay the resolved base, +- Git metadata in the staged workspace for `dotnet-remote-testing`, copying the repository's `.git` directory into the disposable staged copy and resolving a linked worktree's `gitdir:` pointer to the real directory, so version stamping, SourceLink, and any repository-root probe that walks up to a `.git` directory behave as they do on the host instead of silently resolving elsewhere and changing what the tests observe, with `--no-git-metadata` as an explicit, reported opt-out for a repository whose history dominates staging cost, +- `dotnet test`-shaped result reporting in `dotnet-remote-testing`, breaking results down per test assembly and target framework and reporting each failure with its fully-qualified name, target framework, elapsed time, assertion message, stack trace, and test-written output, reporting an infrastructure failure with its own phase's log rather than a tail of the whole run, and adding `--show-log` for the complete container log, - `dotnet-segregated-assets` skill that migrates an ASP.NET Core application to serve deployed static content from Codebelt Static Content Provider (`codebeltnet/web-cdn-origin:2.0.0`) while `wwwroot` remains the authoring root, separating app-owned assets from shared CDN assets and preserving Razor Class Library, framework, and generated Static Web Assets, - `dotnet-segregated-assets` deterministic runner `segregate-assets.cs` that inspects static-asset topology, classifies existing segregation state, escalates Blazor, Razor Class Library, scoped-CSS, and frontend-build risk instead of blindly excluding it, resolves Cuemon TagHelper package versions from the NuGet V3 service index at plan time, reports cache-busting interfaces and registrations without rewriting Razor or C# source, and proves the publish invariant through `verify --run-publish` against an isolated temp directory, - Artifact-first container contract for `dotnet-segregated-assets` in which both application Dockerfiles package an already-published `artifacts/publish/` directory rather than compiling source, with the validator rejecting an SDK stage, a `dotnet build` or `dotnet publish` step, an `mcr.microsoft.com` runtime, or a missing artifact copy, From 1352427a95e0f57cb1b2674f5bd549293ff7e845 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 12:47:36 +0200 Subject: [PATCH 13/31] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20refactor=20sync-skil?= =?UTF-8?q?l-install=20for=20host-tool=20distinction?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Separate host-tool root detection from skill installation path validation. This prevents false positive drift reports when a tool host exists but contains no skill installation yet; previously, both states were treated identically as 'not installed'. The refactor also creates missing skill directories on sync and correctly reports them as drift during verify-only mode. --- scripts/sync-skill-install.ps1 | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/scripts/sync-skill-install.ps1 b/scripts/sync-skill-install.ps1 index 38ef4ba..70d6262 100644 --- a/scripts/sync-skill-install.ps1 +++ b/scripts/sync-skill-install.ps1 @@ -21,14 +21,16 @@ function Get-RepoRoot { return (Resolve-Path (Join-Path $PSScriptRoot '..')).Path } +# HostRoot is the tool's own directory. Its absence means the tool is not installed on this machine and +# there is genuinely nothing to sync; a missing skill directory *under* an installed tool is drift. function Get-InstallRoot { param([string]$SkillName) $home_ = [Environment]::GetFolderPath('UserProfile') return @( - (Join-Path $home_ ".claude/skills/$SkillName"), - (Join-Path $home_ ".agents/skills/$SkillName"), - (Join-Path $home_ ".gemini/antigravity-cli/skills/$SkillName") + [pscustomobject]@{ HostRoot = (Join-Path $home_ '.claude'); Path = (Join-Path $home_ ".claude/skills/$SkillName") }, + [pscustomobject]@{ HostRoot = (Join-Path $home_ '.agents'); Path = (Join-Path $home_ ".agents/skills/$SkillName") }, + [pscustomobject]@{ HostRoot = (Join-Path $home_ '.gemini/antigravity-cli'); Path = (Join-Path $home_ ".gemini/antigravity-cli/skills/$SkillName") } ) } @@ -121,12 +123,26 @@ foreach ($name in $Skill) { $relativeFile = Get-RelativeFile -Root $sourceRoot - foreach ($installRoot in (Get-InstallRoot -SkillName $name)) { - if (-not (Test-Path -LiteralPath $installRoot)) { - Write-Host "[SKIP] $name -> $installRoot (not installed)" + foreach ($install in (Get-InstallRoot -SkillName $name)) { + $installRoot = $install.Path + + if (-not (Test-Path -LiteralPath $install.HostRoot)) { + Write-Host "[SKIP] $name -> $installRoot (host tool not installed)" continue } + # An installed host with no copy of the skill used to be skipped, which let a run where nothing + # was ever installed still report "verified, 0 drift". A sync creates the install; a verify fails. + if (-not (Test-Path -LiteralPath $installRoot)) { + if ($VerifyOnly) { + Write-Host "[FAIL] $name -> $installRoot (not installed)" + $totalDrift += 1 + continue + } + + New-Item -ItemType Directory -Force -Path $installRoot | Out-Null + } + if (-not $VerifyOnly) { Sync-SkillTree -SourceRoot $sourceRoot -InstallRoot $installRoot -RelativeFile $relativeFile } From db92094d209d5369ffbbc47923563b0b35bfc0d5 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 15 Aug 2026 12:47:49 +0200 Subject: [PATCH 14/31] =?UTF-8?q?=F0=9F=90=9B=20preserve=20configured=20us?= =?UTF-8?q?er=20in=20docker=20prepared=20images?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add ResolveUserAsync to extract the configured USER from a docker image, then restore it in the prepared-image Dockerfile after package installation. This ensures a base image configured to run as a non-root user continues to do so in the prepared image, preserving file ownership and permission-sensitive test behavior. Update docker-execution.md to explain the identity-preserving guarantee and the linked-worktree git-directory handling. --- .../references/docker-execution.md | 4 +- .../scripts/remote-test.cs | 149 +++++++++++++++++- 2 files changed, 146 insertions(+), 7 deletions(-) diff --git a/skills/dotnet-remote-testing/references/docker-execution.md b/skills/dotnet-remote-testing/references/docker-execution.md index 173db59..c936c2b 100644 --- a/skills/dotnet-remote-testing/references/docker-execution.md +++ b/skills/dotnet-remote-testing/references/docker-execution.md @@ -34,6 +34,7 @@ Microsoft's SDK images ship `git`; a minimal runner image may not. So after the FROM USER root RUN … install git with whichever package manager the base image ships (apt-get / apk / microdnf / dnf / yum) +USER ``` This preparation layer is: @@ -41,6 +42,7 @@ This preparation layer is: - **outside the repository** — the Dockerfile is written into the run's own temporary directory, never into the working tree, so the "never generate container plumbing" rule is intact; - **content-addressed and cached** — tagged `dotnet-remote-testing/prepared:git-`, so it is built once per base image and reused by every later run (`Reused prepared image providing git.`); - **transparent** — the reported `Image`/`Digest` remain the resolved base image (reproducibility identity), with the preparation reported separately; +- **identity-preserving** — installing packages needs root, but the image's own `USER` is restored afterwards, so a base image that runs as a non-root user keeps doing so and file ownership and permission-sensitive tests behave as they do in the configured image; - **best effort** — if the tooling cannot be added (no package manager, `--offline`, no network), the base image is used anyway and the reason is reported, because a repository that never invokes `git` runs fine without it. ## Git metadata in the workspace @@ -57,7 +59,7 @@ This matters because staging the source without its git metadata is not a neutra The third row is the subtle one: a suite that passes under Visual Studio's Remote Testing and fails here, for reasons unrelated to the container, is very often a repository-root probe landing somewhere else. Installing `git` into the image (above) and staging `.git` are two halves of one guarantee: the build tooling is present *and* the metadata it reads is present. -A linked worktree or submodule stores `.git` as a `gitdir: ` pointer file; the runner resolves the pointer and stages the real directory, because the path it names does not exist inside the container. A source tree with no git metadata at all stages normally and silently — that is an ordinary case, not a degraded one. +A linked worktree or submodule stores `.git` as a `gitdir: ` pointer file; the runner resolves the pointer and stages the real directory, because the path it names does not exist inside the container. A linked worktree's git directory is only half a repository — it holds per-worktree state (`HEAD`, `index`, logs) and points at a shared `commondir` that holds the objects, refs and config — so the runner stages the shared half first, layers the per-worktree files over it, and drops the `commondir`/`gitdir` pointers. The staged copy is then an ordinary standalone repository, which is what git-based versioning and SourceLink need; staging the near half alone would leave a git directory git cannot read, and versioning would fall back to `0.0.0` exactly as if `.git` had been skipped. A source tree with no git metadata at all stages normally and silently — that is an ordinary case, not a degraded one. `--no-git-metadata` opts out for a repository whose `.git` is large enough that copying it dominates the run. The run then reports the fidelity loss rather than hiding it. diff --git a/skills/dotnet-remote-testing/scripts/remote-test.cs b/skills/dotnet-remote-testing/scripts/remote-test.cs index 4a22dba..b456ff1 100644 --- a/skills/dotnet-remote-testing/scripts/remote-test.cs +++ b/skills/dotnet-remote-testing/scripts/remote-test.cs @@ -2268,6 +2268,19 @@ public static Task PullAsync(string image, CancellationToken ct) return r.ExitCode == 0 ? r.StdOut.Trim() : null; } + // The user an image is configured to run as. Empty means the image sets none, i.e. root. + public static async Task ResolveUserAsync(string image, CancellationToken ct) + { + var r = await ProcessRunner.RunAsync("docker", ["inspect", "--format", "{{.Config.User}}", image], null, ct); + if (r.ExitCode != 0) + { + return null; + } + + var user = r.StdOut.Trim(); + return user.Length == 0 ? null : user; + } + public static Task BuildAsync(string dockerfile, string context, string tag, CancellationToken ct) => ProcessRunner.RunAsync("docker", ["build", "-f", dockerfile, "-t", tag, context], null, ct); @@ -2325,9 +2338,15 @@ public static string ProbeCommand() => // A single RUN that adapts to whichever package manager the base image ships. The Dockerfile is // written to the run's own temporary directory — never into the repository being tested. - public static string Dockerfile(string baseReference) + // + // Installing packages needs root, but the identity the tests run under is part of the environment + // being reproduced: a base image that runs as a non-root user must keep doing so, or the prepared + // image writes build and test output with different ownership than the configured image would. + // baseUser is that image's configured user, or null when it sets none (already root). + public static string Dockerfile(string baseReference, string? baseUser = null) { var tools = string.Join(' ', RequiredTools); + var restore = string.IsNullOrWhiteSpace(baseUser) ? string.Empty : $"USER {baseUser.Trim()}\n"; return $""" FROM {baseReference} USER root @@ -2345,7 +2364,7 @@ USER root else \ echo 'No supported package manager in the base image.' >&2; exit 1; \ fi - + {restore} """; } @@ -2375,10 +2394,14 @@ public static async Task EnsureAsync( + "A build that invokes it will fail."); } + // Read the identity off the base image before deriving from it, so the prepared image keeps + // running as whoever the configured image runs as instead of silently switching to root. + var baseUser = await DockerClient.ResolveUserAsync(baseReference, ct); + var contextDir = Path.Combine(workRoot, "image-prep"); Directory.CreateDirectory(contextDir); var dockerfile = Path.Combine(contextDir, "Dockerfile"); - await File.WriteAllTextAsync(dockerfile, Dockerfile(baseReference), ct); + await File.WriteAllTextAsync(dockerfile, Dockerfile(baseReference, baseUser), ct); var build = await DockerClient.BuildAsync(dockerfile, contextDir, tag, ct); if (build.ExitCode != 0) @@ -2475,7 +2498,37 @@ private static GitStagingResult StageGitMetadata(string sourceRoot, string stagi File.Delete(destination); } + // A linked worktree's git directory holds only per-worktree state (HEAD, index, logs). The + // objects, refs and config live in the shared directory its "commondir" points at, outside + // the staged copy. Staging the worktree half alone produces a git directory git cannot read, + // so versioning falls back to 0.0.0 and SourceLink stops embedding — the exact fidelity loss + // staging .git exists to prevent. The shared half is copied first and the per-worktree files + // are layered over it, which collapses the pair into an ordinary standalone repository. + var commonDir = ResolveCommonDirectory(gitDir); + if (commonDir is not null) + { + // "worktrees/" only registers linked worktrees by host path; none of them exist in the + // container, and this worktree's own entry is exactly what is being flattened here. + CopyDirectory(commonDir, destination, excludeTopLevelDirectory: "worktrees"); + } + var bytes = CopyDirectory(gitDir, destination); + if (commonDir is not null) + { + // The staged repository is standalone now; leaving the pointers behind would send git + // back out to host paths that do not exist in the container. + foreach (var pointer in new[] { "commondir", "gitdir" }) + { + var stale = Path.Combine(destination, pointer); + if (File.Exists(stale)) + { + File.Delete(stale); + } + } + + bytes = MeasureDirectory(destination); + } + return new GitStagingResult(true, bytes, null); } catch (Exception ex) @@ -2516,7 +2569,34 @@ private static GitStagingResult StageGitMetadata(string sourceRoot, string stagi return Directory.Exists(target) ? target : null; } - private static long CopyDirectory(string source, string destination) + // A linked worktree's git directory carries a "commondir" file naming the shared repository + // directory that actually holds objects, refs and config. An ordinary clone has no such file. + private static string? ResolveCommonDirectory(string gitDir) + { + var marker = Path.Combine(gitDir, "commondir"); + if (!File.Exists(marker)) + { + return null; + } + + var target = File.ReadAllText(marker).Trim(); + if (target.Length == 0) + { + return null; + } + + if (!Path.IsPathRooted(target)) + { + target = Path.GetFullPath(Path.Combine(gitDir, target)); + } + + return Directory.Exists(target) ? target : null; + } + + private static long MeasureDirectory(string root) => + Directory.EnumerateFiles(root, "*", SearchOption.AllDirectories).Sum(f => new FileInfo(f).Length); + + private static long CopyDirectory(string source, string destination, string? excludeTopLevelDirectory = null) { long bytes = 0; var stack = new Stack<(string Source, string Destination)>(); @@ -2537,7 +2617,15 @@ private static long CopyDirectory(string source, string destination) foreach (var dir in Directory.EnumerateDirectories(from)) { - stack.Push((dir, Path.Combine(to, Path.GetFileName(dir)))); + var name = Path.GetFileName(dir); + if (excludeTopLevelDirectory is not null + && string.Equals(from, source, StringComparison.Ordinal) + && string.Equals(name, excludeTopLevelDirectory, StringComparison.OrdinalIgnoreCase)) + { + continue; + } + + stack.Push((dir, Path.Combine(to, name))); } } @@ -2546,7 +2634,10 @@ private static long CopyDirectory(string source, string destination) private static async Task?> TryGitEnumerateAsync(string sourceRoot, CancellationToken ct) { - if (!Directory.Exists(Path.Combine(sourceRoot, ".git"))) + // .git is a directory in an ordinary clone and a "gitdir:" pointer file in a linked worktree; + // git enumerates both, and skipping the pointer form would stage ignored build output. + var marker = Path.Combine(sourceRoot, ".git"); + if (!Directory.Exists(marker) && !File.Exists(marker)) { return null; } @@ -4003,6 +4094,15 @@ private static void ImagePreparationTests() Check("provisioning adapts to the image's package manager", dockerfile.Contains("command -v apt-get") && dockerfile.Contains("command -v apk") && dockerfile.Contains("command -v microdnf")); Check("provisioning fails loudly on an unknown package manager", dockerfile.Contains("No supported package manager")); + + // Installing needs root, but the prepared image must still run as whoever the base image runs + // as; switching the image to root changes file ownership and permission-sensitive results. + Check("provisioning does not leave a root-only image as root", !dockerfile.TrimEnd().EndsWith("USER root", StringComparison.Ordinal)); + var nonRoot = ImageProvisioner.Dockerfile("acme/runner:1", "app"); + Check("provisioning restores the base image's user", nonRoot.TrimEnd().EndsWith("USER app", StringComparison.Ordinal)); + Check("provisioning still installs as root", nonRoot.Contains("USER root", StringComparison.Ordinal)); + Check("provisioning adds no user line when the base image sets none", + !ImageProvisioner.Dockerfile("acme/runner:1", " ").Contains("USER app", StringComparison.Ordinal)); } private static void SourceStagingTests() @@ -4054,6 +4154,43 @@ private static void SourceStagingTests() Check("gitdir pointer file is resolved to the real git directory", linkedResult.GitMetadataStaged && File.Exists(Path.Combine(linkedStaged, ".git", "HEAD"))); + // A real linked worktree splits its git directory in two: per-worktree state here, objects + // and refs in the shared "commondir". Staging only the near half leaves a git directory git + // cannot read, so MinVer/Nerdbank fall back to 0.0.0 and SourceLink stops embedding. + var common = Path.Combine(root, "main", ".git"); + var worktreeGit = Path.Combine(common, "worktrees", "wt"); + Directory.CreateDirectory(Path.Combine(common, "objects", "pack")); + Directory.CreateDirectory(Path.Combine(common, "refs", "heads")); + Directory.CreateDirectory(worktreeGit); + File.WriteAllText(Path.Combine(common, "HEAD"), "ref: refs/heads/main"); + File.WriteAllText(Path.Combine(common, "config"), "[core]\n\tbare = false"); + File.WriteAllText(Path.Combine(common, "objects", "pack", "pack-1.pack"), "objects"); + File.WriteAllText(Path.Combine(common, "refs", "heads", "main"), "0123456789abcdef"); + File.WriteAllText(Path.Combine(worktreeGit, "HEAD"), "ref: refs/heads/feature"); + File.WriteAllText(Path.Combine(worktreeGit, "commondir"), "../.."); + File.WriteAllText(Path.Combine(worktreeGit, "gitdir"), Path.Combine(root, "wt", ".git")); + + var worktree = Path.Combine(root, "wt"); + Directory.CreateDirectory(worktree); + File.WriteAllText(Path.Combine(worktree, "a.txt"), "a"); + File.WriteAllText(Path.Combine(worktree, ".git"), $"gitdir: {worktreeGit}"); + + var worktreeStaged = Path.Combine(root, "staged-worktree"); + var worktreeResult = SourceStager.StageAsync(worktree, worktreeStaged, CancellationToken.None).GetAwaiter().GetResult(); + var stagedGit = Path.Combine(worktreeStaged, ".git"); + + Check("worktree staging carries the shared objects and refs", + worktreeResult.GitMetadataStaged + && File.Exists(Path.Combine(stagedGit, "objects", "pack", "pack-1.pack")) + && File.Exists(Path.Combine(stagedGit, "refs", "heads", "main")) + && File.Exists(Path.Combine(stagedGit, "config"))); + Check("worktree staging keeps the worktree's own HEAD", + File.ReadAllText(Path.Combine(stagedGit, "HEAD")) == "ref: refs/heads/feature"); + Check("worktree staging drops pointers to host paths", + !File.Exists(Path.Combine(stagedGit, "commondir")) && !File.Exists(Path.Combine(stagedGit, "gitdir"))); + Check("worktree staging does not register host worktrees", !Directory.Exists(Path.Combine(stagedGit, "worktrees"))); + Check("worktree staging reports the merged size", worktreeResult.GitMetadataBytes > 0); + // A repository with no git metadata at all is ordinary, not an error. var plain = Path.Combine(root, "plain"); Directory.CreateDirectory(plain); From e3e2bdcdb51d94c1f24beaf78ceca6b7d494be39 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sun, 16 Aug 2026 15:19:10 +0200 Subject: [PATCH 15/31] =?UTF-8?q?=F0=9F=93=9A=20expand=20git-visual-squash?= =?UTF-8?q?-summary=20catalog=20entry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expands the skill's README entry with additional constraints around immediate action and clarifies that the skill does not ask permission before proceeding. --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 154f582..073731a 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith | [git-keep-a-changelog](skills/git-keep-a-changelog/SKILL.md) | Git-aware Keep a Changelog companion selected only for explicit changelog or release-note intent. Bare yolo/auto and commit-execution requests such as `git bot commit yolo` do not activate it; those words modify autonomy only after changelog intent is established. Bundled deterministic resolvers separate branch-unique commit history from merge-base-to-`HEAD` net diffs, exclude the previous-release or comparison boundary, fail on base-history bleed, and classify explicit path-backed release entities as `Added`, `Removed`, `Changed`, or `Unchanged`. The skill establishes each user-facing release entity against the base before section classification. It asks a mandatory `Yes / No / Custom` question before including pending worktree changes in ordinary concrete-release drafts, includes staged, unstaged, and untracked work automatically only in scoped yolo/auto mode, creates missing changelogs, writes SemVer-aware highlights, maintains compare-link footers, preserves natural prose wrapping, and curates surviving outcomes instead of dumping raw commit logs. | | [git-nuget-release-notes](skills/git-nuget-release-notes/SKILL.md) | Git-aware NuGet release-notes companion for .NET repos that keep cumulative `.nuget/{ProjectName}/PackageReleaseNotes.txt` files. Discovers packable `src/` projects, resolves concrete package version and availability, creates missing files when needed, reduces each package to its surviving base-to-`HEAD` delta before classifying history, and establishes each package capability against the base so pre-release refinements and fixes to a new capability remain one `ADDED` New Feature. It writes per-package `ALM` / `Breaking Changes` / `New Features` / `Improvements` / `Bug Fixes` style notes from final package state plus supporting commit context instead of dumping commit subjects. | | [git-nuget-readme](skills/git-nuget-readme/SKILL.md) | Git-aware NuGet README companion for .NET repos that advertise a package from `src/`. Resolves the real packable project the README should sell, combines git history with actual package metadata, source capabilities, and relevant tests when feasible, preserves honest badge/docs/contributing sections, and writes a forthcoming, adoption-friendly `README.md` with repo-derived branding, clear value, install, framework-support, and quick-start guidance. | -| [git-visual-squash-summary](skills/git-visual-squash-summary/SKILL.md) | Non-mutating grouped-summary companion to `git-visual-commits`. Turns the full current feature branch into a curated set of compact lowercase-start summary lines for PR or squash-and-merge contexts by default, comparing against the repository base branch rather than a same-named tracking remote, including commits from all authors unless explicitly narrowed, reducing the cumulative base-to-`HEAD` delta first so reverted churn disappears, preserving technical identifiers, merging overlap, keeping surviving dependency/version changes separate from build/refactor work when the final diff still shows them, and avoiding changelog-style wording, unsupported claims, yolo prompts, needless commit-range questions, or commit-selection UI for ordinary branch-level squash requests. | +| [git-visual-squash-summary](skills/git-visual-squash-summary/SKILL.md) | Non-mutating grouped-summary companion to `git-visual-commits`. Turns the full current feature branch into a curated set of compact lowercase-start summary lines for PR or squash-and-merge contexts by default, comparing against the repository base branch rather than a same-named tracking remote, including commits from all authors unless explicitly narrowed, reducing the cumulative base-to-`HEAD` delta first so reverted churn disappears, preserving technical identifiers, merging overlap, keeping surviving dependency/version changes separate from build/refactor work when the final diff still shows them, and avoiding changelog-style wording, unsupported claims, yolo prompts, needless commit-range questions, commit-selection UI for ordinary branch-level squash requests, or an instruction recap that asks permission before starting. | | [skill-creator-agnostic](skills/skill-creator-agnostic/SKILL.md) | **⚠️ Deprecated** — no longer maintained and retained only for backward compatibility until **1.0.0**. Do not use it for new skill-authoring work; use Anthropic `skill-creator` together with this repository's `AGENTS.md`. | | [markdown-illustrator](skills/markdown-illustrator/SKILL.md) | Reads a markdown file and answers directly in chat with one document-wide Visual Brief plus one compiled prompt. Infers a compact visual strategy by default, keeps follow-up questions near zero, and only branches when the user explicitly asks for added specificity. | | [git-repo-digest](skills/git-repo-digest/SKILL.md) | Turns any full repository URL into a deterministic digest workspace using the bundled .NET file-based runner `scripts/digest.cs`. Requires explicit `--repo-url`, resolves omitted output paths to `/.bot/digests` and passes that as `--output-root`, maps multiple positional URLs the same way for slash commands, bare pasted URLs, and natural-language requests by treating the first URL as the digest repo and every later URL as repeated `--external-repo-url`, always writes into `{output-root}/{repo-id}/{yyyyMMdd-HHmmssZ}`, accepts repeated curated public consumer repos, derives `{repo-id}`, fixes `result/`, performs shallow git clones, packs local tracked files with the bundled C# packer using `git ls-files`, separates XML evidence into `source.xml`, `tests.xml`, `projects.xml`, editorial `readmes.xml`, and scenario-only `external-usage.xml`, writes package and conceptual overview prompts under `prompts/`, emits public API summaries, engineering signals, evidence indexes, ordered XML chunks, referenced-package evidence maps for aggregate examples, and manifest-backed frontmatter hints, treats previous digest prose as contamination during fresh generation, then guides the agent to fully read the current phase's required evidence before writing package digests and a concept-led `result/Index.md` with YAML frontmatter containing Product-derived overview title metadata, validated documentation URLs resolved from PackageProjectUrl, documentation-host-filtered exact `.nuget//README.md` documentation links including emoji-prefixed Documentation headings and "More documentation..." blocks, DocFX `metadata[].dest` API paths, and source namespace page candidates from `src//**/*.cs`, target frameworks, package/library counts, external links, package-family links, and context glyphs, and validates authored result examples with `--validate-results` as a deterministic API-shape, Codebelt.Extensions.Xunit shape, PascalCase `MethodName_Scenario_ExpectedBehavior` test-method naming, Basic usage quality, and optimized NuGet-backed executable test gate with bounded parallelism. | @@ -307,6 +307,7 @@ Sometimes the history is already written and the only thing you need is the fina - **Whole-branch by default** — for squash-and-merge requests, uses the full current feature branch from merge-base to `HEAD` instead of asking which branch commits to include - **All authors included** — branch-level summaries treat branch topology as the scope and include every contributor's commits unless the user explicitly asks for an author-filtered summary - **Bare invocation means summarize now** — calling `git-visual-squash-summary` directly should resolve the current branch scope automatically and return the grouped lines, not a "what do you want me to summarize?" question +- **The summary is the acknowledgment** — the first response is the grouped lines themselves, never an "I understand the instructions" recap followed by "would you like me to generate it now?" - **Base branch, not tracking copy** — a feature branch that is in sync with `origin/` is still summarized against `origin/HEAD`, `origin/main`, `origin/master`, `main`, or `master` before declaring there is nothing to summarize - **No yolo prompt** — the skill is read-only, so it acts directly without asking for auto-approval language from mutating workflows - **No commit-picker UX** — ordinary branch-level squash requests do not become commit-selection questions or widgets; the skill resolves the branch scope and writes the summary From 87f4a9d28763af46e5932cd18029eaa4a0012932 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sun, 16 Aug 2026 15:19:17 +0200 Subject: [PATCH 16/31] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20clarify=20immediate-?= =?UTF-8?q?action=20contract=20in=20squash-summary=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructures instructions with a new 'Start Here' section clarifying that the skill acts immediately without seeking permission. Adds explicit 'Do not' guidance around permission-seeking questions and adds test case 19 to verify this behavior. --- skills/git-visual-squash-summary/SKILL.md | 26 +++++++++++++++++-- .../evals/evals.json | 12 +++++++++ 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/skills/git-visual-squash-summary/SKILL.md b/skills/git-visual-squash-summary/SKILL.md index 8e2fdd1..a143cf7 100644 --- a/skills/git-visual-squash-summary/SKILL.md +++ b/skills/git-visual-squash-summary/SKILL.md @@ -1,7 +1,7 @@ --- name: git-visual-squash-summary description: > - Turn many commits into a curated grouped squash summary compatible with the opinionated wording style of git-visual-commits. Use when the user asks to squash a branch into a concise summary, write a squash-and-merge summary, summarize this branch, summarize a commit range or PR as grouped lines, clean up noisy commit history, or asks for a curated summary without committing. For normal squash-and-merge requests, default to the full current feature branch from merge-base to HEAD against the base branch instead of a same-named tracking remote, include commits from all authors unless the user explicitly narrows by author, and do not ask for yolo because the skill is read-only. Returns grouped lines only, resolves the cumulative base-to-HEAD diff first so reverted churn disappears, preserves identifiers, merges overlap, drops noise, and avoids changelog wording. + Turn many commits into a curated grouped squash summary compatible with the opinionated wording style of git-visual-commits. Use when the user asks to squash a branch into a concise summary, write a squash-and-merge summary, summarize this branch, summarize a commit range or PR as grouped lines, clean up noisy commit history, or asks for a curated summary without committing. For normal squash-and-merge requests, default to the full current feature branch from merge-base to HEAD against the base branch instead of a same-named tracking remote, include commits from all authors unless the user explicitly narrows by author, and do not ask for yolo because the skill is read-only. Returns grouped lines only, resolves the cumulative base-to-HEAD diff first so reverted churn disappears, preserves identifiers, merges overlap, drops noise, and avoids changelog wording. A bare invocation is already a complete request: run the read-only git commands immediately and reply with the summary lines themselves, never an instruction recap or a would-you-like-me-to-proceed question. --- # Git Visual Squash Summary @@ -16,6 +16,26 @@ This skill has one job: produce a ready-to-paste squash-and-merge summary for th This skill answers one question: **What would this branch effectively do if it were squashed into one commit now?** +## Start Here: The First Response Is the Summary + +Invoking this skill is the request. Nothing needs confirming, because the skill mutates nothing and the scope is derivable on your own: the current branch against its base branch. A confirmation round-trip costs the user a turn and returns no information you could not have resolved yourself with `git`. + +So the first thing to do after loading this skill is run the read-only commands in Step 1 — not compose a reply. The first thing the user sees is the finished grouped summary. + +A response from this skill is one of exactly three things: + +1. The grouped summary lines. This is the normal case and covers nearly every invocation. +2. `No branch changes to summarize.` when every safe base-branch comparison is genuinely empty. +3. One direct question naming the missing base branch or range — only after the Step 1 fallbacks have all been tried and failed. + +Everything else is a failed invocation, including: + +- Reciting these instructions back as "I understand the instructions" or a list of "I will ..." promises. Quoting the rules is not evidence of following them; running the commands is, and the user cannot act on a restatement of your own prompt. +- Offering to do the thing already asked for: "Would you like me to generate a squash summary of your current branch now?" +- Announcing a plan and stopping before any `git` command has run. + +If a sentence you are drafting starts with "I will" or "Would you like", delete it and run `git` instead. The summary is the acknowledgment. + ## Deterministic Reduction Model Use this model for every resolved scope: @@ -74,6 +94,7 @@ Do not classify commit 1, then commit 2, then commit 3 and merge duplicate prose - A bare invocation such as `git-visual-squash-summary` or `/git-visual-squash-summary` is itself a complete request: resolve the current branch against the base branch, then return the grouped summary directly. - Never require, infer, or ask for `yolo` / `auto`. Those modes approve mutating workflows; this skill is read-only and should act directly. - Do not collect commit-set parameters through follow-up questions, widgets, or choice UIs for ordinary squash-and-merge requests. +- Do not answer an invocation with an acknowledgment, a restatement of these rules, or an offer to proceed. Run the commands and return the summary. - Do not ask the user to choose between earlier branch commits and later branch commits such as changelog, version-bump, or release-finalization follow-ups. They are part of the branch unless the user explicitly narrows scope. - Do not stop after comparing `HEAD` to a same-named tracking branch such as `origin/`. That only proves local sync with the remote copy of the feature branch, not that there is nothing to summarize. @@ -93,7 +114,7 @@ Resolve the commit set in this order: Never turn steps 2 or 3 into a user-facing choice. Resolve them automatically and continue. Never add `--author`, `--committer`, current-user, current-email, current-contributor, or identity-mode filters while resolving ordinary branch-level squash summaries. Author metadata may help understand ownership, but it must not narrow the default commit set. Do not stop to ask whether the latest branch commit "should count". If it is on the branch, it is in scope by default. -Do not open with "What would you like me to summarize?" when the user invoked this skill directly or otherwise already asked for a squash summary. +Do not open with "What would you like me to summarize?" or "Would you like me to generate it now?" when the user invoked this skill directly or otherwise already asked for a squash summary. Both questions ask the user to repeat a request they already made. If every safe base-branch comparison is genuinely empty, say `No branch changes to summarize.` and stop. Do not ask for a hypothetical range or demo. Helpful read-only commands: @@ -237,6 +258,7 @@ Output the finished grouped summary lines and stop. Do not run `git commit`, `gi - Chronological narration of each commit in order. - Dumping raw commit subjects line by line. - Preserving reverted dependency or version churn just because it happened in history. +- Restating the skill's own rules as an "I understand the instructions" preamble, then asking permission to start. - Asking the user to choose among commits that are all on the current feature branch when they asked for a squash summary of that branch. - Presenting commit-selection widgets or multiple-choice prompts for ordinary branch-level squash requests. - Filtering the branch to the current user's or current contributor's commits, or treating "my changes" as the default scope. diff --git a/skills/git-visual-squash-summary/evals/evals.json b/skills/git-visual-squash-summary/evals/evals.json index 5d12c1a..9422bc0 100644 --- a/skills/git-visual-squash-summary/evals/evals.json +++ b/skills/git-visual-squash-summary/evals/evals.json @@ -211,6 +211,18 @@ "Produces a small number of high-signal lines independent of commit count", "Does not emit one line per fixup or implementation swap" ] + }, + { + "id": 19, + "prompt": "/git-visual-squash-summary", + "expected_output": "The very first response contains the grouped summary lines for the current branch, with no instruction recap and no request for permission to begin.", + "expectations": [ + "Runs the read-only git resolution commands before composing any user-facing reply", + "Does not restate the skill's rules as an `I understand the instructions` or `I will ...` preamble", + "Does not ask `Would you like me to generate a squash summary of your current branch now?` or any equivalent offer to proceed", + "Does not announce a plan and stop before any git command has run", + "Returns one of only three permitted response shapes: the grouped lines, `No branch changes to summarize.`, or a single base-branch question after every Step 1 fallback failed" + ] } ] } From 699dbaa5b822500c0096d48e8837284a3e413ecc Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sun, 16 Aug 2026 15:19:17 +0200 Subject: [PATCH 17/31] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20clarify=20immediate-?= =?UTF-8?q?action=20contract=20in=20squash-summary=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructures instructions with a new 'Start Here' section clarifying that the skill acts immediately without seeking permission. Adds explicit 'Do not' guidance around permission-seeking questions and adds test case 19 to verify this behavior. --- skills/git-visual-squash-summary/SKILL.md | 26 +++++++++++++++++-- .../evals/evals.json | 12 +++++++++ 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/skills/git-visual-squash-summary/SKILL.md b/skills/git-visual-squash-summary/SKILL.md index 8e2fdd1..71c5ddf 100644 --- a/skills/git-visual-squash-summary/SKILL.md +++ b/skills/git-visual-squash-summary/SKILL.md @@ -1,7 +1,7 @@ --- name: git-visual-squash-summary description: > - Turn many commits into a curated grouped squash summary compatible with the opinionated wording style of git-visual-commits. Use when the user asks to squash a branch into a concise summary, write a squash-and-merge summary, summarize this branch, summarize a commit range or PR as grouped lines, clean up noisy commit history, or asks for a curated summary without committing. For normal squash-and-merge requests, default to the full current feature branch from merge-base to HEAD against the base branch instead of a same-named tracking remote, include commits from all authors unless the user explicitly narrows by author, and do not ask for yolo because the skill is read-only. Returns grouped lines only, resolves the cumulative base-to-HEAD diff first so reverted churn disappears, preserves identifiers, merges overlap, drops noise, and avoids changelog wording. + Turn many commits into a curated grouped squash summary for squash-and-merge contexts. Use when the user asks to squash a branch, summarize PR commits, or clean up history. Defaults to full feature branch against base (not tracking remote), includes all authors unless narrowed, and acts immediately—the skill is read-only with no permission-seeking. Returns grouped lines only, resolving the cumulative diff to drop reverted churn, preserving identifiers and overlap, and avoiding changelog wording. A bare invocation is a complete request: run git commands immediately and return summary lines, never an instruction recap or permission question. --- # Git Visual Squash Summary @@ -16,6 +16,26 @@ This skill has one job: produce a ready-to-paste squash-and-merge summary for th This skill answers one question: **What would this branch effectively do if it were squashed into one commit now?** +## Start Here: The First Response Is the Summary + +Invoking this skill is the request. Nothing needs confirming, because the skill mutates nothing and the scope is derivable on your own: the current branch against its base branch. A confirmation round-trip costs the user a turn and returns no information you could not have resolved yourself with `git`. + +So the first thing to do after loading this skill is run the read-only commands in Step 1 — not compose a reply. The first thing the user sees is the finished grouped summary. + +A response from this skill is one of exactly three things: + +1. The grouped summary lines. This is the normal case and covers nearly every invocation. +2. `No branch changes to summarize.` when every safe base-branch comparison is genuinely empty. +3. One direct question naming the missing base branch or range — only after the Step 1 fallbacks have all been tried and failed. + +Everything else is a failed invocation, including: + +- Reciting these instructions back as "I understand the instructions" or a list of "I will ..." promises. Quoting the rules is not evidence of following them; running the commands is, and the user cannot act on a restatement of your own prompt. +- Offering to do the thing already asked for: "Would you like me to generate a squash summary of your current branch now?" +- Announcing a plan and stopping before any `git` command has run. + +If a sentence you are drafting starts with "I will" or "Would you like", delete it and run `git` instead. The summary is the acknowledgment. + ## Deterministic Reduction Model Use this model for every resolved scope: @@ -74,6 +94,7 @@ Do not classify commit 1, then commit 2, then commit 3 and merge duplicate prose - A bare invocation such as `git-visual-squash-summary` or `/git-visual-squash-summary` is itself a complete request: resolve the current branch against the base branch, then return the grouped summary directly. - Never require, infer, or ask for `yolo` / `auto`. Those modes approve mutating workflows; this skill is read-only and should act directly. - Do not collect commit-set parameters through follow-up questions, widgets, or choice UIs for ordinary squash-and-merge requests. +- Do not answer an invocation with an acknowledgment, a restatement of these rules, or an offer to proceed. Run the commands and return the summary. - Do not ask the user to choose between earlier branch commits and later branch commits such as changelog, version-bump, or release-finalization follow-ups. They are part of the branch unless the user explicitly narrows scope. - Do not stop after comparing `HEAD` to a same-named tracking branch such as `origin/`. That only proves local sync with the remote copy of the feature branch, not that there is nothing to summarize. @@ -93,7 +114,7 @@ Resolve the commit set in this order: Never turn steps 2 or 3 into a user-facing choice. Resolve them automatically and continue. Never add `--author`, `--committer`, current-user, current-email, current-contributor, or identity-mode filters while resolving ordinary branch-level squash summaries. Author metadata may help understand ownership, but it must not narrow the default commit set. Do not stop to ask whether the latest branch commit "should count". If it is on the branch, it is in scope by default. -Do not open with "What would you like me to summarize?" when the user invoked this skill directly or otherwise already asked for a squash summary. +Do not open with "What would you like me to summarize?" or "Would you like me to generate it now?" when the user invoked this skill directly or otherwise already asked for a squash summary. Both questions ask the user to repeat a request they already made. If every safe base-branch comparison is genuinely empty, say `No branch changes to summarize.` and stop. Do not ask for a hypothetical range or demo. Helpful read-only commands: @@ -237,6 +258,7 @@ Output the finished grouped summary lines and stop. Do not run `git commit`, `gi - Chronological narration of each commit in order. - Dumping raw commit subjects line by line. - Preserving reverted dependency or version churn just because it happened in history. +- Restating the skill's own rules as an "I understand the instructions" preamble, then asking permission to start. - Asking the user to choose among commits that are all on the current feature branch when they asked for a squash summary of that branch. - Presenting commit-selection widgets or multiple-choice prompts for ordinary branch-level squash requests. - Filtering the branch to the current user's or current contributor's commits, or treating "my changes" as the default scope. diff --git a/skills/git-visual-squash-summary/evals/evals.json b/skills/git-visual-squash-summary/evals/evals.json index 5d12c1a..9422bc0 100644 --- a/skills/git-visual-squash-summary/evals/evals.json +++ b/skills/git-visual-squash-summary/evals/evals.json @@ -211,6 +211,18 @@ "Produces a small number of high-signal lines independent of commit count", "Does not emit one line per fixup or implementation swap" ] + }, + { + "id": 19, + "prompt": "/git-visual-squash-summary", + "expected_output": "The very first response contains the grouped summary lines for the current branch, with no instruction recap and no request for permission to begin.", + "expectations": [ + "Runs the read-only git resolution commands before composing any user-facing reply", + "Does not restate the skill's rules as an `I understand the instructions` or `I will ...` preamble", + "Does not ask `Would you like me to generate a squash summary of your current branch now?` or any equivalent offer to proceed", + "Does not announce a plan and stop before any git command has run", + "Returns one of only three permitted response shapes: the grouped lines, `No branch changes to summarize.`, or a single base-branch question after every Step 1 fallback failed" + ] } ] } From 6a2738fc97409f4b514feedf071818f6702c192b Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sun, 16 Aug 2026 16:13:05 +0200 Subject: [PATCH 18/31] =?UTF-8?q?=F0=9F=94=A8=20add=20kebab-case=20validat?= =?UTF-8?q?ion=20to=20sync=20script?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds input validation to prevent path traversal attacks via skill name parameter. Ensures skill names follow kebab-case convention (alphanumeric + hyphens only) before constructing file paths in sync operations. --- scripts/sync-skill-install.ps1 | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/scripts/sync-skill-install.ps1 b/scripts/sync-skill-install.ps1 index 70d6262..bf6ba93 100644 --- a/scripts/sync-skill-install.ps1 +++ b/scripts/sync-skill-install.ps1 @@ -115,6 +115,14 @@ if (-not $Skill -or $Skill.Count -eq 0) { $totalDrift = 0 foreach ($name in $Skill) { + # Both roots are built by joining this name onto a trusted prefix, so a separator or `..` in it walks + # the sync out of the skill tree: the source becomes the repo and the install becomes the skills root, + # where -Prune would delete every other installed skill. Skill directories are kebab-case by + # convention, so anything else is malformed input rather than a skill that is merely missing. + if ($name -notmatch '^[a-z0-9]+(-[a-z0-9]+)*$') { + throw "Invalid skill name: '$name'. Expected a kebab-case skill directory name." + } + $sourceRoot = Join-Path $skillsRoot $name if (-not (Test-Path -LiteralPath $sourceRoot)) { Write-Host "[SKIP] $name (not a repo-managed skill)" From 5ecc791d2a12008c41c6ce24739bb16c69933b5e Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 00:37:49 +0200 Subject: [PATCH 19/31] =?UTF-8?q?=F0=9F=94=A7=20strengthen=20dotnet-test?= =?UTF-8?q?=20validator=20contract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add explicit assertions for immediate-action behavior and evidence-first contracts. These validators lock down the regression where a bare invocation produced a capability menu instead of running the first action. Also adds assertions for managed-fixture floor version and async-disposal defense-code guidance. --- scripts/validate-skill-templates.ps1 | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 395681e..b915856 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1252,6 +1252,15 @@ Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'zero remaining `WebApplicationFactory`' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Do not invent an endpoint, service, configuration key, or expected result.' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'An MTP executable run may supplement that gate but never replaces it' + # The skill once answered a bare invocation with a capability menu and inspected nothing; these lock the evidence-first contract. + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'You were invoked. That is the request.' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Forbidden as a first response:' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'The test host comes from Codebelt, not from Microsoft' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'it is the fallback for genuine ambiguity, not an intake wizard' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'do not exist below Codebelt xUnit **11.1.0**' + Assert-Contains -Name 'dotnet-test/FORMS.md' -Content $forms -Needle 'This form is a fallback for genuine ambiguity, not an intake step.' + Assert-Contains -Name 'inspect-dotnet-tests.ps1' -Content $inspect -Needle '$managedFixtureFloor = [version]' + Assert-Contains -Name 'dotnet-test/web-functional-tests.md' -Content $web -Needle 'Probing with `if (_application is IAsyncDisposable d)` is dead defensive code' Assert-Contains -Name 'dotnet-test/FORMS.md' -Content $forms -Needle '### project_selection' Assert-Contains -Name 'dotnet-test/FORMS.md' -Content $forms -Needle '### operation_mode' Assert-Contains -Name 'dotnet-test/FORMS.md' -Content $forms -Needle '### test_role' @@ -1280,12 +1289,16 @@ Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'restore evidence' $evalObject = $evals | ConvertFrom-Json - if (@($evalObject.evals).Count -ne 6) { - throw "dotnet-test must define exactly six requested paired eval scenarios; found $(@($evalObject.evals).Count)" + if (@($evalObject.evals).Count -lt 7) { + throw "dotnet-test must define the six paired role scenarios plus the bare-invocation immediate-action scenario; found $(@($evalObject.evals).Count)" } foreach ($needle in @('attached Acme.Calculator fixture', 'xUnit v2 project', 'web-cdn-origin-style', 'IClassFixture>', 'ApplicationTestFactory pattern', 'ApplicationTest>')) { Assert-Contains -Name 'dotnet-test/evals/evals.json' -Content $evals -Needle $needle } + # A bare invocation must act on inspector evidence instead of answering with a menu; that regression shipped once, so it stays covered. + foreach ($needle in @('Does not present a numbered menu of modes', 'Runs inspect-dotnet-tests.ps1 as the first action')) { + Assert-Contains -Name 'dotnet-test/evals/evals.json' -Content $evals -Needle $needle + } if (@($fixtureFiles | Where-Object { $_ -match '(^|[\\/])(bin|obj)([\\/]|$)' }).Count -gt 0) { throw 'dotnet-test eval fixtures must not include bin/ or obj/ paths' } From d130f087bbb1dbb696726ec70304b3b865b4e618 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 00:38:13 +0200 Subject: [PATCH 20/31] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20refactor=20dotnet-te?= =?UTF-8?q?st=20skill=20with=20immediate-action=20lockdown?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructure SKILL.md to lock down immediate-action behavior: a bare invocation must act on inspector evidence instead of presenting a capability menu. Add form fallback documentation and tighten managed-fixture floor version checks. Harden eval scenarios to cover both paired role selections and the regression case. Update reference docs with async-disposal guidance for test hosts and add fixture version assertions in inspection scripts. --- skills/dotnet-test/FORMS.md | 13 +++- skills/dotnet-test/SKILL.md | 59 ++++++++++++++++--- skills/dotnet-test/evals/evals.json | 23 ++++++++ .../focused-web/Directory.Packages.props | 1 + .../Acme.Cdn.Origin.FunctionalTests.csproj | 1 + .../references/web-functional-tests.md | 20 +++++++ .../scripts/inspect-dotnet-tests.ps1 | 13 ++++ .../scripts/test-inspect-dotnet-tests.ps1 | 15 +++++ 8 files changed, 135 insertions(+), 10 deletions(-) diff --git a/skills/dotnet-test/FORMS.md b/skills/dotnet-test/FORMS.md index 7f65217..a9c2d42 100644 --- a/skills/dotnet-test/FORMS.md +++ b/skills/dotnet-test/FORMS.md @@ -1,6 +1,10 @@ # .NET Test Input Form -Collect only unresolved fields. Prefer native structured controls when the host provides them. Otherwise use the plain-text fallback below without changing field order or defaults. +This form is a fallback for genuine ambiguity, not an intake step. `scripts/inspect-dotnet-tests.ps1` already answers every field below from the repository, so in the normal case you run it, resolve the fields from its JSON, and never open this file. The mapping from inspector output to field is in `SKILL.md` Step 1. + +Ask a field only when the inspector's evidence leaves it genuinely open. When that happens, ask that one field on its own, say what made it ambiguous, and keep the resolved fields silent — re-asking something the JSON already stated reads as if the inspection never ran. + +Prefer native structured controls when the host provides them. Otherwise use the plain-text fallback below without changing field order or defaults. ## Fields @@ -11,6 +15,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - **choices:** Dynamically list discovered test `.csproj` files relative to the repository root - **default:** The only discovered test project, or the project explicitly named by the user (Recommended) - **required:** true +- **resolved_by:** the request naming a project, or `projects[]` holding exactly one ### operation_mode @@ -21,6 +26,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - Bootstrap test coverage (Recommended when no selected test project exists or it has no behavior tests) - **default:** Compute from the selected project - **required:** true +- **resolved_by:** tests present in the selected project (refactor) or absent (bootstrap) — a repository fact, never a preference to poll ### test_role @@ -33,6 +39,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - Console or worker functional test - **default:** Auto-classify from repository evidence (Recommended) - **required:** true +- **resolved_by:** `projects[].role` ### application_adaptation @@ -43,6 +50,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - Application and test code are both in scope - **default:** Test code only; report the required application adaptation (Recommended) - **required:** true +- **resolved_by:** `referencedApplications[].genericHost` — when true, no adaptation is needed and the field does not apply - **show_when:** `test_role` is `Console or worker functional test`, or auto-classification reports a missing Generic Host blocker ### host_ownership @@ -55,6 +63,7 @@ Collect only unresolved fields. Prefer native structured controls when the host - Shared xUnit class fixture - **default:** Auto-classify from current factory/fixture usage and isolation requirements (Recommended) - **required:** true +- **resolved_by:** existing usage — a factory per test method is focused; `IClassFixture` or one shared host is shared - **show_when:** `test_role` is `ASP.NET Core functional test` or `Console or worker functional test`, and repository evidence does not already decide focused versus shared ownership ### confirmation @@ -66,9 +75,11 @@ Collect only unresolved fields. Prefer native structured controls when the host - No - **default:** Yes (Recommended) - **required:** true +- **resolved_by:** the request itself; ask only when mutation would exceed the scope it authorized ## Presentation rules +- `required: true` means the field must be **settled** before mutation, not that it must be asked. A field settled from inspector evidence is satisfied. - Infer explicit answers from the request and inspection output; do not ask them again. - Ask one unresolved field at a time. - Present the recommended/default choice first and suffix it with `(Recommended)`. diff --git a/skills/dotnet-test/SKILL.md b/skills/dotnet-test/SKILL.md index 327e9d1..01786f9 100644 --- a/skills/dotnet-test/SKILL.md +++ b/skills/dotnet-test/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-test description: > - Bootstrap or refactor .NET xUnit test projects to Codebelt conventions. Use for unit-test setup, xUnit v2-to-v3 modernization, Microsoft Testing Platform adoption, ASP.NET Core WebApplicationFactory migration, entrypoint-owned managed fixtures, reusable functional-test harnesses, and in-process console or worker tests. Classify the selected project, preserve behavior and test names, resolve compatible stable packages from NuGet, and validate restore/build/test. Do not use for NUnit/MSTest-only work, production refactoring without a test-project goal, or process-launching end-to-end harnesses. + Move .NET xUnit test projects onto Codebelt's entrypoint-owned test hosts, replacing Microsoft's WebApplicationFactory and hand-rolled host plumbing with WebApplicationTestFactory, WebApplicationTest, ApplicationTestFactory, and ApplicationTest — for ASP.NET Core, console, and worker applications alike. Invoking this skill IS the request: inspect the repository and refactor immediately, never opening with a menu, a capability list, or a questionnaire. Use for WebApplicationFactory migration, xUnit v2-to-v3 modernization, Microsoft Testing Platform adoption, managed fixtures, reusable functional-test harnesses, in-process console or worker tests, and unit-test bootstrap. Preserve behavior, test names, and package ownership, then validate restore/build/test. Do NOT use for NUnit/MSTest-only work, production refactoring without a test-project goal, or process-launching end-to-end harnesses. compatibility: > Requires .NET SDK, PowerShell 7+, and network access to NuGet for dynamic package resolution. --- @@ -10,6 +10,31 @@ compatibility: > Bootstrap and refactor xUnit projects using the tested patterns from [Codebelt xUnit](https://github.com/codebeltnet/xunit) and the matching application-host patterns from [Codebelt Bootstrapper](https://github.com/codebeltnet/bootstrapper). +## This skill has one job + +**The test host comes from Codebelt, not from Microsoft, and not from a builder you write in the test project.** + +Microsoft ships `WebApplicationFactory`, and it only covers ASP.NET Core. Everything else — console apps, workers, hosted services — has no Microsoft equivalent, so teams hand-roll a `HostBuilder` in the test project and end up testing a composition root that no deployed process ever runs. Codebelt xUnit closes both gaps with one family of abstractions where the application's own entry point owns startup: + +| What the test needs | Codebelt gives you | Instead of | +|---|---|---| +| One host per test / per narrow harness (web) | `WebApplicationTestFactory.Create(..., new ManagedWebApplicationFixture())` | `new MyFactory() : WebApplicationFactory` | +| One host shared by a test class (web) | `WebApplicationTest>` | `IClassFixture>` | +| One host per test / per narrow harness (console, worker) | `ApplicationTestFactory.Create(..., new ManagedApplicationFixture())` | a hand-built `HostBuilder` in the test project | +| One host shared by a test class (console, worker) | `ApplicationTest>` | a static host cached in a test helper | + +That substitution is the deliverable. A run that leaves `WebApplicationFactory` in place, or that swaps the type while quietly rebuilding the host in test code, has not done the job no matter how green the test run looks. + +## Do this now + +**You were invoked. That is the request.** Your first action is the inspector in [Step 1](#step-1-gather-evidence-before-asking-anything) — not a question, not a menu, not a plan. + +The inspector answers, from the repository itself, essentially every question you might be tempted to ask: which test projects exist, what role each one plays, whether it is already on xUnit v3 and Microsoft Testing Platform, who owns each package version, every `WebApplicationFactory` and managed-fixture usage with file and line, whether the referenced application has a Generic Host seam, and what the recommended migration is. Asking the developer to hand-type answers the JSON already contains costs them a turn and tells you nothing new. + +**Forbidden as a first response:** a numbered menu of things this skill could do; "bootstrap vs refactor vs improve coverage"; "would you like me to run a diagnostic scan first?"; listing capabilities; asking which project, role, or host ownership to use before the inspector has run. If you are about to write "What do you want to do?", run the inspector instead — its output makes the question obsolete. + +There is exactly one shape of legitimate question, and it comes *after* the evidence: the inspector reported a real blocker, or its evidence genuinely contradicts what the request asked for. See [Step 1](#step-1-gather-evidence-before-asking-anything). + ## Critical - Inspect before editing. Run `scripts/inspect-dotnet-tests.ps1` against the selected project and treat its role, package ownership, `WebApplicationFactory` inventory, and blockers as the starting contract. @@ -18,25 +43,37 @@ Bootstrap and refactor xUnit projects using the tested patterns from [Codebelt x - Use entrypoint-owned `ManagedWebApplicationFixture` and `ManagedApplicationFixture` for new and migrated functional tests. Do not emit their deprecated blocking variants; they are scheduled for removal. - Do not reconstruct an application entry point inside test code with `WebApplication.CreateBuilder`, `WebHostBuilder`, `HostBuilder`, `UseTestServer`, copied service registrations, or copied middleware. That creates a second composition root which can pass while the real `Program` is broken. - Preserve target frameworks, central package management, unrelated MSBuild configuration, existing test names, and test isolation. +- Edit files you are actually changing, in place. Rewriting a file wholesale flips its line endings and makes `git status` report churn that reviewers must read to discover it means nothing; a file with no semantic change must not appear in the diff at all. - Replace every selected `WebApplicationFactory` usage. A partial migration that leaves a selected usage or package reference behind is incomplete. - Keep functional testing in-process. Never add a process-launching fallback for console or worker applications. - If a selected executable has no Generic Host, adapt production startup only when application adaptation is explicitly in scope. Otherwise report the exact missing host seam and stop before changing production startup. - During bootstrap, add at least one test derived from real source behavior. Placeholder assertions such as `Assert.True(true)` do not satisfy the task. - When the request requires restore/build/test, `dotnet test` must discover the expected non-zero test count and report zero failures. An MTP executable run may supplement that gate but never replaces it; if `dotnet test` discovers zero tests, add or restore the repository-appropriate `xunit.runner.visualstudio` adapter and rerun. -## Step 1: Resolve scope and inputs - -Read `FORMS.md`. Infer fields already answered by the request or repository. Ask only for unresolved fields, one at a time, and confirm the final summary before mutation. - -Resolve the repository root and selected `.csproj` path. Do not broaden a single-project request to every test project. +## Step 1: Gather evidence before asking anything -Run: +Run the inspector first. Omit `-ProjectPath` when the request did not name a project — the inspector then discovers and classifies every test project itself: ```powershell -pwsh -NoProfile -File "/scripts/inspect-dotnet-tests.ps1" -RepoRoot "" -ProjectPath "" +pwsh -NoProfile -File "/scripts/inspect-dotnet-tests.ps1" -RepoRoot "" [-ProjectPath ""] ``` -Keep stdout as JSON. Treat a non-zero exit or a reported blocker as a real stop condition. +`` is the directory containing this `SKILL.md`; quote both paths. Keep stdout as JSON. A non-zero exit or a reported blocker is a real stop condition. + +Now read the JSON and resolve the `FORMS.md` fields from it. Almost always, all of them resolve and you proceed straight to Step 2 without asking anything: + +| `FORMS.md` field | Resolved by | Ask only when | +|---|---|---| +| `project_selection` | the request naming a project, or `projects[]` containing exactly one | `projects[]` has several and the request does not narrow them | +| `operation_mode` | tests present in the selected project → refactor; none → bootstrap | never — this is a fact about the repository, not a preference | +| `test_role` | `projects[].role` | `role` contradicts what the request explicitly asked for | +| `application_adaptation` | `referencedApplications[].genericHost` is `true` → not applicable | a missing-Generic-Host blocker is reported | +| `host_ownership` | a factory constructed per test method → focused; `IClassFixture` or one shared host → shared | usage is genuinely mixed and the request does not say | +| `confirmation` | the request itself | mutation would exceed the scope the request authorized | + +Read `FORMS.md` only when a row above actually lands in its "ask" column; it is the fallback for genuine ambiguity, not an intake wizard to run up front. When you do ask, ask that one field alone, state the evidence that made it ambiguous, and carry every already-resolved field forward silently. + +Do not broaden a single-project request to every test project. ## Step 2: Classify the project @@ -62,6 +99,10 @@ The resolver queries NuGet stable versions, tries newer candidates first, and ve When a benchmark or smoke harness provides `DOTNET_TEST_MAXIMUM_CANDIDATES`, `DOTNET_TEST_RESOLVER_CACHE_DIR`, or `DOTNET_TEST_RESOLVER_TRACE_FILE`, honor that measured scope instead of widening the live search again. Use a small explicit candidate limit for ordinary eval smoke runs; fallback and combined-package behavior stay covered by `scripts/test-resolve-test-package-versions.ps1`. +The managed fixtures this skill targets do not exist below Codebelt xUnit **11.1.0**. A project pinned under that floor restores and builds fine today and then fails to compile the moment you write the pattern, so treat the inspector's version-floor recommendation as a prerequisite edit rather than advice — raise it in the owning props file before Step 4. + +Apply the smallest change that makes the target pattern compile. The resolver returns a complete latest-compatible set because it verifies the set as a whole, not because every member needs to move; adopting all of it turns a test-host migration into a repo-wide dependency bump the request never asked for. Bump what the pattern requires, leave working pins alone, and mention the rest as available rather than applying it. + Preserve package ownership: - Central Package Management: update or add `PackageVersion` in the owning `Directory.Packages.props`; keep project `PackageReference` items versionless. diff --git a/skills/dotnet-test/evals/evals.json b/skills/dotnet-test/evals/evals.json index 3303209..64b8f41 100644 --- a/skills/dotnet-test/evals/evals.json +++ b/skills/dotnet-test/evals/evals.json @@ -53,6 +53,7 @@ "Bootstraps the real Program entry point and does not call WebApplication.CreateBuilder, UseTestServer, new TestServer, or otherwise reconstruct the application pipeline in test source", "Preserves Production environment, in-memory settings, one application per test, and disposes both IHostTest and temporary content through matching synchronous and asynchronous Test hooks", "Keeps CompressionTest and both existing method names unchanged", + "Raises Codebelt.Extensions.Xunit.App above the 11.1.0 managed-fixture floor in the owning props file, without bumping unrelated pinned packages", "Removes Microsoft.AspNetCore.Mvc.Testing when no longer needed", "The focused inspector postcondition succeeds, search finds no WebApplicationFactory in the selected migration, and restore/build/test succeed" ], @@ -134,6 +135,28 @@ "evals/files/worker-functional/test/Acme.QueuePump.FunctionalTests/Acme.QueuePump.FunctionalTests.csproj", "evals/files/worker-functional/test/Acme.QueuePump.FunctionalTests/QueuePumpTest.cs" ] + }, + { + "id": 7, + "prompt": "/dotnet-test", + "expected_output": "The first action is scripts/inspect-dotnet-tests.ps1 against the repository, followed immediately by the migration work its evidence implies. No menu, capability list, questionnaire, or plan-then-stop response is produced.", + "expectations": [ + "Runs inspect-dotnet-tests.ps1 as the first action rather than replying with a question", + "Does not present a numbered menu of modes such as bootstrap versus refactor versus improve coverage", + "Does not ask which project, role, or host ownership to use when the inspector evidence already resolves them", + "Proceeds to classify and migrate using the inspector output without an intervening confirmation round-trip", + "Asks a single scoped question only if the inspector reports a real blocker or its evidence contradicts the request" + ], + "files": [ + "evals/files/focused-web/Directory.Build.props", + "evals/files/focused-web/Directory.Packages.props", + "evals/files/focused-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj", + "evals/files/focused-web/src/Acme.Cdn.Origin/Program.cs", + "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj", + "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs", + "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs", + "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs" + ] } ] } diff --git a/skills/dotnet-test/evals/files/focused-web/Directory.Packages.props b/skills/dotnet-test/evals/files/focused-web/Directory.Packages.props index bd580fc..e1796e3 100644 --- a/skills/dotnet-test/evals/files/focused-web/Directory.Packages.props +++ b/skills/dotnet-test/evals/files/focused-web/Directory.Packages.props @@ -1,6 +1,7 @@ true + diff --git a/skills/dotnet-test/evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj b/skills/dotnet-test/evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj index 32f4530..38670cd 100644 --- a/skills/dotnet-test/evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj +++ b/skills/dotnet-test/evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj @@ -1,6 +1,7 @@ Acme.Cdn.Origin + diff --git a/skills/dotnet-test/references/web-functional-tests.md b/skills/dotnet-test/references/web-functional-tests.md index 5e1de76..efe1f0b 100644 --- a/skills/dotnet-test/references/web-functional-tests.md +++ b/skills/dotnet-test/references/web-functional-tests.md @@ -35,6 +35,26 @@ The test must bootstrap the real `Program` entry point. Do not reproduce `Progra When setup is repeated across many focused tests, a narrow `Test`-derived harness may own the factory result and temporary resources. Accept `ITestOutputHelper`, keep one harness per intended isolation scope, and expose a client/host rather than a second composition root. Override both `OnDisposeManagedResources` and `OnDisposeManagedResourcesAsync`: dispose the `IHostTest` and owned resources in each matching path, then call the base hook. Overriding only the synchronous hook is insufficient when callers use `await using`. +`IHostTest` derives from `ITest`, which implements both `IDisposable` and `IAsyncDisposable`, so call `_field.Dispose()` and `await _field.DisposeAsync()` directly on the field: + +```csharp +protected override void OnDisposeManagedResources() +{ + _application.Dispose(); + Content.Dispose(); + base.OnDisposeManagedResources(); +} + +protected override async ValueTask OnDisposeManagedResourcesAsync() +{ + await _application.DisposeAsync().ConfigureAwait(false); + Content.Dispose(); + await base.OnDisposeManagedResourcesAsync().ConfigureAwait(false); +} +``` + +Probing with `if (_application is IAsyncDisposable d)` is dead defensive code — the interface already guarantees it — and it hides the disposal behind a local, which the `inspect-dotnet-tests.ps1` ownership check reads as a harness that never disposes what it owns. + ## Shared xUnit fixture ownership Use `WebApplicationTest` when all tests in a class share one initialized host: diff --git a/skills/dotnet-test/scripts/inspect-dotnet-tests.ps1 b/skills/dotnet-test/scripts/inspect-dotnet-tests.ps1 index c2863c4..edd7240 100644 --- a/skills/dotnet-test/scripts/inspect-dotnet-tests.ps1 +++ b/skills/dotnet-test/scripts/inspect-dotnet-tests.ps1 @@ -366,6 +366,19 @@ $reports = foreach ($project in $projects) { } $recommendations = [System.Collections.Generic.List[string]]::new() + # The managed fixtures this skill targets were introduced in Codebelt xUnit 11.1.0. A project pinned below that + # floor restores fine and reports no usage problem, then fails to compile the moment the pattern is written, so + # surface the required bump as evidence during inspection instead of as a build error after the edits. + $managedFixtureFloor = [version]'11.1.0' + foreach ($package in $packages) { + if ($package.id -notmatch '^Codebelt\.Extensions\.Xunit(\.App|\.Hosting(\.AspNetCore)?)?$') { continue } + $normalizedVersion = ([string]$package.version -split '-', 2)[0] + $parsedVersion = $null + if (-not [version]::TryParse($normalizedVersion, [ref]$parsedVersion)) { continue } + if ($parsedVersion -lt $managedFixtureFloor) { + $recommendations.Add("Raise $($package.id) from $($package.version) to at least $managedFixtureFloor in $($package.versionOwner); ManagedWebApplicationFixture and ManagedApplicationFixture do not exist below that version, so the required pattern cannot compile until the version is raised.") + } + } if ($xunitGeneration -eq 'v2') { $recommendations.Add('Modernize the selected project to xUnit v3 and Microsoft Testing Platform while preserving target frameworks and package ownership.') } if ([string]$properties.UseMicrosoftTestingPlatformRunner -ne 'true') { $recommendations.Add('Enable UseMicrosoftTestingPlatformRunner for the selected xUnit v3 test project, preferably in its existing shared test-project property owner.') } if ($webUsages.Count -gt 0) { $recommendations.Add('Replace every selected WebApplicationFactory usage and preserve configuration, start behavior, clients, services, disposal, and isolation.') } diff --git a/skills/dotnet-test/scripts/test-inspect-dotnet-tests.ps1 b/skills/dotnet-test/scripts/test-inspect-dotnet-tests.ps1 index fc96b2b..121053a 100644 --- a/skills/dotnet-test/scripts/test-inspect-dotnet-tests.ps1 +++ b/skills/dotnet-test/scripts/test-inspect-dotnet-tests.ps1 @@ -152,6 +152,21 @@ using Codebelt.Extensions.Xunit.Hosting; public class ConsoleTest : ApplicationT $sharedApplicationReport = $sharedApplicationJson | ConvertFrom-Json if ($sharedApplicationReport.projects[0].sharedApplicationTestUsages.Count -ne 1) { throw 'Expected one shared ApplicationTest usage.' } + Write-File -Path (Join-Path $workspace 'Directory.Packages.props') -Content @' +true +'@ + Write-File -Path (Join-Path $workspace 'test/App.FunctionalTests/App.FunctionalTests.csproj') -Content @' +net10.0true +'@ + $belowFloorReport = & pwsh -NoProfile -File $scriptPath -RepoRoot $workspace -ProjectPath 'test/App.FunctionalTests/App.FunctionalTests.csproj' | ConvertFrom-Json + if (@($belowFloorReport.projects[0].recommendations | Where-Object { $_ -match 'Raise Codebelt\.Extensions\.Xunit\.App from 11\.0\.10 to at least 11\.1\.0' }).Count -ne 1) { throw 'Expected a managed-fixture version-floor recommendation for a below-floor Codebelt package.' } + + Write-File -Path (Join-Path $workspace 'Directory.Packages.props') -Content @' +true +'@ + $atFloorReport = & pwsh -NoProfile -File $scriptPath -RepoRoot $workspace -ProjectPath 'test/App.FunctionalTests/App.FunctionalTests.csproj' | ConvertFrom-Json + if (@($atFloorReport.projects[0].recommendations | Where-Object { $_ -match 'version-floor|to at least 11\.1\.0' }).Count -ne 0) { throw 'Did not expect a version-floor recommendation for a package at or above the managed-fixture floor.' } + Write-Host 'inspect-dotnet-tests.ps1 regression: PASS' } finally { if (Test-Path -LiteralPath $workspace) { Remove-Item -LiteralPath $workspace -Recurse -Force } From 53180852d3fa7c5b6c4ce52b0905db3996fac243 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 00:38:30 +0200 Subject: [PATCH 21/31] =?UTF-8?q?=F0=9F=92=AC=20update=20README=20for=20do?= =?UTF-8?q?tnet-test=20availability?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add dotnet-test skill to the Available Skills table and the project scaffold examples. Include both installation and discovery sections to help users locate the skill. Clarifies the skill's role in transitioning WebApplicationFactory-based test projects to Codebelt's entrypoint-owned test host pattern. --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 073731a..aaebbe2 100644 --- a/README.md +++ b/README.md @@ -128,7 +128,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill agent-smith | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | -| [dotnet-test](skills/dotnet-test/SKILL.md) | Bootstraps and refactors xUnit projects to Codebelt conventions. It deterministically inspects project roles, target frameworks, xUnit generation, package ownership, inheritance, application entry points—including Bootstrapper `MinimalConsoleProgram`, `MinimalWorkerProgram`, and `MinimalWebProgram` hosts—and every selected `WebApplicationFactory` usage; classifies ordinary unit, ASP.NET Core functional, and console/worker functional tests; modernizes xUnit v2 projects to xUnit v3 plus Microsoft Testing Platform without moving package ownership or changing frameworks; and resolves current stable compatible packages through NuGet-backed isolated compatibility-project restores, including the selected combined package set. Focused web tests use `WebApplicationTestFactory` with an explicit entrypoint-owned `ManagedWebApplicationFixture`, directly or through a narrow `Test`-derived harness; shared web fixtures use `WebApplicationTest` with `ManagedWebApplicationFixture`; focused console/worker tests use `ApplicationTestFactory` with `ManagedApplicationFixture`; and shared non-web fixtures use `ApplicationTest` with `ManagedApplicationFixture`. Deprecated blocking fixtures are migration inputs only and are never emitted because they are scheduled for removal. Functional migrations fail closed unless the chosen Codebelt pattern and managed fixture are present, the legacy or blocking fixture is absent, and test code does not reconstruct the production composition root with its own `WebApplication`, `TestServer`, or `HostBuilder`. Migrations preserve entrypoint-owned startup, host configuration, lazy start, clients, services, configuration, sync/async disposal, isolation, and existing test names, while fresh bootstraps add source-grounded behavior tests. Non-web tests stay in-process and require a resolvable Generic Host; test-only scope reports the exact production adaptation instead of silently rewriting startup or launching a process. | +| [dotnet-test](skills/dotnet-test/SKILL.md) | Moves xUnit projects onto Codebelt's entrypoint-owned test hosts, replacing Microsoft's ASP.NET-only `WebApplicationFactory`—and the hand-rolled `HostBuilder` that console and worker tests reach for because Microsoft ships no equivalent—with one family of abstractions where the application's own entry point owns startup. Invocation is the request: it inspects and refactors immediately instead of opening with a menu or a questionnaire. It deterministically inspects project roles, target frameworks, xUnit generation, package ownership, inheritance, application entry points—including Bootstrapper `MinimalConsoleProgram`, `MinimalWorkerProgram`, and `MinimalWebProgram` hosts—and every selected `WebApplicationFactory` usage; classifies ordinary unit, ASP.NET Core functional, and console/worker functional tests; modernizes xUnit v2 projects to xUnit v3 plus Microsoft Testing Platform without moving package ownership or changing frameworks; and resolves current stable compatible packages through NuGet-backed isolated compatibility-project restores, including the selected combined package set. Focused web tests use `WebApplicationTestFactory` with an explicit entrypoint-owned `ManagedWebApplicationFixture`, directly or through a narrow `Test`-derived harness; shared web fixtures use `WebApplicationTest` with `ManagedWebApplicationFixture`; focused console/worker tests use `ApplicationTestFactory` with `ManagedApplicationFixture`; and shared non-web fixtures use `ApplicationTest` with `ManagedApplicationFixture`. Deprecated blocking fixtures are migration inputs only and are never emitted because they are scheduled for removal. Functional migrations fail closed unless the chosen Codebelt pattern and managed fixture are present, the legacy or blocking fixture is absent, and test code does not reconstruct the production composition root with its own `WebApplication`, `TestServer`, or `HostBuilder`. Migrations preserve entrypoint-owned startup, host configuration, lazy start, clients, services, configuration, sync/async disposal, isolation, and existing test names, while fresh bootstraps add source-grounded behavior tests. Non-web tests stay in-process and require a resolvable Generic Host; test-only scope reports the exact production adaptation instead of silently rewriting startup or launching a process. | | [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, requires exact per-case correctness oracles plus a semantic preflight for truthful workload labels, hard-gates interpretation on a complete valid BenchmarkDotNet summary, preserves workload invariants such as selectivity and hit/miss ratios as sizes scale, distinguishes deferred pipeline creation from terminal/materialization work, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash; after the first valid full result it stops unless deeper diagnostics could change a real engineering decision. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | | [dotnet-remote-testing](skills/dotnet-remote-testing/SKILL.md) | Run .NET tests inside a resolved remote Docker environment and return concise, structured results — Visual Studio's Remote Testing experience (choose an environment → run tests → see results) with the container plumbing hidden behind a deterministic runner (`scripts/remote-test.cs`) the skill orchestrates instead of composing ad-hoc `docker run` commands. Invoking it is the request: with one applicable Docker environment it runs immediately — no capability menu, no parameter questionnaire, no confirmation — and when several channels are derived, the repository's own highest target framework selects the matching one and the choice is reported. The runner decides when a question is unavoidable, exiting `SelectionRequired` (16) with the exact candidates, and every exit code maps to exactly one next action so behavior is identical across models. It honors Microsoft's existing `testenvironments.json` version-1 contract (`name`, `localRoot`, `dockerImage`, `dockerFile`, with the either/or Docker-source rule), treats that file as authoritative when present, and reports WSL/SSH/unknown types as unsupported rather than converting or silently ignoring them. When no `testenvironments.json` exists it provides a zero-configuration experience built exclusively on official `mcr.microsoft.com/dotnet/sdk` images, discovering the currently supported LTS and STS channels plus the current preview from Microsoft's live `releases-index.json` using `support-phase`/`release-type` (never hardcoded version numbers or even/odd assumptions) and caching that metadata outside the repository for offline reuse. It prefers an exact `latest-sdk` image tag (stripping preview build metadata), validates the tag against Microsoft's registry, and pins each execution to the resolved immutable digest so results are reproducible across environment, image, digest, SDK, and architecture. Execution stages the source into an isolated workspace so container builds never leave Linux `bin`/`obj` in the working tree, mounts a persistent NuGet cache outside the repo, runs restore → build → test with structured TRX collection, classifies failures into distinct kinds (configuration, unsupported environment, Docker unavailable, image resolution, SDK incompatibility, staging, restore, compilation, test-host, test failure, result-processing, cleanup, cancellation, release-metadata) so infrastructure problems are never reported as failing unit tests, and always cleans up transient Docker resources. It never generates a `Dockerfile`, dev container, compose file, or editor configuration (an existing configured `dockerFile` is honored, never created), never runs privileged containers or mounts the Docker socket, and never silently falls back to running tests on the host. Docker is the only transport for now, designed so WSL/SSH can be added later without disturbing the deterministic Docker path, which is covered by a comprehensive built-in `--self-test` plus a PowerShell harness. | | [dotnet-segregated-assets](skills/dotnet-segregated-assets/SKILL.md) | Migrate or configure ASP.NET Core static delivery with `codebeltnet/web-cdn-origin:2.0.0` while keeping `wwwroot` as the authoring root. The deterministic runner inspects and verifies topology, publish exclusion, Static Web Assets risks, Cuemon signals, competing `AppAssetOptions`-style abstractions, actual `app-*`/`cdn-*` markup, and scheme-safe local origins; the agent performs semantic edits. For an existing Cuemon package reference, its plan resolves the highest stable version from NuGet.org at execution time, preserves Central Package Management versus inline ownership, excludes prereleases, and fails rather than copying an old fixture or example version. It reuses Cuemon `AppTagHelperOptions`/`CdnTagHelperOptions`, `BaseUrlMode`, and the public `app-link`, `app-script`, `app-img`, `cdn-link`, `cdn-script`, and `cdn-img` helpers when already available, otherwise reuses a suitable project abstraction without adding Cuemon. It keeps App and shared CDN ownership separate, preserves ordinary Project-based Development, adds opt-in segregated Development through a root Docker Compose profile, and makes `compose.assets.yml` directly build artifact-first `LocalDevelopment.Dockerfile` and `Assets.Dockerfile` images. Every generated file comes from a literal template in `assets/` and lands in one fixed location — the three Dockerfiles beside the web `.csproj`, orchestration at the repository root — and `verify --check-local` proves that placement along with the artifact-first contract: no SDK stage or `dotnet publish` inside an application image, a `.dockerignore` that still carries `artifacts/`, `LocalPublishDirectory` behind a guarded post-build target, Compose host ports derived from the ordinary Project profile, and a CI job that produces the artifact those images copy. Production CI publishes the same application artifact for the shell-less runtime `Dockerfile`. The skill excludes app-owned `wwwroot` with targeted MSBuild metadata, preserves `_content`/`_framework` and generated Static Web Assets, and proves publish/local invariants deterministically and idempotently. | @@ -646,8 +646,12 @@ API documentation rots the moment code changes. A new public type ships without Test-project refactoring is deceptively lifecycle-sensitive. A `WebApplicationFactory` wrapper may own temporary directories, defer host startup until the first client, replace services in a specific order, or isolate settings per test. Console and worker tests have a different boundary: they need a resolvable in-process Generic Host, not a child process hidden behind a test helper. -**dotnet-test** begins with machine-readable inspection, then chooses the Codebelt pattern that matches the selected project's role and ownership model. It preserves package ownership and frameworks, migrates xUnit v2 to v3/Microsoft Testing Platform when needed, and makes the chosen focused/shared web or application pattern, entrypoint-owned managed fixture, zero remaining selected `WebApplicationFactory` usages, zero deprecated blocking fixtures, zero replacement composition roots, and restore/build/test explicit gates. +The skill has one job: the test host comes from Codebelt, not from Microsoft, and not from a builder written in the test project. `WebApplicationFactory` covers ASP.NET Core and nothing else, so console and worker tests end up hand-rolling a host that no deployed process ever runs. Codebelt closes both gaps with `WebApplicationTestFactory`, `WebApplicationTest`, `ApplicationTestFactory`, and `ApplicationTest`. +**dotnet-test** begins with machine-readable inspection, then chooses the Codebelt pattern that matches the selected project's role and ownership model. Because that inspection already answers which project, role, and ownership apply, the skill acts on the evidence instead of asking the developer to retype what the JSON says. It preserves package ownership and frameworks, migrates xUnit v2 to v3/Microsoft Testing Platform when needed, and makes the chosen focused/shared web or application pattern, entrypoint-owned managed fixture, zero remaining selected `WebApplicationFactory` usages, zero deprecated blocking fixtures, zero replacement composition roots, and restore/build/test explicit gates. + +- **Evidence before questions** — the bundled inspector resolves project, role, mode, host ownership, and package owner, so a bare invocation starts working instead of returning a capability menu, +- **Managed-fixture version floor** — inspection flags a Codebelt xUnit package pinned below 11.1.0, where the managed fixtures do not exist yet, before the pattern is written rather than after it fails to compile, - **Three explicit roles** — ordinary unit, ASP.NET Core functional, and console/worker functional tests route to separate references and assets, - **Lifecycle-preserving functional migration** — focused factories or narrow `Test`-derived harnesses and shared managed fixtures retain configuration, lazy start, client/service access, synchronous/asynchronous disposal, and isolation, - **Real entry-point coverage** — focused and shared postconditions reject test-owned `WebApplication`/`TestServer` pipelines that can pass while the production `Program` is broken, From 4c8092622cd9365e4400d8763a1582396dabe1fa Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 21:18:57 +0200 Subject: [PATCH 22/31] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20restructure=20dotnet?= =?UTF-8?q?-test=20skill=20with=20xunit=20anchor=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The skill contract now documents the xunit anchor mechanism that prevents xunit* packages from outrunning the Codebelt release the skill targets. The resolver uses the Codebelt xUnit package's published nuspec to determine which xunit* ids pin 1:1 to declared versions and which cap at the anchor's major version. Updated SKILL.md with constraints documentation and xunit-v3-modernization.md with version distinction between package name and actual version numbers. --- skills/dotnet-test/SKILL.md | 13 +++++++++++-- .../references/xunit-v3-modernization.md | 2 +- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/skills/dotnet-test/SKILL.md b/skills/dotnet-test/SKILL.md index 01786f9..9b0213f 100644 --- a/skills/dotnet-test/SKILL.md +++ b/skills/dotnet-test/SKILL.md @@ -43,6 +43,7 @@ There is exactly one shape of legitimate question, and it comes *after* the evid - Use entrypoint-owned `ManagedWebApplicationFixture` and `ManagedApplicationFixture` for new and migrated functional tests. Do not emit their deprecated blocking variants; they are scheduled for removal. - Do not reconstruct an application entry point inside test code with `WebApplication.CreateBuilder`, `WebHostBuilder`, `HostBuilder`, `UseTestServer`, copied service registrations, or copied middleware. That creates a second composition root which can pass while the real `Program` is broken. - Preserve target frameworks, central package management, unrelated MSBuild configuration, existing test names, and test isolation. +- Never take an `xunit*` package past the major the Codebelt xUnit release depends on. The resolver anchors that ceiling in [Step 3](#step-3-resolve-packages-without-hardcoding-latest); newest-on-NuGet is not it. - Edit files you are actually changing, in place. Rewriting a file wholesale flips its line endings and makes `git status` report churn that reviewers must read to discover it means nothing; a file with no semantic change must not appear in the diff at all. - Replace every selected `WebApplicationFactory` usage. A partial migration that leaves a selected usage or package reference behind is incomplete. - Keep functional testing in-process. Never add a process-launching fallback for console or worker applications. @@ -92,11 +93,19 @@ If the evidence conflicts with the requested role, report the conflict and ask b Run the resolver for the selected target frameworks and role: ```powershell -pwsh -NoProfile -File "/scripts/resolve-test-package-versions.ps1" -TargetFramework -Role +pwsh -NoProfile -File "/scripts/resolve-test-package-versions.ps1" -TargetFramework -Role [-XunitAnchorVersion ] ``` The resolver queries NuGet stable versions, tries newer candidates first, and verifies each candidate against the selected package set through isolated compatibility-project restores; it emits only a set whose combined package restore passes. If it fails, report the package, target frameworks, and restore evidence instead of guessing. +**Newest is not the ceiling for `xunit*`.** xUnit versions its own packages on its own schedule — `xunit.v3` and `xunit.runner.visualstudio` are both past 4.0.0 while [Codebelt xUnit](https://github.com/codebeltnet/xunit) still builds against the 3.x line — so "latest stable" would push a test project a whole xUnit generation past the Codebelt API it is supposed to use. The resolver therefore anchors every `xunit*` id to the Codebelt package for the role (`Codebelt.Extensions.Xunit` for `Unit`, `Codebelt.Extensions.Xunit.App` otherwise), reading the anchor's own published nuspec dependencies: + +- an id the anchor declares — today `xunit.v3.assert` and `xunit.v3.extensibility.core` — resolves **1:1** to the exact version the anchor declares; +- every other `xunit*` id resolves to the newest minor/patch **at or below the anchor's major**; +- nothing else is capped, and the anchored evidence is reported back under `xunitAnchor` plus a per-package `constraint`. + +Pass `-XunitAnchorVersion` with the `Codebelt.Extensions.Xunit*` version the repository already references — the inspector reports it under `packageOwnership` — whenever that pin is being kept, so the resolved xUnit generation matches the Codebelt release actually in use rather than the newest one on NuGet. Omit it to anchor on the newest Codebelt release. Never hand-pick an `xunit*` version above the reported anchor major; if a project genuinely needs the next xUnit generation, the Codebelt package has to move there first. + When a benchmark or smoke harness provides `DOTNET_TEST_MAXIMUM_CANDIDATES`, `DOTNET_TEST_RESOLVER_CACHE_DIR`, or `DOTNET_TEST_RESOLVER_TRACE_FILE`, honor that measured scope instead of widening the live search again. Use a small explicit candidate limit for ordinary eval smoke runs; fallback and combined-package behavior stay covered by `scripts/test-resolve-test-package-versions.ps1`. The managed fixtures this skill targets do not exist below Codebelt xUnit **11.1.0**. A project pinned under that floor restores and builds fine today and then fails to compile the moment you write the pattern, so treat the inspector's version-floor recommendation as a prerequisite edit rather than advice — raise it in the owning props file before Step 4. @@ -200,7 +209,7 @@ Report: - selected project and classified role; - mode and whether production application adaptation was in scope; -- package ownership and resolved versions; +- package ownership and resolved versions, including the Codebelt xUnit anchor that bounded the `xunit*` versions; - preserved migration invariants; - behavior test added or existing tests retained; - exact restore/build/test and zero-usage-search results; diff --git a/skills/dotnet-test/references/xunit-v3-modernization.md b/skills/dotnet-test/references/xunit-v3-modernization.md index b231cfc..8ec20e5 100644 --- a/skills/dotnet-test/references/xunit-v3-modernization.md +++ b/skills/dotnet-test/references/xunit-v3-modernization.md @@ -4,7 +4,7 @@ Modernize the selected project without rewriting unrelated project infrastructur ## Required project shape -- Replace xUnit v2 packages with `xunit.v3` and the repository's runner packages. +- Replace xUnit v2 packages with `xunit.v3` and the repository's runner packages, at the versions the Step 3 resolver anchored to the Codebelt xUnit release. "v3" names the package, not the version: `xunit.v3` has its own majors above 3, and this modernization targets the one Codebelt xUnit depends on. - Set test projects to executable output when not inherited: `Exe`. - Enable Microsoft Testing Platform: `true`. - Remove `Xunit.Abstractions`; import `Xunit` for `ITestOutputHelper`. From 6a6a8f759d20fbfb65a0cdf1983fb815ebad46aa Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 21:19:06 +0200 Subject: [PATCH 23/31] =?UTF-8?q?=F0=9F=94=A8=20implement=20xunit=20anchor?= =?UTF-8?q?=20resolution=20in=20dotnet-test?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The resolver now anchors every xunit* package to the Codebelt xUnit release (Codebelt.Extensions.Xunit or Codebelt.Extensions.Xunit.App by role). It queries the anchor's published nuspec dependencies to determine which xunit ids resolve 1:1 to the declared version and which cap at the anchor's major version. New functions: Resolve-XunitAnchor reads nuspec and returns major/dependencies, Select-AnchoredCandidate filters candidates by anchored major, Get-AnchorConstraint reports the constraint for each resolved package. The resolver caches and reports anchoring evidence. --- .../scripts/resolve-test-package-versions.ps1 | 178 ++++++++++++++++-- 1 file changed, 159 insertions(+), 19 deletions(-) diff --git a/skills/dotnet-test/scripts/resolve-test-package-versions.ps1 b/skills/dotnet-test/scripts/resolve-test-package-versions.ps1 index db4c6c8..bc1df02 100644 --- a/skills/dotnet-test/scripts/resolve-test-package-versions.ps1 +++ b/skills/dotnet-test/scripts/resolve-test-package-versions.ps1 @@ -8,6 +8,10 @@ param( [string[]]$PackageId, + [string]$XunitAnchorPackageId, + + [string]$XunitAnchorVersion, + [int]$MaximumCandidates, [string]$CacheDirectory = $env:DOTNET_TEST_RESOLVER_CACHE_DIR, @@ -37,12 +41,20 @@ if ($MaximumCandidates -lt 1 -or $MaximumCandidates -gt 100) { throw "MaximumCandidates must be between 1 and 100. Found '$MaximumCandidates'." } +# xUnit versions its packages independently of this skill: xunit.v3 4.0.0 and xunit.runner.visualstudio 4.0.0 are stable +# on NuGet while Codebelt.Extensions.Xunit still builds against 3.2.2. "Newest stable" would therefore drag a test project +# a whole xUnit generation past the Codebelt API it is meant to use, so every id matching this pattern is anchored to the +# Codebelt package instead of resolved freely. +$xunitPackagePattern = '^xunit(\.|$)' +$stableVersionPattern = '^\d+(?:\.\d+){1,3}$' + function Get-ResolverCacheKey { param( [Parameter(Mandatory = $true)] [string]$RoleName, [Parameter(Mandatory = $true)] [string[]]$Frameworks, [Parameter(Mandatory = $true)] [string[]]$Packages, - [Parameter(Mandatory = $true)] [int]$CandidateLimit + [Parameter(Mandatory = $true)] [int]$CandidateLimit, + [string]$Anchor ) $seed = [ordered]@{ @@ -50,6 +62,7 @@ function Get-ResolverCacheKey { targetFrameworks = @($Frameworks | Sort-Object) packageIds = @($Packages | Sort-Object) maximumCandidates = $CandidateLimit + xunitAnchor = [string]$Anchor } | ConvertTo-Json -Compress return [System.BitConverter]::ToString(([System.Security.Cryptography.SHA256]::HashData($utf8NoBom.GetBytes($seed)))).Replace('-', '').ToLowerInvariant() } @@ -62,7 +75,8 @@ function Write-ResolverTrace { [string[]]$Packages, [int]$CandidateLimit, [bool]$CacheHit, - [double]$DurationSeconds + [double]$DurationSeconds, + [string]$Anchor ) if ([string]::IsNullOrWhiteSpace($TraceFilePath)) { return } @@ -76,6 +90,7 @@ function Write-ResolverTrace { targetFrameworks = @($Frameworks) packageIds = @($Packages) maximumCandidates = $CandidateLimit + xunitAnchor = [string]$Anchor cacheHit = $CacheHit durationSeconds = [math]::Round($DurationSeconds, 3) timestamp = [DateTimeOffset]::UtcNow.ToString('O') @@ -94,6 +109,117 @@ function Get-VersionKey { } } +function Compare-VersionText { + param([string]$Left, [string]$Right) + + $leftKey = Get-VersionKey -Version $Left + $rightKey = Get-VersionKey -Version $Right + foreach ($part in @('major', 'minor', 'patch', 'revision')) { + $leftPart = [int]$leftKey.$part + $rightPart = [int]$rightKey.$part + if ($leftPart -ne $rightPart) { return $leftPart.CompareTo($rightPart) } + } + return 0 +} + +function Get-PackageBaseAddress { + $serviceIndex = Invoke-RestMethod -Uri 'https://api.nuget.org/v3/index.json' + $address = $serviceIndex.resources | + Where-Object { $_.'@type' -eq 'PackageBaseAddress/3.0.0' } | + Select-Object -First 1 -ExpandProperty '@id' + if ([string]::IsNullOrWhiteSpace($address)) { throw 'NuGet service index did not expose PackageBaseAddress/3.0.0.' } + return $address +} + +function Get-StableVersion { + param([string]$BaseAddress, [string]$PackageId) + + $indexUrl = '{0}{1}/index.json' -f $BaseAddress, $PackageId.ToLowerInvariant() + try { $index = Invoke-RestMethod -Uri $indexUrl } catch { throw "NuGet lookup failed for '$PackageId' at '$indexUrl': $($_.Exception.Message)" } + $versions = @($index.versions | + Where-Object { $_ -match $stableVersionPattern } | + ForEach-Object { Get-VersionKey -Version $_ } | + Sort-Object major, minor, patch, revision -Descending) + return [pscustomobject]@{ source = $indexUrl; versions = $versions } +} + +function Resolve-XunitAnchor { + param([string]$BaseAddress, [string]$PackageId, [string]$Version) + + $anchorVersion = $Version + if ([string]::IsNullOrWhiteSpace($anchorVersion)) { + $index = Get-StableVersion -BaseAddress $BaseAddress -PackageId $PackageId + if (@($index.versions).Count -eq 0) { throw "NuGet returned no stable versions for the xUnit anchor package '$PackageId'." } + $anchorVersion = @($index.versions)[0].text + } + if ($anchorVersion -notmatch $stableVersionPattern) { + throw "XunitAnchorVersion must be a stable version such as '11.2.1'. Found '$anchorVersion'." + } + + $nuspecUrl = '{0}{1}/{2}/{1}.nuspec' -f $BaseAddress, $PackageId.ToLowerInvariant(), $anchorVersion.ToLowerInvariant() + try { $nuspec = Invoke-RestMethod -Uri $nuspecUrl } catch { throw "NuGet nuspec lookup failed for '$PackageId' $anchorVersion at '$nuspecUrl': $($_.Exception.Message)" } + + $pins = [ordered]@{} + foreach ($node in @($nuspec.GetElementsByTagName('dependency'))) { + $dependencyId = [string]$node.GetAttribute('id') + if ($dependencyId -notmatch $xunitPackagePattern) { continue } + $declared = (([string]$node.GetAttribute('version')).Trim('[', ']', '(', ')', ' ') -split ',')[0].Trim() + if ($declared -notmatch $stableVersionPattern) { continue } + $key = $dependencyId.ToLowerInvariant() + if ($pins.Contains($key) -and (Compare-VersionText -Left ([string]$pins[$key]) -Right $declared) -ge 0) { continue } + $pins[$key] = $declared + } + if ($pins.Count -eq 0) { + throw "'$PackageId' $anchorVersion declares no stable xunit* dependency, so the xUnit ceiling cannot be anchored to it. Pass -XunitAnchorPackageId with a Codebelt xUnit package that does." + } + + $major = @(@($pins.Values) | ForEach-Object { [int]((Get-VersionKey -Version ([string]$_)).major) } | Sort-Object -Descending)[0] + return [pscustomobject]@{ + packageId = $PackageId + version = $anchorVersion + major = $major + source = $nuspecUrl + pins = $pins + } +} + +function Select-AnchoredCandidate { + param([string]$PackageId, [object[]]$Candidates, [object]$Anchor) + + if ($null -eq $Anchor -or $PackageId -notmatch $xunitPackagePattern) { return @($Candidates) } + + $allowed = @(@($Candidates) | Where-Object { [int]($_.major) -le [int]($Anchor.major) }) + if ($allowed.Count -eq 0) { + throw "No stable '$PackageId' version at or below major $($Anchor.major) exists. That ceiling comes from $($Anchor.packageId) $($Anchor.version); raise the anchor package before raising the xUnit generation." + } + + $key = $PackageId.ToLowerInvariant() + if ($Anchor.pins.Contains($key)) { + $pinned = [string]$Anchor.pins[$key] + $exact = @($allowed | Where-Object { $_.text -eq $pinned }) + if ($exact.Count -gt 0) { + $allowed = @($exact) + @($allowed | Where-Object { $_.text -ne $pinned }) + } + } + return @($allowed) +} + +function Get-AnchorConstraint { + param([string]$PackageId, [string]$Version, [object]$Anchor) + + if ($null -eq $Anchor -or $PackageId -notmatch $xunitPackagePattern) { return 'unanchored' } + + $key = $PackageId.ToLowerInvariant() + if (-not $Anchor.pins.Contains($key)) { + return "capped at major $($Anchor.major) by $($Anchor.packageId) $($Anchor.version)" + } + $pinned = [string]$Anchor.pins[$key] + if ($pinned -eq $Version) { + return "matched 1:1 to the $pinned dependency declared by $($Anchor.packageId) $($Anchor.version)" + } + return "capped at major $($Anchor.major) by $($Anchor.packageId) $($Anchor.version) which declares $pinned" +} + function Test-PackageCompatibility { param([object[]]$Packages, [string[]]$Frameworks, [string]$Workspace) @@ -133,14 +259,24 @@ foreach ($framework in $TargetFramework) { } } +$codebeltPackage = if ($Role -eq 'Unit') { 'Codebelt.Extensions.Xunit' } else { 'Codebelt.Extensions.Xunit.App' } $packageIds = if ($PackageId -and $PackageId.Count -gt 0) { @($PackageId) } else { - $codebeltPackage = if ($Role -eq 'Unit') { 'Codebelt.Extensions.Xunit' } else { 'Codebelt.Extensions.Xunit.App' } @('Microsoft.NET.Test.Sdk', 'xunit.v3', 'xunit.v3.runner.console', 'xunit.runner.visualstudio', $codebeltPackage) } -$cacheKey = if ([string]::IsNullOrWhiteSpace($CacheDirectory)) { $null } else { Get-ResolverCacheKey -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates } +$anchorPackageId = if ([string]::IsNullOrWhiteSpace($XunitAnchorPackageId)) { $codebeltPackage } else { $XunitAnchorPackageId } +$anchoredPackageIds = @($packageIds | Where-Object { $_ -match $xunitPackagePattern }) +$flatContainerAddress = $null +$anchor = $null +if ($anchoredPackageIds.Count -gt 0) { + $flatContainerAddress = Get-PackageBaseAddress + $anchor = Resolve-XunitAnchor -BaseAddress $flatContainerAddress -PackageId $anchorPackageId -Version $XunitAnchorVersion +} +$anchorKey = if ($null -eq $anchor) { '' } else { '{0}/{1}' -f $anchor.packageId, $anchor.version } + +$cacheKey = if ([string]::IsNullOrWhiteSpace($CacheDirectory)) { $null } else { Get-ResolverCacheKey -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -Anchor $anchorKey } $cachePath = if ($null -eq $cacheKey) { $null } else { Join-Path $CacheDirectory ($cacheKey + '.json') } $startedAt = [DateTimeOffset]::UtcNow @@ -157,16 +293,12 @@ if ($null -ne $cachePath -and (Test-Path -LiteralPath $cachePath -PathType Leaf) $cached.cache | Add-Member -NotePropertyName hit -NotePropertyValue $true -Force $cached.cache | Add-Member -NotePropertyName key -NotePropertyValue $cacheKey -Force $cached.timing | Add-Member -NotePropertyName durationSeconds -NotePropertyValue $durationSeconds -Force - Write-ResolverTrace -TraceFilePath $TraceFile -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -CacheHit $true -DurationSeconds $durationSeconds + Write-ResolverTrace -TraceFilePath $TraceFile -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -CacheHit $true -DurationSeconds $durationSeconds -Anchor $anchorKey $cached | ConvertTo-Json -Depth 8 return } -$serviceIndex = Invoke-RestMethod -Uri 'https://api.nuget.org/v3/index.json' -$packageBaseAddress = $serviceIndex.resources | - Where-Object { $_.'@type' -eq 'PackageBaseAddress/3.0.0' } | - Select-Object -First 1 -ExpandProperty '@id' -if ([string]::IsNullOrWhiteSpace($packageBaseAddress)) { throw 'NuGet service index did not expose PackageBaseAddress/3.0.0.' } +if ($null -eq $flatContainerAddress) { $flatContainerAddress = Get-PackageBaseAddress } $workspace = Join-Path ([System.IO.Path]::GetTempPath()) ('dotnet-test-package-resolution-' + [Guid]::NewGuid().ToString('N')) New-Item -ItemType Directory -Path $workspace -Force | Out-Null @@ -174,15 +306,13 @@ New-Item -ItemType Directory -Path $workspace -Force | Out-Null try { $packageCandidates = [System.Collections.Generic.List[object]]::new() foreach ($id in @($packageIds | Sort-Object -Unique)) { - $indexUrl = '{0}{1}/index.json' -f $packageBaseAddress, $id.ToLowerInvariant() - try { $index = Invoke-RestMethod -Uri $indexUrl } catch { throw "NuGet lookup failed for '$id' at '$indexUrl': $($_.Exception.Message)" } - $candidates = @($index.versions | - Where-Object { $_ -match '^\d+(?:\.\d+){1,3}$' } | - ForEach-Object { Get-VersionKey -Version $_ } | - Sort-Object major, minor, patch, revision -Descending | + $index = Get-StableVersion -BaseAddress $flatContainerAddress -PackageId $id + if (@($index.versions).Count -eq 0) { throw "NuGet returned no stable versions for '$id'." } + # Anchor first, trim second: a package whose newest candidates are all above the anchored major would otherwise + # arrive here with nothing left to try. + $candidates = @(Select-AnchoredCandidate -PackageId $id -Candidates @($index.versions) -Anchor $anchor | Select-Object -First $MaximumCandidates) - if ($candidates.Count -eq 0) { throw "NuGet returned no stable versions for '$id'." } - $packageCandidates.Add([pscustomobject]@{ packageId = $id; source = $indexUrl; candidates = $candidates }) + $packageCandidates.Add([pscustomobject]@{ packageId = $id; source = $index.source; candidates = $candidates }) } function Resolve-PackageSet { @@ -239,6 +369,7 @@ try { version = $package.version source = $package.source compatibility = 'combined restore passed' + constraint = Get-AnchorConstraint -PackageId $package.packageId -Version $package.version -Anchor $anchor }) } @@ -247,6 +378,15 @@ try { role = $Role targetFrameworks = @($TargetFramework) maximumCandidates = $MaximumCandidates + xunitAnchor = if ($null -eq $anchor) { $null } else { + [ordered]@{ + packageId = $anchor.packageId + version = $anchor.version + major = $anchor.major + source = $anchor.source + declaredDependencies = $anchor.pins + } + } cache = [ordered]@{ enabled = $null -ne $cachePath hit = $false @@ -266,7 +406,7 @@ try { $result | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $cachePath -Encoding utf8 } - Write-ResolverTrace -TraceFilePath $TraceFile -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -CacheHit $false -DurationSeconds $durationSeconds + Write-ResolverTrace -TraceFilePath $TraceFile -RoleName $Role -Frameworks $TargetFramework -Packages $packageIds -CandidateLimit $MaximumCandidates -CacheHit $false -DurationSeconds $durationSeconds -Anchor $anchorKey $result | ConvertTo-Json -Depth 8 } finally { if (Test-Path -LiteralPath $workspace) { Remove-Item -LiteralPath $workspace -Recurse -Force } From 23127de5921eb87162d2dea2063610854045772b Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 21:19:10 +0200 Subject: [PATCH 24/31] =?UTF-8?q?=E2=9C=85=20add=20xunit=20anchor=20resolu?= =?UTF-8?q?tion=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Test harness now covers xunit anchor anchoring logic: mocks for nuspec queries, anchor resolution from Codebelt xUnit packages, candidate selection filtered by anchored major, and constraint reporting. Validates that candidates above the anchored major are excluded, that no candidate above the anchored major may reach a restore, and that the anchor evidence is correctly returned in the resolver output. --- .../test-resolve-test-package-versions.ps1 | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/skills/dotnet-test/scripts/test-resolve-test-package-versions.ps1 b/skills/dotnet-test/scripts/test-resolve-test-package-versions.ps1 index 922c998..8845d83 100644 --- a/skills/dotnet-test/scripts/test-resolve-test-package-versions.ps1 +++ b/skills/dotnet-test/scripts/test-resolve-test-package-versions.ps1 @@ -10,6 +10,7 @@ $packageBaseAddress = 'https://mock.nuget/flatcontainer/' function Reset-ResolverMock { $global:DotnetTestResolverVersions = @{} + $global:DotnetTestResolverNuspecs = @{} $global:DotnetTestResolverRestoreRequests = [System.Collections.Generic.List[object]]::new() $global:DotnetTestResolverHttpRequests = [System.Collections.Generic.List[string]]::new() $global:DotnetTestResolverFailureMode = 'Success' @@ -31,6 +32,27 @@ function Set-TestPackageVersions { $global:DotnetTestResolverVersions[$Id.ToLowerInvariant()] = @($Versions) } +function Set-TestPackageNuspec { + param( + [Parameter(Mandatory = $true)] + [string]$Id, + + [Parameter(Mandatory = $true)] + [string]$Version, + + [Parameter(Mandatory = $true)] + [hashtable]$Dependencies + ) + + $entries = (@($Dependencies.GetEnumerator() | Sort-Object Key) | ForEach-Object { + '' -f $_.Key, $_.Value + }) -join '' + # Real nuspecs repeat the same dependency once per target-framework group, so the mock does too. + $groups = (@('net10.0', 'net9.0') | ForEach-Object { '{1}' -f $_, $entries }) -join '' + $xml = '{0}{1}{2}' -f $Id, $Version, $groups + $global:DotnetTestResolverNuspecs[('{0}/{1}' -f $Id.ToLowerInvariant(), $Version.ToLowerInvariant())] = $xml +} + function Invoke-RestMethod { param([Parameter(Mandatory = $true)][string]$Uri) @@ -47,6 +69,15 @@ function Invoke-RestMethod { if ($Uri.StartsWith($packageBaseAddress, [System.StringComparison]::Ordinal)) { $segments = $Uri.TrimEnd('/') -split '/' + if ($Uri.EndsWith('.nuspec', [System.StringComparison]::OrdinalIgnoreCase)) { + $nuspecKey = '{0}/{1}' -f $segments[$segments.Count - 3], $segments[$segments.Count - 2] + if (-not $global:DotnetTestResolverNuspecs.ContainsKey($nuspecKey)) { + throw "Unexpected nuspec lookup: $Uri" + } + + return [xml]$global:DotnetTestResolverNuspecs[$nuspecKey] + } + $id = $segments[$segments.Count - 2] if (-not $global:DotnetTestResolverVersions.ContainsKey($id)) { throw "Unexpected package lookup: $Uri" @@ -108,6 +139,8 @@ function Invoke-TestResolver { [string]$CacheDirectory, + [string]$XunitAnchorVersion, + [switch]$UseDefaultCandidateLimit ) @@ -120,6 +153,8 @@ function Invoke-TestResolver { } else { $output = @(& $resolver -TargetFramework net10.0 -Role Unit -PackageId $PackageId -CacheDirectory $CacheDirectory 2>&1) } + } elseif (-not [string]::IsNullOrWhiteSpace($XunitAnchorVersion)) { + $output = @(& $resolver -TargetFramework net10.0 -Role Unit -PackageId $PackageId -MaximumCandidates $MaximumCandidates -XunitAnchorVersion $XunitAnchorVersion 2>&1) } elseif ([string]::IsNullOrWhiteSpace($CacheDirectory)) { $output = @(& $resolver -TargetFramework net10.0 -Role Unit -PackageId $PackageId -MaximumCandidates $MaximumCandidates 2>&1) } else { @@ -291,10 +326,61 @@ try { Remove-Item Env:DOTNET_TEST_MAXIMUM_CANDIDATES -ErrorAction SilentlyContinue } + # xUnit shipped stable 4.0.0 packages while Codebelt.Extensions.Xunit still declared 3.2.2, so "newest stable" + # silently jumped the test project a whole xUnit generation past the Codebelt API it targets. + Reset-ResolverMock + Set-TestPackageVersions -Id 'Codebelt.Extensions.Xunit' -Versions @('11.2.1', '11.1.0') + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '11.2.1' -Dependencies @{ 'xunit.v3.assert' = '3.2.2'; 'xunit.v3.extensibility.core' = '3.2.2' } + Set-TestPackageVersions -Id 'xunit.v3.assert' -Versions @('4.0.0', '3.3.0', '3.2.2') + Set-TestPackageVersions -Id 'xunit.v3' -Versions @('4.0.0', '3.3.0', '3.2.2') + $anchored = Invoke-TestResolver -PackageId @('Codebelt.Extensions.Xunit', 'xunit.v3', 'xunit.v3.assert') -MaximumCandidates 1 + $anchoredResult = Get-ResolverJsonObject -Text $anchored.text + $anchoredPackages = @($anchoredResult.packages) + Assert-Equal -Actual $anchored.exitCode -Expected 0 -Because 'anchored resolution should succeed' + Assert-Equal -Actual $anchoredResult.xunitAnchor.packageId -Expected 'Codebelt.Extensions.Xunit' -Because 'the unit role must anchor to the Codebelt xUnit package' + Assert-Equal -Actual $anchoredResult.xunitAnchor.major -Expected 3 -Because 'the xUnit ceiling must come from the Codebelt package dependency major' + Assert-Equal -Actual (($anchoredPackages | Where-Object packageId -eq 'xunit.v3.assert').version) -Expected '3.2.2' -Because 'an id the anchor declares must match it 1:1 even when a newer same-major version exists' + Assert-ContainsText -Text (($anchoredPackages | Where-Object packageId -eq 'xunit.v3.assert').constraint) -Expected 'matched 1:1' -Because 'the output must report the 1:1 anchor match' + Assert-Equal -Actual (($anchoredPackages | Where-Object packageId -eq 'xunit.v3').version) -Expected '3.3.0' -Because 'an id the anchor does not declare may take the newest minor or patch below the anchored major' + Assert-Equal -Actual (($anchoredPackages | Where-Object packageId -eq 'Codebelt.Extensions.Xunit').version) -Expected '11.2.1' -Because 'the anchor package itself stays unconstrained' + Assert-True -Condition (@($global:DotnetTestResolverRestoreRequests | Where-Object { @($_.references | Where-Object { $_.version -eq '4.0.0' }).Count -gt 0 }).Count -eq 0) -Because 'no candidate above the anchored major may reach a restore' + + # Anchoring has to precede the candidate trim, otherwise a package whose newest versions are all above the ceiling + # arrives at resolution with nothing left to try. + Reset-ResolverMock + Set-TestPackageVersions -Id 'Codebelt.Extensions.Xunit' -Versions @('11.2.1') + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '11.2.1' -Dependencies @{ 'xunit.v3.assert' = '3.2.2' } + Set-TestPackageVersions -Id 'xunit.v3' -Versions @('4.0.1', '4.0.0', '3.2.2') + $trimmed = Invoke-TestResolver -PackageId @('Codebelt.Extensions.Xunit', 'xunit.v3') -MaximumCandidates 2 + $trimmedResult = Get-ResolverJsonObject -Text $trimmed.text + Assert-Equal -Actual $trimmed.exitCode -Expected 0 -Because 'the candidate limit must apply after the anchored ceiling' + Assert-Equal -Actual ((@($trimmedResult.packages) | Where-Object packageId -eq 'xunit.v3').version) -Expected '3.2.2' -Because 'the newest candidate below the anchored major must survive the candidate trim' + + Reset-ResolverMock + Set-TestPackageVersions -Id 'Codebelt.Extensions.Xunit' -Versions @('11.2.1') + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '11.2.1' -Dependencies @{ 'xunit.v3.assert' = '3.2.2' } + Set-TestPackageVersions -Id 'xunit.runner.visualstudio' -Versions @('4.0.0') + $ceiling = Invoke-TestResolver -PackageId @('Codebelt.Extensions.Xunit', 'xunit.runner.visualstudio') -MaximumCandidates 2 + Assert-True -Condition ($ceiling.exitCode -ne 0) -Because 'an xunit package with no version below the anchored major must fail closed' + Assert-ContainsText -Text $ceiling.text -Expected 'at or below major 3' -Because 'the ceiling failure must name the anchored major' + + # A repository that deliberately pins an older Codebelt xUnit must resolve the xUnit generation that release declared. + Reset-ResolverMock + Set-TestPackageVersions -Id 'Codebelt.Extensions.Xunit' -Versions @('12.0.0', '11.2.1') + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '12.0.0' -Dependencies @{ 'xunit.v3.assert' = '4.0.0' } + Set-TestPackageNuspec -Id 'Codebelt.Extensions.Xunit' -Version '11.2.1' -Dependencies @{ 'xunit.v3.assert' = '3.2.2' } + Set-TestPackageVersions -Id 'xunit.v3' -Versions @('4.0.0', '3.2.2') + $pinned = Invoke-TestResolver -PackageId @('xunit.v3') -MaximumCandidates 1 -XunitAnchorVersion '11.2.1' + $pinnedResult = Get-ResolverJsonObject -Text $pinned.text + Assert-Equal -Actual $pinned.exitCode -Expected 0 -Because 'an explicit anchor version should resolve' + Assert-Equal -Actual $pinnedResult.xunitAnchor.version -Expected '11.2.1' -Because 'the explicit anchor version must be honored over the newest anchor release' + Assert-Equal -Actual ((@($pinnedResult.packages) | Where-Object packageId -eq 'xunit.v3').version) -Expected '3.2.2' -Because 'the ceiling must follow the pinned anchor release rather than the newest one' + Write-Output 'resolve-test-package-versions.ps1 regression: PASS' } finally { foreach ($name in @( 'DotnetTestResolverVersions', + 'DotnetTestResolverNuspecs', 'DotnetTestResolverRestoreRequests', 'DotnetTestResolverHttpRequests', 'DotnetTestResolverFailureMode', From 7dcc121bdd2edcbe484b44387c73437b06cb4a03 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 21:19:15 +0200 Subject: [PATCH 25/31] =?UTF-8?q?=F0=9F=94=A8=20add=20xunit=20anchor=20val?= =?UTF-8?q?idation=20to=20skill=20templates?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validation now confirms that resolve-test-package-versions.ps1 implements xunit anchor resolution with Resolve-XunitAnchor, Select-AnchoredCandidate, and constraint reporting. Also validates that SKILL.md documents the anchor ceiling mechanism and that test-resolve-test-package-versions.ps1 includes regression tests for the anchoring logic. This ensures dotnet-test skill templates remain compliant with xunit generation safety constraints. --- scripts/validate-skill-templates.ps1 | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index b915856..b2696aa 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1284,6 +1284,13 @@ Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'https://api.nuget.org/v3/index.json' Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'Test-PackageCompatibility -Packages $trial' Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'combined restore passed' + # xunit.v3 and xunit.runner.visualstudio shipped stable 4.0.0 releases ahead of Codebelt xUnit; "newest stable" must + # never be allowed to outrun the Codebelt package the skill targets. + Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'Resolve-XunitAnchor -BaseAddress' + Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'Select-AnchoredCandidate -PackageId' + Assert-Contains -Name 'resolve-test-package-versions.ps1' -Content $resolve -Needle 'at or below major' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Newest is not the ceiling for `xunit*`.' + Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'no candidate above the anchored major may reach a restore' Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'stable candidate resolution should succeed' Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'combined package set' Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'restore evidence' From 62dcdf4527ebdb36289ab9a885e13c7b34a2f3c5 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 21:19:19 +0200 Subject: [PATCH 26/31] =?UTF-8?q?=F0=9F=92=AC=20document=20xunit=20anchor?= =?UTF-8?q?=20feature=20in=20release=20notes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated v0.9.0 release notes to describe the xunit anchor mechanism in dotnet-test tooling. The resolver now caps every xunit* package at the major version declared by the Codebelt xUnit release, preventing unanchored NuGet searches from pulling an xUnit generation ahead of the Codebelt API the skill targets. This is a safety constraint that ensures resolved test packages remain compatible with the framework the skill was designed for. --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c674038..ceaa825 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,7 @@ This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-r ### Added - `dotnet-test` skill that bootstraps and refactors xUnit test projects to Codebelt conventions, classifying each selected project as an ordinary unit test, an ASP.NET Core functional test, or a console/worker functional test, then applying the matching focused or shared fixture pattern while preserving test names, lifecycle behavior, package ownership, and target frameworks, -- `dotnet-test` bundled tooling: `inspect-dotnet-tests.ps1` emits machine-readable role, framework, xUnit-generation, inheritance, migration, and blocker evidence before any mutation, and `resolve-test-package-versions.ps1` resolves stable NuGet candidates and proves the combined package set restores across the selected target frameworks, each covered by its own PowerShell regression harness, +- `dotnet-test` bundled tooling: `inspect-dotnet-tests.ps1` emits machine-readable role, framework, xUnit-generation, inheritance, migration, and blocker evidence before any mutation, and `resolve-test-package-versions.ps1` resolves stable NuGet candidates and proves the combined package set restores across the selected target frameworks while anchoring every `xunit*` package to the Codebelt xUnit release — an id that release declares resolves 1:1 to the declared version and every other `xunit*` id stays at or below its major, so a new xUnit generation on NuGet cannot outrun the Codebelt API the skill targets — each covered by its own PowerShell regression harness, - `dotnet-test` assets and reference documentation covering unit-test behavior patterns, focused and shared web-application fixtures, application-focused fixtures, bootstrapper hosts for console and worker services in both minimal and Program/Startup form, xUnit v2-to-v3 modernization, and migration-invariant preservation, - `dotnet-remote-testing` skill that runs .NET tests inside Docker using official `mcr.microsoft.com/dotnet/sdk` images, honoring an existing `testenvironments.json` as authoritative when present and otherwise deriving environments from Microsoft's live release index, while reporting WSL and SSH as unsupported instead of silently falling back to the host, - `dotnet-remote-testing` deterministic runner `remote-test.cs` owning configuration discovery, release parsing, digest-pinned image resolution, isolated source staging, NuGet caching, execution, result parsing, distinct failure classification, and cleanup behind a single entry point, with a built-in `--self-test` alongside a PowerShell harness, From 6e74f4e72b2917679f204075a48bd5563372ff91 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 21:47:03 +0200 Subject: [PATCH 27/31] =?UTF-8?q?=F0=9F=93=9D=20add=20dotnet-test=20migrat?= =?UTF-8?q?ion=20recovery=20documentation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add examples and guidance for catching false-positive migrations where WebApplicationFactory is wrapped, renamed, or repackaged instead of replaced. Update references and add test scenario for the laundered-migration recovery workflow. --- skills/dotnet-test/SKILL.md | 42 ++++++++--- skills/dotnet-test/evals/evals.json | 27 ++++++- .../files/laundered-web/Directory.Build.props | 13 ++++ .../laundered-web/Directory.Packages.props | 11 +++ .../Acme.Cdn.Origin/Acme.Cdn.Origin.csproj | 2 + .../src/Acme.Cdn.Origin/Program.cs | 7 ++ .../Acme.Cdn.Origin.FunctionalTests.csproj | 12 ++++ .../CdnOriginTestApplication.cs | 70 +++++++++++++++++++ .../CompressionTest.cs | 25 +++++++ .../TempContent.cs | 18 +++++ .../application-functional-tests.md | 2 +- .../references/migration-invariants.md | 2 +- .../references/web-functional-tests.md | 2 +- 13 files changed, 220 insertions(+), 13 deletions(-) create mode 100644 skills/dotnet-test/evals/files/laundered-web/Directory.Build.props create mode 100644 skills/dotnet-test/evals/files/laundered-web/Directory.Packages.props create mode 100644 skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj create mode 100644 skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Program.cs create mode 100644 skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj create mode 100644 skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs create mode 100644 skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs create mode 100644 skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs diff --git a/skills/dotnet-test/SKILL.md b/skills/dotnet-test/SKILL.md index 9b0213f..c095854 100644 --- a/skills/dotnet-test/SKILL.md +++ b/skills/dotnet-test/SKILL.md @@ -25,6 +25,19 @@ Microsoft ships `WebApplicationFactory`, and it only covers ASP.NET That substitution is the deliverable. A run that leaves `WebApplicationFactory` in place, or that swaps the type while quietly rebuilding the host in test code, has not done the job no matter how green the test run looks. +### What finishing looks like + +One thing has to be true at the end: the Codebelt abstraction constructs the host, and nothing in the selected project derives from `WebApplicationFactory` any more. How you get there — file names, helper shapes, where settings come from — is yours to choose. + +These rewrites feel like migrations and change nothing: + +- **Wrapping the factory.** Keeping `WebApplicationFactory` as a private nested class, a renamed facade, or a field inside a new `...TestApplication` type. Microsoft's host still starts the application; the wrapper only hides that from the diff. +- **Renaming the seam.** Turning `new CdnOriginTestApplication()` into `CdnOriginTestApplication.Create()` across every test file. Every call site changes and the composition root does not. +- **Importing the namespace.** Adding `using Codebelt.Extensions.Xunit;` without ever calling `WebApplicationTestFactory.Create` or deriving from `WebApplicationTest<,>`. +- **Bumping packages instead.** Raising `xunit*` or unrelated pins produces a busy diff that reads as effort. It is not the deliverable, and moving `xunit*` past the anchor in [Step 3](#step-3-resolve-packages-without-hardcoding-latest) breaks the very API you are migrating onto. + +Every one of these shipped from a real run of this skill and was reported back as a successful migration, which is the point: from inside the run, a wrapper looks like progress, and the tests stay green because the host never changed. That is why [Step 7](#step-7-validate-and-loop) ends in a verdict a script produces rather than a summary you write. Either `WebApplicationTestFactory`, `WebApplicationTest<,>`, `ApplicationTestFactory`, or `ApplicationTest<,>` appears in the project's own source, or the migration did not happen. + ## Do this now **You were invoked. That is the request.** Your first action is the inspector in [Step 1](#step-1-gather-evidence-before-asking-anything) — not a question, not a menu, not a plan. @@ -45,7 +58,8 @@ There is exactly one shape of legitimate question, and it comes *after* the evid - Preserve target frameworks, central package management, unrelated MSBuild configuration, existing test names, and test isolation. - Never take an `xunit*` package past the major the Codebelt xUnit release depends on. The resolver anchors that ceiling in [Step 3](#step-3-resolve-packages-without-hardcoding-latest); newest-on-NuGet is not it. - Edit files you are actually changing, in place. Rewriting a file wholesale flips its line endings and makes `git status` report churn that reviewers must read to discover it means nothing; a file with no semantic change must not appear in the diff at all. -- Replace every selected `WebApplicationFactory` usage. A partial migration that leaves a selected usage or package reference behind is incomplete. +- Replace every selected `WebApplicationFactory` usage. A partial migration that leaves a selected usage or package reference behind is incomplete, and a usage moved inside a wrapper type is still a usage. +- Finish on the verdict from `scripts/verify-dotnet-test-migration.ps1`, quoted as it printed. Reporting a migration complete without it leaves the one claim that matters unverified. - Keep functional testing in-process. Never add a process-launching fallback for console or worker applications. - If a selected executable has no Generic Host, adapt production startup only when application adaptation is explicitly in scope. Otherwise report the exact missing host seam and stop before changing production startup. - During bootstrap, add at least one test derived from real source behavior. Placeholder assertions such as `Assert.True(true)` do not satisfy the task. @@ -192,14 +206,23 @@ Replace every placeholder with repository evidence. Do not invent an endpoint, s ## Step 7: Validate and loop -Run the narrowest authoritative sequence that covers the selected change: +For a migration, run the gate before anything expensive — it is static analysis and it fails in seconds, so there is no reason to spend a restore and a full test run discovering that the host never moved: + +```powershell +pwsh -NoProfile -File "/scripts/verify-dotnet-test-migration.ps1" -RepoRoot "" -ProjectPath "" -ExpectedWebPattern +``` + +Use `-ExpectedApplicationPattern ` instead for a console or worker migration. The gate reruns the inspector under that postcondition and adds the checks that can only exist once the edits do: a type still deriving from `WebApplicationFactory`, a retained `Microsoft.AspNetCore.Mvc.Testing` reference, `xunit*` pins past the major the restored Codebelt package declares, and files changed under the project while the target pattern appears zero times. Exit 0 prints `PASSED`, exit 1 prints `FAILED` with numbered violations and their file and line, and exit 2 means the gate itself could not run. + +Treat `FAILED` as the answer to "is this done", not as advice. Each violation names what to change; fix them and rerun. Do not restate the verdict in your own words, and do not move on to the completion report while it still says `FAILED`. + +Then run the narrowest authoritative sequence that covers the selected change: -1. rerun `inspect-dotnet-tests.ps1`; for a web migration, pass `-ExpectedWebPattern Focused` or `-ExpectedWebPattern Shared`; for a console/worker migration, pass `-ExpectedApplicationPattern Focused` or `-ExpectedApplicationPattern Shared`. These gates require the selected Codebelt pattern with an entrypoint-owned managed fixture and reject legacy factories, deprecated blocking fixtures, and direct replacement-host construction; -2. restore the selected test project; -3. build the selected test project; -4. run `dotnet test` when restore/build/test was requested and confirm the expected non-zero test count with zero failures; a zero-discovery exit code is a failure and `dotnet run` is not a substitute; -5. for migrations, search the selected scope and confirm zero remaining `WebApplicationFactory` usages; do not treat that zero count as sufficient without the expected-pattern postcondition; -6. inspect the final diff for target-framework, package-owner, test-name, and unrelated-change drift. +1. restore the selected test project; +2. build the selected test project; +3. run `dotnet test` when restore/build/test was requested and confirm the expected non-zero test count with zero failures; a zero-discovery exit code is a failure and `dotnet run` is not a substitute; +4. inspect the final diff for target-framework, package-owner, test-name, and unrelated-change drift; +5. rerun the gate as the last action, after the final edit. An earlier `PASSED` describes an earlier state of the files, and the verdict you quote has to describe the ones you are handing over. If tests expose a migration regression, repair the preserved lifecycle or configuration behavior rather than weakening assertions. @@ -212,5 +235,6 @@ Report: - package ownership and resolved versions, including the Codebelt xUnit anchor that bounded the `xunit*` versions; - preserved migration invariants; - behavior test added or existing tests retained; -- exact restore/build/test and zero-usage-search results; +- exact restore/build/test results; +- the verdict block from the final `verify-dotnet-test-migration.ps1` run, pasted as it printed. It is the evidence for the migration claim, so a paraphrase or a remembered result from earlier in the session does not stand in for it; - blockers or validation limits. diff --git a/skills/dotnet-test/evals/evals.json b/skills/dotnet-test/evals/evals.json index 64b8f41..12d5f89 100644 --- a/skills/dotnet-test/evals/evals.json +++ b/skills/dotnet-test/evals/evals.json @@ -55,7 +55,7 @@ "Keeps CompressionTest and both existing method names unchanged", "Raises Codebelt.Extensions.Xunit.App above the 11.1.0 managed-fixture floor in the owning props file, without bumping unrelated pinned packages", "Removes Microsoft.AspNetCore.Mvc.Testing when no longer needed", - "The focused inspector postcondition succeeds, search finds no WebApplicationFactory in the selected migration, and restore/build/test succeed" + "verify-dotnet-test-migration.ps1 with -ExpectedWebPattern Focused reports PASSED, and restore/build/test succeed" ], "files": [ "evals/files/focused-web/Directory.Build.props", @@ -157,6 +157,31 @@ "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs", "evals/files/focused-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs" ] + }, + { + "id": 8, + "prompt": "A previous run reported the attached Acme.Cdn.Origin functional tests as migrated off WebApplicationFactory, but the tests still exercise Microsoft's host. Finish the migration onto the focused Codebelt pattern: CdnOriginTestApplication must stop wrapping WebApplicationFactory, WebApplicationTestFactory.Create with an explicit ManagedWebApplicationFixture must own the host, Microsoft.AspNetCore.Mvc.Testing must go, and the xunit pins must return to the generation the referenced Codebelt release declares. Preserve the Production environment, per-test settings, temporary content ownership, and both existing test names.", + "expected_output": "A focused Test-derived harness owns an IHostTest created through WebApplicationTestFactory with an explicit ManagedWebApplicationFixture, no type derives from WebApplicationFactory, Microsoft.AspNetCore.Mvc.Testing is gone, xunit* is back inside the Codebelt anchor major, and verify-dotnet-test-migration.ps1 reports PASSED.", + "expectations": [ + "Recognizes that the nested private CdnOriginApplicationFactory means the migration never happened, rather than accepting the prior run's report", + "Leaves no type deriving from WebApplicationFactory, including nested, private, and renamed facades", + "Adds WebApplicationTestFactory.Create with an explicit ManagedWebApplicationFixture so Program owns startup", + "Does not keep the static Create factory method as the only change, and does not reconstruct the pipeline with WebApplication.CreateBuilder, UseTestServer, or new TestServer", + "Removes the Microsoft.AspNetCore.Mvc.Testing package reference and its version entry", + "Returns xunit.v3 and xunit.v3.runner.console to the major that Codebelt.Extensions.Xunit.App 11.2.1 declares instead of leaving them on 4.0.0", + "Preserves Production environment, per-test settings, one application per test, temporary content disposal through matching synchronous and asynchronous Test hooks, and both existing method names", + "Runs verify-dotnet-test-migration.ps1 with -ExpectedWebPattern Focused and quotes the PASSED verdict block rather than paraphrasing it" + ], + "files": [ + "evals/files/laundered-web/Directory.Build.props", + "evals/files/laundered-web/Directory.Packages.props", + "evals/files/laundered-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj", + "evals/files/laundered-web/src/Acme.Cdn.Origin/Program.cs", + "evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj", + "evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs", + "evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs", + "evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs" + ] } ] } diff --git a/skills/dotnet-test/evals/files/laundered-web/Directory.Build.props b/skills/dotnet-test/evals/files/laundered-web/Directory.Build.props new file mode 100644 index 0000000..380c35c --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/Directory.Build.props @@ -0,0 +1,13 @@ + + + net10.0 + enable + enable + $(MSBuildProjectName.EndsWith('Tests')) + + + Exe + true + + + diff --git a/skills/dotnet-test/evals/files/laundered-web/Directory.Packages.props b/skills/dotnet-test/evals/files/laundered-web/Directory.Packages.props new file mode 100644 index 0000000..47f3deb --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/Directory.Packages.props @@ -0,0 +1,11 @@ + + true + + + + + + + + + diff --git a/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj b/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj new file mode 100644 index 0000000..dd2327a --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Acme.Cdn.Origin.csproj @@ -0,0 +1,2 @@ + + diff --git a/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Program.cs b/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Program.cs new file mode 100644 index 0000000..93ef5de --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/src/Acme.Cdn.Origin/Program.cs @@ -0,0 +1,7 @@ +var builder = WebApplication.CreateBuilder(args); +var app = builder.Build(); +app.MapGet("/compression", (IConfiguration configuration) => configuration.GetValue("Compression:Enabled", false) ? "br" : "identity"); +app.Run(); + +public partial class Program; + diff --git a/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj new file mode 100644 index 0000000..38670cd --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/Acme.Cdn.Origin.FunctionalTests.csproj @@ -0,0 +1,12 @@ + + Acme.Cdn.Origin + + + + + + + + + + diff --git a/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs new file mode 100644 index 0000000..abd1051 --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CdnOriginTestApplication.cs @@ -0,0 +1,70 @@ +using Codebelt.Extensions.Xunit; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Mvc.Testing; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.Hosting; + +namespace Acme.Cdn.Origin; + +/// +/// Hosts the real origin pipeline over an isolated temporary content directory. +/// Implements the Codebelt managed fixture pattern with entrypoint-owned startup. +/// +public sealed class CdnOriginTestApplication : IAsyncDisposable, IDisposable +{ + private readonly WebApplicationFactory _factory; + private readonly TempContent _content; + + private CdnOriginTestApplication(WebApplicationFactory factory, TempContent content) + { + _factory = factory; + _content = content; + } + + public TempContent Content => _content; + + public static CdnOriginTestApplication Create(IDictionary? settings = null) + { + var content = new TempContent(); + var merged = settings is null ? new Dictionary() : new Dictionary(settings); + return new CdnOriginTestApplication(new CdnOriginApplicationFactory(merged), content); + } + + public HttpClient CreateClient() => _factory.CreateClient(); + + public void Dispose() + { + _factory.Dispose(); + _content.Dispose(); + } + + public async ValueTask DisposeAsync() + { + if (_factory is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + else + { + _factory.Dispose(); + } + + _content.Dispose(); + } + + private sealed class CdnOriginApplicationFactory : WebApplicationFactory + { + private readonly Dictionary _settings; + + public CdnOriginApplicationFactory(Dictionary settings) + { + _settings = settings; + } + + protected override void ConfigureWebHost(IWebHostBuilder builder) + { + builder.UseEnvironment(Environments.Production); + builder.ConfigureAppConfiguration((_, configuration) => configuration.AddInMemoryCollection(_settings)); + } + } +} diff --git a/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs new file mode 100644 index 0000000..46e737a --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/CompressionTest.cs @@ -0,0 +1,25 @@ +using Xunit; + +namespace Acme.Cdn.Origin; + +public class CompressionTest +{ + [Fact] + public async Task Get_ShouldNotCompress_WhenCompressionDisabled() + { + await using var application = CdnOriginTestApplication.Create(); + using var client = application.CreateClient(); + + Assert.Equal("identity", await client.GetStringAsync("/compression")); + } + + [Fact] + public async Task Get_ShouldCompress_WhenEnabled() + { + await using var application = CdnOriginTestApplication.Create(new Dictionary { ["Compression:Enabled"] = "true" }); + using var client = application.CreateClient(); + + Assert.Equal("br", await client.GetStringAsync("/compression")); + } +} + diff --git a/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs new file mode 100644 index 0000000..c70d5fc --- /dev/null +++ b/skills/dotnet-test/evals/files/laundered-web/test/Acme.Cdn.Origin.FunctionalTests/TempContent.cs @@ -0,0 +1,18 @@ +namespace Acme.Cdn.Origin; + +public sealed class TempContent : IDisposable +{ + public TempContent() + { + Root = Path.Combine(Path.GetTempPath(), "acme-cdn-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(Root); + } + + public string Root { get; } + + public void Dispose() + { + if (Directory.Exists(Root)) { Directory.Delete(Root, true); } + } +} + diff --git a/skills/dotnet-test/references/application-functional-tests.md b/skills/dotnet-test/references/application-functional-tests.md index d6c0607..33d2620 100644 --- a/skills/dotnet-test/references/application-functional-tests.md +++ b/skills/dotnet-test/references/application-functional-tests.md @@ -31,7 +31,7 @@ Pass `ManagedApplicationFixture` explicitly for focused tests and use i A repeated focused setup may be encapsulated in a narrow `Test`-derived harness. It must accept `ITestOutputHelper`, retain the `IHostTest`, and dispose that host test plus every owned resource in both the synchronous and asynchronous `Test` disposal hooks. -After migration, run `inspect-dotnet-tests.ps1` with `-ExpectedApplicationPattern Focused` or `-ExpectedApplicationPattern Shared`. A non-zero exit is a migration failure even when restore, build, and tests pass. +After migration, run `verify-dotnet-test-migration.ps1` for the selected project with `-ExpectedApplicationPattern Focused` or `-ExpectedApplicationPattern Shared`. A `FAILED` verdict is a migration failure even when restore, build, and tests all pass, because green tests only prove the host that ran still works, not that it is the one you were asked to move to. ## Host seam gate diff --git a/skills/dotnet-test/references/migration-invariants.md b/skills/dotnet-test/references/migration-invariants.md index 99b9cf4..0b1ff73 100644 --- a/skills/dotnet-test/references/migration-invariants.md +++ b/skills/dotnet-test/references/migration-invariants.md @@ -37,4 +37,4 @@ Prefer focused `WebApplicationTestFactory` ownership when the old test construct For either ownership shape, use the entrypoint-owned `ManagedWebApplicationFixture` and pass it explicitly to factories. Migrate deprecated `BlockingManagedWebApplicationFixture` input; never emit it as a target. -After migration, search the authorized scope. Zero selected `WebApplicationFactory` identifiers is a completion gate, but it is not sufficient by itself: the chosen Codebelt pattern must be present, direct replacement-host construction must be absent, restore/build/test must pass, and lifecycle invariants must still hold. Enforce the web-pattern checks by rerunning `inspect-dotnet-tests.ps1` with `-ExpectedWebPattern Focused` or `-ExpectedWebPattern Shared`. +After migration, search the authorized scope. Zero selected `WebApplicationFactory` identifiers is a completion gate, but it is not sufficient by itself: the chosen Codebelt pattern must be present, direct replacement-host construction must be absent, restore/build/test must pass, and lifecycle invariants must still hold. Enforce all of that with `verify-dotnet-test-migration.ps1` and its `-ExpectedWebPattern` or `-ExpectedApplicationPattern` postcondition; its verdict, not a self-assessment, is what closes the migration. diff --git a/skills/dotnet-test/references/web-functional-tests.md b/skills/dotnet-test/references/web-functional-tests.md index efe1f0b..7340508 100644 --- a/skills/dotnet-test/references/web-functional-tests.md +++ b/skills/dotnet-test/references/web-functional-tests.md @@ -87,4 +87,4 @@ public class HealthTest : WebApplicationTest Date: Tue, 18 Aug 2026 21:47:17 +0200 Subject: [PATCH 28/31] =?UTF-8?q?=E2=9C=85=20implement=20verify-dotnet-tes?= =?UTF-8?q?t-migration=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduce verify-dotnet-test-migration.ps1 as a static-analysis gate that closes migration runs with a PASSED/FAILED verdict. Detects wrapped factories, retained Microsoft.AspNetCore.Mvc.Testing references, xunit version anchoring breaches, and churn-without-conversion patterns. Add test harness with positive-control scenario validating a completed migration. --- .../test-verify-dotnet-test-migration.ps1 | 160 +++++++++++ skills/dotnet-test/scripts/validate-skill.ps1 | 15 +- .../scripts/verify-dotnet-test-migration.ps1 | 269 ++++++++++++++++++ 3 files changed, 440 insertions(+), 4 deletions(-) create mode 100644 skills/dotnet-test/scripts/test-verify-dotnet-test-migration.ps1 create mode 100644 skills/dotnet-test/scripts/verify-dotnet-test-migration.ps1 diff --git a/skills/dotnet-test/scripts/test-verify-dotnet-test-migration.ps1 b/skills/dotnet-test/scripts/test-verify-dotnet-test-migration.ps1 new file mode 100644 index 0000000..1c46ae0 --- /dev/null +++ b/skills/dotnet-test/scripts/test-verify-dotnet-test-migration.ps1 @@ -0,0 +1,160 @@ +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) +$workspace = Join-Path ([System.IO.Path]::GetTempPath()) ('dotnet-test-verify-' + [Guid]::NewGuid().ToString('N')) + +function Write-File { + param([string]$Path, [string]$Content) + $directory = Split-Path -Path $Path -Parent + if (-not (Test-Path -LiteralPath $directory)) { New-Item -ItemType Directory -Path $directory -Force | Out-Null } + [System.IO.File]::WriteAllText($Path, $Content, $utf8NoBom) +} + +$gate = Join-Path $PSScriptRoot 'verify-dotnet-test-migration.ps1' +$projectPath = 'test/App.FunctionalTests/App.FunctionalTests.csproj' + +function Invoke-Gate { + $output = @(& pwsh -NoProfile -File $gate -RepoRoot $workspace -ProjectPath $projectPath -ExpectedWebPattern Focused 2>&1) + return [pscustomobject]@{ + exitCode = $LASTEXITCODE + text = ($output -join [Environment]::NewLine) + } +} + +function Assert-Codes { + param([string]$Scenario, [object]$Run, [int]$ExpectedExit, [string[]]$Expected = @(), [string[]]$Forbidden = @()) + + if ($Run.exitCode -ne $ExpectedExit) { + throw "[$Scenario] expected exit $ExpectedExit, found $($Run.exitCode).`n$($Run.text)" + } + foreach ($code in $Expected) { + if ($Run.text -notmatch [regex]::Escape("[$code]")) { throw "[$Scenario] expected violation $code.`n$($Run.text)" } + } + foreach ($code in $Forbidden) { + if ($Run.text -match [regex]::Escape("[$code]")) { throw "[$Scenario] did not expect $code.`n$($Run.text)" } + } +} + +$packagesPath = Join-Path $workspace 'Directory.Packages.props' +$testProjectPath = Join-Path $workspace 'test/App.FunctionalTests/App.FunctionalTests.csproj' +$harnessPath = Join-Path $workspace 'test/App.FunctionalTests/AppTestApplication.cs' +$assetsPath = Join-Path $workspace 'test/App.FunctionalTests/obj/project.assets.json' + +$anchoredPackages = @' +true +'@ + +$legacyProjectReferences = @' +net10.0true +'@ + +$migratedProjectReferences = @' +net10.0true +'@ + +# The exact shape the failed web-cdn-origin run produced: the legacy factory survives as a private +# nested class behind a renamed facade, so every test file changes while the host never moves. +$launderedHarness = @' +using Microsoft.AspNetCore.Mvc.Testing; + +public sealed class AppTestApplication : IDisposable +{ + private readonly WebApplicationFactory _factory; + + private AppTestApplication(WebApplicationFactory factory) { _factory = factory; } + + public static AppTestApplication Create() => new AppTestApplication(new AppApplicationFactory()); + + public HttpClient CreateClient() => _factory.CreateClient(); + + public void Dispose() { _factory.Dispose(); } + + private sealed class AppApplicationFactory : WebApplicationFactory + { + } +} +'@ + +$migratedHarness = @' +using Codebelt.Extensions.Xunit; +using Codebelt.Extensions.Xunit.Hosting; +using Codebelt.Extensions.Xunit.Hosting.AspNetCore; + +public class HealthTest : Test +{ + private readonly IHostTest _application = WebApplicationTestFactory.Create(hostFixture: new ManagedWebApplicationFixture()); + + protected override void OnDisposeManagedResources() { _application.Dispose(); base.OnDisposeManagedResources(); } + + protected override async ValueTask OnDisposeManagedResourcesAsync() { await _application.DisposeAsync(); await base.OnDisposeManagedResourcesAsync(); } +} +'@ + +function Write-Assets { + param([string]$XunitAssertVersion = '3.2.2') + Write-File -Path $assetsPath -Content ('{"version":3,"targets":{"net10.0":{"Codebelt.Extensions.Xunit.App/11.2.1":{"type":"package","dependencies":{"Codebelt.Extensions.Xunit":"11.2.1","xunit.v3.assert":"' + $XunitAssertVersion + '","xunit.v3.extensibility.core":"' + $XunitAssertVersion + '"}}}},"libraries":{}}') +} + +New-Item -ItemType Directory -Path $workspace -Force | Out-Null +try { + Write-File -Path (Join-Path $workspace 'app/App.csproj') -Content @' +net10.0Exe +'@ + Write-File -Path (Join-Path $workspace 'app/Program.cs') -Content @' +public class Program { public static void Main(string[] args) { var builder = WebApplication.CreateBuilder(args); builder.Build().Run(); } } +'@ + Write-File -Path $packagesPath -Content $anchoredPackages + Write-File -Path $testProjectPath -Content $legacyProjectReferences + Write-File -Path $harnessPath -Content $launderedHarness + Write-Assets + + # A repository with no git history must not crash the churn check; it simply has nothing to compare. + $noGit = Invoke-Gate + Assert-Codes -Scenario 'laundered facade without git' -Run $noGit -ExpectedExit 1 ` + -Expected @('LAUNDERED-FACADE', 'WAF-RETAINED', 'PATTERN-MISSING', 'FIXTURE-MISSING', 'LEGACY-PACKAGE-RETAINED') ` + -Forbidden @('CHURN-WITHOUT-CONVERSION') + + # `git init` alone is enough to make every file report as untracked, which is what the churn + # check reads. No commit, and therefore no identity configuration, is involved. + & git -C $workspace init --quiet 2>&1 | Out-Null + $laundered = Invoke-Gate + Assert-Codes -Scenario 'laundered facade' -Run $laundered -ExpectedExit 1 -Expected @('LAUNDERED-FACADE', 'CHURN-WITHOUT-CONVERSION') + if ($laundered.text -notmatch 'result\s+:\s+FAILED') { throw "[laundered facade] expected a FAILED verdict line.`n$($laundered.text)" } + + # Positive control: a real migration has to pass, otherwise the gate is noise rather than signal. + Remove-Item -LiteralPath $harnessPath -Force + Write-File -Path (Join-Path $workspace 'test/App.FunctionalTests/HealthTest.cs') -Content $migratedHarness + Write-File -Path $testProjectPath -Content $migratedProjectReferences + $migrated = Invoke-Gate + Assert-Codes -Scenario 'completed migration' -Run $migrated -ExpectedExit 0 ` + -Forbidden @('LAUNDERED-FACADE', 'WAF-RETAINED', 'PATTERN-MISSING', 'FIXTURE-MISSING', 'LEGACY-PACKAGE-RETAINED', 'CHURN-WITHOUT-CONVERSION', 'XUNIT-ANCHOR-BREACH') + if ($migrated.text -notmatch 'result\s+:\s+PASSED') { throw "[completed migration] expected a PASSED verdict line.`n$($migrated.text)" } + if ($migrated.text -notmatch 'anchor\s+:\s+Codebelt\.Extensions\.Xunit\.App 11\.2\.1') { throw "[completed migration] expected the resolved anchor in the verdict.`n$($migrated.text)" } + + # Bumping an unanchored xunit id past the anchor major is the version-drift half of the same slop. + Write-File -Path $packagesPath -Content ($anchoredPackages -replace 'Include="xunit.v3" Version="3.2.2"', 'Include="xunit.v3" Version="4.0.0"') + $anchorBreach = Invoke-Gate + Assert-Codes -Scenario 'xunit major breach' -Run $anchorBreach -ExpectedExit 1 -Expected @('XUNIT-ANCHOR-BREACH') + if ($anchorBreach.text -notmatch 'xunit\.v3 is pinned to 4\.0\.0') { throw "[xunit major breach] expected the offending id and version.`n$($anchorBreach.text)" } + + # An id the anchor names itself has to match exactly, not merely stay inside the major. + Write-File -Path $packagesPath -Content ($anchoredPackages -replace 'Include="xunit.v3.assert" Version="3.2.2"', 'Include="xunit.v3.assert" Version="3.1.0"') + $exactBreach = Invoke-Gate + Assert-Codes -Scenario 'anchored id drift' -Run $exactBreach -ExpectedExit 1 -Expected @('XUNIT-ANCHOR-BREACH') + if ($exactBreach.text -notmatch 'declares 3\.2\.2') { throw "[anchored id drift] expected the declared anchor version.`n$($exactBreach.text)" } + + # Without a restored anchor the versions are unproven, which is a warning about missing evidence + # rather than a violation: reporting an unverifiable breach would be a guess. + Write-File -Path $packagesPath -Content $anchoredPackages + Remove-Item -LiteralPath $assetsPath -Force + $unverified = Invoke-Gate + Assert-Codes -Scenario 'unrestored anchor' -Run $unverified -ExpectedExit 0 -Forbidden @('XUNIT-ANCHOR-BREACH') + if ($unverified.text -notmatch 'XUNIT-ANCHOR-UNVERIFIED') { throw "[unrestored anchor] expected the unverified warning.`n$($unverified.text)" } + + $missingExpectation = @(& pwsh -NoProfile -File $gate -RepoRoot $workspace -ProjectPath $projectPath 2>&1) + if ($LASTEXITCODE -ne 2) { throw "Expected a usage error without an expected pattern, found $LASTEXITCODE.`n$($missingExpectation -join [Environment]::NewLine)" } + + Write-Output 'verify-dotnet-test-migration.ps1 checks passed.' +} finally { + if (Test-Path -LiteralPath $workspace) { Remove-Item -LiteralPath $workspace -Recurse -Force -ErrorAction SilentlyContinue } +} diff --git a/skills/dotnet-test/scripts/validate-skill.ps1 b/skills/dotnet-test/scripts/validate-skill.ps1 index b706269..2464bd1 100644 --- a/skills/dotnet-test/scripts/validate-skill.ps1 +++ b/skills/dotnet-test/scripts/validate-skill.ps1 @@ -4,7 +4,8 @@ $skillRoot = (Resolve-Path (Join-Path $PSScriptRoot '..')).Path $required = @( 'SKILL.md', 'FORMS.md', 'evals/evals.json', - 'scripts/inspect-dotnet-tests.ps1', 'scripts/resolve-test-package-versions.ps1', 'scripts/test-inspect-dotnet-tests.ps1', 'scripts/test-resolve-test-package-versions.ps1', + 'scripts/inspect-dotnet-tests.ps1', 'scripts/resolve-test-package-versions.ps1', 'scripts/verify-dotnet-test-migration.ps1', + 'scripts/test-inspect-dotnet-tests.ps1', 'scripts/test-resolve-test-package-versions.ps1', 'scripts/test-verify-dotnet-test-migration.ps1', 'references/unit-tests.md', 'references/web-functional-tests.md', 'references/application-functional-tests.md', 'references/bootstrapper-hosts.md', 'references/xunit-v3-modernization.md', 'references/migration-invariants.md', 'assets/unit/BehaviorTest.cs', 'assets/web/FocusedWebApplicationTest.cs', 'assets/web/SharedWebApplicationTest.cs', @@ -20,20 +21,26 @@ foreach ($relative in $required) { } $skill = [System.IO.File]::ReadAllText((Join-Path $skillRoot 'SKILL.md')) -foreach ($needle in @('WebApplicationTestFactory', 'ApplicationTestFactory', 'ManagedWebApplicationFixture', 'ManagedApplicationFixture', 'zero remaining `WebApplicationFactory`', '-ExpectedWebPattern', '-ExpectedApplicationPattern', 'second composition root')) { +foreach ($needle in @('WebApplicationTestFactory', 'ApplicationTestFactory', 'ManagedWebApplicationFixture', 'ManagedApplicationFixture', 'verify-dotnet-test-migration.ps1', '-ExpectedWebPattern', '-ExpectedApplicationPattern', 'second composition root')) { if (-not $skill.Contains($needle, [System.StringComparison]::Ordinal)) { throw "SKILL.md is missing required contract: $needle" } } if (-not $skill.Contains('An MTP executable run may supplement that gate but never replaces it', [System.StringComparison]::Ordinal)) { throw 'SKILL.md must reject MTP executable substitution for requested dotnet test validation.' } +foreach ($needle in @('What finishing looks like', 'Wrapping the factory', 'Renaming the seam', 'Bumping packages instead')) { + if (-not $skill.Contains($needle, [System.StringComparison]::Ordinal)) { throw "SKILL.md must keep the named laundering failure mode: $needle" } +} & pwsh -NoProfile -File (Join-Path $PSScriptRoot 'test-resolve-test-package-versions.ps1') if ($LASTEXITCODE -ne 0) { throw "Resolver regression failed with exit code $LASTEXITCODE." } $evals = [System.IO.File]::ReadAllText((Join-Path $skillRoot 'evals/evals.json')) -foreach ($needle in @('WebApplicationTestFactory.Create', 'ManagedWebApplicationFixture', 'WebApplication.CreateBuilder', 'focused inspector postcondition')) { - if (-not $evals.Contains($needle, [System.StringComparison]::Ordinal)) { throw "Focused-web eval is missing regression contract: $needle" } +foreach ($needle in @('WebApplicationTestFactory.Create', 'ManagedWebApplicationFixture', 'WebApplication.CreateBuilder', 'verify-dotnet-test-migration.ps1', 'nested private CdnOriginApplicationFactory')) { + if (-not $evals.Contains($needle, [System.StringComparison]::Ordinal)) { throw "Web eval is missing regression contract: $needle" } } & pwsh -NoProfile -File (Join-Path $PSScriptRoot 'test-inspect-dotnet-tests.ps1') if ($LASTEXITCODE -ne 0) { throw "Inspection regression failed with exit code $LASTEXITCODE." } +& pwsh -NoProfile -File (Join-Path $PSScriptRoot 'test-verify-dotnet-test-migration.ps1') +if ($LASTEXITCODE -ne 0) { throw "Migration gate regression failed with exit code $LASTEXITCODE." } + Write-Host 'dotnet-test skill validation: PASS' diff --git a/skills/dotnet-test/scripts/verify-dotnet-test-migration.ps1 b/skills/dotnet-test/scripts/verify-dotnet-test-migration.ps1 new file mode 100644 index 0000000..3d684dc --- /dev/null +++ b/skills/dotnet-test/scripts/verify-dotnet-test-migration.ps1 @@ -0,0 +1,269 @@ +<# +.SYNOPSIS + Blocking completion gate for dotnet-test migrations. + +.DESCRIPTION + Answers one question with evidence instead of narration: did this run actually move the selected + test project onto the Codebelt entrypoint-owned host, or did it only rearrange code around the + host it was supposed to replace? + + The inspector already knows how to recognize the target pattern, so this wraps + inspect-dotnet-tests.ps1 rather than reimplementing its regexes, then adds the checks that only + make sense after the edits exist: the xUnit anchor the resolver established, retained legacy + packages, laundered WebApplicationFactory facades, and edits that produced churn without + conversion. It renders one verdict a reviewer can read without parsing JSON. +#> +param( + [string]$RepoRoot = (Get-Location).Path, + [Parameter(Mandatory)] + [string]$ProjectPath, + [ValidateSet('Focused', 'Shared')] + [string]$ExpectedWebPattern, + [ValidateSet('Focused', 'Shared')] + [string]$ExpectedApplicationPattern, + [switch]$Json +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' +$utf8NoBom = [System.Text.UTF8Encoding]::new($false) +[Console]::OutputEncoding = $utf8NoBom +$OutputEncoding = $utf8NoBom + +# Usage problems exit 2 so a caller can tell "the gate could not run" from "the migration failed". +# Write-Error would terminate under the Stop preference above and surface as exit 1, collapsing that +# distinction into the failure code. +function Exit-WithUsageError { + param([string]$Message) + [Console]::Error.WriteLine($Message) + exit 2 +} + +if ([string]::IsNullOrWhiteSpace($ExpectedWebPattern) -and [string]::IsNullOrWhiteSpace($ExpectedApplicationPattern)) { + Exit-WithUsageError 'Specify -ExpectedWebPattern or -ExpectedApplicationPattern. The gate verifies a named target pattern; without one there is nothing to verify.' +} +if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern) -and -not [string]::IsNullOrWhiteSpace($ExpectedApplicationPattern)) { + Exit-WithUsageError 'ExpectedWebPattern and ExpectedApplicationPattern are mutually exclusive.' +} + +$violations = [System.Collections.Generic.List[object]]::new() +$warnings = [System.Collections.Generic.List[object]]::new() + +function Add-Violation { + param([string]$Code, [string]$Message, [string]$Evidence) + $violations.Add([pscustomobject]@{ code = $Code; message = $Message; evidence = $Evidence }) +} + +function Add-Warning { + param([string]$Code, [string]$Message, [string]$Evidence) + $warnings.Add([pscustomobject]@{ code = $Code; message = $Message; evidence = $Evidence }) +} + +function Get-MajorVersion { + param([string]$Version) + if ([string]::IsNullOrWhiteSpace($Version)) { return $null } + $core = ($Version -split '-', 2)[0] + $first = ($core -split '\.')[0] + $parsed = 0 + if ([int]::TryParse($first, [ref]$parsed)) { return $parsed } + return $null +} + +$repoRootPath = (Resolve-Path -LiteralPath $RepoRoot).Path +$scriptRoot = Split-Path -Path $PSCommandPath -Parent +$inspector = Join-Path $scriptRoot 'inspect-dotnet-tests.ps1' +if (-not (Test-Path -LiteralPath $inspector -PathType Leaf)) { + Exit-WithUsageError "The inspector was not found next to this gate: $inspector" +} + +# --- Run the inspector under the expected-pattern postcondition ------------------------------- +$inspectorArguments = @('-NoProfile', '-File', $inspector, '-RepoRoot', $repoRootPath, '-ProjectPath', $ProjectPath) +if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern)) { $inspectorArguments += @('-ExpectedWebPattern', $ExpectedWebPattern) } +if (-not [string]::IsNullOrWhiteSpace($ExpectedApplicationPattern)) { $inspectorArguments += @('-ExpectedApplicationPattern', $ExpectedApplicationPattern) } + +$inspectorOutput = @(& pwsh @inspectorArguments 2>&1) +$inspectorExit = $LASTEXITCODE +$inspectorText = ($inspectorOutput -join [Environment]::NewLine) +$report = $null +try { + $report = ($inspectorText | ConvertFrom-Json).projects[0] +} catch { + Write-Output '================ DOTNET-TEST MIGRATION VERDICT ================' + Write-Output "project : $ProjectPath" + Write-Output 'result : ERROR - the inspector did not return parseable JSON' + Write-Output '' + Write-Output $inspectorText + Write-Output '===============================================================' + exit 2 +} + +foreach ($blocker in @($report.blockers)) { + $code = switch -Regex ($blocker) { + 'still contains WebApplicationFactory' { 'WAF-RETAINED'; break } + 'constructs a replacement host' { 'REPLACEMENT-HOST'; break } + 'deprecated Blocking' { 'BLOCKING-FIXTURE'; break } + 'must explicitly use Managed' { 'FIXTURE-MISSING'; break } + 'does not dispose it through both' { 'DISPOSAL-INCOMPLETE'; break } + 'postcondition requires' { 'PATTERN-MISSING'; break } + 'cannot be applied because' { 'ROLE-MISMATCH'; break } + default { 'INSPECTOR-BLOCKER' } + } + Add-Violation -Code $code -Message $blocker -Evidence 'inspect-dotnet-tests.ps1' +} + +# --- Laundered facade ------------------------------------------------------------------------- +# Wrapping the legacy factory in a new type - a private nested subclass, a renamed facade, a +# constructor turned into a static Create - keeps the Microsoft host in charge while the diff looks +# like a migration. Name it separately from the generic retained-usage blocker so the report says +# what actually happened rather than leaving the reader to infer it from a line number. +foreach ($declaration in @($report.inheritance)) { + if ($declaration.baseTypes -match '\bWebApplicationFactory\s*<') { + Add-Violation -Code 'LAUNDERED-FACADE' ` + -Message "Type '$($declaration.type)' still derives from WebApplicationFactory. Wrapping, nesting, or renaming the legacy factory keeps Microsoft's host in charge; the deliverable is that the Codebelt abstraction owns the host instead." ` + -Evidence "$($declaration.path):$($declaration.line)" + } +} + +# --- Retained legacy packages ----------------------------------------------------------------- +$legacyWebPackages = @('Microsoft.AspNetCore.Mvc.Testing') +if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern)) { + foreach ($package in @($report.packageOwnership)) { + if ($legacyWebPackages -notcontains $package.id) { continue } + Add-Violation -Code 'LEGACY-PACKAGE-RETAINED' ` + -Message "$($package.id) is still referenced. It exists to supply WebApplicationFactory; keeping it after the migration leaves the replaced host one using directive away from returning." ` + -Evidence "$($package.referenceOwner) (version owner: $($package.versionOwner))" + } +} + +# --- xUnit anchor breach ---------------------------------------------------------------------- +# The resolver anchors xunit* to the Codebelt release in use. Nothing re-checks that after the +# edits, so a well-meant "bump everything to latest" can silently push the project a whole xUnit +# generation past the API it was migrated onto. project.assets.json records the anchor's own +# declared dependencies, which makes this verifiable offline from what actually restored. +$projectFullPath = if ([System.IO.Path]::IsPathRooted($ProjectPath)) { $ProjectPath } else { Join-Path $repoRootPath $ProjectPath } +$assetsPath = Join-Path (Split-Path -Path $projectFullPath -Parent) 'obj/project.assets.json' +$anchorId = $null +$anchorVersion = $null +$anchorMajor = $null +$anchorDeclared = @{} +if (Test-Path -LiteralPath $assetsPath -PathType Leaf) { + try { + $assets = [System.IO.File]::ReadAllText($assetsPath, $utf8NoBom) | ConvertFrom-Json + foreach ($targetProperty in $assets.targets.PSObject.Properties) { + foreach ($libraryProperty in $targetProperty.Value.PSObject.Properties) { + if ($libraryProperty.Name -notmatch '^Codebelt\.Extensions\.Xunit(?:\.App)?/(?.+)$') { continue } + $library = $libraryProperty.Value + if ($null -eq $library.PSObject.Properties['dependencies']) { continue } + foreach ($dependency in $library.dependencies.PSObject.Properties) { + if ($dependency.Name -notlike 'xunit*') { continue } + $anchorId = ($libraryProperty.Name -split '/')[0] + $anchorVersion = $Matches['version'] + $anchorDeclared[$dependency.Name] = [string]$dependency.Value + } + } + } + } catch { + Add-Warning -Code 'XUNIT-ANCHOR-UNREADABLE' -Message "project.assets.json could not be parsed, so the xUnit anchor was not verified: $($_.Exception.Message)" -Evidence $assetsPath + } +} + +if ($anchorDeclared.Count -gt 0) { + $anchorMajor = Get-MajorVersion -Version (@($anchorDeclared.Values)[0]) + foreach ($package in @($report.packageOwnership)) { + if ($package.id -notlike 'xunit*') { continue } + $major = Get-MajorVersion -Version $package.version + if ($null -eq $major) { continue } + if ($anchorDeclared.ContainsKey($package.id)) { + $expected = $anchorDeclared[$package.id] + if ($package.version -ne $expected) { + Add-Violation -Code 'XUNIT-ANCHOR-BREACH' ` + -Message "$($package.id) is pinned to $($package.version) but $anchorId $anchorVersion declares $expected. An id the anchor names resolves 1:1 to the version it declares." ` + -Evidence $package.versionOwner + } + } elseif ($null -ne $anchorMajor -and $major -gt $anchorMajor) { + Add-Violation -Code 'XUNIT-ANCHOR-BREACH' ` + -Message "$($package.id) is pinned to $($package.version), past major $anchorMajor of the $anchorId $anchorVersion anchor. Newest-on-NuGet is not the ceiling; the Codebelt package has to move to the next xUnit generation first." ` + -Evidence $package.versionOwner + } + } +} else { + Add-Warning -Code 'XUNIT-ANCHOR-UNVERIFIED' ` + -Message 'No restored Codebelt.Extensions.Xunit anchor was found, so xunit* versions were not bounded. Restore the project and rerun this gate to verify them.' ` + -Evidence $assetsPath +} + +# --- Churn without conversion ----------------------------------------------------------------- +# The most honest single signal that a run produced motion instead of migration: files under the +# selected project changed, yet not one line of the target pattern exists. Renaming a constructor +# to a static factory method reads as progress in a summary and as nothing at all in a diff. +$patternUsageCount = if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern)) { + @($report.focusedWebApplicationTestFactoryUsages).Count + @($report.sharedWebApplicationTestUsages).Count +} else { + @($report.focusedApplicationTestFactoryUsages).Count + @($report.sharedApplicationTestUsages).Count +} +$projectDirectoryRelative = Split-Path -Path $report.project -Parent +if ($patternUsageCount -eq 0 -and -not [string]::IsNullOrWhiteSpace($projectDirectoryRelative)) { + $changed = @() + try { + $changed = @(& git -C $repoRootPath status --porcelain -- $projectDirectoryRelative 2>$null | Where-Object { $_ -notmatch '[\\/](bin|obj)[\\/]' }) + } catch { + $changed = @() + } + if ($changed.Count -gt 0) { + Add-Violation -Code 'CHURN-WITHOUT-CONVERSION' ` + -Message "$($changed.Count) file(s) under the selected project changed, yet the target pattern appears zero times. Edits that rename, wrap, or reformat the existing host produce a reviewable diff without performing the migration." ` + -Evidence (($changed | Select-Object -First 8) -join '; ') + } +} + +# --- Verdict ---------------------------------------------------------------------------------- +$result = if ($violations.Count -eq 0) { 'PASSED' } else { 'FAILED' } +$expectation = if (-not [string]::IsNullOrWhiteSpace($ExpectedWebPattern)) { + "$ExpectedWebPattern ASP.NET Core web pattern" +} else { + "$ExpectedApplicationPattern console/worker application pattern" +} + +Write-Output '================ DOTNET-TEST MIGRATION VERDICT ================' +Write-Output "project : $($report.project)" +Write-Output "role : $($report.role)" +Write-Output "expected : $expectation" +if ($null -ne $anchorId) { Write-Output "anchor : $anchorId $anchorVersion (xunit major $anchorMajor)" } +Write-Output "result : $result ($($violations.Count) violation(s), $($warnings.Count) warning(s))" +if ($violations.Count -gt 0) { + Write-Output '' + Write-Output 'VIOLATIONS' + $index = 1 + foreach ($violation in $violations) { + Write-Output (" {0}. [{1}] {2}" -f $index, $violation.code, $violation.message) + Write-Output (" evidence: {0}" -f $violation.evidence) + $index++ + } +} +if ($warnings.Count -gt 0) { + Write-Output '' + Write-Output 'WARNINGS' + $index = 1 + foreach ($warning in $warnings) { + Write-Output (" {0}. [{1}] {2}" -f $index, $warning.code, $warning.message) + Write-Output (" evidence: {0}" -f $warning.evidence) + $index++ + } +} +Write-Output '===============================================================' + +if ($Json) { + Write-Output ([ordered]@{ + project = $report.project + role = $report.role + expected = $expectation + result = $result + inspectorExitCode = $inspectorExit + xunitAnchor = [ordered]@{ id = $anchorId; version = $anchorVersion; major = $anchorMajor; declared = $anchorDeclared } + violations = @($violations) + warnings = @($warnings) + } | ConvertTo-Json -Depth 6) +} + +if ($violations.Count -gt 0) { exit 1 } +exit 0 From f7f6e9d338e36376b5b21f68ccadd45067a41c80 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 21:47:30 +0200 Subject: [PATCH 29/31] =?UTF-8?q?=F0=9F=94=A7=20add=20dotnet-test=20migrat?= =?UTF-8?q?ion=20gate=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enhance repository validator to check for verify-dotnet-test-migration.ps1 presence and coverage of migration-failure modes: wrapped/nested factories, renamed facades, package-bump-only churn. Require the minimum test scenario count of 8, including the laundered-migration recovery scenario with positive control validation. --- scripts/validate-skill-templates.ps1 | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index b2696aa..01763f2 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -1249,7 +1249,20 @@ Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'ApplicationTest>' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Never emit them in generated or refactored code.' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Never add a process-launching fallback' - Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'zero remaining `WebApplicationFactory`' + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'verify-dotnet-test-migration.ps1' + # A wrapped, renamed, or repackaged factory once shipped as a completed migration. The named failure + # modes and the script-produced verdict are what stop that from reading as success again. + foreach ($needle in @('What finishing looks like', 'Wrapping the factory', 'Renaming the seam', 'Bumping packages instead', 'a verdict a script produces rather than a summary you write')) { + Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle $needle + } + $verify = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-test/scripts/verify-dotnet-test-migration.ps1' -GitRef $Ref + $verifyTest = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-test/scripts/test-verify-dotnet-test-migration.ps1' -GitRef $Ref + foreach ($needle in @('LAUNDERED-FACADE', 'LEGACY-PACKAGE-RETAINED', 'XUNIT-ANCHOR-BREACH', 'CHURN-WITHOUT-CONVERSION', 'project.assets.json')) { + Assert-Contains -Name 'verify-dotnet-test-migration.ps1' -Content $verify -Needle $needle + } + # A gate that only ever fails is noise, so the positive control is part of the contract. + Assert-Contains -Name 'test-verify-dotnet-test-migration.ps1' -Content $verifyTest -Needle 'Positive control' + Assert-Contains -Name 'test-verify-dotnet-test-migration.ps1' -Content $verifyTest -Needle 'completed migration' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'Do not invent an endpoint, service, configuration key, or expected result.' Assert-Contains -Name 'dotnet-test/SKILL.md' -Content $skill -Needle 'An MTP executable run may supplement that gate but never replaces it' # The skill once answered a bare invocation with a capability menu and inspected nothing; these lock the evidence-first contract. @@ -1296,8 +1309,8 @@ Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific Assert-Contains -Name 'test-resolve-test-package-versions.ps1' -Content $resolveTest -Needle 'restore evidence' $evalObject = $evals | ConvertFrom-Json - if (@($evalObject.evals).Count -lt 7) { - throw "dotnet-test must define the six paired role scenarios plus the bare-invocation immediate-action scenario; found $(@($evalObject.evals).Count)" + if (@($evalObject.evals).Count -lt 8) { + throw "dotnet-test must define the six paired role scenarios, the bare-invocation immediate-action scenario, and the laundered-migration recovery scenario; found $(@($evalObject.evals).Count)" } foreach ($needle in @('attached Acme.Calculator fixture', 'xUnit v2 project', 'web-cdn-origin-style', 'IClassFixture>', 'ApplicationTestFactory pattern', 'ApplicationTest>')) { Assert-Contains -Name 'dotnet-test/evals/evals.json' -Content $evals -Needle $needle @@ -1306,6 +1319,11 @@ Add-ValidationResult -Results $results -Name 'dotnet-test encodes role-specific foreach ($needle in @('Does not present a numbered menu of modes', 'Runs inspect-dotnet-tests.ps1 as the first action')) { Assert-Contains -Name 'dotnet-test/evals/evals.json' -Content $evals -Needle $needle } + # A run once wrapped WebApplicationFactory in a private nested class and reported the migration done; + # the recovery scenario keeps that exact outcome in the eval set rather than only in a postmortem. + foreach ($needle in @('nested private CdnOriginApplicationFactory', 'including nested, private, and renamed facades', 'verify-dotnet-test-migration.ps1')) { + Assert-Contains -Name 'dotnet-test/evals/evals.json' -Content $evals -Needle $needle + } if (@($fixtureFiles | Where-Object { $_ -match '(^|[\\/])(bin|obj)([\\/]|$)' }).Count -gt 0) { throw 'dotnet-test eval fixtures must not include bin/ or obj/ paths' } From 9051b4b324341344fd1d6a9846c549ecf19863f5 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 21:47:41 +0200 Subject: [PATCH 30/31] =?UTF-8?q?=F0=9F=92=AC=20update=20changelog=20for?= =?UTF-8?q?=20dotnet-test=20migration=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add v0.9.0 release notes documenting the verify-dotnet-test-migration.ps1 gate that produces a PASSED/FAILED verdict, validates the expected pattern postcondition, and catches wrapped/renamed factories, retained testing packages, xunit anchor breaches, and churn-without-conversion patterns. --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ceaa825..066081d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ This is a minor release that adds three .NET skills — `dotnet-test`, `dotnet-r - `dotnet-test` skill that bootstraps and refactors xUnit test projects to Codebelt conventions, classifying each selected project as an ordinary unit test, an ASP.NET Core functional test, or a console/worker functional test, then applying the matching focused or shared fixture pattern while preserving test names, lifecycle behavior, package ownership, and target frameworks, - `dotnet-test` bundled tooling: `inspect-dotnet-tests.ps1` emits machine-readable role, framework, xUnit-generation, inheritance, migration, and blocker evidence before any mutation, and `resolve-test-package-versions.ps1` resolves stable NuGet candidates and proves the combined package set restores across the selected target frameworks while anchoring every `xunit*` package to the Codebelt xUnit release — an id that release declares resolves 1:1 to the declared version and every other `xunit*` id stays at or below its major, so a new xUnit generation on NuGet cannot outrun the Codebelt API the skill targets — each covered by its own PowerShell regression harness, +- `dotnet-test` migration gate `verify-dotnet-test-migration.ps1`, which closes a migration with a `PASSED`/`FAILED` verdict rather than a self-assessment: it reruns the inspector under the expected focused or shared postcondition and adds the checks that only exist once the edits do — a type still deriving from `WebApplicationFactory` behind a wrapper or rename, a retained `Microsoft.AspNetCore.Mvc.Testing` reference, `xunit*` pins past the major the restored Codebelt package declares in `project.assets.json`, and files changed under the selected project while the target pattern appears nowhere in it — each violation reported with its file and line, and covered by a regression harness that includes a completed-migration positive control, - `dotnet-test` assets and reference documentation covering unit-test behavior patterns, focused and shared web-application fixtures, application-focused fixtures, bootstrapper hosts for console and worker services in both minimal and Program/Startup form, xUnit v2-to-v3 modernization, and migration-invariant preservation, - `dotnet-remote-testing` skill that runs .NET tests inside Docker using official `mcr.microsoft.com/dotnet/sdk` images, honoring an existing `testenvironments.json` as authoritative when present and otherwise deriving environments from Microsoft's live release index, while reporting WSL and SSH as unsupported instead of silently falling back to the host, - `dotnet-remote-testing` deterministic runner `remote-test.cs` owning configuration discovery, release parsing, digest-pinned image resolution, isolated source staging, NuGet caching, execution, result parsing, distinct failure classification, and cleanup behind a single entry point, with a built-in `--self-test` alongside a PowerShell harness, From a2ceb4ea9b282b2996472b9956745d70904573e7 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Tue, 18 Aug 2026 21:47:55 +0200 Subject: [PATCH 31/31] =?UTF-8?q?=F0=9F=93=9D=20document=20dotnet-test=20m?= =?UTF-8?q?igration=20verification=20capabilities?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update README with a clear explanation of the hardest migration failure to catch: wrapped WebApplicationFactory, renamed static factories, or package-bump-only changes that produce a substantial diff while the host never actually moves. Document the verify-dotnet-test-migration.ps1 verdict gate that re-checks the expected pattern and validates post-edit conditions offline. --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index aaebbe2..e36df5b 100644 --- a/README.md +++ b/README.md @@ -650,6 +650,8 @@ The skill has one job: the test host comes from Codebelt, not from Microsoft, an **dotnet-test** begins with machine-readable inspection, then chooses the Codebelt pattern that matches the selected project's role and ownership model. Because that inspection already answers which project, role, and ownership apply, the skill acts on the evidence instead of asking the developer to retype what the JSON says. It preserves package ownership and frameworks, migrates xUnit v2 to v3/Microsoft Testing Platform when needed, and makes the chosen focused/shared web or application pattern, entrypoint-owned managed fixture, zero remaining selected `WebApplicationFactory` usages, zero deprecated blocking fixtures, zero replacement composition roots, and restore/build/test explicit gates. +The hardest failure to catch is the migration that only looks like one. Wrapping `WebApplicationFactory` in a private nested class, renaming a constructor to a static `Create`, or bumping package pins all produce a substantial diff while Microsoft's host still starts the application and every test stays green. `verify-dotnet-test-migration.ps1` closes the run with a verdict instead of a self-assessment: it re-checks the expected pattern and adds the post-edit checks for a surviving factory base type, a retained `Microsoft.AspNetCore.Mvc.Testing` reference, `xunit*` pins past the major the restored Codebelt package declares, and files changed under the project while the target pattern appears nowhere in it. + - **Evidence before questions** — the bundled inspector resolves project, role, mode, host ownership, and package owner, so a bare invocation starts working instead of returning a capability menu, - **Managed-fixture version floor** — inspection flags a Codebelt xUnit package pinned below 11.1.0, where the managed fixtures do not exist yet, before the pattern is written rather than after it fails to compile, - **Three explicit roles** — ordinary unit, ASP.NET Core functional, and console/worker functional tests route to separate references and assets, @@ -659,7 +661,9 @@ The skill has one job: the test host comes from Codebelt, not from Microsoft, an - **Bootstrapper host fidelity** — Startup-based hosts and `MinimalConsoleProgram`, `MinimalWorkerProgram`, or `MinimalWebProgram` hosts remain in their established family instead of being rewritten for test convenience, - **Dynamic compatibility** — stable package versions come from NuGet and must pass isolated compatibility-project restores, including the selected combined package set and target frameworks, - **Source-grounded bootstrap** — new projects receive at least one behavior test derived from real source instead of a placeholder, -- **Deterministic evidence** — inspection JSON reports roles, frameworks, xUnit generation, package owners, inheritance, migrations, recommendations, and blockers before mutation. +- **Deterministic evidence** — inspection JSON reports roles, frameworks, xUnit generation, package owners, inheritance, migrations, recommendations, and blockers before mutation, +- **A verdict, not a summary** — the migration gate exits `PASSED` or `FAILED` with numbered violations and their file and line, so a wrapped, renamed, or repackaged factory is reported as an unfinished migration rather than a completed one, +- **Anchored xUnit versions** — the gate reads the restored Codebelt package's own declared dependencies from `project.assets.json`, so an after-the-fact bump past that xUnit generation is caught offline instead of at the next compile. ### Why dotnet-benchmark?