diff --git a/Directory.Packages.props b/Directory.Packages.props index a79eb43ab4..bd5ca6dbc9 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -66,6 +66,15 @@ + + @@ -135,17 +144,23 @@ + view option, and by src/Opc.Ua.Vision.OpenUsd which renders the + Vision server's simulated camera views offscreen via Silk (D3D12/Vulkan). + Published on nuget.org. The runtime packages ship win-x64, linux-x64 and + osx-arm64; the RID-agnostic metapackages resolve the correct native asset + per RID, so no RID-conditional reference is required. All of these move + together: 0.11.0-alpha carries Storm child native ABI 8, and the managed + and RID-specific runtime packages have to match, so a partial bump fails + when the natives load rather than when the solution builds. --> - - - - - - + + + + + + + + diff --git a/README.md b/README.md index 268765a3e6..a591c3f3cd 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,21 @@ Each sample has its own `README.md` with build and run instructions. - [Intent Viewer Client](samples/Robotics/IntentViewerClient/README.md) — click a target in an OpenUSD viewport and watch the arm execute the resulting intent; also runs headless. +- [Bin Picking Cell](samples/Robotics/BinPickingCell/README.md) — Robot Intent, + Vision, an eye-in-hand camera, and an in-address-space OpenUSD scene in one + reference cell. +- [Bin Picking Client](samples/Robotics/BinPickingClient/README.md) — closes the + Vision-to-Robot-Intent loop, with optional MCP hosting and an OpenUSD viewport. +- [Visual Inspection Cell](samples/Vision/VisualInspectionCell/README.md) — hosts + Vision, AI Model Management, ISA-95 Job Control V2, and an operator dialog for + deterministic machined-bracket inspection. +- [Visual Inspection Agent](samples/Vision/VisualInspectionAgent/README.md) — drives + the inspection loop with typed clients, routes inference through deployment + `Invoke`, applies recipe verdicts, schedules allowlisted jobs, and records + operator ground truth. +- [AI Model Management sample](samples/AI/README.md) — `ModelManagementServer` + publishes the draft AI Model Management catalogue and routes inference; + `ModelManagementClient` discovers deployments and exercises the Methods. - [Minimal ISA-95 Server](samples/Isa95/MinimalIsa95Server/README.md) — minimal server hosting the OPC-10030 ISA-95 Common Model together with OPC-10031-4 Job Control V1 and V2, using the typed common-model @@ -188,4 +203,3 @@ vulnerabilities via the process documented in companion repository with more sample applications. - [Preview NuGet feed](https://opcfoundation.visualstudio.com/opcua-netstandard/_packaging?_a=feed&feed=opcua-preview%40Local) — prerelease builds from Azure DevOps. - diff --git a/UA.slnx b/UA.slnx index 042e99b806..700487d9bb 100644 --- a/UA.slnx +++ b/UA.slnx @@ -8,8 +8,16 @@ + + + + + + + + @@ -103,6 +111,10 @@ + + + + @@ -122,6 +134,10 @@ + + + + @@ -272,6 +288,9 @@ + + + @@ -300,6 +319,7 @@ + @@ -328,6 +348,7 @@ + diff --git a/docs/AiIntegration.md b/docs/AiIntegration.md new file mode 100644 index 0000000000..d63e67d779 --- /dev/null +++ b/docs/AiIntegration.md @@ -0,0 +1,258 @@ +# AI Model Management developer guide + +This guide documents the `Opc.Ua.AI`, `Opc.Ua.AI.Server`, +`Opc.Ua.AI.Client` and `Opc.Ua.AI.Inference` package family — the .NET +implementation of the draft *OPC UA — AI Model Management and Inference* +companion specification. + +> **Draft.** The namespace `http://opcfoundation.org/UA/AI/` and every NodeId +> in it are provisional. The model is neither official nor endorsed by the OPC +> Foundation until the working group publishes it. + +AI Model Management publishes model sources, models, datasets, deployments and +inference methods through OPC UA. The control plane is OPC UA: clients discover +what is available, read the provenance and trust-boundary metadata, call +`Invoke` or `InvokeAsync`, and transfer large artefacts through the standard +file-transfer types. + +## Packages + +| Package | What it gives you | Depends on | +|---|---|---| +| `OPCFoundation.NetStandard.Opc.Ua.AI` | Source-generated AI model — ObjectTypes, ReferenceTypes, DataTypes, enums, node states and model loader | `Opc.Ua.Core` | +| `OPCFoundation.NetStandard.Opc.Ua.AI.Server` | `AINodeManager`, `AIOptions`, fallback reporting, transfer and job support, and `AddAI` hosting extensions | `Opc.Ua.AI`, `Opc.Ua.Server`, `Opc.Ua.AI.Inference` | +| `OPCFoundation.NetStandard.Opc.Ua.AI.Client` | `AIClient`, `AIClientFactory` and `AddAIClient()` DI registration | `Opc.Ua.AI`, `Opc.Ua.Client` | +| `OPCFoundation.NetStandard.Opc.Ua.AI.Inference` | `IInferenceBackend`, `ChatClientInferenceBackend`, `RestChatCompletionsBackend`, credential resolvers and backend options | `Opc.Ua.AI`, `Microsoft.Extensions.AI` | + +The AI libraries target modern .NET TFMs used by the sample (`net8.0`, +`net9.0` and `net10.0`). The inference assembly intentionally has no Azure, +OpenAI or other vendor SDK dependency. + +## Model + +The Server publishes one AI root below the Server object. Under it are model +sources, deployments, models, datasets and jobs. A deployment describes where +inference runs, whether egress is permitted, whether input may be retained, the +maximum inline payload size, and the model source it uses. + +Two properties are especially important: + +- `ModelUsed` is returned with an inference result so a fallback cannot answer + silently. A caller can distinguish "the primary model answered" from "a + degraded fallback answered". +- `CredentialReference` is a name only. The credential value is resolved inside + the Server process by an `ICredentialResolver` and is never placed in the + address space. + +Large payloads use the standard Part 5 `FileType` transfer flow. Asynchronous +inference jobs use the Part 10 program lifecycle so clients can monitor state +instead of polling a private API. + +## Minimal hosted server + +`AddAI` registers the node manager, options and default backend composition. +The host supplies the `IChatClient` through `IChatClientFactory`; that keeps +vendor packages in the host and out of `Opc.Ua.AI.Inference`. + +```csharp +using Microsoft.Extensions.Hosting; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; +using Opc.Ua.AI.Server.Hosting; +using Opc.Ua.Server.Fluent; + +HostApplicationBuilder builder = Host.CreateApplicationBuilder(args); + +builder.Services.AddRestChatCompletionsAIChatClientFactory(); + +builder.Services + .AddOpcUa() + .AddServer(options => + { + options.ApplicationName = "AIServer"; + options.ApplicationUri = "urn:localhost:OPCFoundation:AIServer"; + options.AutoAcceptUntrustedCertificates = true; + options.EndpointUrls.Add("opc.tcp://localhost:62640/AIServer"); + }) + .AddAI( + ai => builder.Configuration.GetSection(AIOptions.SectionName).Bind(ai), + backend => builder.Configuration + .GetSection(InferenceBackendOptions.SectionName) + .Bind(backend), + fallback => builder.Configuration + .GetSection(InferenceBackendOptions.FallbackSectionName) + .Bind(fallback)); + +using IHost app = builder.Build(); +await app.RunAsync().ConfigureAwait(false); +``` + +`AddRestChatCompletionsAIChatClientFactory()` is the sample-friendly factory: +it creates an `IChatClient` over the configured OpenAI-compatible endpoint +without adding a vendor SDK. A production host can instead register its own +`IChatClientFactory` that creates `IChatClient` instances from Azure, OpenAI, +Ollama or an on-device runtime package. + +The direct construction path remains available for hosts that do not use the +generic hosting stack: + +```csharp +var backends = new InferenceBackends(primaryBackend, fallbackBackend); +var factory = new AINodeManagerFactory( + backends, + Options.Create(new AIOptions()), + Options.Create(new InferenceBackendOptions())); +``` + +## Hosting API + +The extension method on `IOpcUaServerBuilder` is: + +| Method | Purpose | +|---|---| +| `AddAI(Action?, Action?, Action?)` | Registers `AINodeManagerFactory`, `AIOptions`, primary and fallback `InferenceBackendOptions`, an `InferenceBackends` singleton, and the OPC UA node-manager registration | + +`AddAI` composes the backend from `InferenceBackendOptions.Kind`: + +- `ChatClient` (default) creates `ChatClientInferenceBackend` from + `IChatClientFactory`. +- `RestChatCompletions` creates `RestChatCompletionsBackend` directly for + endpoints whose wire contract is the OpenAI-compatible REST shape. + +### `AIOptions` + +| Property | Purpose | +|---|---| +| `PrimaryDeploymentId` / `FallbackDeploymentId` | Deployment identifiers published in the address space | +| `EnableFallback` | Publishes the fallback deployment and `FallsBackTo` reference | +| `EnableCatalogue` | Publishes catalogue and import-job nodes | +| `EnableLearningLoop` | Publishes a `LearningJobType` node for ground-truth sample accounting | +| `TransferExpiry`, `MaxTransferSize`, `MaxConcurrentTransfers`, `TransferInferenceTimeout` | Bounds for chunked transfers | +| `AsyncInferenceDelay`, `MaxRetainedJobs` | Bounds and timing for asynchronous inference jobs | +| `SourceId` | Identifier of the model source | + +When `EnableLearningLoop` is true, `AINodeManager` publishes one +`LearningJobType` under `LearningJobs`. Host-level coordinators report +ground-truth corrections through `RecordLearningSampleAsync(sampleId, +sampleKind)`. The stable `sampleId` makes retries idempotent, and +`AILearningSampleKind.Negative` counts empty or retracted observations exactly +as positive examples count. + +### `InferenceBackendOptions` + +| Property | Purpose | +|---|---| +| `Enabled` | Enables the backend; most useful for disabling fallback | +| `Kind` | `ChatClient` or `RestChatCompletions` | +| `EndpointUri`, `ChatCompletionsPath`, `ProbePath` | Endpoint and paths for REST-shaped clients | +| `Authentication`, `CredentialReference`, `ApiKeyHeader`, `CredentialDirectory`, `TokenAudience` | Server-to-backend authentication | +| `Site`, `DataJurisdiction`, `EgressPermitted`, `RetainsInput` | Trust-boundary metadata published to clients | +| `MaxInlinePayloadSize` | Inline `Invoke` payload limit | +| `Models` | Configured model catalogue entries | + +## Client surface + +`AddAIClient()` mirrors the other companion-family client extensions. It +registers an `AIClientFactory` and a +`Func>` over the managed session. + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Opc.Ua; +using Opc.Ua.AI.Client; +using Opc.Ua.Client; + +HostApplicationBuilder builder = Host.CreateApplicationBuilder(args); + +builder.Services + .AddOpcUa() + .AddClient(options => + { + options.ApplicationName = "AIClient"; + options.ApplicationUri = "urn:localhost:OPCFoundation:AIClient"; + options.AutoAcceptUntrustedCertificates = true; + }) + .AddDiscoveryAndConnect(options => + { + options.DiscoveryUrl = "opc.tcp://localhost:62640/ModelManagementServer"; + options.SecurityMode = MessageSecurityMode.SignAndEncrypt; + options.SecurityPolicyUri = SecurityPolicies.Basic256Sha256; + }) + .AddAIClient(); + +using IHost app = builder.Build(); +await app.StartAsync().ConfigureAwait(false); + +var createClient = app.Services + .GetRequiredService>>(); +AIClient? client = await createClient(CancellationToken.None) + .ConfigureAwait(false); + +if (client is null) +{ + Console.WriteLine("The Server does not implement AI Model Management."); +} +``` + +For one-off code, `AIClient.TryCreate(session)` is the direct fallback +when you already own an `ISession`. + +## Inference backends + +`IInferenceBackend` is the server-side contract: list models, invoke a model +and probe reachability. Two implementations ship: + +- `ChatClientInferenceBackend` wraps `Microsoft.Extensions.AI.IChatClient`. + This is the default because it lets the host choose any SDK or local runtime + that implements the abstraction without changing the OPC UA address space. +- `RestChatCompletionsBackend` speaks the OpenAI-compatible REST + chat-completions contract directly. Use it when that REST shape is the actual + wire contract and no `IChatClient` is available. + +Both hosted and on-device deployments use the same OPC UA nodes. The difference +is configuration: endpoint, credentials, data jurisdiction and egress. + +## Example + +The sample in +[`samples/AI/ModelManagementServer`](../samples/AI/ModelManagementServer) hosts +the node manager with `AddAI`. By default it uses the `ChatClient` path and the +sample composition root supplies a small `IChatClient` over the +OpenAI-compatible endpoint. `verify_backend.py` is a throwaway endpoint that +speaks enough of that contract for local validation: + +```powershell +python samples/AI/verify_backend.py 5273 +dotnet run --project samples/AI/ModelManagementServer +dotnet run --project samples/AI/ModelManagementClient +``` + +Set `InferenceBackend__Kind=RestChatCompletions` when testing an endpoint that +must be reached through the REST backend directly. Configure +`FallbackInferenceBackend__Kind` independently when the fallback uses a +different wire contract from the primary. + +## Limitations + +- The companion specification is a draft, so namespace URIs and NodeIds can + change. +- The sample publishes a real `LearningJobType` instance and a real + `SamplesCollected` counter. Host-level code can call the server-side + accounting API when ground-truth corrections arrive, including empty or + retracted observations. Retraining, candidate generation and promotion are + deliberately not simulated. +- `IChatClient` has no standard model-enumeration method, so hosts that need a + catalogue should configure `InferenceBackendOptions.Models`. +- The libraries do not reference vendor SDKs. If a provider package is needed, + add it in the hosting application and expose it through `IChatClientFactory`. +- Native AOT is disabled for the AI inference/sample projects because the + `Microsoft.Extensions.AI` ecosystem uses reflection in areas this repository + builds with warnings as errors. + +## See also + +- [AI sample README](../samples/AI/README.md) +- [Developer guide](DeveloperGuide.md) +- [Vision developer guide](Vision.md) +- [Robotics developer guide](Robotics.md) diff --git a/docs/McpServer.md b/docs/McpServer.md index d2c556d667..56afe1cfaa 100644 --- a/docs/McpServer.md +++ b/docs/McpServer.md @@ -4,7 +4,7 @@ The OPC UA MCP Server exposes all OPC UA Part 4 service calls as [Model Context ## What It Does -The MCP server wraps the OPC UA .NET Standard client library, translating between JSON-based MCP tool calls and OPC UA binary protocol operations. The server exposes tools through a [tool profile](#tool-profiles) — a named, bounded catalog selected at startup — rather than a single fixed tool count. The default `full` profile currently registers every tool below; running a narrower profile (`core`, `services`, `administration`, `pubsub`, or `diagnostics`) exposes only the subset relevant to that workflow. The tables below list the complete tool surface, organized by OPC UA Part 4 service set: +The MCP server wraps the OPC UA .NET Standard client library, translating between JSON-based MCP tool calls and OPC UA binary protocol operations. The server exposes tools through a [tool profile](#tool-profiles) — a named, bounded catalog selected at startup — rather than a single fixed tool count. The default `full` profile currently registers every tool below; running a narrower profile (`core`, `services`, `administration`, `pubsub`, `diagnostics`, `robotics`, or `vision`) exposes only the subset relevant to that workflow. The tables below list the complete tool surface, organized by OPC UA Part 4 service set: | Service Set | Tools | Description | |---|---|---| @@ -57,7 +57,8 @@ The server selects its tool catalog through a **tool profile** — a bounded set | `administration` | Configuration, Connection, NodeSet Export, PKI Management | Certificate trust management and NodeSet export | | `pubsub` | PubSub runtime, discovery, action, and capture tools (plus PubSub decode when diagnostics tools are enabled) | Part 14 PubSub publish/subscribe, discovery, and capture workflows | | `diagnostics` | Connection, Packet Capture (plus decode/replay when diagnostics tools are enabled) | OPC UA-aware packet capture, offline decode, and replay | -| `robotics` | Connection plus the Robot Intent discovery, monitoring, control and mission tools | Commanding and monitoring a Robot Intent controller | +| `robotics` | Connection plus Robot Intent discovery, paged monitoring, control, mission and Vision-guided Pick tools | Commanding and monitoring a Robot Intent controller | +| `vision` | Connection plus the Vision discovery, monitoring, seeing, inference, feedback and geometry tools | Perception-driven agents that need to see through a Vision server, compose poses across the §5.12 frame graph, run or submit inference, and (when composed with `robotics`) act on what they see — see the [Vision developer guide](Vision.md) | | `full` (default) | Every tool class above | Unrestricted access; the current-major default so existing integrations keep working unchanged | `full` is the default for the current major version — `core` and the other bounded profiles are opt-in. Select a profile with: @@ -66,8 +67,40 @@ The server selects its tool catalog through a **tool profile** — a bounded set - The `McpServer:ToolProfile` configuration value - The `OPCUA_MCP_TOOL_PROFILE` environment variable +**Profiles compose.** A `--profile` value can name more than one bounded profile at a time — the `BinPickingClient` sample runs `--profile vision,robotics` and exposes both catalogs from the same MCP host, deduplicating the shared `Connection` tools. The composed set uses the `WithOpcUaCoreTools(McpToolProfileSet)` / `WithOpcUaVisionTools(McpToolProfileSet)` / `WithOpcUaRoboticsTools(McpToolProfileSet)` overloads, and the core-tools overload owns the single `ConnectionTools` registration across every package that references the same MCP server. See the [Vision developer guide](Vision.md#mcp-tools) for the composed 64-tool example and the [BinPickingClient sample](../samples/Robotics/BinPickingClient) for the running catalog. + Because the exact number of tools in each profile (and in `full`) changes as tools are added or removed, this document intentionally does not hard-code a tool count. Use the tables above (or `tools/list`) to enumerate the tools actually exposed by a running server. +### Vision-guided Robotics + +`robotics_vision_pick` closes the common perception-to-action path without +making an agent copy a result NodeId and several resource NodeIds between tool +calls. It runs one detection inference through `Opc.Ua.Vision.Client`, on the +same named OPC UA session as the Robot Intent controller, then submits either +one Pick or a two-step Pick/Place mission: + +```json +{ + "request": { + "controller": "BinPickingController", + "pipeline": "BinPickingPipeline", + "source": "Bin", + "tool": "ParallelGripper", + "destination": "Fixture", + "classLabel": "RedCube", + "minimumConfidence": 0.9, + "missionId": "place-red-cube" + } +} +``` + +Detection selection is deterministic: exact DetectionId/ClassLabel filters and +the confidence threshold are applied first, then highest confidence wins with +ordinal DetectionId and original result order as tie-breakers. The result +contains Vision result/pipeline/sensor/model/frame provenance, the selected +pose, and the authoritative intent or mission handle. The helper does not +request authority, wait, retry, cancel, or reinterpret a server refusal. + ## Resources The MCP server exposes connected sessions as **MCP resources**, enabling the LLM to discover, inspect, and subscribe to session state. @@ -411,7 +444,7 @@ runtime and collect publisher responses): ## Architecture -The MCP tools ship as five libraries plus the executable that composes them. +The MCP tools ship as six libraries plus the executable that composes them. The executable owns only transport, logging and CLI plumbing; every tool lives in a library that an application can reference on its own. @@ -445,12 +478,16 @@ tools/ │ └── Tools/{PacketCapture,PacketDecode,PacketReplay}Tools.cs ├── Opc.Ua.Mcp.PubSub.Diagnostics/ # PubSub capture and decode │ └── Tools/{PubSubCapture,PubSubDecode}Tools.cs -├── Opc.Ua.Mcp.Robotics/ # Robot Intent discovery, monitoring, control, missions +├── Opc.Ua.Mcp.Robotics/ # Robot Intent plus same-session Vision-guided picking │ ├── RoboticsIntentManager.cs -│ └── Tools/Robotics{Discovery,Monitoring,Control,Mission}Tools.cs +│ ├── VisionGuidedRoboticsManager.cs +│ └── Tools/Robotics{Discovery,Monitoring,Control,Mission,Vision}Tools.cs +├── Opc.Ua.Mcp.Vision/ # Vision discovery, monitoring, seeing, inference, feedback, geometry +│ ├── VisionClientAccessor.cs +│ └── Tools/Vision{Discovery,Monitoring,Seeing,Inference,Feedback,Geometry}Tools.cs └── Opc.Ua.Mcp/ # .NET 10 project, packaged as dotnet tool ├── Program.cs # Entry point, stdio + Streamable HTTP transport (/mcp) - ├── McpHostBuilder.cs # Composes the five libraries + ├── McpHostBuilder.cs # Composes the six libraries ├── Opc.Ua.Mcp.Config.xml # OPC UA client application config └── .mcp/server.json # MCP server manifest for NuGet discovery ``` @@ -463,7 +500,8 @@ tools/ | `OPCFoundation.NetStandard.Opc.Ua.Mcp.PubSub` | PubSub runtime, actions, discovery | Core + `Opc.Ua.PubSub` | | `OPCFoundation.NetStandard.Opc.Ua.Mcp.Diagnostics` | UA-TCP capture, decode, replay | Core + `Opc.Ua.Core.Diagnostics` | | `OPCFoundation.NetStandard.Opc.Ua.Mcp.PubSub.Diagnostics` | PubSub capture, decode | Core + `Opc.Ua.PubSub.Diagnostics` | -| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Robotics` | Robot Intent discovery, monitoring, control, missions | Core + `Opc.Ua.Robotics.Client` | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Robotics` | Robot Intent discovery, typed control/missions, paged monitoring, Vision-guided Pick | Core + `Opc.Ua.Robotics.Client` + `Opc.Ua.Vision.Client` | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Vision` | Vision discovery, monitoring, seeing (`vision_get_frame` returns an MCP `ImageContentBlock`), inference, off-server feedback, §5.12 pose composition | Core + `Opc.Ua.Vision.Client` | | `OPCFoundation.NetStandard.Opc.Ua.Mcp` | the ready-to-run `opcua-mcp` tool | all of the above | The libraries multi-target `net8.0;net9.0;net10.0`; the executable targets @@ -495,11 +533,13 @@ builder.Services.AddOpcUaMcpPubSub(); builder.Services.AddOpcUaMcpDiagnostics(); builder.Services.AddOpcUaMcpPubSubDiagnostics(); builder.Services.AddOpcUaMcpRobotics(); +builder.Services.AddOpcUaMcpVision(); mcp.WithOpcUaPubSubTools(profile) .WithOpcUaDiagnosticsTools(profile, diagnosticsEnabled) .WithOpcUaPubSubDiagnosticsTools(profile, diagnosticsEnabled) - .WithOpcUaRoboticsTools(profile); + .WithOpcUaRoboticsTools(profile) + .WithOpcUaVisionTools(profile); ``` `WithOpcUaMcpFilters` registers the request and schema filters that make tool @@ -512,6 +552,17 @@ nothing rather than failing, so the same profile value can be passed to every package a host references — `Full` in a host that never referenced `Opc.Ua.Mcp.Diagnostics` simply yields no capture tools. +Bounded profiles that own their own connection tools — `Vision` in the +example above — carry `ConnectionTools` themselves in the single-profile +overload of `WithOpcUa...Tools`. When two or more bounded profiles are +composed through the `McpToolProfileSet` overloads (for example the +[`vision,robotics` composition](Vision.md#mcp-tools) the BinPickingClient +sample runs), the corresponding `WithOpcUaCoreTools(McpToolProfileSet)` +overload owns and deduplicates `ConnectionTools` across every package. +Each Vision or Robotics package's `McpToolProfileSet` overload never +registers `ConnectionTools` directly, so the composed catalog contains +one Connection surface, not several. + The capture tool classes stay `internal` and are reachable only through their registration extensions. diff --git a/docs/MigrationGuide.md b/docs/MigrationGuide.md index 5aa5d80fe3..8a259303fb 100644 --- a/docs/MigrationGuide.md +++ b/docs/MigrationGuide.md @@ -43,6 +43,125 @@ Looking for the broader narrative (non-prescriptive overview of what changed in a release)? See [What's New in 2.0](WhatsNewIn2.0.md). +## Migrating Robotics and Vision MCP requests + +The Robotics and Vision MCP tool names remain stable, but their request schemas +are now strongly typed. Robotics tools no longer accept JSON encoded inside a +string, and controller-scoped values can use an exact, unambiguous published +name instead of copying every NodeId. + +For example, the old Pick request nested one JSON document inside another: + +```json +{ + "controllerId": "ns=3;s=7001_Controllers_BinPickingController", + "intentJson": "{\"intentId\":\"pick-red\",\"source\":\"ns=3;s=Bin\",\"tool\":\"ns=3;s=Gripper\",\"objectClass\":\"RedCube\"}" +} +``` + +Pass the typed object directly now: + +```json +{ + "controller": "BinPickingController", + "input": { + "intentId": "pick-red", + "source": "Bin", + "tool": "ParallelGripper", + "objectClass": "RedCube" + } +} +``` + +The same change applies to every `robotics_submit_*` tool. Motion poses, +trajectory points, process attributes and program arguments are nested typed +objects or arrays. Values that become OPC UA Variants use an explicit +`dataType` plus `value`; they are never inferred through an `object`-typed API. + +Mission steps and transitions are arrays rather than stringified arrays. The +intent `kind` is a closed discriminator and exactly one matching payload is +required: + +```json +{ + "controller": "BinPickingController", + "missionId": "move-red", + "missionUpdateId": 1, + "steps": [ + { + "stepId": "pick", + "released": true, + "intent": { + "kind": "Pick", + "pick": { + "source": "Bin", + "tool": "ParallelGripper", + "objectClass": "RedCube" + } + } + }, + { + "stepId": "place", + "released": true, + "intent": { + "kind": "Place", + "place": { + "destination": "Fixture", + "tool": "ParallelGripper" + } + } + } + ], + "transitions": [] +} +``` + +`robotics_list_operations` and `robotics_list_missions` now return bounded +pages. Their optional `query` selects active or terminal work, filters by +identifier/state, chooses `Summary` or `Full`, and carries an opaque +continuation cursor. Use `robotics_wait_mission` with the MissionId and mission +operation NodeId returned by `robotics_submit_mission`; timeout returns the +current snapshot with `completed=false`, just like +`robotics_wait_operation`. + +One-shot Vision inference also uses one structured request. Replace the old +`pipelineNodeId` scalar: + +```json +{ + "pipelineNodeId": "ns=3;s=Vision/Pipelines/BinPickingPipeline" +} +``` + +with: + +```json +{ + "request": { + "pipeline": "BinPickingPipeline", + "expectedKind": "Detection", + "detail": "Summary", + "maxItems": 20 + } +} +``` + +The result still includes the ResultId and result NodeId, and now also includes +authoritative result kind/provenance plus a bounded detection, inspection or +segmentation summary. The `vision_read_*_result` tools remain available when a +caller needs the complete result. + +Callers that previously chained inference, detection selection, Pick and Place +can instead use `robotics_vision_pick`. Its one structured request names the +controller, pipeline, source, tool and optional destination plus detection +filters. The result carries the selected detection provenance and either the +intent operation or mission operation needed by the corresponding bounded wait +tool. Command authority is still requested separately. + +Name matching is exact and ordinal after trimming. A missing or ambiguous name +is an error that lists the matching candidates and NodeIds; the MCP layer never +chooses the first candidate or requests command authority as a side effect. + ## Migrating code that used the exposed diagnostics locks `IServerInternal`, `ISession` and `ISubscription` no longer expose their diff --git a/docs/OpenUsd.md b/docs/OpenUsd.md index 29ba2646cd..658769e465 100644 --- a/docs/OpenUsd.md +++ b/docs/OpenUsd.md @@ -32,6 +32,37 @@ requires the other. | `OPCFoundation.NetStandard.Opc.Ua.OpenUsdScene` | Part 2: the source-generated companion model, the scene document model, the `.usda` reader/writer, and the value-type map. | | `OPCFoundation.NetStandard.Opc.Ua.OpenUsdScene.Server` | Part 2: materializer, exporter, discovery, and Part 1 interop. | +The runtime path is intentionally narrow: the connector is an OPC UA client that +discovers binding descriptors, subscribes to source Variables, converts values to +USD-shaped `Variant`s, and writes them into one or more sinks. The optional viewer +is just another sink, so the on-disk override layer and the picture receive the +same updates. + +```mermaid +flowchart LR + Server["OPC UA Server
OpenUSD representations + bindings"] + Registry["Server/OpenUSD/Representations"] + Sources["Bound source Variables
telemetry + alarms"] + History["Historized source Variables"] + Connector["OpenUsdConnector
discover + subscribe + compose"] + Descriptor["RepresentationInfo + BindingInfo
prim path + property + conversion"] + Sink["IUsdSink"] + File["UsdFileSink
live.usda override layer"] + Viewer["Viewport sink
optional --view"] + Stage["USD stage assets
fetched and digest-verified"] + + Server --> Registry --> Connector + Sources -->|"MonitoredItems"| Connector + History -->|"ReplayHistoryAsync"| Connector + Connector -->|"discovers"| Descriptor + Descriptor -.->|"selects prim + conversion"| Connector + Connector -->|"converted Variant"| Sink + Connector -->|"FetchServedAssetsAsync"| Stage + Sink --> File + Sink --> Viewer + Stage --> Viewer +``` + ## The connector `OpenUsdConnector` is a **client**: it discovers a server's `OpenUsdRepresentation` instances through the Part 1 @@ -240,7 +271,7 @@ and the callback fires when that target changes. > The viewport requires .NET 10 and the OpenUSD packages (`OpenUsd`, `OpenUsd.Viewer`, `OpenUsd.Runtime.Imaging`), > which are published on nuget.org, so a plain restore is enough. The RID-agnostic runtime metapackages resolve the -> correct native payload per RID; `win-x64`, `linux-x64` and `osx-arm64` are all supported. With `0.7.0-alpha`, a +> correct native payload per RID; `win-x64`, `linux-x64` and `osx-arm64` are all supported. With `0.8.0-alpha`, a > RID-less build or publish on a supported host copies that host's OpenUSD native payload. Use an explicit RID when > publishing for another platform. Publish the connector and the viewport into the *same* directory, substituting your > own RID: diff --git a/docs/README.md b/docs/README.md index 2ee8711987..ed6965dec5 100644 --- a/docs/README.md +++ b/docs/README.md @@ -40,6 +40,8 @@ Here is a list of available documentation for different topics: * [Device Integration (DI) developer guide](DeviceIntegration.md) - End-to-end documentation for the `Opc.Ua.Di*` library trio: fluent `IDeviceBuilder`, device sub-type extensions (`AddSoftware`, `AddBlock`, `AddConfigurableObject`, `AddLifetimeIndication`, `WithSupportInfo`), hosting integration (`AddOpcUaDi` / `ConfigureDevicesFor`), lock service, software-update package store, and client helpers (`DiLockClient`, `DiTopologyClient`, `SoftwareUpdateClient`). Includes a section enumerating supported OPC 10000-100 features against the spec. * [OpenUSD](OpenUsd.md) — bridge an OPC UA address space to an OpenUSD stage, in two parts. **Part 1 — bindings**: the generic domain-agnostic `OpenUsdConnector` (discovers `Server/OpenUSD/Representations`, subscribes, composes, verifies stage/asset digests, replays history), the `Variant`-based `IUsdSink` with `UsdFileSink` / `MockUsdSink`, the fluent/DI `AddOpenUsdConnector` extensions + `OpenUsdConnectorFactory` / `OpenUsdConnectorOptions`, server-side `UsdAssetDelivery`, and the optional `--view` viewport. **Part 2 — scene materialization**: materializes a composed USD stage *inside* the address space so the prim tree is the node hierarchy — the `Opc.Ua.OpenUsdScene` companion model plus scene document model, `.usda` reader/writer and the §6.2 `UsdValueTypeMap` (USD roles as DataTypes subtyping their built-in), and `Opc.Ua.OpenUsdScene.Server` with `MaterializeUsdStage` / `ExportUsdStage`, unknown-type fallbacks, Mode-A live attributes, portable Cesium georeference dual-authoring, discovery and Part 1 binding-target resolution. (Both companion models are currently drafts.) * [Robotics developer guide](Robotics.md) — the `Opc.Ua.Robotics` / `Opc.Ua.Robotics.Server` / `Opc.Ua.Robotics.Client` trio for OPC 40010 Robotics 1.02 over OPC 40001-1 IA and OPC 10000-100 DI: source-generated models, `AddRobotics` / `AddRoboticsModel` / `ConfigureRobotics(For)` hosting, `IRoboticsModelProvider` / `IRoboticsConfigurator` / `IRoboticsBuildContext`, validated fluent topology builders (`AddMotionDeviceSystemAsync` down to axes, power trains, motors, gears, drives, safety states, and task controls), semantic references, `ArrayOf` snapshot contracts, and `RoboticsClient` discovery. It also covers [Robot Intent](Robotics.md#robot-intent), the draft task-level motion verbs OPC 40010 leaves undefined, including the MCP surface for LLM agents. +* [Vision developer guide](Vision.md) — the `Opc.Ua.Vision` / `Opc.Ua.Vision.Server` / `Opc.Ua.Vision.Client` / `Opc.Ua.Vision.OpenUsd` package family for the draft *OPC UA — Vision* companion specification: source-generated Vision model, `AddVision` / `ConfigureVision` hosting with `IVisionMediaProvider` / `IVisionInferenceProvider` / `IVisionFeedbackSink`, the two perception paths behind one contract (`OnServer` deterministic detector vs `EdgeOffServer` agent submissions), fluent topology builders for frames, sensors, calibrations, media endpoints and inference pipelines, `VisionClient` discovery + `VisionFrameGraph` §5.12 pose composition + `VisionResultReader` streaming detections + `VisionFeedbackClient` for off-server VLM agents, the §6.4 media-gating states, the `NoRenderingBackend` degrade path, facet derivation, and the composed [`vision` MCP profile](McpServer.md) with the [BinPickingCell / BinPickingClient](../samples/Robotics/BinPickingCell) example. +* [AI Model Management developer guide](AiIntegration.md) — the `Opc.Ua.AI` / `Opc.Ua.AI.Inference` / `Opc.Ua.AI.Server` / `Opc.Ua.AI.Client` package family for the draft *OPC UA — AI Model Management and Inference* companion specification over xRegistry: source-generated catalogues, datasets, deployments, inference endpoints and learning jobs, the `IInferenceBackend` contract with `Microsoft.Extensions.AI` `IChatClient` and OpenAI-compatible REST backends, `AINodeManagerFactory` hosting via `AddNodeManager`, `Invoke` routing, standard file-transfer artefact streaming, credential resolvers that keep secret material out of the address space, and the [ModelManagementServer / ModelManagementClient](../samples/AI/README.md) example. * [Relative Spatial Location and Global Positioning](Positioning.md) — source-generated OPC 10000-210 RSL and OPC 10000-211 GPOS models, standalone/composed server hosting, provider contracts, high-level clients, frame-chain resolution, WGS84/ENU conversion, and ground-control-point fitting. * [Generators (generating sets)](../samples/OpenUsd/GeneratorServer/Generators.md) — the draft Generators companion specification realised end to end by [GeneratorServer](../samples/OpenUsd/GeneratorServer): a datasheet-driven simulation in which load fraction is the only independent variable, DI + Machinery integration, and one independent OpenUSD twin per configured set. Includes [SiteCompositionServer](../samples/OpenUsd/SiteCompositionServer), a supervisory server that owns no devices and composes the pump and generator servers into a single scene through cross-server components. * [ISA-95 developer guide](ISA95.md) - End-to-end documentation for the `Opc.Ua.ISA95*` library trio: the OPC-10030 Common Model and OPC-10031-4 Job Control V1/V2, the two transparently-documented normative NodeSet repairs, `AddIsa95Server`/`AddIsa95Client` hosting, the typed common-model builder, the shared Job Control state engine and its `Uncertain`/Annex-B `ReturnStatus` result model, the provider-backed `GeoSpatialLocationType` seam and planned Part 210/211 RSL/GPOS integration, and a conformance matrix distinguishing static NodeSet structure from runtime-tested behavior. @@ -89,4 +91,3 @@ Starting with version 1.5.375.XX the Windows Forms reference client & reference * [Role-Based Security](RoleBasedUserManagement.md) — Part 18 roles and claim-based identity-mapping rules. * [Identity Providers](IdentityProviders.md) — server and client identity-provider architecture. * [Dependency Injection](DependencyInjection.md) — dependency injection hosting and identity registration extensions. - diff --git a/docs/Robotics.md b/docs/Robotics.md index ad04b4387c..42fe6eac28 100644 --- a/docs/Robotics.md +++ b/docs/Robotics.md @@ -11,9 +11,9 @@ dependency order. > **Status: draft companion model.** The namespace `http://opcfoundation.org/UA/RobotIntent/` and every > NodeId in it are **provisional**. This implements the working-group draft -> [*OPC UA — Robot Intent*](https://github.com/marcschier/opcua-drafts/blob/main/metaverse-specs/robot-intent/OPC-UA-Robot-Intent.md); -> nothing here is official or endorsed by the OPC Foundation. Do not deploy it on a production robot -> and expect the identifiers to survive. +> *OPC UA — Robot Intent*; nothing here is official or endorsed by the OPC +> Foundation. Do not deploy it on a production robot and expect the identifiers +> to survive. OPC 40010 describes a robot in detail — its motion device system, its axes, its power trains, its controller, its safety states — and defines **no motion verbs at all**. Its whole actuation surface is @@ -40,6 +40,24 @@ Robot Intent without pulling in OPC 40010, OPC 10000-100 DI, or anything else. | `OPCFoundation.NetStandard.Opc.Ua.Robotics.Server` | Stock Robotics node manager, Robot Intent node manager, model providers, hosting extensions (`AddRobotics`, `AddRobotIntent`, `ConfigureRobotics`, `ConfigureRobotIntent`), validated fluent topology builders, `IntentControllerHost`, safety binding, real-time channel declarations and facet calculation. | | `OPCFoundation.NetStandard.Opc.Ua.Robotics.Client` | Continuation-safe, subtype-aware discovery of Robotics instances over the DI client, Robotics type classification, Robot Intent discovery, the awaitable operation handle, command authority, real-time-channel leases, missions and `RobotIntentBuilder`. | +```mermaid +graph TD + Di["Opc.Ua.Di
Device Integration base model"] + Model["Opc.Ua.Robotics
OPC 40010 + Robot Intent model"] + Server["Opc.Ua.Robotics.Server
node managers + builders + IntentControllerHost"] + Client["Opc.Ua.Robotics.Client
discovery + RobotIntentClient"] + Mcp["Opc.Ua.Mcp.Robotics
agent tools"] + Executor["IIntentExecutor"] + Safety["IRobotIntentSafetySource"] + + Di --> Model + Model --> Server + Model --> Client + Client --> Mcp + Executor -.->|"executes admitted intents"| Server + Safety -.->|"guards admission"| Server +``` + Generated OPC 40010 model types stay in the specification namespaces `Opc.Ua.Robotics` and `Opc.Ua.IA`; hand-written APIs compose the generated NodeStates, factories, enums, and ObjectType clients instead of replacing or inheriting from them. @@ -126,6 +144,35 @@ await host.Build().RunAsync(); `AddOpcUaDi()`; both register the shared `DiAddressSpaceOwnership` marker and the second call throws with the name of the conflicting extension. +The two address-space shapes are separate but composable. OPC 40010 Robotics +instances live below the DI `DeviceSet`; Robot Intent lives below +`Server/RobotIntent` and exposes the command surface a client or MCP agent uses. + +```mermaid +flowchart TD + Objects["Objects"] --> DeviceSet["DeviceSet
OPC 10000-100 DI"] + DeviceSet --> Mds["MotionDeviceSystemType"] + Mds --> C40010["Controllers"] + Mds --> Motion["MotionDevices"] + Mds --> Safety40010["SafetyStates"] + Motion --> Axes40010["Axes"] + Motion --> Power["PowerTrains"] + + ServerObj["Server"] --> Root["RobotIntent
RobotIntentRootType"] + Root --> IntentControllers["Controllers"] + IntentControllers --> Controller["IntentControllerType"] + Controller --> Frames["Frames"] + Controller --> Tools["Tools"] + Controller --> Locations["Locations"] + Controller --> Axes["Axes"] + Controller --> Intents["Intents
IntentOperationType instances"] + Controller --> Missions["Missions
MissionType instances"] + Controller --> Channels["RealTimeChannels"] + Controller --> Capabilities["Capabilities
SupportedIntents + facets"] + Frames -.->|"HasFrameParent tree"| Frames + Controller -.->|"optional HasIntentController"| Mds +``` + ## Hosting API | Method | Builder | Purpose | @@ -420,7 +467,7 @@ model, and the configured instance namespace before returning. The manager's must be thread-safe, must reserve unique NodeIds for unregistered nodes, and must allocate Robotics instances in the configured instance namespace. -[`MinimalRobotServer`](../samples/Robotics/MinimalRobotServer) is the worked example of +[`MinimalRobotServer`](../samples/Robotics/MinimalRobotServer) is the example of the custom-manager route: it composes Robotics, IA, DI, the draft OpenUSD binding, and RSL/GPOS in one `DiNodeManager` subclass. @@ -1361,12 +1408,53 @@ adds four tool groups: * discovery: list controllers and read a controller's declared `SupportedIntents`, `SupportedFacets` and lookup tables; -* monitoring: read live state, list operations and missions, and wait for an operation with a bounded - timeout; +* monitoring: read live state, page/filter concise operation and mission summaries, and wait for an + operation or mission with a bounded timeout; * direct control: request and release authority, cancel, pause, resume, retry, and submit one tool per intent kind; * missions: submit, update the horizon of, and cancel missions. +Every controller argument is a selector: a NodeId remains valid, while an exact, +unambiguous published name avoids copying NodeIds between calls. Tool, frame, +location, output and program selectors inside an intent are resolved from the +controller snapshot read for that call. Resolution is read-only and ambiguity +is an error; it never requests command authority or submits exploratory work. + +Intent payloads are structured MCP objects rather than JSON text embedded in a +string: + +```json +{ + "controller": "BinPickingController", + "input": { + "intentId": "pick-red", + "source": "Bin", + "tool": "ParallelGripper", + "objectClass": "RedCube" + } +} +``` + +Mission `steps` and `transitions` are typed arrays. Each step carries a closed +`kind` discriminator and exactly one matching payload. The server preserves a +caller-supplied step IntentId, generates one only when omitted, and publishes +the resulting `StepId -> IntentId -> operation NodeId -> state` correlation. +`robotics_wait_mission` observes the mission operation NodeId returned by +submission; it does not rediscover work through mutable active state. + +The list tools return bounded pages. Summary detail omits poses and full output +payloads; request Full detail only for the page that needs it. A timeout from +either wait tool is not a failure: it returns `completed=false` and a refreshed +snapshot. + +When Vision and Robotics are composed, `robotics_vision_pick` runs one +detection inference, applies exact class/detection filters and a confidence +threshold, and deterministically chooses the highest-confidence candidate. It +submits one Pick when no destination is supplied, or a two-step Pick/Place +mission when a destination is supplied, then returns detection provenance and +the authoritative submission handles. It never takes authority, retries, +waits, cancels or converts a refusal into a success-shaped result. + The sample `IntentViewerClient --mcp` is one host for these tools. In headless mode it defaults to MCP stdio. With `--view`, it automatically uses Streamable HTTP because MCP stdio carries protocol frames on stdout and the in-process OpenUSD viewport shares that stream. An explicit `--transport stdio @@ -1551,6 +1639,26 @@ it, because `Queued`, `Cancelling` and the three distinct terminal outcomes cann | `Cancelled` | `Halted` | Terminal. Ended early because a cancel was accepted. | | `Retriable` | `Halted` | Terminal for now; `Retry` may re-attempt it. | +```mermaid +stateDiagram-v2 + [*] --> Accepted: SubmitIntent admitted + Accepted --> Queued: queued or buffered + Queued --> Executing: dispatch starts + Accepted --> Executing: no predecessor + Executing --> Succeeded: executor returns success + Executing --> Failed: executor fault or failure + Executing --> Retriable: executor returns retriable + Executing --> Cancelling: CancelIntent accepted + Queued --> Cancelled: CancelIntent accepted + Accepted --> Cancelled: CancelIntent on an admitted intent that is not current + Cancelling --> Cancelled: controlled stop completes + Cancelling --> Failed: executor fails while stopping + Retriable --> Accepted: Retry creates a new operation + Succeeded --> [*] + Failed --> [*] + Cancelled --> [*] +``` + `Cancelling` is **not** terminal. A client that treats acceptance of a cancel as the end of motion acts too early. diff --git a/docs/Vision.md b/docs/Vision.md new file mode 100644 index 0000000000..4c900ed9f0 --- /dev/null +++ b/docs/Vision.md @@ -0,0 +1,1177 @@ +# Vision developer guide + +This guide documents the `Opc.Ua.Vision`, `Opc.Ua.Vision.Server`, +`Opc.Ua.Vision.Client`, `Opc.Ua.Vision.OpenUsd` and `Opc.Ua.Mcp.Vision` package +family — the .NET implementation of the working-group draft *OPC UA — Vision* +companion specification, plus the OpenUSD offscreen capture adapter and the +Model Context Protocol tool package that lets a language-model agent see through +a Vision server and act on what it sees. + +> **Draft.** The namespace `http://opcfoundation.org/UA/Vision/` and every +> NodeId in it are provisional. The model is a working-group draft and is +> neither official nor endorsed by the OPC Foundation. The API is stable within +> this repository but every ObjectType, DataType and BrowseName can still change +> when the specification is published. + +Vision layers on top of the base OPC UA namespace only — it does not require +Devices, Machinery or Robotics. It composes cleanly with Robotics, as shown in the +[`samples/Robotics/BinPickingCell`](../samples/Robotics/BinPickingCell) and +[`samples/Robotics/BinPickingClient`](../samples/Robotics/BinPickingClient) +samples (`Vision` + `Robot Intent` in one server, `vision_*` + +`robotics_*` MCP tools in one agent). + +## Packages + +| Package | What it gives you | Depends on | +|---|---|---| +| `OPCFoundation.NetStandard.Opc.Ua.Vision` | Source-generated Vision model — ObjectTypes, ReferenceTypes, DataTypes, enums, node states, typed client proxies, `AddOpcUaVision` model loader | `Opc.Ua.Core` | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Server` | `VisionNodeManager`, `IVisionBuildContext`, fluent topology builders, `IVisionMediaProvider` / `IVisionInferenceProvider` / `IVisionFeedbackSink`, facet derivation, `AddVision` / `ConfigureVision` hosting extensions | `Opc.Ua.Vision`, `Opc.Ua.Server` | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Client` | `VisionClient` discovery, `VisionSensorClient`, `VisionPipelineClient`, `VisionResultReader`, `VisionMediaClient`, `VisionFrameGraph`, `VisionFeedbackClient`, `session.Vision(...)` extension, `AddVisionClient()` DI | `Opc.Ua.Vision`, `Opc.Ua.Client` | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.OpenUsd` | `ISceneCameraCaptureProvider` implementation that renders a `UsdGeomCamera` offscreen and reports `NoRenderingBackend` gracefully when no graphics device is available | `Opc.Ua.Types`, native OpenUSD renderer payload (optional per-RID) | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Vision` | 22 MCP tools split across discovery, monitoring, seeing, inference, feedback and geometry, plus the `vision` bounded profile and composition entry point | `Opc.Ua.Mcp.Core`, `Opc.Ua.Vision.Client` | + +```mermaid +graph TD + Model["Opc.Ua.Vision
source-generated model"] + Server["Opc.Ua.Vision.Server
VisionNodeManager + builders"] + Client["Opc.Ua.Vision.Client
typed discovery and readers"] + OpenUsd["Opc.Ua.Vision.OpenUsd
scene-camera capture provider"] + Mcp["Opc.Ua.Mcp.Vision
agent tools"] + Media["IVisionMediaProvider"] + Inference["IVisionInferenceProvider"] + Feedback["IVisionFeedbackSink"] + + Model --> Server + Model --> Client + Client --> Mcp + OpenUsd -.->|captures frames for| Media + Media -.->|plugs into| Server + Inference -.->|plugs into| Server + Feedback -.->|plugs into| Server +``` + +`Opc.Ua.Vision.OpenUsd` deliberately sits outside that dependency chain: it +depends on `Opc.Ua.Types` alone and knows nothing about the Vision server. It +offers a camera, and a host writes the `IVisionMediaProvider` that hands the +resulting frames to a pipeline — which is what +[`BinPickingCell`](../samples/Robotics/BinPickingCell) does. A host with a real +camera writes the same interface over its own SDK and never references OpenUSD +at all. + +The runtime address space is rooted under the standard Server object. +Configurators add frames, sensors and pipelines through the fluent builder; +providers plug into the sensor or pipeline nodes that the builder creates. + +```mermaid +flowchart TD + ServerObj["Server"] --> VisionRoot["Vision
VisionRootType"] + VisionRoot --> Sensors["Sensors"] + VisionRoot --> Frames["Frames"] + VisionRoot --> Pipelines["Pipelines"] + + Frames --> Frame["CoordinateFrameType
FrameId + Transform"] + Sensors --> Sensor["VisionSensorType
ImageSensorType / Depth3DSensorType"] + Sensor --> Calibrations["Calibrations"] + Calibrations --> Intrinsic["IntrinsicCalibrationType"] + Calibrations --> Extrinsic["ExtrinsicCalibrationType"] + Sensor --> Media["Media
VisionMediaManagementType"] + Media --> Streams["StreamEndpoints"] + Media --> Clips["ClipEndpoints"] + Sensor -.->|MountedOn| Frame + Sensor -.->|HasScenePrim| Scene["OpenUSD scene prim"] + + Pipelines --> Pipeline["InferencePipelineType"] + Pipeline --> Results["Results"] + Pipeline --> Feedback["Feedback
VisionFeedbackType"] + Pipeline -.->|Sensor| Sensor + Pipeline -.->|Deployment| Deployment["AI deployment NodeId"] +``` + +The libraries multi-target `net8.0;net9.0;net10.0` and `netstandard2.0` +where applicable; the MCP tool package multi-targets `net8.0;net9.0;net10.0`. + +## Two perception paths behind one contract + +Every pipeline advertises exactly one of two inference locations, and a +client reads a `DetectionResultType` identically regardless of which is in +force: + +- **`InferenceLocation = OnServer`** — the Server holds an + `IVisionInferenceProvider` and computes results locally. This is the + deterministic path: it needs no model, no network and no GPU, and it is + the default for CI and offline validation. `RunInference`, + `StartContinuous` and `Stop` all delegate to the provider; the Server + publishes the resulting `DetectionResultType` / `InspectionResultType` / + `SegmentationResultType` under the pipeline's `Results` folder and + advertises `VIS-Inference-OnServer`. +- **`InferenceLocation = EdgeOffServer`** — the pipeline exposes a + `VisionFeedbackType` object bound to an `IVisionFeedbackSink`; an + off-Server agent (a vision-language model over MCP, an edge inference + service, another Server) is expected to look at the current frame and + call `SubmitDetections` / `SubmitInspectionResult` / + `SubmitCorrection` / `SubmitImageReference`. The Server publishes those + results into the address space unchanged, and advertises + `VIS-Inference-OffServer`. + +Choose `OnServer` when a deterministic algorithm answers the question +(vision-guided screwdriver alignment against a known fiducial; presence-or- +absence in a clean scene) or when reproducibility on CI matters. Choose +`EdgeOffServer` when a language model or a heavier out-of-process model is +what actually sees the world — the Server publishes what it did not +compute, and safety validation still applies (§9 refusals, class-label / +box / pose / confidence checks; see [Feedback validation](#feedback-validation)). + +The two paths are exclusive per pipeline by design: mixing a running +`OnServer` provider with a `SubmitDetections` sink would let a computed +and a submitted result publish on the same pipeline out of any known +order. + +```mermaid +flowchart LR + Client["Client"] + Server["Vision Server"] + Provider["IVisionInferenceProvider"] + Edge["Off-server agent"] + Sink["IVisionFeedbackSink"] + Results["Pipeline Results folder"] + + Client -->|RunInference| Server + Server -->|OnServer delegates| Provider + Provider -->|result id + result node| Results + + Edge -->|reads frame| Server + Edge -->|SubmitDetections / SubmitInspectionResult| Server + Server -->|EdgeOffServer delegates| Sink + Sink -->|published result| Results + + Client -->|reads same result types| Results +``` + +## Minimal hosted server + +The example below hosts a single simulated eye-in-hand camera, one +inference pipeline, and a two-frame tree (`world` → `flange`). It is the +smallest useful shape; the [`BinPickingCell`](../samples/Robotics/BinPickingCell) +sample is the full end-to-end version. + +```csharp +using Microsoft.Extensions.Hosting; +using Opc.Ua; +using Opc.Ua.Server; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Builders; + +HostApplicationBuilder builder = Host.CreateApplicationBuilder(args); + +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); + +builder.Services + .AddOpcUa() + .AddServer(options => + { + options.ApplicationName = "VisionServer"; + options.ApplicationUri = "urn:localhost:OPCFoundation:VisionServer"; + options.AutoAcceptUntrustedCertificates = true; + options.EndpointUrls.Add("opc.tcp://localhost:62855/VisionServer"); + }) + .AddVision(options => + { + options.InstanceNamespaceUri = "urn:example:vision:instances"; + }) + .AddVisionMediaProvider(sensorBrowseName: "Camera01") + .AddVisionInferenceProvider( + pipelineBrowseName: "Detector", + onServer: true) + .ConfigureVision((context, ct) => + { + IVisionNodeBuilder nodes = context.Nodes; + // Nodes is the fluent address-space builder: everything it creates + // becomes real OPC UA nodes under Server/Vision that any client can + // browse. The full build context is described further down. + + nodes.AddFrame("World", frame => frame + .WithFrameId("world") + .WithRole(VisionFrameRoleEnum.World)); + + nodes.AddFrame("Flange", frame => frame + .WithFrameId("flange") + .WithRole(VisionFrameRoleEnum.MechanicalInterface) + .WithParent("world")); + + nodes.AddImageSensor("Camera01", sensor => sensor + .WithSensorId("cam-01") + .WithModality(VisionSensorModalityEnum.Area2D) + .WithRealityKind(VisionRealityKindEnum.Physical) + .WithFrameId("flange") + .WithResolution(1920u, 1080u) + .WithPixelFormat("Mono8") + .AddClipEndpoint("Clips", ep => ep + .WithEndpointId("clip-01") + .WithEndpointUri("opcua-inline://visionserver/clips") + .WithClipFormat(VisionClipFormatEnum.Png) + .WithResolution(1920u, 1080u) + .WithInlineDelivery(enabled: true, maxInlineClipSize: 8_388_608u))); + + // The pipeline is bound to its inference provider by + // AddVisionInferenceProvider("Detector", ...) + // above; the ConfigureVision delegate just creates the pipeline + // node. A real cell resolves the sensor node the pipeline is + // reading from off Server/Vision/Sensors — see the BinPickingCell + // sample for the walk. + nodes.AddPipeline("Detector", pipe => pipe + .WithPipelineId("pipe-01") + .WithSensor(NodeId.Null)); + + return ValueTask.CompletedTask; + }); + +using IHost app = builder.Build(); +await app.RunAsync().ConfigureAwait(false); +``` + +`AddVision` never modifies the Server's existing NodeManagers — it adds a +standalone `VisionNodeManager` under the well-known `Server/Vision` object +(§4.2). It also composes with `AddRobotIntent` and `AddRobotics`; both +sample cells run all three side by side without any coupling in code. + +The `AddVisionMediaProvider` / `AddVisionInferenceProvider` / +`AddVisionFeedbackSink` extensions resolve the provider from the DI +container at build time and bind it to the sensor or pipeline whose +BrowseName is passed. The equivalent `UseMediaProvider(provider)` / +`UseInferenceProvider(provider, onServer)` / `UseFeedbackSink(sink)` +methods on the fluent builders let a configurator bind a provider it +holds directly. + +## Hosting API + +The extension methods on `IOpcUaServerBuilder` that make up the Vision +hosting surface: + +| Method | Purpose | +|---|---| +| `AddVision(Action?)` | Registers the standalone `VisionNodeManager` and its factory; accepts an optional options delegate | +| `AddVisionMediaProvider(string sensorBrowseName)` | Registers a media provider type — resolved from DI — for the sensor with the given BrowseName | +| `AddVisionMediaProvider(string sensorBrowseName, IVisionMediaProvider provider)` | Registers a media provider instance for the sensor with the given BrowseName | +| `AddVisionInferenceProvider(string pipelineBrowseName, bool onServer)` | Registers an inference provider type — resolved from DI — for the pipeline; `onServer` controls the advertised `VIS-Inference-OnServer` / `VIS-Inference-OffServer` facet (§8.2) | +| `AddVisionInferenceProvider(string pipelineBrowseName, bool onServer, IVisionInferenceProvider provider)` | Registers an inference provider instance | +| `AddVisionFeedbackSink(string pipelineBrowseName)` | Registers a feedback sink type — resolved from DI — for the pipeline's `Feedback` object | +| `AddVisionFeedbackSink(string pipelineBrowseName, IVisionFeedbackSink sink)` | Registers a feedback sink instance | +| `ConfigureVision(Func)` | Async configurator, run on server start against the standalone `VisionNodeManager` | +| `ConfigureVision(Action)` | Sync configurator overload | +| `ConfigureVisionFor(...)` | Configurator targeting a specific Vision node-manager type. Currently only `VisionNodeManager` is supported | + +The Robotics guide's `ConfigureFor<...>` pattern also applies here — any +of these hosting extensions can be called from a class-based configurator +that reads `IServiceProvider`, keeps its own logger, and does not put a +lambda in `Program.cs`. + +### `VisionServerOptions` + +| Property | Purpose | +|---|---| +| `InstanceNamespaceUri` | The application-owned namespace URI used for the instances the configurator materialises. Must be distinct from the OPC UA base namespace and from `http://opcfoundation.org/UA/Vision/`. Defaults to `urn:opcfoundation:UA:Vision:Instances`. | +| `SpecificationVersion` | The value the Server reports on `Vision.SpecificationVersion`. Defaults to `"0.1.0"`. | +| `AdditionalFacets` | The facets the Server declares beyond those the facet calculator derives structurally — the escape hatch for facets whose requirements are behavioural (an interop facet that the host meets by contract). | + +## Build context + +`ConfigureVision(...)` receives an `IVisionBuildContext`. Here `Nodes` is +the fluent address-space builder rooted at the well-known `Server/Vision` +object; it creates the frame, sensor, calibration, media-endpoint and +pipeline nodes that ordinary OPC UA clients browse and read. + + +| Member | Purpose | +|---|---| +| `Nodes` | The fluent `IVisionNodeBuilder` rooted at the well-known `Server/Vision` object | +| `Manager` | The active `AsyncCustomNodeManager`, for the rare case a configurator must fall back to raw node authoring | +| `Context` | The active `ISystemContext` | +| `Root` | The `VisionRootState` (§4.2) | +| `InstanceNamespaceIndex` | The namespace index of `VisionServerOptions.InstanceNamespaceUri` | +| `VisionNamespaceIndex` | The namespace index of `http://opcfoundation.org/UA/Vision/` | +| `CancellationToken` | The startup cancellation token | +| `GetRequiredService()` | Application-scoped DI resolution | + +Everything a Vision cell needs (frames, sensors, pipelines, calibrations, +media endpoints) is authored through `Nodes`. The low-level members are +present for interop with hand-written NodeManagers and for the vendor +extension patterns the Robotics guide describes. + +### Without DI + +A `VisionNodeManager` created by hand exposes the same fluent surface +through `ConfigureVisionAsync`: + +```csharp +await manager.ConfigureVisionAsync(context => +{ + context.Nodes.AddFrame("World", f => f + .WithFrameId("world") + .WithRole(VisionFrameRoleEnum.World)); +}); +``` + +Prefer it over `CreateVisionBuildContext()`. The node manager indexes the +Vision root when it creates the address space, so anything the builder +grafts on afterwards has to be registered as well before it can be +browsed or read **by its own NodeId** — which is how an ordinary client +and the MCP discovery tools navigate. `ConfigureVisionAsync` does that +registration when the delegate returns; a context obtained from +`CreateVisionBuildContext()` never does, so nodes built through it stay +reachable only by browsing forward from their parent. + +## Topology builders + +### Frames + +```csharp +nodes.AddFrame("World", f => f + .WithFrameId("world") + .WithRole(VisionFrameRoleEnum.World)); + +nodes.AddFrame("RobotBase", f => f + .WithFrameId("robot_base") + .WithRole(VisionFrameRoleEnum.Base) + .WithParent("world") + .WithTransform(new VisionPose3DDataType + { + FrameId = "world", + Position = new[] { 0.0, 0.0, 0.829 }.ToArrayOf(), + Orientation = new[] { 0.0, 0.0, 0.0, 1.0 }.ToArrayOf(), + Covariance = ArrayOf.Empty, + })); +``` + +The `Transform.FrameId` names the parent frame per the §5.12 frame- +precedence rule. When the parent is added later in the configurator, +`WithParent("world")` resolves at finalise time. Passing a `NodeId` +overload (`WithParent(NodeId)`) skips the deferred resolution when the +caller already has one. + +Roles: `World`, `Base`, `MechanicalInterface`, `Tool`, `Object`, `Station`, +`Camera`, `Custom`. `Camera` is the non-ISO addition Vision introduces +for the sensor's own frame. + +### Sensors + +`IVisionNodeBuilder` exposes three sensor entry points: + +- `AddImageSensor(browseName, configure)` for `ImageSensorType`; +- `AddDepth3DSensor(browseName, configure)` for `Depth3DSensorType`; +- `AddSensor(browseName, configure)` for the abstract `VisionSensorType` + when a vendor subtype is materialised through a `IVisionModelProvider`. + +All sensor builders share the members on `IVisionSensorBuilder`: +identity (`WithSensorId`, `WithManufacturer`, `WithModel`, +`WithSerialNumber`, `WithDeviceUri`), the frame binding (`WithFrameId`, +`MountedOn`, `HasScenePrim`), the reality kind (`WithRealityKind`), the +modality (`WithModality`), and the nested builders — `WithOptics(...)`, +`WithIllumination(...)`, `AddIntrinsicCalibration(...)`, +`AddExtrinsicCalibration(...)`, `AddStreamEndpoint(...)`, +`AddClipEndpoint(...)`, `UseMediaProvider(...)`. + +`WithFrameId(frameId)` also adds a `MountedOn` reference to the +`CoordinateFrameType` instance with the matching `FrameId` when one has +been registered under `Vision/Frames`. This is the recommended way to +attach a sensor to a frame; `MountedOn(NodeId)` and `HasScenePrim(NodeId)` +are the fallbacks for mounts that are not vision frames. + +### Pipelines + +A pipeline is the Vision object that connects one sensor, an optional AI +deployment reference, and the result/feedback methods for one perception task +such as detection or inspection. Clients discover pipelines first, then run +or observe the task through that pipeline. + +```csharp +nodes.AddPipeline("Detector", pipe => pipe + .WithPipelineId("pipe-01") + .WithSensor(cameraNodeId) + .WithDeployment(deploymentNodeId) + .ProducedBy(controllerNodeId) + .UseInferenceProvider(inferenceProvider, onServer: true) + .UseFeedbackSink(feedbackSink)); +``` + +`WithSensor(NodeId)` points the pipeline at the sensor that supplies the +frames. `WithDeployment(NodeId)` then records which model deployment, if +any, is responsible for the task. The specification deliberately keeps +`Deployment` typed as `NodeId` so a Server never has to depend on the AI +Model Management companion; a host that implements that companion can point +at its deployment node, and a host that does not can leave the value null. +`ProducedBy(NodeId)` adds the `ProducedBy` semantic reference to a controller +or process instance. + +`UseFeedbackSink(sink)` is optional for `OnServer` pipelines. `OnServer` +pipelines without a feedback sink still expose a `Feedback` object +whose `Submit*` methods return `Bad_NotSupported` — a client cannot +publish detections into a pipeline where nothing consumes them. +`EdgeOffServer` pipelines almost always want both a provider (whose +`RunInference` explains the mode with `Bad_NotSupported`) and a sink +that receives the agent's submissions. + +### Providers + +The provider abstractions live in `Opc.Ua.Vision.Server`: + +- **`IVisionMediaProvider`** — supplies media without putting pixels on + OPC UA. `GetStreamAsync` returns a leased URI; `GetClipAsync` returns + a `VisionImageReferenceDataType` and, when the caller asked for it and + the encoded bytes fit the effective inline limit, an inline + `ByteString`. Providers implement the by-reference default (§6.4) — + Servers keep pixel bytes off the OPC UA wire. +- **`IVisionInferenceProvider`** — binds a pipeline to whatever actually + computes results. The Server publishes the result nodes and applies + the spec's method conventions regardless of whether the provider runs + a deterministic detector, a GPU inference engine, an in-process + simulation or refuses everything with `Bad_NotSupported` (the + `EdgeOffServer` case). +- **`IVisionFeedbackSink`** — receives `SubmitDetections`, + `SubmitInspectionResult`, `SubmitCorrection` and + `SubmitImageReference`. Off-Server agents publish through this path + and the Server records what it did not compute. + +Every provider is registered as a DI singleton and constructed with the +same lifetime as the server host — the framework never re-creates them +per call. + +## §5.12 conventions + +Vision inherits and adds a small number of numerical conventions that are +silently wrong if misread. Every client, provider and configurator in +this repository respects them, and public API points that carry pose or +image geometry document them explicitly: + +- **Quaternion order is `(x, y, z, w)`.** Every `Orientation` array in + `VisionPose3DDataType` is a unit quaternion ordered `(x, y, z, w)`. The + frame graph checks `‖q‖ = 1` within tolerance `1e-6` and refuses a + zero-norm quaternion with `Bad_InvalidArgument`. +- **Positions are metres.** Every `Position` array is a 3-vector in + metres. `MinDepth`, `MaxDepth`, `Baseline`, `WorkingDistance` and every + distance-typed member are metres. +- **The principal point is corner-datum.** `VisionIntrinsicsDataType.Cx` + and `Cy` are measured from the top-left corner of the image (pixel + centre `(0.5, 0.5)`). A client bridging to a library that uses + centre-datum coordinates subtracts `0.5` from `Cx` and `Cy`. +- **An empty covariance array is the sentinel for "not reported".** A + pose that reports no covariance uses `Covariance = ArrayOf.Empty` + — not a 6×6 zero matrix, which would misrepresent the pose as having + been measured with perfect certainty. + +The `VisionFrameGraph` composes transforms strictly per these rules: +right-handed frames, `(x, y, z, w)`-ordered quaternions, no +substitutions. + +## §6.4 media gating + +Vision separates the by-reference default path (a `VisionImageReference` +descriptor with URI, timestamp and digest) from the optional inline +delivery of encoded still image bytes. §6.4 fixes what a Server returns +in each state, and `VisionMediaClient` classifies the raw `StatusCode` +into a `VisionInlineClipState` enum so a caller can branch cleanly: + +| State | `StatusCode` | Meaning | +|---|---|---| +| `Available` | `Good` | The encoded image fits the inline limit and is returned in `VisionInlineClipReading.Bytes` | +| `NotYetAvailable` | `Bad_NoDataAvailable` | The Server has not published a clip yet — §6.4 rule 5 requires this before the first acquisition | +| `InlineDisabled` | `Bad_NotSupported` | `InlineDeliveryEnabled = false` on the endpoint; §6.4 rule 5 requires this exact code | +| `Overflow` | `Bad_EncodingLimitsExceeded` | The last acquisition exceeded the effective inline size limit; §6.4 rule 3 requires no truncation | +| `Faulted` | Other | The endpoint reported a different error | + +`GetClip` is always the safe path — it returns the by-reference +descriptor, and returns the inline bytes as well when the caller passes +`requestInline: true` and the still fits. `LatestClip` and +`LatestClipMetadata` are the "read the latest" variant of the same +contract. Crucially, `LatestClipMetadata` remains readable even when +`LatestClip` reports `Bad_NotSupported` — the metadata carries the URI, +timestamp, digest and pixel format the caller needs to walk the still +out of band, and reporting `Bad_NotSupported` on the metadata read would +be wrong. + +```mermaid +stateDiagram-v2 + [*] --> NotYetAvailable: no acquisition yet + NotYetAvailable --> Available: clip captured and fits + Available --> Overflow: next clip exceeds limit + Overflow --> Available: later clip fits + Available --> InlineDisabled: inline delivery disabled + InlineDisabled --> Available: inline delivery enabled + Available --> Faulted: provider error + Overflow --> Faulted: provider error + Faulted --> Available: provider recovers +``` + +## Rendering without pixels + +`Opc.Ua.Vision.OpenUsd` renders a `UsdGeomCamera` from a USD stage to an +encoded still, and is the reference `ISceneCameraCaptureProvider` +implementation the sample cell registers with +`services.AddOpenUsdSceneCameraCaptureProvider()`. When rendering is not +possible it reports the unavailable backend and leaves the address space +walkable instead of throwing from browse/read paths. It supports the +following behaviour explicitly: + +- On a machine with the native OpenUSD renderer payload present and a + usable graphics device, it renders normally and returns encoded PNG / + JPEG bytes. +- On a machine with no graphics device — the normal case on CI — it + reports `SceneCameraCaptureBackend.NoRenderingBackend` on + `Backend.UnavailableReason` and returns `Bad_NoDataAvailable` for + every capture. The sensor still exists in the address space and every + browse still works; only the pixel bytes are absent. + +The intent is that a client can rely on the address space always being +walkable, even when the process has no way to produce pixels. The +`--demo` client path in the sample skips its compose step gracefully +when the frame is unavailable rather than falsely reporting a rendering +bug. + +## Facets supported + +`VisionServerOptions.AdditionalFacets` is additive on top of the facets +the address-space calculator derives structurally. **Supported** means the +stock builder and calculator can claim the facet from the materialised address +space. **Partial** means the model surface exists, but the host must attest the +facet through `AdditionalFacets` because the calculator cannot verify the +behaviour or provider-owned result nodes. **Not supported** means the stock +server does not currently claim that facet. + +| Facet | Support | Structural requirement or limitation | +|---|---|---| +| `VIS-Base` | Supported | A registered sensor contributes the base Vision server shape. | +| `VIS-Sensor-Params` | Supported | A sensor includes manufacturer, model or serial-number parameters. | +| `VIS-Optics` | Supported | A sensor has an `Optics` child. | +| `VIS-Media-Rtsp` | Supported | A stream endpoint uses `VisionStreamProtocolEnum.Rtsp`. | +| `VIS-Media-Jpeg` | Supported | A clip endpoint uses `VisionClipFormatEnum.Jpeg`. | +| `VIS-Media-Inline` | Supported | A clip endpoint has `InlineDeliveryEnabled = true`. | +| `VIS-Media-DataChannel` | Not supported | There is no stock data-channel endpoint builder or calculator rule. | +| `VIS-Endpoint-Config` | Supported | At least one stream endpoint is materialised. | +| `VIS-Calibration` | Supported | An intrinsic or extrinsic calibration is materialised. | +| `VIS-Result-Detection` | Partial | Providers and sinks can publish detection results, but result ownership is provider-side and not structurally derived. | +| `VIS-Result-Inspection` | Partial | `SubmitInspectionResult` exists, but the calculator does not infer inspection-result publication. | +| `VIS-Result-Segmentation` | Not supported | The stock conformance URI list and feedback surface do not claim a segmentation-result facet. | +| `VIS-Feedback` | Supported | A pipeline has a `Feedback` object bound to a sink. | +| `VIS-Inference-OnServer` | Supported | A pipeline was registered with `onServer: true`. | +| `VIS-Inference-OffServer` | Supported | A pipeline was registered with `onServer: false`. | +| `VIS-Simulation` | Supported | A sensor has `RealityKind = Simulated` or `Hybrid`. | +| `VIS-Learning` | Partial | Learning jobs are modelled by reference; the host owns the job and any `SamplesCollected` accounting. | +| `VIS-Interop-Scene` | Supported | A sensor carries a `HasScenePrim` reference. | +| `VIS-Interop-40100` | Partial | The host must attest cross-model behaviour through `AdditionalFacets`. | +| `VIS-Interop-RobotIntent` | Partial | The host must attest cross-model behaviour through `AdditionalFacets`. | + +The Server publishes the composed set on +`Server.ServerCapabilities.ServerProfileArray`. + +## Using the client libraries + +### Registration + +The client hosting extensions register a `VisionClientFactory` and the +factory function downstream services request: + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Opc.Ua.Client; +using Opc.Ua.Vision.Client; + +builder.Services + .AddOpcUa() + .AddClient(options => { /* endpoint and application options */ }) + .AddVisionClient(); +``` + +`AddVisionClient()` requires `AddClient(...)` to have been called first +so the shared `ManagedSession` factory is available. + +Without DI, a `VisionClient` can be opened directly from any connected +`ISession`: + +```csharp +using Opc.Ua; +using Opc.Ua.Client; +using Opc.Ua.Vision.Client; + +VisionClient vision = session.Vision(telemetry); +if (!vision.IsVisionNamespaceAvailable) +{ + // The Server does not implement the Vision companion. + return; +} +``` + +### Discovery + +```csharp +await foreach (VisionNodeEntry sensor in vision.EnumerateSensorsAsync(ct)) +{ + Console.WriteLine($"{sensor.BrowseName} ({sensor.TypeDefinition})"); +} + +await foreach (VisionNodeEntry pipeline in vision.EnumeratePipelinesAsync(ct)) +{ + Console.WriteLine($"{pipeline.BrowseName} ({pipeline.TypeDefinition})"); +} +``` + +Both `EnumerateSensorsAsync` and `EnumeratePipelinesAsync` are subtype +aware: a Server that specialises `ImageSensorType` with a vendor +subtype, or `DetectionResultType` with a domain result subtype, is +enumerated as an instance of the closest declared Vision base type. The +lower-level `DiscoverSensorsAsync` / `DiscoverPipelinesAsync` / +`DiscoverFramesAsync` return `ArrayOf` for callers that already +know how to render the picker themselves. + +### Reading a detection + +```csharp +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +NodeId pipelineNodeId = /* discovered above */; +VisionPipelineClient pipe = vision.Pipeline(pipelineNodeId); + +VisionPipelineSnapshot snapshot = await pipe.ReadAsync(ct); +string runId = await pipe.RunInferenceAsync(cancellationToken: ct); + +await foreach (VisionNodeEntry result in pipe.EnumerateResultsAsync(ct)) +{ + VisionDetectionResultSnapshot detection = await vision + .Result(result.NodeId) + .ReadDetectionAsync(ct); + + Console.WriteLine( + $"result={detection.ResultId} frame={detection.FrameId} " + + $"count={detection.Detections.Count}"); + for (int ii = 0; ii < detection.Detections.Count; ii++) + { + VisionDetectionDataType d = detection.Detections[ii]; + Console.WriteLine($" [{ii}] {d.ClassLabel} conf={d.Confidence:0.###}"); + } +} +``` + +`VisionPipelineClient.RunInferenceAsync(default, ct)` lets the Server +acquire the frame "now"; passing a non-default `DateTimeUtc` requests a +specific acquisition timestamp — the Server may honour it or refuse per +§8. + +### Composing a pose + +The Vision-side calibrations and frame names match the Robotics-side +frame ids by convention — a client can walk from the vision-side pose to +the robot-side world frame without any translation table: + +```csharp +VisionFrameGraph frames = vision.Frames(); + +NodeId cameraFrameId = /* from EnumerateFramesAsync, matching FrameId "camera_eih" */; +NodeId worldFrameId = /* likewise, matching "world" */; + +VisionDetectionResultSnapshot detection = + await vision.Result(resultNodeId).ReadDetectionAsync(ct); + +for (int ii = 0; ii < detection.Detections.Count; ii++) +{ + VisionDetectionDataType d = detection.Detections[ii]; + if (!d.HasPose) + { + continue; + } + + // Compose the detection's pose (expressed in the camera frame) into + // the world frame; the frame graph walks the parent chain and + // multiplies transforms per §5.12. + VisionPose3DDataType inWorld = await frames.ComposeAsync( + d.Pose, cameraFrameId, worldFrameId, ct); + + Console.WriteLine( + $"{d.ClassLabel}: pos=[{inWorld.Position[0]:0.###}, " + + $"{inWorld.Position[1]:0.###}, {inWorld.Position[2]:0.###}]"); +} +``` + +`ComposeAsync` walks up to 32 frames from each side, throws +`Bad_NoMatch` when the two frames share no common ancestor, and refuses +a non-unit quaternion within tolerance `1e-6` — none of which are +substituted with an identity transform, because a silent substitution +would make an incorrect pose look correct. + +### Submitting feedback + +```csharp +VisionFeedbackClient? feedback = await pipe.OpenFeedbackAsync(ct); +if (feedback is null) +{ + // The pipeline does not expose a Feedback object — nothing to submit into. + return; +} + +ArrayOf detections = new[] +{ + new VisionDetectionDataType + { + ClassLabel = "RedCube", + Confidence = 0.94, + HasBoundingBox2D = true, + BoundingBox2D = new VisionBoundingBox2DDataType + { + CenterX = 812.0, CenterY = 604.0, Width = 96.0, Height = 96.0, + }, + HasPose = true, + Pose = new VisionPose3DDataType + { + FrameId = "camera_eih", + Position = new[] { 0.031, -0.017, 0.412 }.ToArrayOf(), + Orientation = new[] { 0.0, 0.0, 0.0, 1.0 }.ToArrayOf(), + Covariance = ArrayOf.Empty, + }, + }, +}.ToArrayOf(); + +await feedback.SubmitDetectionsAsync( + VisionFeedbackPurposeEnum.Reconciliation, + detections, + frameReference: null, + inlineImage: ByteString.Empty, + cancellationToken: ct); +``` + +The `Purpose` values `Overlay`, `Reconciliation`, `GroundTruthLabel` +and `Trigger` are all defined; a Server refuses the ones it does not +permit with `Bad_NotSupported`. `SubmitCorrection` accepts *at most +one* of the two `corrected*` arrays non-empty — passing both is an +argument error (§9.5). `SubmitInspectionResult` requires at least one +characteristic. + +> **Reporting an empty scene, and retracting a false positive.** +> `SubmitDetections` takes a `SceneIsEmpty` flag and `SubmitCorrection` +> a `RetractAll` flag. They exist because an empty observation is a real +> one: "I examined this frame and there is nothing in it" is the +> terminating condition of a bin-picking task and a valid negative +> training label, and a false positive is corrected by asserting that +> nothing replaces it. Neither statement can be made by submitting an +> array, because both *are* the empty array. +> +> The pairing is checked in both directions. An empty `Detections` +> without `SceneIsEmpty` is refused — the flag is what distinguishes a +> deliberate observation from a lost payload — and `SceneIsEmpty` with +> detections attached is refused because it asserts two contradictory +> things about one frame. `RetractAll` behaves the same way against the +> corrected arrays. §9.4 requires a Server to count a negative example +> in `SamplesCollected` exactly as it counts one carrying geometry — +> that counter belongs to the learning job, so see +> [Limitations](#limitations) for where this implementation draws the +> line and what it hands the host instead. + +### Streaming detections + +`VisionResultReader.ObserveDetectionsAsync(IStreamingSubscription, ct)` +publishes each `DetectionResultType` change as it arrives — either a new +result or a mutation of an existing one. Use it when you need to react +to detections rather than poll: + +```csharp +using Opc.Ua.Client; + +// A ManagedSession exposes a shared default IStreamingSubscription +// that a caller can hand to any Observe*Async method. +IStreamingSubscription streaming = session.DefaultStreaming; + +VisionResultReader reader = vision.Result(resultNodeId); +await foreach (VisionDetectionResultSnapshot snapshot in + reader.ObserveDetectionsAsync(streaming, ct)) +{ + Console.WriteLine($"result {snapshot.ResultId}: {snapshot.Detections.Count} detections"); +} +``` + +`ObserveInspectionAsync` and `ObserveSegmentationAsync` do the same for +inspection and segmentation results. + +## MCP tools + +The `Opc.Ua.Mcp.Vision` package contributes 22 tools split across six +categories, and the bounded `vision` profile in `docs/McpServer.md` +carries them plus the four `ConnectionTools` — every Vision tool +resolves a named OPC UA session, and only the connection tools can open +one. + +- Discovery — `vision_list_sensors`, `vision_list_pipelines`, + `vision_list_frames`, `vision_list_calibrations`. +- Monitoring — `vision_read_sensor`, `vision_read_extrinsic_calibration`, + `vision_read_pipeline`, `vision_read_detection_result`, + `vision_read_inspection_result`, `vision_read_segmentation_result`. +- Seeing — `vision_get_frame` (returns the encoded still as an MCP + `ImageContentBlock` with the correct MIME type, so a model actually + sees pixels rather than a description of them), `vision_get_frame_metadata`. +- Inference — `vision_run_inference`, `vision_start_continuous_inference`, + `vision_stop_inference`. +- Feedback — `vision_submit_detections`, `vision_submit_inspection_result`, + `vision_submit_correction`, `vision_submit_image_reference`. +- Geometry — `vision_read_frame`, `vision_compose_pose`, + `vision_compose_transform`. + +`vision_run_inference` accepts one structured `request`. Its `pipeline` +selector is an exact BrowseName, DisplayName or NodeId; `expectedKind` can +require a Detection, Inspection or Segmentation result; and `detail=Summary` +returns a bounded typed summary together with the ResultId, result NodeId, +sensor, requested/published pipeline, model and frame provenance. Use +`detail=HandleOnly` when only the addressable result is needed, or call a +`vision_read_*_result` tool for the complete payload. + +Profiles compose. A host that wires the `Vision` and `Robotics` profile +sets together — the [BinPickingClient sample](../samples/Robotics/BinPickingClient) +does this — exposes 64 tools in total, measured as +`22 Vision + 4 Connection + 42 Robotics − 4 shared Connection = 64`. +The `WithOpcUaVisionTools(McpToolProfileSet)` overload never registers +`ConnectionTools` directly; the corresponding +`WithOpcUaCoreTools(McpToolProfileSet)` overload owns and deduplicates +that registration across every OPC UA MCP package a host references. + +See [`docs/McpServer.md`](McpServer.md) for the full profile table and +composition rules. + +## Sample: bin-picking + +[`samples/Robotics/BinPickingCell`](../samples/Robotics/BinPickingCell) is +the reference from the *OPC UA Robotics-Vision Addendum*: a +branch-stable four-axis palletizer with a parallel gripper, an +eye-in-hand camera parented to the flange, a bin of five parts, a +fixture, and the frame tree +`world → robot_base → flange → gripper_tcp` with `camera_eih` on the +flange. It hosts `Robot Intent`, the Vision companion and the +`OpenUsdScene` companion side by side, and either the on-server +deterministic detector (`--inferenceLocation OnServer`, the default) +or the off-server agent path (`--inferenceLocation EdgeOffServer`). +The selected detection identifies the workpiece and preserves its pose as +provenance. The Pick carries the selected class plus the source Location and +tool; the cell's executor resolves that workpiece at its observed world pose +rather than substituting the Location centre. + +[`samples/Robotics/BinPickingClient`](../samples/Robotics/BinPickingClient) +is the paired client: `--demo` runs the whole loop without an agent, +`--mcp` exposes the composed 64-tool MCP catalogue for a language-model +agent, and `--view` opens the in-process OpenUSD viewport so a human +sees the same scene the agent sees. + +The composed Robotics package also contributes `robotics_vision_pick`. It runs +one inference through the Vision client, selects one detection +deterministically, and submits either a Pick operation or a two-step Pick/Place +mission. It returns the selected detection provenance and the authoritative +operation or mission handles. Command authority remains explicit: the helper +never requests ownership, retries, waits, cancels or hides a refusal. + +### Scene lighting + +The scene is lit by a single `DomeLight` with `intensity = 1000`. +Do **not** reintroduce a `DistantLight` or any other bright directional or +point light for a Vision demo: at any intensity that shows geometry, a +`DistantLight` blows every surface to pure white regardless of +`displayColor`, and any agent looking at the frame sees a uniform +white blur. Under a `DomeLight` the five sample parts measure +`red (220, 37, 37)`, `green (37, 208, 49)`, `blue (37, 73, 233)` — distinct +enough for a vision-language model to reason about ("pick the red +cube"). The `Cell.usda` header records this contract explicitly. +Anyone authoring their own scene from scratch needs to know this or +their agent will see nothing they can act on. + +### Feedback validation + +When the cell runs in `EdgeOffServer` mode the agent sends detections +through `SubmitDetections`. The sample's feedback sink refuses malformed +submissions with `Bad_InvalidArgument` and a message the agent can act +on: + +- **Unknown class label** — refused with the exact list of parts that + do exist. `Detection 0 class 'PurplePyramid' is not a part in this cell. + Known classes: RedCube, GreenCylinder, BlueSphere, YellowSlab, OrangeBrick.` +- **Confidence outside `[0, 1]`** — refused with the observed value. +- **Bounding box outside the image** — refused with the box coordinates + and the image dimensions the box was measured against. +- **Zero-norm quaternion or pose with fewer than three position + components** — refused with the detection index. + +An **empty** detection set is refused too, with `Bad_InvalidArgument`, +because §9.5 states it plainly: "`Detections` empty" is an argument +error. So is a `SubmitCorrection` whose corrected arrays are both empty +or both populated — §9.5 requires *exactly one* to be non-empty. + +That is worth dwelling on, because it means two useful statements cannot +be made at all. An agent that has emptied the bin cannot report "I looked +and there is nothing there"; it must either invent a detection or say +nothing. And a false positive — the model saw something that was not +there — cannot be retracted by correcting the result down to an empty +set, which is one of the more valuable labels a correction could carry. +The implementation conforms rather than deviating, and the gap is raised +against the draft; see [Limitations](#limitations). + +## Limitations + +- **The Vision specification is a draft.** The namespace URI and every + NodeId are provisional; every ObjectType and BrowseName can change + when the working group publishes. +- **No vendor drivers ship.** The reference `IVisionMediaProvider` + covered here renders an OpenUSD stage offscreen. There is no GigE + Vision, USB3 Vision, GenICam or vendor-native driver in the box; a + host implementing `IVisionMediaProvider` for a real camera is the + supported extension point. +- **`ConfigureVisionFor` only accepts `VisionNodeManager`.** + Custom Vision node-manager types are not yet supported; vendor + extension follows the same class-based-configurator pattern the + Robotics guide describes. +- **Learning jobs are modelled but not driven, and `SamplesCollected` is + not counted here.** `InferencePipelineType` carries a `LearningJob` + optional child and the facet calculator publishes `VIS-Learning` when + one is bound, but the standalone `VisionNodeManager` does not itself + run training — a host provides a learning-job provider through the + extension pattern. This has one named conformance consequence: §9.4 + requires a Server to count a negative example (`SceneIsEmpty` or + `RetractAll` carrying a `GroundTruthLabel`) in `SamplesCollected` + exactly as it counts one carrying geometry, and `VisionNodeManager` + does not do that counting. It cannot: `SamplesCollected` is a property + of `LearningJobType`, which the *AI Model Management* companion + defines, and Vision reaches it through a `NodeId` value rather than a + Reference precisely so this model takes no dependency on the model + that defines the job. What the Server does guarantee is that the + negative example survives the hop intact — `SceneIsEmpty` and + `RetractAll` are carried verbatim on + `VisionSubmitDetectionsRequest` / `VisionSubmitCorrectionRequest`, so + a host that binds a learning job has everything it needs to satisfy + §9.4 on the counter it owns. +- **AOT compatibility of the OpenUSD capture provider depends on the + native renderer payload.** The managed layer is AOT-friendly; the + native payload ships per-RID and its presence at runtime is what + distinguishes a rendering capture from a `NoRenderingBackend` one. + +## Visual inspection: a cross-companion cell + +Inspection is the part of the specification that composes with the most +other models, so it is documented here in full rather than in isolation. +The [Vision samples](../samples/Vision) build a cell where a camera +photographs a machined bracket, a model measures it, deterministic code +judges those measurements against a recipe, the verdict drives an ISA-95 job +order, and anything the machine cannot decide escalates to an operator whose +answer is captured as ground truth and counted as a learning sample. The +sample READMEs cover how to run it; what follows is why it is built this way. + +### The model never decides + +The central safety property is that model output is evidence, not authority. A +model may be involved in producing measured characteristics and a confidence, +but deterministic code applies the recipe tolerances and computes the verdict. +That matters because a plant that let a language model schedule production from +what it thought it saw in a photograph would have an image-shaped path straight +into job control. In this sample, image content can influence the measured +values, but it cannot become a free-form job-control instruction. + +The sample also routes inference through the AI companion's deployment +`Invoke` method instead of letting the agent call a model privately in its own +process. That keeps the deployment node, model version, and usage accounting in +the address space and in the provenance trail. A private model call would make +the most important part of the loop invisible. + +### Address-space composition + +`VisualInspectionCell` composes four companion areas in one server process: + +- Vision publishes `BracketFixtureCamera`, `FixtureImages`, + `BracketInspectionPipeline`, inspection results, and feedback. +- AI Model Management publishes the primary deployment + `visual-inspection-primary`, its model metadata, and the learning job whose + `SamplesCollected` value is incremented by host code. +- ISA-95 Job Control V2 publishes the fixed job-order catalogue and the V2 + Methods the agent calls. +- Alarms & Conditions publishes the `OperatorDispositionDialog` condition. + +`BracketInspectionPipeline` points at the AI deployment through `Deployment` and +at the learning job through `LearningJob`. The Vision companion deliberately +uses `NodeId` values for those bindings; it does not take a compile-time +dependency on the AI companion. This sample is the host that binds both models +and can therefore satisfy the learning-job counter semantics that standalone +Vision cannot. + +### Recipe and verdict rule + +The inspected part is a machined bracket with three dimensional characteristics +in millimetres: + +| Characteristic | Nominal | Tolerance | +|---|---:|---:| +| `BoreDiameter` | 12.00 | ± 0.20 | +| `SlotWidth` | 8.00 | ± 0.15 | +| `EdgeOffset` | 20.00 | ± 0.25 | + +For each characteristic, the rule builds an interval from the measured value and +physical uncertainty: + +```text +measurement interval = actual ± uncertainty +tolerance interval = [nominal - lowerTol, nominal + upperTol] +``` + +Then it classifies the characteristic: + +- wholly inside the tolerance interval -> `Ok` +- wholly outside the tolerance interval -> `NotOk` +- straddling either tolerance limit -> `NotDecidable` + +The part verdict is the worst characteristic verdict, with `NotOk` worse than +`NotDecidable`, and `NotDecidable` worse than `Ok`. + +```mermaid +flowchart TD + Measurement["Measured characteristic
actual and uncertainty"] --> Interval["Build actual +/- uncertainty"] + Interval --> Compare{"Compare with tolerance interval"} + Compare -->|"wholly inside"| Ok["Ok"] + Compare -->|"wholly outside"| NotOk["NotOk"] + Compare -->|"straddles a limit"| NotDecidable["NotDecidable"] + Ok --> Worst["Part takes worst characteristic verdict"] + NotOk --> Worst + NotDecidable --> Worst +``` + +### Why uncertainty is physical + +The fixture images are 800 x 600 pixels at 10 px/mm. A feature edge can only land +on a pixel boundary, so a dimensional measurement carries one-pixel quantisation +uncertainty: 0.10 mm. That is the camera's pixel pitch. It is what makes +`VisionCharacteristicDataType.Uncertainty` meaningful, and it is why +`NotDecidable` arises naturally instead of being contrived. + +The three fixtures exercise all branches: + +| Fixture | Decisive characteristic | Interval | Verdict | +|---|---|---|---| +| `bracket-ok.png` | `BoreDiameter = 12.00` | `[11.90, 12.10]` is wholly inside `[11.80, 12.20]` | `Ok` | +| `bracket-not-ok.png` | `BoreDiameter = 12.60` | `[12.50, 12.70]` is wholly outside the bore tolerance | `NotOk` | +| `bracket-ambiguous.png` | `SlotWidth = 8.10` | `[8.00, 8.20]` straddles the 8.15 upper limit | `NotDecidable` | + +The ambiguous fixture is intentionally mundane: the intended 8.15 mm slot is +81.5 pixels at 10 px/mm and cannot be drawn exactly. The raster image therefore +measures as 8.10 mm, and one pixel of uncertainty crosses the tolerance limit. +That is precisely the case the Vision `NotDecidable` value exists for. + +### Inspection loop + +The agent drives the process from outside the server. It discovers the Vision +pipeline, opens the media endpoint, follows the pipeline's `Deployment` to the +AI companion, discovers ISA-95 V2 endpoints, and finds the operator dialog. + +```mermaid +flowchart TD + Discover["Discover pipeline, media, deployment, jobs, dialog"] --> Capture["Get fixture PNG"] + Capture --> Measure["Measure bracket geometry"] + Measure --> Invoke["Call AI deployment Invoke"] + Invoke --> Judge["Apply recipe rule"] + Judge --> Submit["Submit inspection result to Vision Feedback"] + Submit --> Verdict{"Verdict"} + Verdict -->|"Ok"| CloseOk["Start, stop, and clear inspection job"] + CloseOk --> Next["StoreAndStart inspection order"] + Verdict -->|"NotOk"| CloseBad["Start, stop, and clear inspection job"] + CloseBad --> Reject["StoreAndStart rework/reject order"] + Verdict -->|"NotDecidable"| Hold["Hold for operator"] +``` + +The important separation is that quality outcome and job execution state are +separate facts. A defective part does not mean the inspection job failed. + +| Verdict | Inspection job | Next job | +|---|---|---| +| `Ok` | complete, close | schedule next inspection | +| `NotOk` | complete, close | schedule rework/reject order | +| `NotDecidable` | hold | none until the operator answers | + +Scheduling selects an order from a fixed allowlisted catalogue and calls V2 +`StoreAndStart`. `InspectionJobControlProvider` accepts only +`VIS-INSP-BRACKET-001` and `VIS-REWORK-REJECT-001`; the agent never invents a +job payload. + +### Escalation and ground truth + +`NotDecidable` activates the human path. The design dispositions are +`AcceptAsOk`, `AcceptAsNotOk`, `Reinspect`, and `Stop`. The implementation maps +those dispositions onto the dialog response and a bounded timeout: it holds or +stops, but it does not auto-approve and does not block forever. + +```mermaid +sequenceDiagram + participant Agent as VisualInspectionAgent + participant Vision as Vision Feedback + participant Dialog as Operator Dialog + participant Operator as Human Operator + participant AI as AI Learning Job + + Agent->>Vision: SubmitInspectionResult NotDecidable + Agent->>Dialog: Wait for disposition + Dialog->>Operator: Request Accept, Reinspect, Reject, or Stop + Operator-->>Dialog: Disposition + Dialog-->>Agent: Response or timeout + alt Accept as ground truth + Agent->>Vision: SubmitCorrection GroundTruthLabel + Vision->>AI: RecordLearningSampleAsync + AI-->>Vision: Idempotent count result + else Reinspect + Agent->>Vision: No correction + Agent->>Agent: Schedule inspection order + else Stop or timeout + Agent->>Agent: Hold or stop without approval + end +``` + +The operator answer becomes a Vision §9 ground-truth correction. The cell's +feedback sink calls `AiNodeManager.RecordLearningSampleAsync` and uses a stable +sample id, so a retry does not count the same label twice. A negative example is +still a learning sample: it counts exactly once even when it carries no geometry. + +This closes a limitation called out in the [Vision developer guide](Vision.md#limitations): +`SamplesCollected` belongs to the AI companion's `LearningJobType`, while Vision +only names the learning job by `NodeId`. A standalone Vision node manager cannot +increment a counter owned by another companion. A host binding Vision and AI +Model Management together can, and this cell is that host. See also the +[AI Model Management developer guide](AiIntegration.md) for the deployment +and learning-job model. + +### Modes + +`VisualInspectionAgent` supports three modes: + +- `scripted` — deterministic analyser, scripted operator policy, and a finite + `--cycles N`. This is the unattended path. +- `live-ai` — a real model path. It requires `--ai-endpoint`; if no endpoint is + configured, the agent exits before creating any job and never silently falls + back to the simulated analyser. +- `human` — a real dialog subscriber path with a bounded + `--operator-timeout`. + +The no-silent-fallback rule is part of the sample's safety story. A sample that +quietly degrades can look green while proving neither model connectivity nor the +provenance path it claims to demonstrate. + +### What is deliberately not implemented + +The sample does not implement retraining or model promotion. It records learning +samples honestly and increments the AI learning-job count, but it does not fake +an MLOps workflow. A simulated retraining integration that appeared to work +would mislead readers about the one part of the specification a sample cannot +honestly demonstrate. The [AI Model Management sample](../samples/AI/README.md) +takes the same line. + +## See also + +- [Robotics developer guide](Robotics.md) — the sibling companion + implementation Vision composes with; the `BinPickingCell` sample is + the cross-companion example. +- [OpenUSD guide](OpenUsd.md) — the OpenUSD connector and scene + materialisation used to bind the Vision sample cell to a live USD + stage. +- [MCP Server guide](McpServer.md) — the `vision` MCP profile and its + composition with `robotics`. +- [AI Model Management developer guide](AiIntegration.md) — the + companion that owns the model, dataset and deployment a Vision + pipeline's `Deployment` and `LearningJob` point at, and the counter + §9.4 asks a learning job to keep. +- [Dependency Injection](DependencyInjection.md) — the `AddOpcUa()` + builder surface that hosts `AddVision`, `AddVisionClient`, and every + other component. \ No newline at end of file diff --git a/docs/WhatsNewIn2.0.md b/docs/WhatsNewIn2.0.md index ade36ef101..5918e9802e 100644 --- a/docs/WhatsNewIn2.0.md +++ b/docs/WhatsNewIn2.0.md @@ -194,6 +194,22 @@ server- and client-side implementations: gears, drives, safety states, and task controls with the correct companion-spec references. See [Robotics](Robotics.md), including the draft Robot Intent task-level command model. +- **OPC UA — Vision** (draft): the `Opc.Ua.Vision` / + `Opc.Ua.Vision.Server` / `Opc.Ua.Vision.Client` / + `Opc.Ua.Vision.OpenUsd` package family, with a source-generated Vision model, + `AddVision` / `ConfigureVision` hosting, media/inference/feedback providers, + fluent frame/sensor/calibration/pipeline builders, typed `VisionClient` + discovery, result streaming, off-server feedback, facet derivation, and + OpenUSD camera capture. See [Vision](Vision.md), including the Robotics + + Vision bin-picking example. +- **OPC UA — AI Model Management and Inference** (draft): the `Opc.Ua.AI` / + `Opc.Ua.AI.Inference` / `Opc.Ua.AI.Server` / `Opc.Ua.AI.Client` package + family over xRegistry, with a source-generated catalogue/deployment/inference + model, the `IInferenceBackend` contract, `Microsoft.Extensions.AI` + `IChatClient` and OpenAI-compatible REST backends, `AINodeManagerFactory` + hosting through `AddNodeManager`, `Invoke` routing, + learning jobs, and standard file-transfer artefact streaming. See + [AI Model Management](AiIntegration.md). - **OPC 10100-1 — WoT Connectivity**: model, server, and client libraries for surfacing OPC UA servers as Web of Things Thing Descriptions, with the `WoTAssetConnectionManagement` server methods gated by a @@ -509,6 +525,8 @@ coverage service; see [Alias Names](AliasNames.md), [Device Integration](DeviceIntegration.md), [Relative Spatial Location and Global Positioning](Positioning.md), + [Vision](Vision.md), + [AI Model Management](AiIntegration.md), [Software Update](SoftwareUpdate.md), [WoT Connectivity](WoTConnectivity.md), [Subscriptions and Monitored Items](Subscriptions.md), diff --git a/docs/migrate/2.0.x/packages.md b/docs/migrate/2.0.x/packages.md index f7d604e7b6..3df2c7f4bb 100644 --- a/docs/migrate/2.0.x/packages.md +++ b/docs/migrate/2.0.x/packages.md @@ -33,6 +33,7 @@ The minimum SDK is the **.NET 10 SDK**, and projects compile with **`LangVersion | `Microsoft.CodeAnalysis.Analyzers` 4.14.0 | Added (pinned) | Centralised pin only, no direct reference; holds the analyzer closure on the `roslyn.props` band | | `Microsoft.CodeAnalysis.Common` 5.0.0 | Added | `tools/SourceGeneratorVariant.targets`, `tools/MigrationAnalyzerVariant.targets` | | `Microsoft.CodeAnalysis.CSharp` 5.0.0 | Added | `tools/SourceGeneratorVariant.targets`, `tools/MigrationAnalyzerVariant.targets` | +| `Microsoft.Extensions.Caching.Abstractions` 10.0.10 | Added (pinned) | Introduced as a transitive dependency by the ModelContextProtocol 2.x SDK | | `Microsoft.Extensions.Configuration.Abstractions` 10.0.10 | Added | `src/Opc.Ua.Client.ComplexTypes`, `src/Opc.Ua.PubSub` | | `Microsoft.Extensions.Diagnostics` 10.0.10 | Added | `src/Opc.Ua.Core/Opc.Ua.Core.csproj` | | `Microsoft.Extensions.Hosting` 10.0.10 | Added | Samples and tools that host a server or client | diff --git a/samples/AI/ModelManagementClient/AiScenarioRunner.cs b/samples/AI/ModelManagementClient/AiScenarioRunner.cs new file mode 100644 index 0000000000..bf28dde757 --- /dev/null +++ b/samples/AI/ModelManagementClient/AiScenarioRunner.cs @@ -0,0 +1,290 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua; +using Opc.Ua.AI.Client; +using Opc.Ua.Client; + +namespace Opc.Ua.AI.Client +{ + internal sealed class AIScenarioRunner + { + private AIScenarioRunner(AIClient client) + { + m_client = client; + } + + public static AIScenarioRunner? TryCreate(ISession session, ITelemetryContext telemetry) + { + var client = new AIClient(session, telemetry); + return client.IsAINamespaceAvailable ? new AIScenarioRunner(client) : null; + } + + public async Task RunAsync(CancellationToken ct) + { + Console.WriteLine("AI root: {0}", m_client.AIRootId); + Console.WriteLine(); + Console.WriteLine("--- model catalogue"); + await ReportModelsAsync(ct).ConfigureAwait(false); + + ArrayOf deployments = await m_client.DiscoverDeploymentsAsync(ct) + .ConfigureAwait(false); + if (deployments.Count == 0) + { + Console.WriteLine("No deployments are published."); + return; + } + + for (int ii = 0; ii < deployments.Count; ii++) + { + await DescribeDeploymentAsync(deployments[ii], ct).ConfigureAwait(false); + } + + AIDeploymentClient deployment = m_client.Deployment(deployments[0]); + await RunCapabilitiesAsync(deployment, ct).ConfigureAwait(false); + await RunInlineInferenceAsync(deployment, ct).ConfigureAwait(false); + await RunTransferAsync(deployment, ct).ConfigureAwait(false); + await RunAsynchronousInferenceAsync(deployment, ct).ConfigureAwait(false); + await RunSourceAsync(ct).ConfigureAwait(false); + } + + private async Task ReportModelsAsync(CancellationToken ct) + { + await foreach (AINodeEntry entry in m_client.EnumerateModelsAsync(ct).ConfigureAwait(false)) + { + AIModelSnapshot model = await m_client.Model(entry.NodeId).ReadAsync(ct) + .ConfigureAwait(false); + Console.WriteLine( + " {0} {1} {2} ({3})", + model.ModelId, + model.Name, + model.Version, + model.NodeId); + } + } + + private async Task DescribeDeploymentAsync(NodeId deploymentNodeId, CancellationToken ct) + { + AIDeploymentClient deployment = m_client.Deployment(deploymentNodeId); + AIDeploymentSnapshot snapshot = await deployment.ReadAsync(ct).ConfigureAwait(false); + + Console.WriteLine(); + Console.WriteLine("--- deployment {0}", snapshot.NodeId); + Console.WriteLine(" DeploymentId {0}", snapshot.DeploymentId); + Console.WriteLine(" InferenceLocation {0}", snapshot.InferenceLocation); + Console.WriteLine(" State {0}", snapshot.State); + Console.WriteLine(" DataJurisdiction {0}", snapshot.DataJurisdiction); + Console.WriteLine(" EgressPermitted {0}", snapshot.EgressPermitted); + Console.WriteLine(" MaxInlinePayloadSize {0}", snapshot.MaxInlinePayloadSize); + Console.WriteLine(" EndpointUri {0}", snapshot.EndpointUri); + + AIModelClient? model = await deployment.OpenModelAsync(ct).ConfigureAwait(false); + if (model is not null) + { + AIModelSnapshot modelSnapshot = await model.ReadAsync(ct).ConfigureAwait(false); + Console.WriteLine( + " uses model {0} ({1})", + modelSnapshot.ModelId, + modelSnapshot.NodeId); + Console.WriteLine( + " digest {0}", + modelSnapshot.Digest.Length > 0 + ? Convert.ToHexString(modelSnapshot.Digest.Span) + : "(none declared)"); + } + + if (!snapshot.FallbackDeploymentId.IsNull) + { + Console.WriteLine(" falls back to {0}", snapshot.FallbackDeploymentId); + } + } + + private static async Task RunCapabilitiesAsync(AIDeploymentClient deployment, CancellationToken ct) + { + Console.WriteLine(); + Console.WriteLine("--- GetCapabilities"); + ArrayOf capabilities = await deployment.GetCapabilitiesAsync(ct) + .ConfigureAwait(false); + for (int ii = 0; ii < capabilities.Count; ii++) + { + Console.WriteLine( + " {0}: {1}", + capabilities[ii].Name, + capabilities[ii].Supported); + } + } + + private static async Task RunInlineInferenceAsync(AIDeploymentClient deployment, CancellationToken ct) + { + Console.WriteLine(); + Console.WriteLine("--- Invoke"); + ByteString payload = ByteString.From(Encoding.UTF8.GetBytes( + "{\"messages\":[{\"role\":\"user\",\"content\":\"Summarise the last shift.\"}]}")); + AIInvokeResult result = await deployment.InvokeAsync( + payload, + "application/json", + ArrayOf.Empty, + 5000, + cancellationToken: ct).ConfigureAwait(false); + ReportInvokeOutputs(result); + } + + private async Task RunTransferAsync(AIDeploymentClient deployment, CancellationToken ct) + { + Console.WriteLine(); + Console.WriteLine("--- BeginTransfer"); + ByteString payload = ByteString.From(Encoding.UTF8.GetBytes( + "{\"messages\":[{\"role\":\"user\",\"content\":\"" + + new string('x', 4096) + + "\"}]}")); + AIBeginTransferResult begun = await deployment.BeginTransferAsync( + "application/json", (ulong)payload.Length, ct).ConfigureAwait(false); + if (!begun.Accepted) + { + Console.WriteLine(" refused"); + return; + } + Console.WriteLine(" transfer {0}", begun.TransferId); + + AIInferenceTransferClient transfer = m_client.Transfer(begun.TransferId); + await transfer.WriteRequestAsync(payload, cancellationToken: ct).ConfigureAwait(false); + bool accepted = await transfer.ExecuteAsync(ct).ConfigureAwait(false); + AITransferSnapshot snapshot = await transfer.ReadAsync(ct).ConfigureAwait(false); + Console.WriteLine(" accepted {0}", accepted); + Console.WriteLine(" state {0}", snapshot.State); + Console.WriteLine(" ModelUsed {0}", snapshot.ModelUsed); + + ByteString answer = await transfer.ReadResponseAsync(cancellationToken: ct) + .ConfigureAwait(false); + if (answer.Length > 0) + { + Console.WriteLine(" response {0}", Encoding.UTF8.GetString(answer.Span)); + } + } + + private async Task RunAsynchronousInferenceAsync(AIDeploymentClient deployment, CancellationToken ct) + { + Console.WriteLine(); + Console.WriteLine("--- InvokeAsync"); + ByteString payload = ByteString.From(Encoding.UTF8.GetBytes( + "{\"messages\":[{\"role\":\"user\",\"content\":\"Explain the trend.\"}]}")); + NodeId jobId = await deployment.InvokeAsyncAsync( + payload, + "application/json", + ArrayOf.Empty, + cancellationToken: ct).ConfigureAwait(false); + if (jobId.IsNull) + { + return; + } + Console.WriteLine(" job {0}", jobId); + + AIInferenceJobClient job = m_client.InferenceJob(jobId); + AIInferenceJobSnapshot snapshot = new(); + for (int attempt = 0; attempt < 50; attempt++) + { + snapshot = await job.ReadAsync(ct).ConfigureAwait(false); + if (snapshot.ResponsePayload.Length > 0 || !snapshot.ModelUsed.IsNull) + { + break; + } + await Task.Delay(200, ct).ConfigureAwait(false); + } + if (snapshot.ResponsePayload.Length > 0) + { + Console.WriteLine(" Response {0}", Encoding.UTF8.GetString(snapshot.ResponsePayload.Span)); + } + Console.WriteLine(" ModelUsed {0}", snapshot.ModelUsed); + Console.WriteLine(" FinishReason {0}", snapshot.FinishReason); + } + + private async Task RunSourceAsync(CancellationToken ct) + { + ArrayOf sourceIds = await m_client.DiscoverSourcesAsync(ct).ConfigureAwait(false); + if (sourceIds.Count == 0) + { + return; + } + + Console.WriteLine(); + Console.WriteLine("--- model source"); + AIModelSourceClient source = m_client.Source(sourceIds[0]); + AIModelSourceSnapshot snapshot = await source.ReadAsync(ct).ConfigureAwait(false); + Console.WriteLine(" SourceId {0}", snapshot.SourceId); + Console.WriteLine(" EndpointUri {0}", snapshot.EndpointUri); + Console.WriteLine(" ApiDialect {0}", snapshot.ApiDialect); + Console.WriteLine(" AuthenticationKind {0}", snapshot.AuthenticationKind); + Console.WriteLine(" CredentialReference {0}", snapshot.CredentialReference); + + AISourceConnectionResult connection = await source.TestConnectionAsync(ct) + .ConfigureAwait(false); + Console.WriteLine(" reachable {0} ({1})", connection.Reachable, connection.Detail); + + AISourceModelListResult models = await source.ListModelsAsync(maxResults: 20, cancellationToken: ct) + .ConfigureAwait(false); + for (int ii = 0; ii < models.Models.Count; ii++) + { + ModelReferenceDataType model = models.Models[ii]; + Console.WriteLine( + " offers {0}/{1}/{2}", + model.Publisher, + model.Name, + model.Version); + } + } + + private static void ReportInvokeOutputs(AIInvokeResult result) + { + if (result.ResponsePayload.Length > 0) + { + Console.WriteLine(" response {0}", Encoding.UTF8.GetString(result.ResponsePayload.Span)); + } + Console.WriteLine(" ModelUsed {0}", result.ModelUsed); + if (result.Usage is not null) + { + Console.WriteLine( + " Usage {0} in, {1} out {2}", + result.Usage.InputUnits, + result.Usage.OutputUnits, + result.Usage.UnitKind); + } + Console.WriteLine(" FinishReason {0}", result.FinishReason); + if (result.TransferRequired) + { + Console.WriteLine(" payload too large; transfer at {0}", result.TransferId); + } + } + + private readonly AIClient m_client; + } +} diff --git a/samples/AI/ModelManagementClient/AssemblyInfo.cs b/samples/AI/ModelManagementClient/AssemblyInfo.cs new file mode 100644 index 0000000000..bb6d2173b8 --- /dev/null +++ b/samples/AI/ModelManagementClient/AssemblyInfo.cs @@ -0,0 +1,33 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +// The OPC UA surface this builds on is not CLS compliant. +[assembly: CLSCompliant(false)] diff --git a/samples/AI/ModelManagementClient/ModelManagementClient.csproj b/samples/AI/ModelManagementClient/ModelManagementClient.csproj new file mode 100644 index 0000000000..55f96c092b --- /dev/null +++ b/samples/AI/ModelManagementClient/ModelManagementClient.csproj @@ -0,0 +1,26 @@ + + + Exe + ModelManagementClient + + net10.0;net9.0;net8.0 + $(CustomTestTarget) + net10.0 + true ModelManagement.Client + $(NoWarn);CS1591;CS0108 + enable + false + false + false + + + + + + + + + + + diff --git a/samples/AI/ModelManagementClient/Program.cs b/samples/AI/ModelManagementClient/Program.cs new file mode 100644 index 0000000000..d4365506fe --- /dev/null +++ b/samples/AI/ModelManagementClient/Program.cs @@ -0,0 +1,136 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.AI.Client; +using Opc.Ua.Client; + +try +{ + string endpoint = args.Length > 0 && !args[0].StartsWith("--", StringComparison.Ordinal) + ? args[0] + : "opc.tcp://localhost:62640/ModelManagementServer"; + + bool insecure = Array.IndexOf(args, "--insecure") >= 0; + + Console.WriteLine("OPC UA AI Model Management client"); + Console.WriteLine("Endpoint: {0}", endpoint); + + if (insecure) + { + Console.Error.WriteLine( + "WARNING: --insecure selects an endpoint without message security."); + } + + Console.WriteLine(); + + HostApplicationBuilder builder = Host.CreateApplicationBuilder(); + builder.Logging.ClearProviders(); + builder.Logging.AddConsole(); + builder.Logging.SetMinimumLevel(LogLevel.Warning); + + builder.Services + .AddOpcUa() + .AddClient(options => + { + const string applicationName = "ModelManagementClient"; + options.ApplicationName = applicationName; + options.ApplicationUri = "urn:localhost:OPCFoundation:ModelManagementClient"; + options.ProductUri = "uri:opcfoundation.org:ModelManagementClient"; + options.PkiRoot = Path.Combine( + Path.GetTempPath(), "OPC Foundation", applicationName, "pki"); + // Sample convenience only; never auto-accept in production. + options.AutoAcceptUntrustedCertificates = true; + options.RejectSHA1SignedCertificates = true; + options.MinimumCertificateKeySize = 2048; + options.Session = new ManagedSessionOptions + { + SessionName = applicationName, + SessionTimeout = TimeSpan.FromSeconds(60) + }; + }) + .AddDiscoveryAndConnect(options => + { + options.DiscoveryUrl = endpoint; + options.SecurityMode = insecure + ? MessageSecurityMode.None + : MessageSecurityMode.SignAndEncrypt; + options.SecurityPolicyUri = insecure + ? SecurityPolicies.None + : SecurityPolicies.Basic256Sha256; + }) + .AddAIClient(); + + using IHost host = builder.Build(); + await host.StartAsync(CancellationToken.None).ConfigureAwait(false); + + try + { + Func> connect = host.Services + .GetRequiredService>>(); + + ManagedSession session = await connect(CancellationToken.None).ConfigureAwait(false); + + await using (session) + { + Console.WriteLine("Connected."); + Console.WriteLine(); + + ITelemetryContext telemetry = host.Services.GetRequiredService(); + AIScenarioRunner? runner = AIScenarioRunner.TryCreate(session, telemetry); + + if (runner is null) + { + Console.Error.WriteLine( + "This Server does not implement OPC UA - AI Model Management."); + return 2; + } + + await runner.RunAsync(CancellationToken.None).ConfigureAwait(false); + } + } + finally + { + await host.StopAsync(CancellationToken.None).ConfigureAwait(false); + } + + return 0; +} +catch (Exception ex) +{ + Console.Error.WriteLine(ex); + return 1; +} diff --git a/samples/AI/ModelManagementServer/AssemblyInfo.cs b/samples/AI/ModelManagementServer/AssemblyInfo.cs new file mode 100644 index 0000000000..be83f31ace --- /dev/null +++ b/samples/AI/ModelManagementServer/AssemblyInfo.cs @@ -0,0 +1,35 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +// The OPC UA stack surface this builds on is not CLS compliant (unsigned +// integers appear throughout the specification's own data types), so claiming +// compliance here would be false. +[assembly: CLSCompliant(false)] diff --git a/samples/AI/ModelManagementServer/Dockerfile b/samples/AI/ModelManagementServer/Dockerfile new file mode 100644 index 0000000000..249a579055 --- /dev/null +++ b/samples/AI/ModelManagementServer/Dockerfile @@ -0,0 +1,108 @@ +# ModelManagementServer container image. +# +# Multi-stage, framework-dependent build on the .NET AzureLinux 3 base +# images, following samples/DI/PumpDeviceIntegrationServer/Dockerfile. +# +# IMPORTANT: the build context MUST be the repository ROOT (not this +# folder), because the project pulls in the whole source tree. Build it +# like this: +# +# # from the repository root: +# docker build -f samples/AI/ModelManagementServer/Dockerfile \ +# -t modelmanagementserver:local . +# +# The final image runs as a non-root user and listens on 0.0.0.0:62640 +# (override host/port via the `host` / `port` environment variables). + +FROM mcr.microsoft.com/dotnet/sdk:10.0-azurelinux3.0 AS build +WORKDIR /src + +# JIT is disabled for the source generators unless W^X is turned off on +# some kernels; match the existing sample Dockerfiles. +ENV DOTNET_EnableWriteXorExecute=0 + +# Copy the full repository. The project pulls in many transitive +# ProjectReferences, a source-generator analyzer, and an AdditionalFiles +# NodeSet2 XML input, so a per-csproj copy list would be fragile. +COPY . . + +# Fail fast with a clear message when the build context is not the +# repository root (UA.slnx is the root marker). +RUN test -f UA.slnx || { \ + echo ''; \ + echo 'ERROR: build the repository ROOT as the Docker context, not this folder.'; \ + echo ' The image needs the full source tree (src/, tools/).'; \ + echo ''; \ + echo ' From the repository root run:'; \ + echo ' docker build -f samples/AI/ModelManagementServer/Dockerfile -t modelmanagementserver .'; \ + echo ''; \ + exit 1; \ + } + +# The project imports use lower-case tools/, but the repository directory is +# Tools/. Windows hides the difference; the Linux container does not. +RUN if [ ! -d tools ] && [ -d Tools ]; then ln -s Tools tools; fi + +# MSBuild in the Linux SDK needs a Unix separator and stable logical names here +# so the source generator embeds the design resources it reads during publish. +RUN sed -i 's###' \ + Tools/Opc.Ua.SourceGeneration.Core/Opc.Ua.SourceGeneration.Core.csproj + +ARG Version +ARG SimpleVersion +ARG InformationalVersion + +RUN dotnet restore "samples/AI/ModelManagementServer/ModelManagementServer.csproj" \ + --force -p:NoHttps=true + +# PublishAot is off for this sample. The workload-identity credential path +# resolves tokens through a library that serialises by reflection, which the +# AOT analyzer warns on and this repository builds warnings as errors. Nothing +# in the OPC UA surface needs AOT; see the sample README. +RUN dotnet publish "samples/AI/ModelManagementServer/ModelManagementServer.csproj" \ + --no-restore \ + -c Release \ + -f net10.0 \ + -p:PublishAot=false \ + -p:NoHttps=true \ + -p:Dockerbuild=true \ + -p:Version=${Version} \ + -p:AssemblyVersion=${SimpleVersion} \ + -p:FileVersion=${Version} \ + -p:InformationalVersion=${InformationalVersion} \ + -o /app/publish + +FROM mcr.microsoft.com/dotnet/runtime:10.0-azurelinux3.0 AS final + +# AzureLinux forces the SymCrypt OpenSSL provider as the default for all +# EVP operations. SymCrypt's ECDSA SignHash path rejects the self-signing +# call the OPC UA stack makes when creating its ECC application-instance +# certificate. Clearing the forced default lets the standard OpenSSL +# default provider handle ECDSA. +RUN sed -i 's/^default_properties = .*/default_properties = ""/' /etc/pki/tls/openssl.cnf + +WORKDIR /app +COPY --from=build /app/publish . + +# The .NET base images ship a predefined non-root user `app` (UID 1654, +# exposed via APP_UID). The Server creates its certificate store under +# /app at runtime. +# +# /var/run/secrets/ai is where a mounted credential is read from. It is +# created empty so the anonymous path works without a Secret, and so the +# directory permissions are right when one is mounted. +ENV HOME=/app +RUN mkdir -p /app/pki /var/run/secrets/ai \ + && chown -R $APP_UID:0 /app /var/run/secrets/ai \ + && chmod -R g=u /app /var/run/secrets/ai + +USER $APP_UID + +# Bind to all interfaces so the endpoint is reachable from outside the +# container. Override with `-e host=...` / `-e port=...`. +ENV host=0.0.0.0 \ + port=62640 + +EXPOSE 62640 + +ENTRYPOINT ["dotnet", "ModelManagementServer.dll"] diff --git a/samples/AI/ModelManagementServer/ModelManagementServer.csproj b/samples/AI/ModelManagementServer/ModelManagementServer.csproj new file mode 100644 index 0000000000..6a629e6ee0 --- /dev/null +++ b/samples/AI/ModelManagementServer/ModelManagementServer.csproj @@ -0,0 +1,30 @@ + + + Exe + ModelManagementServer + + net10.0;net9.0;net8.0 + $(CustomTestTarget) + net10.0 + true ModelManagement.Server + $(NoWarn);CS1591;CS0108 + enable + false + false + false + + + + + + + + + + + + + + + diff --git a/samples/AI/ModelManagementServer/Program.cs b/samples/AI/ModelManagementServer/Program.cs new file mode 100644 index 0000000000..4a0da2cd92 --- /dev/null +++ b/samples/AI/ModelManagementServer/Program.cs @@ -0,0 +1,81 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; +using Opc.Ua.AI.Server.Hosting; +using Opc.Ua.Server.Fluent; + +HostApplicationBuilder builder = Host.CreateApplicationBuilder(args); + +builder.Logging.ClearProviders(); +builder.Logging.AddConsole(); + +int port = int.TryParse(builder.Configuration["port"], out int p) ? p : 62640; + +// 0.0.0.0 so the Server is reachable from outside a container. Override with +// --host for local-only development. +string host = builder.Configuration["host"] is { Length: > 0 } h ? h : "0.0.0.0"; + +builder.Services.AddRestChatCompletionsAIChatClientFactory(); + +// InferenceBackend:Kind defaults to ChatClient, the Microsoft.Extensions.AI +// path. Set it to RestChatCompletions only for endpoints where the host cannot +// supply an IChatClient and the OpenAI-compatible REST contract is the wire +// contract itself. +builder.Services + .AddOpcUa() + .AddServer(o => + { + o.ApplicationName = "ModelManagementServer"; + o.ApplicationUri = "urn:localhost:OPCFoundation:ModelManagementServer"; + o.ProductUri = "uri:opcfoundation.org:ModelManagementServer"; + // Sample convenience only; never auto-accept untrusted certificates in + // production. + o.AutoAcceptUntrustedCertificates = true; + o.PkiRoot = Path.Combine(AppContext.BaseDirectory, "pki"); + o.RejectSHA1Certificates = true; + o.MinCertificateKeySize = 2048; + o.EndpointUrls.Add($"opc.tcp://{host}:{port}/ModelManagementServer"); + }) + .AddAI( + ai => builder.Configuration.GetSection(AIOptions.SectionName).Bind(ai), + backend => builder.Configuration.GetSection(InferenceBackendOptions.SectionName).Bind(backend), + fallback => builder.Configuration + .GetSection(InferenceBackendOptions.FallbackSectionName) + .Bind(fallback)); + +await builder.Build().RunAsync().ConfigureAwait(false); +return 0; diff --git a/samples/AI/README.md b/samples/AI/README.md new file mode 100644 index 0000000000..04cddb1de6 --- /dev/null +++ b/samples/AI/README.md @@ -0,0 +1,282 @@ +# AI Model Management sample + +An OPC UA Server that exposes AI models — hosted somewhere else, or running on +the machine — through the draft companion specification *OPC UA — AI Model +Management and Inference*. Plus a client that exercises it, and a Helm chart +that runs it on Kubernetes. + +The point of the sample is the claim the specification makes: **where inference +runs should not change how it is called.** A client discovers a deployment, +calls `Invoke`, and gets an answer together with the identity of the model that +produced it — whether that model was a hosted service in another jurisdiction or +a quantized copy on the same box. + +> The specification is a **draft**. Namespace URIs and NodeIds are provisional +> and nothing here is endorsed by the OPC Foundation. + +## What is in here + +The reusable parts live under `src/` as the `Opc.Ua.AI` package family, in the +same shape as the Robotics and Vision families: a model assembly, a server-side +assembly, a client-side assembly, and the inference backends. What remains here +is the two sample applications that compose them. + +| | | +|---|---| +| `../../src/Opc.Ua.AI/` | The companion specification's NodeSet, source-generated at build time | +| `../../src/Opc.Ua.AI.Inference/` | Reaching a model: the `IInferenceBackend` contract, a `Microsoft.Extensions.AI` `IChatClient` backend, an OpenAI-compatible REST backend, credential resolution | +| `../../src/Opc.Ua.AI.Server/` | The node manager: publishes the address space and serves the specification's Methods | +| `../../src/Opc.Ua.AI.Client/` | Discovery, typed reads, Method calls and artefact transfer | +| `ModelManagementServer/` | The Server sample. Hosts the node manager and picks a backend | +| `ModelManagementClient/` | A console client that browses from the entry point and exercises what it finds | +| `deploy/helm/` | The chart, its tests, and a cluster smoke test | +| `../../tests/Opc.Ua.AI.Tests/` | Unit and integration tests against a fake backend | + +**No cloud-vendor SDK is referenced anywhere.** The default backend goes +through `Microsoft.Extensions.AI`, which a hosted service and an on-device +runtime both implement, and workload identity is read from the token the +platform projects — the mechanism every platform implements underneath its own +SDK. A sample that only ran against one vendor would not be demonstrating a +platform-independent Server. `RestChatCompletionsBackend` remains available +when the endpoint only exposes the OpenAI-compatible REST contract and the host +cannot supply an `IChatClient`; set `InferenceBackend:Kind` (or +`FallbackInferenceBackend:Kind`) to `RestChatCompletions` for that wire +contract. + +## Running it + +The Server needs something to infer with. The quickest is the throwaway endpoint +in `verify_backend.py`, which speaks just enough of the OpenAI-compatible +contract to answer: + +```powershell +python samples/AI/verify_backend.py 5273 +``` + +Then, in two more terminals: + +```powershell +dotnet run --project samples/AI/ModelManagementServer +dotnet run --project samples/AI/ModelManagementClient +``` + +The client browses to the AI root under the Server Object, walks every +deployment, follows `UsesModel` to the model and its digest, and then calls +`GetCapabilities`, `Invoke`, `BeginTransfer`, `InvokeAsync` and the source's +`TestConnection` and `ListModels`. + +Point it at a real endpoint by configuration: + +```powershell +$env:InferenceBackend__Kind = "ChatClient" +$env:InferenceBackend__EndpointUri = "https://.services.ai.azure.com/openai/" +$env:InferenceBackend__Authentication = "ApiKey" +$env:InferenceBackend__CredentialReference = "inference-api-key" +$env:InferenceBackend__Site = "Cloud" +$env:InferenceBackend__EgressPermitted = "true" +``` + +`ChatClient` is the default. This sample registers a small `IChatClient` over +the OpenAI-compatible endpoint so no vendor package is needed. A production +host can replace `IChatClientFactory` with one that creates the chat client +from its own SDK or local runtime. Use `RestChatCompletions` for deployments +whose contract is the REST shape itself rather than the +`Microsoft.Extensions.AI` abstraction. + +## The control plane is OPC UA + +There is no second management API, and that is a decision rather than an +omission. The specification defines Methods for the things an operator needs to +do — test a source, list what it offers, invoke a model, start a job, promote a +candidate — so adding an HTTP surface beside them would create two ways to do +the same thing that could disagree. + +Everything else is startup configuration: which endpoint to reach, which +deployments to publish, and which credential to present. + +## What the address space says, and why + +A few members are worth reading carefully, because they answer questions that +are asked *before* a call rather than after one. + +**`ModelUsed`** names the model that actually produced a result. It exists for +the fallback case: a caller that cannot see which model answered cannot tell a +degraded answer from a good one, and a fallback that answers silently looks +exactly like a healthy primary. This is the single thing in the sample most +worth getting right, and the one the tests press hardest on. + +**`EgressPermitted`, `DataJurisdiction`, `RetainsInput`** say where the data +goes. Egress is not made false by encryption — that answers who can read data in +flight, not where the data went. + +**`MaxInlinePayloadSize`** is published before a client calls, rather than +discovered from a rejection, because the real bound is the smallest of several +limits a client can see none of. + +**`CredentialReference`** names the credential; it never carries one. A client +is entitled to know *which* credential is configured so it can tell whether the +right one is. A client that could read the value could use it. + +**`Digest`** is empty when the backend declares none. A hosted endpoint that +will not say which weights answered cannot be made to say so by hashing its +name, and a digest that looks like an artefact digest but is not one is worse +than none, because something will eventually compare it. + +## What is real and what is not + +Being clear about this matters more than it looks, because a sample is the thing +people copy. + +**Real.** The address space, the Methods, the provenance references, the +chunked transfer over Part 5 `FileType`, the asynchronous job on the Part 10 +program lifecycle, the fallback and its reporting, the credential handling, and +the HTTP client that reaches an OpenAI-compatible endpoint. The learning job +node is also real: the Server publishes a `LearningJobType` instance, and +`SamplesCollected` is incremented only when host-level code records a distinct +ground-truth sample. Empty or retracted observations count the same way as +samples carrying geometry. All of the inference pieces have been run end to end +against a live endpoint, in a container, and in a Kubernetes cluster. + +**Not real.** There is no retraining loop. A sample cannot retrain a model, and +`PromoteModel` is not wired to a simulated MLOps integration — faking candidate +generation or timed promotion would mislead a reader about the one part of the +specification a sample cannot honestly demonstrate. + +**A test double, not a provider.** `FakeInferenceBackend` lives in the test +project and is not configurable from the Server. It exists so CI needs no +inference service; if it shipped as a supported option the sample could look +healthy while never having reached a model. + +## Kubernetes + +```powershell +docker build -f samples/AI/ModelManagementServer/Dockerfile ` + -t modelmanagementserver:local . + +helm install ai samples/AI/deploy/helm/ai-model-management ` + --set image.repository=modelmanagementserver ` + --set image.tag=local --set image.pullPolicy=Never +``` + +The defaults describe an on-device runtime on loopback with no credential, +because that is the shape that installs and runs without anything else existing. +`deploy/helm/values-cloud.yaml` is the hosted shape: a remote endpoint with a +mounted credential, and a local fallback. + +**Supply the credential as a Secret you manage**, not through Helm: + +```powershell +kubectl create secret generic inference-credentials ` + --from-literal=inference-api-key= + +helm install ai ./ai-model-management -f values-cloud.yaml ` + --set credentials.existingSecret=inference-credentials +``` + +`credentials.create` exists for local clusters and puts the value in the release +history, where `helm get values` will show it. The chart says so when you use it. + +### The chart refuses some configurations + +Each refusal corresponds to a deployment that would come up green and describe +itself wrongly, which is worse than one that fails to start because nobody +investigates a healthy pod: + +- `ApiKey` authentication with no credential mounted. +- A fallback deployment with no fallback endpoint — it would always fail. +- A fallback pointing at the primary's endpoint — that is a retry, and it fails + for every reason the primary just failed for. +- `EgressPermitted: false` with a backend that is not on the machine — the + Server would publish a promise it does not keep. + +### Probes + +The probes are TCP against the OPC UA port. That proves the listener accepts +connections; it does **not** prove the Server is serving the address space. An +exec probe running a real OPC UA client would prove it and costs a process +launch every period. The trade is deliberate, and it is the reason the smoke +test drives a real session rather than trusting readiness. + +### Testing the chart + +```powershell +helm lint samples/AI/deploy/helm/ai-model-management +python samples/AI/deploy/helm/chart_tests.py + +# slow, needs Docker and kind; creates and deletes a cluster +pwsh samples/AI/deploy/helm/smoke-test.ps1 +``` + +`chart_tests.py` renders the chart and asserts what the manifests must and must +not say — including that a credential passed inline appears exactly once, inside +the Secret it belongs in. Every refusal is asserted to fire, because a guardrail +that never triggers is indistinguishable from one that does not work. + +`smoke-test.ps1` builds the image, creates a cluster, deploys a stub endpoint +alongside, installs the chart, and opens a real OPC UA session from outside. It +is the slowest and most fragile check here — image build, image load, chart +install and a live session — so run it deliberately rather than letting it gate +everything else. + +## Why AOT is disabled + +`PublishAot` is off for this sample. The workload-identity credential path +resolves tokens through a library that serialises by reflection, which the AOT +and trimming analyzers warn on, and this repository builds warnings as errors. +Nothing in the OPC UA surface requires AOT, and disabling it for one sample is +preferable to weakening the analyzer settings for the whole repository. + +## Notes for anyone extending this + +Six things cost real time to find. All six produce a Server that builds, starts, +and is wrong in a way nothing reports: + +1. **Dynamic NodeIds must not be numeric here.** The model occupies + `ns=2;i=1001`…`ns=2;i=7001`, and a counter issuing numeric ids in the same + namespace walks straight into it. The predefined-node index takes the last + writer, so a Server that had served a few hundred transfers would quietly have + replaced `AiRootType` with an inference job's `FinishedAt` property. String + identifiers cannot collide with numeric ones at all, which is a stronger + guarantee than any seed value. + +2. **The NodeSet may already declare the entry point.** This one declares + `ns=2;i=7001` parented to the Server Object. Building another leaves two + Objects with the same BrowseName under the Server — one populated, one empty — + and which a client finds depends on browse order. + +3. **A NodeSet is not necessarily in supertype order.** NodeIds are assigned in + declaration order, so a model that gains an abstract base after its first + concrete subtype carries a higher NodeId for the base — and the type table + refuses a type whose supertype it has not seen. + +4. **Passing a parent to a `NodeState` constructor and then calling `AddChild` + leaves `ReferenceTypeId` null**, because `AddChild` only assigns it when the + parent actually changes. The node is indexed, readable by NodeId, and + invisible to `Browse`. + +5. **A child materialised by `CreateChild` is never initialised**, so a Method + has no `InputArguments`. It browses, accepts a call, and rejects it with + `BadTooManyArguments` whatever is passed. `FindChild` does not create at all, + so an optional Method reached that way is simply absent. + +6. **A member — or a folder — created after `AddPredefinedNode` is not + published.** The lazily created `Jobs` folder appeared in a Browse of the root + while returning `BadNodeIdUnknown` when browsed itself, because it was on the + NodeState tree and not in the index. + +The common thread is that none of them fails. Run a real client against a real +Server before believing a green build — every one of these was found that way, +and three of them only after a review agent browsed the running Server. + +### Concurrency + +Method calls are **not** serialised by the Server, so two handlers can run at +once on the same node. Two consequences worth knowing: + +- The transfer buffers are owned by `StreamFileManager` and reached only through + `Snapshot`/`Replace`. Holding the node manager's own lock while touching them + would look careful and guarantee nothing, because the FileType methods take a + different lock. +- A transfer can be aborted or expire while its inference is in flight. The + completing call re-checks that the transfer is still live under the same lock + that removal takes, and drops the answer if it is not. diff --git a/samples/AI/deploy/helm/ai-model-management/Chart.yaml b/samples/AI/deploy/helm/ai-model-management/Chart.yaml new file mode 100644 index 0000000000..09de5b0b20 --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/Chart.yaml @@ -0,0 +1,15 @@ +apiVersion: v2 +name: ai-model-management +description: | + An OPC UA Server that exposes AI models - hosted or on-device - through the + OPC UA AI Model Management and Inference companion specification. +type: application +version: 0.1.0 +appVersion: "0.2.0" +keywords: + - opcua + - ai + - inference +home: https://github.com/OPCFoundation/UA-.NETStandard +sources: + - https://github.com/OPCFoundation/UA-.NETStandard diff --git a/samples/AI/deploy/helm/ai-model-management/templates/NOTES.txt b/samples/AI/deploy/helm/ai-model-management/templates/NOTES.txt new file mode 100644 index 0000000000..9e81c6394a --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/templates/NOTES.txt @@ -0,0 +1,39 @@ +The AI Model Management sample Server is deploying. + + Release: {{ .Release.Name }} + Namespace: {{ .Release.Namespace }} + Endpoint: opc.tcp://{{ include "ai-model-management.fullname" . }}.{{ .Release.Namespace }}.svc:{{ .Values.service.port }}/ModelManagementServer + +Reach it from your machine: + + kubectl --namespace {{ .Release.Namespace }} port-forward svc/{{ include "ai-model-management.fullname" . }} {{ .Values.service.port }}:{{ .Values.service.port }} + dotnet run --project samples/AI/ModelManagementClient -- opc.tcp://localhost:{{ .Values.service.port }}/ModelManagementServer + +What this Server will tell a client about itself: + + Inference runs {{ .Values.backend.site }} + Jurisdiction {{ .Values.backend.dataJurisdiction }} + Egress permitted {{ .Values.backend.egressPermitted }} + Input retained {{ .Values.backend.retainsInput }} +{{- if .Values.ai.enableFallback }} + Fallback published, reaching {{ .Values.fallbackBackend.endpointUri }} +{{- else }} + Fallback none; a failed inference fails +{{- end }} + +{{- if not (include "ai-model-management.hasCredential" .) }} + +No credential is mounted. That is correct for an anonymous endpoint or for +workload identity, and wrong for anything else. +{{- end }} +{{- if .Values.credentials.create }} + +WARNING: credentials.create put the secret through Helm, so it is in the release +history and in `helm get values`. Use credentials.existingSecret for anything +that is not a local cluster. +{{- end }} +{{- if not .Values.persistence.enabled }} + +WARNING: persistence is off, so the Server generates a new application instance +certificate on every restart and every client has to trust it again. +{{- end }} diff --git a/samples/AI/deploy/helm/ai-model-management/templates/_helpers.tpl b/samples/AI/deploy/helm/ai-model-management/templates/_helpers.tpl new file mode 100644 index 0000000000..6d742ced23 --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/templates/_helpers.tpl @@ -0,0 +1,69 @@ +{{/* +Chart name and fullname, per the standard Helm conventions. +*/}} +{{- define "ai-model-management.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{- define "ai-model-management.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{- define "ai-model-management.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{- define "ai-model-management.labels" -}} +helm.sh/chart: {{ include "ai-model-management.chart" . }} +{{ include "ai-model-management.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{- define "ai-model-management.selectorLabels" -}} +app.kubernetes.io/name: {{ include "ai-model-management.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{- define "ai-model-management.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "ai-model-management.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} + +{{/* +The Secret holding the credential, whichever way it was supplied. + +Returns an empty string when there is none, which is what the anonymous and +workload-identity paths want: neither stores a secret at all, so mounting an +empty volume for them would only invite someone to fill it. +*/}} +{{- define "ai-model-management.credentialSecretName" -}} +{{- if .Values.credentials.existingSecret }} +{{- .Values.credentials.existingSecret }} +{{- else if .Values.credentials.create }} +{{- printf "%s-credentials" (include "ai-model-management.fullname" .) }} +{{- end }} +{{- end }} + +{{/* +Whether a credential is mounted at all. +*/}} +{{- define "ai-model-management.hasCredential" -}} +{{- if include "ai-model-management.credentialSecretName" . -}} +true +{{- end }} +{{- end }} diff --git a/samples/AI/deploy/helm/ai-model-management/templates/deployment.yaml b/samples/AI/deploy/helm/ai-model-management/templates/deployment.yaml new file mode 100644 index 0000000000..2736cc47eb --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/templates/deployment.yaml @@ -0,0 +1,176 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "ai-model-management.fullname" . }} + labels: + {{- include "ai-model-management.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + {{- include "ai-model-management.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "ai-model-management.selectorLabels" . | nindent 8 }} + annotations: + {{- with .Values.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- if and .Values.credentials.create .Values.credentials.value }} + # Rolls the pods when the credential changes. The annotation carries a + # digest of the credential's NAME and reference, never of the value: + # Pod annotations are readable by anyone with `get pods`, which is granted + # far more widely than `get secrets`, and a digest of a low-entropy secret + # is recoverable offline. The name changes when the release does, which is + # the case this exists to cover on the demo-only inline path. + checksum/credentials: {{ printf "%s/%s" (include "ai-model-management.credentialSecretName" .) .Values.backend.credentialReference | sha256sum }} + {{- end }} + spec: + serviceAccountName: {{ include "ai-model-management.serviceAccountName" . }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + securityContext: + {{- toYaml .Values.podSecurityContext | nindent 8 }} + containers: + - name: server + image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + securityContext: + {{- toYaml .Values.securityContext | nindent 12 }} + ports: + - name: opcua + containerPort: {{ .Values.server.port }} + protocol: TCP + env: + - name: host + value: {{ .Values.server.host | quote }} + - name: port + value: {{ .Values.server.port | quote }} + + # What this Server publishes. + - name: AiModelManagement__PrimaryDeploymentId + value: {{ .Values.ai.primaryDeploymentId | quote }} + - name: AiModelManagement__FallbackDeploymentId + value: {{ .Values.ai.fallbackDeploymentId | quote }} + - name: AiModelManagement__EnableFallback + value: {{ .Values.ai.enableFallback | quote }} + - name: AiModelManagement__EnableCatalogue + value: {{ .Values.ai.enableCatalogue | quote }} + - name: AiModelManagement__SourceId + value: {{ .Values.ai.sourceId | quote }} + + # The endpoint the primary deployment reaches. Note what is NOT here: + # the credential. Only its name travels, and the value is read from a + # mounted file by the one component that needs it. + - name: InferenceBackend__Site + value: {{ .Values.backend.site | quote }} + - name: InferenceBackend__EndpointUri + value: {{ .Values.backend.endpointUri | quote }} + - name: InferenceBackend__ChatCompletionsPath + value: {{ .Values.backend.chatCompletionsPath | quote }} + - name: InferenceBackend__ProbePath + value: {{ .Values.backend.probePath | quote }} + - name: InferenceBackend__Authentication + value: {{ .Values.backend.authentication | quote }} + - name: InferenceBackend__CredentialReference + value: {{ .Values.backend.credentialReference | quote }} + - name: InferenceBackend__ApiKeyHeader + value: {{ .Values.backend.apiKeyHeader | quote }} + - name: InferenceBackend__CredentialDirectory + value: {{ .Values.credentials.mountPath | quote }} + - name: InferenceBackend__DataJurisdiction + value: {{ .Values.backend.dataJurisdiction | quote }} + - name: InferenceBackend__EgressPermitted + value: {{ .Values.backend.egressPermitted | quote }} + - name: InferenceBackend__RetainsInput + value: {{ .Values.backend.retainsInput | quote }} + - name: InferenceBackend__MaxInlinePayloadSize + value: {{ .Values.backend.maxInlinePayloadSize | quote }} + {{- range $index, $model := .Values.backend.models }} + - name: InferenceBackend__Models__{{ $index }}__Publisher + value: {{ $model.publisher | quote }} + - name: InferenceBackend__Models__{{ $index }}__Name + value: {{ $model.name | quote }} + - name: InferenceBackend__Models__{{ $index }}__Version + value: {{ $model.version | quote }} + {{- if $model.taskKind }} + - name: InferenceBackend__Models__{{ $index }}__TaskKind + value: {{ $model.taskKind | quote }} + {{- end }} + {{- end }} + + # The fallback's own endpoint, which is the point of it. + - name: FallbackInferenceBackend__Enabled + value: {{ .Values.fallbackBackend.enabled | quote }} + {{- if .Values.fallbackBackend.enabled }} + - name: FallbackInferenceBackend__Site + value: {{ .Values.fallbackBackend.site | quote }} + - name: FallbackInferenceBackend__EndpointUri + value: {{ .Values.fallbackBackend.endpointUri | quote }} + - name: FallbackInferenceBackend__Authentication + value: {{ .Values.fallbackBackend.authentication | quote }} + - name: FallbackInferenceBackend__DataJurisdiction + value: {{ .Values.fallbackBackend.dataJurisdiction | quote }} + - name: FallbackInferenceBackend__EgressPermitted + value: {{ .Values.fallbackBackend.egressPermitted | quote }} + - name: FallbackInferenceBackend__RetainsInput + value: {{ .Values.fallbackBackend.retainsInput | quote }} + {{- end }} + {{- with .Values.extraEnv }} + {{- toYaml . | nindent 12 }} + {{- end }} + {{- if .Values.probes.liveness.enabled }} + livenessProbe: + tcpSocket: + port: opcua + initialDelaySeconds: {{ .Values.probes.liveness.initialDelaySeconds }} + periodSeconds: {{ .Values.probes.liveness.periodSeconds }} + failureThreshold: {{ .Values.probes.liveness.failureThreshold }} + {{- end }} + {{- if .Values.probes.readiness.enabled }} + readinessProbe: + tcpSocket: + port: opcua + initialDelaySeconds: {{ .Values.probes.readiness.initialDelaySeconds }} + periodSeconds: {{ .Values.probes.readiness.periodSeconds }} + failureThreshold: {{ .Values.probes.readiness.failureThreshold }} + {{- end }} + resources: + {{- toYaml .Values.resources | nindent 12 }} + volumeMounts: + - name: pki + mountPath: /app/pki + {{- if include "ai-model-management.hasCredential" . }} + - name: credentials + mountPath: {{ .Values.credentials.mountPath }} + readOnly: true + {{- end }} + volumes: + - name: pki + {{- if .Values.persistence.enabled }} + persistentVolumeClaim: + claimName: {{ include "ai-model-management.fullname" . }}-pki + {{- else }} + emptyDir: {} + {{- end }} + {{- if include "ai-model-management.hasCredential" . }} + - name: credentials + secret: + secretName: {{ include "ai-model-management.credentialSecretName" . }} + defaultMode: 0400 + {{- end }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} diff --git a/samples/AI/deploy/helm/ai-model-management/templates/pvc.yaml b/samples/AI/deploy/helm/ai-model-management/templates/pvc.yaml new file mode 100644 index 0000000000..476d9ceb6f --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/templates/pvc.yaml @@ -0,0 +1,17 @@ +{{- if .Values.persistence.enabled }} +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ include "ai-model-management.fullname" . }}-pki + labels: + {{- include "ai-model-management.labels" . | nindent 4 }} +spec: + accessModes: + - {{ .Values.persistence.accessMode }} + {{- if .Values.persistence.storageClass }} + storageClassName: {{ .Values.persistence.storageClass | quote }} + {{- end }} + resources: + requests: + storage: {{ .Values.persistence.size }} +{{- end }} diff --git a/samples/AI/deploy/helm/ai-model-management/templates/secret.yaml b/samples/AI/deploy/helm/ai-model-management/templates/secret.yaml new file mode 100644 index 0000000000..68cd33df90 --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/templates/secret.yaml @@ -0,0 +1,16 @@ +{{- if and .Values.credentials.create (not .Values.credentials.existingSecret) }} +{{- if not .Values.credentials.value }} +{{- fail "credentials.create is set but credentials.value is empty. Supply the value, or point credentials.existingSecret at a Secret you manage." }} +{{- end }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ include "ai-model-management.fullname" . }}-credentials + labels: + {{- include "ai-model-management.labels" . | nindent 4 }} +type: Opaque +stringData: + # The key is the name the address space publishes as CredentialReference, so + # a client can tell WHICH credential is configured without being able to read it. + {{ .Values.backend.credentialReference }}: {{ .Values.credentials.value | quote }} +{{- end }} diff --git a/samples/AI/deploy/helm/ai-model-management/templates/service.yaml b/samples/AI/deploy/helm/ai-model-management/templates/service.yaml new file mode 100644 index 0000000000..295fe59478 --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/templates/service.yaml @@ -0,0 +1,19 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "ai-model-management.fullname" . }} + labels: + {{- include "ai-model-management.labels" . | nindent 4 }} + {{- with .Values.service.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + type: {{ .Values.service.type }} + ports: + - port: {{ .Values.service.port }} + targetPort: opcua + protocol: TCP + name: opcua + selector: + {{- include "ai-model-management.selectorLabels" . | nindent 4 }} diff --git a/samples/AI/deploy/helm/ai-model-management/templates/serviceaccount.yaml b/samples/AI/deploy/helm/ai-model-management/templates/serviceaccount.yaml new file mode 100644 index 0000000000..3e80421a3d --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/templates/serviceaccount.yaml @@ -0,0 +1,12 @@ +{{- if .Values.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "ai-model-management.serviceAccountName" . }} + labels: + {{- include "ai-model-management.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/samples/AI/deploy/helm/ai-model-management/templates/validate.yaml b/samples/AI/deploy/helm/ai-model-management/templates/validate.yaml new file mode 100644 index 0000000000..114d6960b7 --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/templates/validate.yaml @@ -0,0 +1,35 @@ +{{/* +Refuses configurations that would deploy something misleading. + +Each of these is a real mistake that produces a Server which looks healthy and +answers questions wrongly - which is worse than one that fails to start, because +nobody investigates a green pod. +*/}} + +{{- if and (ne .Values.backend.authentication "Anonymous") (not (include "ai-model-management.hasCredential" .)) }} +{{- fail "backend.authentication expects a credential but none is supplied. Set credentials.existingSecret, or use authentication: Anonymous for an endpoint that needs none. This applies to BearerToken and WorkloadIdentity as much as ApiKey: without a credential the Server sends no Authorization header at all, while the address space still says it authenticates." }} +{{- end }} + +{{- if and (ne .Values.backend.authentication "Anonymous") (not .Values.backend.credentialReference) }} +{{- fail "backend.credentialReference is empty, so the Server would authenticate with nothing while publishing AuthenticationKind as though it did. Name the key within the Secret (or the token scope, for WorkloadIdentity)." }} +{{- end }} + +{{- if and .Values.fallbackBackend.enabled (ne .Values.fallbackBackend.site "OnServer") (not .Values.fallbackBackend.egressPermitted) }} +{{- fail "fallbackBackend.egressPermitted is false but its site is not OnServer, so the fallback deployment would publish EgressPermitted=false while sending payloads off the machine." }} +{{- end }} + +{{- if and .Values.credentials.create .Values.credentials.existingSecret }} +{{- fail "Set either credentials.existingSecret or credentials.create, not both." }} +{{- end }} + +{{- if and .Values.ai.enableFallback (not .Values.fallbackBackend.enabled) }} +{{- fail "ai.enableFallback publishes a fallback deployment, but fallbackBackend.enabled is false, so it has nothing to reach. A fallback that always fails is worse than none: set fallbackBackend.enabled, or turn ai.enableFallback off." }} +{{- end }} + +{{- if and .Values.fallbackBackend.enabled (eq .Values.fallbackBackend.endpointUri .Values.backend.endpointUri) }} +{{- fail "The fallback endpoint is the same as the primary. That is a retry, not a fallback: it fails for every reason the primary just failed for." }} +{{- end }} + +{{- if and (not .Values.backend.egressPermitted) (ne .Values.backend.site "OnServer") }} +{{- fail "backend.egressPermitted is false but backend.site is not OnServer, so this Server would publish EgressPermitted=false while sending payloads off the machine. Set egressPermitted, or point the backend at a local runtime." }} +{{- end }} diff --git a/samples/AI/deploy/helm/ai-model-management/values.yaml b/samples/AI/deploy/helm/ai-model-management/values.yaml new file mode 100644 index 0000000000..77e8b01bee --- /dev/null +++ b/samples/AI/deploy/helm/ai-model-management/values.yaml @@ -0,0 +1,156 @@ +# Values for the AI Model Management sample. +# +# The two shapes worth knowing about are a hosted endpoint reached with a +# credential, and an on-device runtime reached over loopback with none. Both are +# the same chart with a different `backend` block; see README.md. + +replicaCount: 1 + +image: + repository: ghcr.io/opcfoundation/modelmanagementserver + tag: latest + pullPolicy: IfNotPresent + +imagePullSecrets: [] +nameOverride: "" +fullnameOverride: "" + +server: + # Port the OPC UA endpoint listens on. + port: 62640 + # 0.0.0.0 so the endpoint is reachable from outside the pod. + host: "0.0.0.0" + +service: + type: ClusterIP + port: 62640 + annotations: {} + +# What this Server publishes. +ai: + primaryDeploymentId: primary + fallbackDeploymentId: fallback + # A fallback deployment and the FallsBackTo reference from the primary to it. + # Off by default because the default backend is already on the machine: there + # is nowhere useful to fall back TO, and a fallback that always fails is worse + # than none. Turn it on together with fallbackBackend.enabled. + enableFallback: false + enableCatalogue: true + sourceId: model-source + +# The endpoint the primary deployment reaches. +# +# The default is an on-device runtime on loopback with no credential, because +# that is the shape that installs and runs without anything else existing. For a +# hosted endpoint see values-cloud.yaml. +backend: + # OnServer, EdgeOffServer or Cloud. This is what a client reads from + # InferenceLocation, so it should describe where inference actually runs - + # it is an operational fact, not a label. + site: OnServer + endpointUri: "http://localhost:5273/" + chatCompletionsPath: "v1/chat/completions" + probePath: "v1/models" + # Anonymous, ApiKey, BearerToken or WorkloadIdentity. + authentication: Anonymous + # Names the key WITHIN the mounted Secret - or, for WorkloadIdentity, the token + # scope being requested. Never the secret itself: this value is published in the + # address space so a client can tell whether the right credential is configured, + # and a client that could read the value could use it. + credentialReference: "" + apiKeyHeader: api-key + # Where the data goes. Egress is not made false by encryption, which answers + # who can read data in flight and not where the data went. + dataJurisdiction: "on-premises" + egressPermitted: false + retainsInput: false + # Published before a client calls rather than discovered from a rejection. + maxInlinePayloadSize: 65536 + models: [] + # - publisher: contoso + # name: weld-inspect + # version: 2.1.0 + # taskKind: classification + +# The fallback deployment's own endpoint. Deliberately a separate block: a +# fallback reached through the same endpoint and credentials as the primary is a +# retry, not a fallback, and fails for every reason the primary just failed for. +fallbackBackend: + enabled: false + site: OnServer + endpointUri: "http://localhost:5273/" + authentication: Anonymous + dataJurisdiction: "on-premises" + egressPermitted: false + retainsInput: false + +# The credential, mounted as a file. +# +# `existingSecret` is the option to use in anything real: the value never passes +# through Helm, so it is not in the release history, not in `helm get values`, +# and not in whatever holds the values file. +credentials: + existingSecret: "" + # Creates a Secret from `value`. For local clusters and demonstrations only. + create: false + value: "" + mountPath: /var/run/secrets/ai + +# The certificate store. A Server that regenerates its application instance +# certificate on every restart has to be re-trusted on every restart. +persistence: + enabled: true + size: 1Gi + storageClass: "" + accessMode: ReadWriteOnce + +# A TCP probe proves the listener accepts connections. It does NOT prove the +# Server is serving the address space - only an OPC UA client can do that, and +# an exec probe running one costs a process launch per probe. The trade is +# deliberate; see README.md. +probes: + liveness: + enabled: true + initialDelaySeconds: 20 + periodSeconds: 20 + failureThreshold: 3 + readiness: + enabled: true + initialDelaySeconds: 10 + periodSeconds: 10 + failureThreshold: 3 + +resources: + requests: + cpu: 100m + memory: 256Mi + limits: + memory: 1Gi + +# UID 1654 is the `app` user the .NET base images ship. +podSecurityContext: + runAsNonRoot: true + runAsUser: 1654 + fsGroup: 1654 + fsGroupChangePolicy: OnRootMismatch + +securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: false + capabilities: + drop: + - ALL + +nodeSelector: {} +tolerations: [] +affinity: {} +podAnnotations: {} + +serviceAccount: + create: true + name: "" + # Workload identity is annotated here rather than configured in the app, + # because which identity a pod runs as is the platform's business. + annotations: {} + +extraEnv: [] diff --git a/samples/AI/deploy/helm/chart_tests.py b/samples/AI/deploy/helm/chart_tests.py new file mode 100644 index 0000000000..be3860e61f --- /dev/null +++ b/samples/AI/deploy/helm/chart_tests.py @@ -0,0 +1,188 @@ +#!/usr/bin/env python3 +"""Renders the chart and asserts what the rendered manifests must and must not say. + +`helm lint` proves a chart is well formed. It does not prove the chart deploys +what it claims to, and the mistakes worth catching here all produce a valid +manifest: a Server that reports the wrong data residency, a credential that +reaches the address space, a fallback pointing at the endpoint it is supposed to +cover for. + +Each negative case is also asserted to FAIL, because a guardrail that never +fires is indistinguishable from one that does not work. + + python chart_tests.py [path-to-helm] +""" + +import json +import shutil +import subprocess +import sys +from pathlib import Path + +CHART = Path(__file__).resolve().parent / "ai-model-management" +HELM = sys.argv[1] if len(sys.argv) > 1 else shutil.which("helm") +if HELM is None: + print("helm was not found on PATH. Pass path-to-helm as the first argument.", file=sys.stderr) + sys.exit(1) + +failures = [] +checks = 0 + + +def render(*args, values=None): + """Runs `helm template` and returns (ok, output).""" + command = [HELM, "template", "test", str(CHART)] + if values: + command += ["-f", str(Path(__file__).resolve().parent / values)] + command += list(args) + result = subprocess.run( + command, capture_output=True, text=True, check=False + ) + return result.returncode == 0, result.stdout + result.stderr + + +def check(name, condition, detail=""): + global checks + checks += 1 + if condition: + print(f" ok {name}") + else: + print(f" FAIL {name}{(': ' + detail) if detail else ''}") + failures.append(name) + + +def expect_render(name, *args, values=None): + ok, output = render(*args, values=values) + if not ok: + check(name, False, output.strip().splitlines()[-1] if output else "") + return "" + return output + + +def expect_refusal(name, fragment, *args, values=None): + """Asserts the chart refuses a configuration, and refuses it for the stated reason.""" + ok, output = render(*args, values=values) + check(name, not ok and fragment in output, + "rendered successfully" if ok else "refused for a different reason") + + +print("Default values") +out = expect_render("renders") +check("names the OPC UA port", "containerPort: 62640" in out) +check("runs as a non-root user", "runAsNonRoot: true" in out) +check("drops all capabilities", "- ALL" in out) +check("keeps the certificate store on a volume", + "kind: PersistentVolumeClaim" in out) +check("mounts no credential when none is configured", + "secretName:" not in out) +check("publishes on-premises residency", + 'value: "on-premises"' in out) +check("publishes egress as false", + "InferenceBackend__EgressPermitted" in out and 'value: "false"' in out) +check("publishes no fallback by default", + 'name: FallbackInferenceBackend__Enabled' in out and + out.count('value: "false"') >= 1) + +print() +print("Cloud values") +out = expect_render("renders", "--set", "credentials.existingSecret=my-secret", + values="values-cloud.yaml") +check("mounts the named Secret", "secretName: my-secret" in out) +check("mounts it read only", "readOnly: true" in out) +check("mounts it unreadable to the group", "defaultMode: 400" in out or + "defaultMode: 0400" in out or "defaultMode: 256" in out) +check("publishes the credential REFERENCE, not a value", + "InferenceBackend__CredentialReference" in out) +check("publishes egress as true when the payload leaves the machine", + 'value: "true"' in out) +check("reaches the fallback at a different endpoint", + "FallbackInferenceBackend__EndpointUri" in out) + +print() +print("The credential never appears in a rendered manifest") +out = expect_render("renders with an inline credential", + "--set", "credentials.create=true", + "--set", "credentials.value=super-secret-value", + "--set", "backend.authentication=ApiKey", + "--set", "backend.credentialReference=inference-api-key") +# It appears exactly once, in the Secret it belongs in, and nowhere else - +# not in an env var, not in an annotation, not in a ConfigMap. +check("appears only inside the Secret", + out.count("super-secret-value") == 1) +check("is not passed as an environment variable", + "value: \"super-secret-value\"" not in out) +check("rolls the pods when it changes", + "checksum/credentials:" in out) + +# A digest of a low-entropy secret is recoverable offline, and Pod annotations +# are readable by anyone with `get pods` - which is granted far more widely than +# `get secrets`. So the annotation must not be a digest OF THE VALUE. +import hashlib # noqa: E402 +value_digest = hashlib.sha256(b"super-secret-value").hexdigest() +check("the annotation is not a digest of the value", + value_digest not in out) + +print() +print("Refusals") +expect_refusal( + "an ApiKey endpoint with no credential", + "expects a credential but none is supplied", + "--set", "backend.authentication=ApiKey") +expect_refusal( + "a BearerToken endpoint with no credential", + "expects a credential but none is supplied", + "--set", "backend.authentication=BearerToken") +expect_refusal( + "a WorkloadIdentity endpoint with no credential", + "expects a credential but none is supplied", + "--set", "backend.authentication=WorkloadIdentity") +expect_refusal( + "a mounted Secret with no key named", + "credentialReference is empty", + "--set", "backend.authentication=ApiKey", + "--set", "credentials.existingSecret=some-secret", + "--set", "backend.credentialReference=") +expect_refusal( + "a fallback claiming no egress while calling off the machine", + "fallbackBackend.egressPermitted is false", + "--set", "ai.enableFallback=true", + "--set", "fallbackBackend.enabled=true", + "--set", "fallbackBackend.site=Cloud", + "--set", "fallbackBackend.egressPermitted=false") +expect_refusal( + "a fallback with nowhere to fall back to", + "nothing to reach", + "--set", "ai.enableFallback=true") +expect_refusal( + "a fallback pointing at the primary's endpoint", + "That is a retry, not a fallback", + "--set", "ai.enableFallback=true", + "--set", "fallbackBackend.enabled=true", + "--set", "fallbackBackend.endpointUri=http://localhost:5273/", + "--set", "backend.endpointUri=http://localhost:5273/") +expect_refusal( + "claiming no egress while calling off the machine", + "would publish EgressPermitted=false", + "--set", "backend.site=Cloud", + "--set", "backend.egressPermitted=false", + "--set", "backend.authentication=Anonymous") +expect_refusal( + "both credential sources at once", + "not both", + "--set", "credentials.create=true", + "--set", "credentials.value=x", + "--set", "credentials.existingSecret=y") +expect_refusal( + "creating a Secret with no value", + "credentials.value is empty", + "--set", "credentials.create=true", + "--set", "backend.authentication=Anonymous") + +print() +if failures: + print(f"{len(failures)} of {checks} checks failed:") + for name in failures: + print(f" - {name}") + sys.exit(1) + +print(f"All {checks} checks passed.") diff --git a/samples/AI/deploy/helm/smoke-test.ps1 b/samples/AI/deploy/helm/smoke-test.ps1 new file mode 100644 index 0000000000..dca4fec7b0 --- /dev/null +++ b/samples/AI/deploy/helm/smoke-test.ps1 @@ -0,0 +1,157 @@ +#!/usr/bin/env pwsh +<# +.SYNOPSIS + Builds the image, deploys the chart to a kind cluster, and drives it with a + real OPC UA client. + +.DESCRIPTION + The chart tests prove the manifests say the right things. They cannot prove + the image runs, that the Server comes up inside a pod, or that an OPC UA + session survives the network between them - and every one of those has + failed for reasons no template rendering would reveal. + + A stub inference endpoint is deployed alongside, so the round trip completes + without a hosted service or a local model runtime. It is a test fixture and + lives outside the chart: a stub shipped in the chart would eventually be + deployed by someone who thought it was a feature. + + This is the slowest and most fragile check in the suite - image build, image + load, chart install and a live session. Run it deliberately, and quarantine + it rather than letting it gate everything else. + +.PARAMETER Cluster + Name of the kind cluster to create. Deleted on exit unless -Keep is given. + +.PARAMETER Keep + Leaves the cluster running, for investigating a failure. + +.EXAMPLE + pwsh deploy/helm/smoke-test.ps1 +#> +[CmdletBinding()] +param( + [string]$Cluster = 'ai-sample-smoke', + [switch]$Keep +) + +$ErrorActionPreference = 'Stop' +$here = Split-Path -Parent $MyInvocation.MyCommand.Path +$repoRoot = Resolve-Path (Join-Path $here '..' '..' '..' '..') + +function Step($message) { + Write-Host '' + Write-Host "==> $message" -ForegroundColor Cyan +} + +function Require($tool) { + if (-not (Get-Command $tool -ErrorAction SilentlyContinue)) { + throw "$tool is required and was not found on PATH." + } +} + +Require docker +Require kind +Require kubectl +Require helm + +$image = 'modelmanagementserver:smoke' +$release = 'ai-smoke' +$created = $false + +try { + Step "Building $image" + docker build ` + -f (Join-Path $repoRoot 'samples/AI/ModelManagementServer/Dockerfile') ` + -t $image ` + $repoRoot + if ($LASTEXITCODE -ne 0) { throw 'The image build failed.' } + + Step "Creating the kind cluster '$Cluster'" + kind create cluster --name $Cluster --wait 180s + if ($LASTEXITCODE -ne 0) { throw 'The cluster did not come up.' } + $created = $true + + Step 'Loading the image into the cluster' + kind load docker-image $image --name $Cluster + if ($LASTEXITCODE -ne 0) { throw 'The image did not load.' } + + Step 'Deploying the stub inference endpoint' + kubectl apply -f (Join-Path $here 'test-fixtures/stub-backend.yaml') + kubectl wait --for=condition=available deployment/stub-inference-backend --timeout=180s + if ($LASTEXITCODE -ne 0) { throw 'The stub endpoint did not become available.' } + + Step 'Installing the chart' + helm install $release (Join-Path $here 'ai-model-management') ` + --set image.repository=modelmanagementserver ` + --set image.tag=smoke ` + --set image.pullPolicy=Never ` + --set backend.endpointUri=http://stub-inference-backend:5273/ ` + --set backend.site=EdgeOffServer ` + --set backend.egressPermitted=true ` + --wait --timeout 240s + if ($LASTEXITCODE -ne 0) { throw 'The chart did not install.' } + + Step 'Opening a session from outside the cluster' + $service = "svc/$release-ai-model-management" + $forward = Start-Process kubectl ` + -ArgumentList 'port-forward', $service, '62640:62640' ` + -NoNewWindow -PassThru + try { + Start-Sleep -Seconds 6 + + $output = dotnet run ` + --project (Join-Path $repoRoot 'samples/AI/ModelManagementClient') ` + -f net10.0 -- opc.tcp://localhost:62640/ModelManagementServer 2>&1 | + Out-String + } + finally { + Stop-Process -Id $forward.Id -Force -ErrorAction SilentlyContinue + } + + Write-Host $output + + Step 'Checking what came back' + + # Each of these is a distinct claim, and a run that satisfies some and not + # others is a more useful report than a single pass or fail. + $checks = [ordered]@{ + 'the Server published an AI root' = 'AI root: ' + 'the inference reached the endpoint' = 'SMOKE-TEST-OK' + 'the result named the model that ran' = 'ModelUsed ns=' + 'the chunked transfer completed' = 'transfer ns=' + 'the asynchronous job produced a result' = 'job ns=' + 'the source reported itself reachable' = 'reachable True' + } + + $failed = @() + + foreach ($check in $checks.GetEnumerator()) { + if ($output -match [regex]::Escape($check.Value)) { + Write-Host " ok $($check.Key)" -ForegroundColor Green + } + else { + Write-Host " FAIL $($check.Key)" -ForegroundColor Red + $failed += $check.Key + } + } + + if ($failed.Count -gt 0) { + Write-Host '' + kubectl logs "deployment/$release-ai-model-management" --tail=60 + throw "$($failed.Count) of $($checks.Count) checks failed." + } + + Write-Host '' + Write-Host "All $($checks.Count) checks passed." -ForegroundColor Green +} +finally { + if ($created -and -not $Keep) { + Step "Deleting the cluster '$Cluster'" + kind delete cluster --name $Cluster | Out-Null + } + elseif ($created) { + Write-Host '' + Write-Host "The cluster '$Cluster' was left running. Delete it with:" -ForegroundColor Yellow + Write-Host " kind delete cluster --name $Cluster" + } +} diff --git a/samples/AI/deploy/helm/test-fixtures/stub-backend.yaml b/samples/AI/deploy/helm/test-fixtures/stub-backend.yaml new file mode 100644 index 0000000000..7d6dcb2a80 --- /dev/null +++ b/samples/AI/deploy/helm/test-fixtures/stub-backend.yaml @@ -0,0 +1,95 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: stub-inference-backend + labels: + app.kubernetes.io/name: stub-inference-backend +data: + # A minimal OpenAI-compatible endpoint. This is a TEST FIXTURE, not part of the + # sample: it exists so a cluster smoke test can prove the whole path - OPC UA + # client, Server, backend, HTTP and back - without a hosted service or a local + # model runtime. Nothing here should be copied into anything real. + server.py: | + import json + from http.server import BaseHTTPRequestHandler, HTTPServer + + class Handler(BaseHTTPRequestHandler): + def _send(self, payload): + body = json.dumps(payload).encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self): + self._send({"data": [{"id": "smoke-model", "object": "model", + "owned_by": "smoke"}]}) + + def do_POST(self): + self.rfile.read(int(self.headers.get("Content-Length", "0"))) + self._send({ + "id": "chatcmpl-smoke", + "model": "smoke-model", + "choices": [{"index": 0, "finish_reason": "stop", + "message": {"role": "assistant", + "content": "SMOKE-TEST-OK"}}], + "usage": {"prompt_tokens": 3, "completion_tokens": 4, + "total_tokens": 7}, + }) + + def log_message(self, fmt, *args): + pass + + HTTPServer(("0.0.0.0", 5273), Handler).serve_forever() +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: stub-inference-backend + labels: + app.kubernetes.io/name: stub-inference-backend +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: stub-inference-backend + template: + metadata: + labels: + app.kubernetes.io/name: stub-inference-backend + spec: + containers: + - name: stub + image: python:3.13-alpine + command: ["python", "/app/server.py"] + ports: + - name: http + containerPort: 5273 + volumeMounts: + - name: script + mountPath: /app + readinessProbe: + httpGet: + path: /v1/models + port: http + initialDelaySeconds: 2 + periodSeconds: 2 + volumes: + - name: script + configMap: + name: stub-inference-backend +--- +apiVersion: v1 +kind: Service +metadata: + name: stub-inference-backend + labels: + app.kubernetes.io/name: stub-inference-backend +spec: + ports: + - port: 5273 + targetPort: http + name: http + selector: + app.kubernetes.io/name: stub-inference-backend diff --git a/samples/AI/deploy/helm/values-cloud.yaml b/samples/AI/deploy/helm/values-cloud.yaml new file mode 100644 index 0000000000..5ec566d32e --- /dev/null +++ b/samples/AI/deploy/helm/values-cloud.yaml @@ -0,0 +1,41 @@ +# A hosted endpoint, with a fallback on the machine. +# +# helm install ai ./ai-model-management -f values-cloud.yaml \ +# --set backend.endpointUri=https://.services.ai.azure.com/openai/ \ +# --set credentials.existingSecret=inference-credentials +# +# The credential is NOT here. Create the Secret separately and name it, so the +# value never passes through Helm and never reaches the release history: +# +# kubectl create secret generic inference-credentials \ +# --from-literal=inference-api-key= + +backend: + site: Cloud + endpointUri: "https://example-resource.services.ai.azure.com/openai/" + authentication: ApiKey + credentialReference: inference-api-key + # The payload leaves the machine. Saying so is the point of the member: a + # client refusing to send process data off-site needs to know before it calls, + # and encryption does not make this false. + egressPermitted: true + retainsInput: false + dataJurisdiction: "eu-west" + +# A local model to fall back to when the link or the service is unavailable. +# Reached separately from the primary on purpose - through the same endpoint and +# credentials it would be a retry, and would fail for the same reason. +ai: + enableFallback: true + +fallbackBackend: + enabled: true + site: OnServer + endpointUri: "http://localhost:5273/" + authentication: Anonymous + egressPermitted: false + retainsInput: false + dataJurisdiction: "on-premises" + +credentials: + existingSecret: "inference-credentials" diff --git a/samples/AI/verify_backend.py b/samples/AI/verify_backend.py new file mode 100644 index 0000000000..1214450005 --- /dev/null +++ b/samples/AI/verify_backend.py @@ -0,0 +1,67 @@ +"""A minimal OpenAI-compatible endpoint, for verifying the sample end to end. + +This is not part of the sample. It exists so that a developer without access to a +hosted inference service, and without a local runtime installed, can still see the +whole path work: OPC UA client, Server, backend, HTTP, and back. + + python verify_backend.py 5273 +""" + +import json +import sys +from http.server import BaseHTTPRequestHandler, HTTPServer + + +class Handler(BaseHTTPRequestHandler): + def _send(self, payload): + body = json.dumps(payload).encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self): + if self.path.rstrip("/").endswith("models"): + self._send( + { + "data": [ + {"id": "verify-model", "object": "model"}, + ] + } + ) + else: + self.send_error(404) + + def do_POST(self): + length = int(self.headers.get("Content-Length", "0")) + self.rfile.read(length) + self._send( + { + "id": "chatcmpl-verify", + "model": "verify-model", + "choices": [ + { + "index": 0, + "finish_reason": "stop", + "message": { + "role": "assistant", + "content": "The last shift ran without incident.", + }, + } + ], + "usage": { + "prompt_tokens": 11, + "completion_tokens": 8, + "total_tokens": 19, + }, + } + ) + + def log_message(self, fmt, *args): + sys.stderr.write("stub: " + (fmt % args) + "\n") + + +if __name__ == "__main__": + port = int(sys.argv[1]) if len(sys.argv) > 1 else 5273 + HTTPServer(("127.0.0.1", port), Handler).serve_forever() diff --git a/samples/OpenUsd/README.md b/samples/OpenUsd/README.md new file mode 100644 index 0000000000..3cabb01e8c --- /dev/null +++ b/samples/OpenUsd/README.md @@ -0,0 +1,9 @@ +# OpenUSD samples + +Runnable samples that publish OpenUSD representations from OPC UA servers. + +| Sample | What it is | What it shows | +|---|---|---| +| [`GeneratorServer`](GeneratorServer) | A server for simulated generating sets | Generators companion modelling, datasheet-driven simulation, and one independent OpenUSD twin per configured set | + +See [OpenUSD](../../docs/OpenUsd.md) for the binding model, connector tool and viewport. diff --git a/samples/Robotics/BinPickingCell/Assets/Cell.usda b/samples/Robotics/BinPickingCell/Assets/Cell.usda new file mode 100644 index 0000000000..7127030e0e --- /dev/null +++ b/samples/Robotics/BinPickingCell/Assets/Cell.usda @@ -0,0 +1,417 @@ +#usda 1.0 +( + doc = """Bin-picking work cell for the OPC UA Vision guided-picking sample. + + The scene uses a branch-stable four-axis palletizer sized for this cell and + the parallel gripper shared with IntentEnabledRobot. It adds a bin of mixed + parts, a fixture to place them onto, and an eye-in-hand camera parented to + the flange so the view moves with the arm. This is the worked example + reproduced from the OPC UA Robotics-Vision Addendum: frame tree + world -> robot_base -> flange -> gripper_tcp with camera_eih on the flange. + + Rendering contract: + + The scene is lit by a single DomeLight (intensity 1000). Do NOT reintroduce + a DistantLight or any other bright directional/point light: at intensities + that show geometry a DistantLight blows every surface to white regardless of + displayColor, and the demo relies on the agent being able to pick "the red + part" by looking at the frame. Under a DomeLight the same materials measure + red (220, 37, 37), green (37, 208, 49), blue (37, 73, 233) - distinct enough + for a language model to reason about. + + Frame tree contract: + + - /World is the WorldFrame origin. + - /World/Robot/Palletizer/Base is the robot_base frame. + - /World/Robot/Palletizer/Base/J1/J2/J3/Leveling/J4/Flange is the flange + (mechanical interface). + - /World/Robot/Palletizer/.../Flange/Camera is the eye-in-hand camera prim + (UsdGeomCamera) that the Vision sensor renders from. + - /World/Robot/Palletizer/.../Flange/Gripper/Tcp is the gripper_tcp frame. + + The camera is a child of the flange with no stack reset, so it travels with + the arm as a real eye-in-hand sensor does: rotateY = -90 turns its view axis + onto the flange +X tool direction. The authored joint angles put the flange + over the bin at the sample's start pose, so it opens looking into the bin. + """ + defaultPrim = "World" + metersPerUnit = 1 + upAxis = "Z" +) + +def Xform "World" +{ + # A fixed observer for the viewport, distinct from the eye-in-hand sensor on the + # flange. Opening the viewer on /World/Robot/.../Camera shows what the tool sees; this + # one shows the cell working, which is what someone watching the robot wants. + # + # It sits on the -Y side so +X reads left to right, which is what puts the fixture on + # the left and the bin on the right. This is deliberately a low, centred front view: + # the table surface is nearly edge-on, its legs stay visible, and the complete wrist, + # gripper, bin and stack can be judged against one another throughout a cycle. + # + # The numbers were fitted to a reference framing and then corrected against what the + # viewer actually rendered, because the analytic placement and the rendered result did + # not agree - the framing came out offset and over-scaled. Treat them as measured + # rather than derived: change one and the framing has to be re-checked against a + # capture, not just recomputed. + def Camera "ObserverCamera" + { + token projection = "perspective" + float focalLength = 31.6 + float horizontalAperture = 20.955 + float verticalAperture = 16.989 + float2 clippingRange = (0.1, 100.0) + double3 xformOp:translate = (0.500, -5.400, 1.300) + double xformOp:rotateX = 75.50 + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:rotateX"] + } + + def Xform "Lights" + { + def DomeLight "Sky" + { + float inputs:intensity = 1000 + color3f inputs:color = (1.0, 1.0, 1.0) + } + } + + def Xform "Robot" + { + def Cylinder "PedestalRiser" + { + uniform token axis = "Z" + double height = 0.2000 + double radius = 0.1650 + color3f[] primvars:displayColor = [(0.16, 0.17, 0.18)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (0, 0, 0.8200) + uniform token[] xformOpOrder = ["xformOp:translate"] + } + + def Xform "Palletizer" ( + references = @palletizer-arm.usda@ + ) + { + double3 xformOp:translate = (0, 0, 0.9200) + uniform token[] xformOpOrder = ["xformOp:translate"] + + over "Base" + { + over "J1" + { + double xformOp:rotateZ = 0.000 + over "J2" + { + double xformOp:rotateY = 31.513 + over "J3" + { + double xformOp:rotateY = -65.890 + over "Leveling" + { + double xformOp:rotateY = 124.377 + over "J4" + { + double xformOp:rotateX = 90.000 + over "Flange" + { + def Xform "Gripper" ( + references = @palletizer-gripper.usda@ + ) + { + } + + def Camera "Camera" + { + token projection = "perspective" + float focalLength = 12.24 + float horizontalAperture = 14.688 + float verticalAperture = 12.288 + float2 clippingRange = (0.02, 5.0) + double3 xformOp:translate = (0.0200, 0.0000, 0.1600) + double xformOp:rotateY = -90 + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:rotateY"] + } + } + } + } + } + } + } + } + } + } + + def Xform "Floor" + { + def Cube "Slab" + { + double size = 1 + color3f[] primvars:displayColor = [(0.36, 0.37, 0.40)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (0, 0, -0.0250) + double3 xformOp:scale = (2.4000, 1.8000, 0.0250) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + } + + def Xform "Bench" + { + def Cube "Top" + { + double size = 1 + color3f[] primvars:displayColor = [(0.58, 0.60, 0.60)] ( + interpolation = "constant" + ) + # The work surface is z = 0.720. The robot base is independently raised + # to z = 0.920 on PedestalRiser, so the wrist no longer has to fold back + # on itself to reach parts lying close to the table. + double3 xformOp:translate = (0, 0, 0.6850) + double3 xformOp:scale = (1.4000, 0.9000, 0.0700) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + def Cube "LegA" + { + double size = 1 + color3f[] primvars:displayColor = [(0.22, 0.23, 0.24)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (0.5800, 0.3300, 0.3250) + double3 xformOp:scale = (0.0250, 0.0250, 0.6500) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + def Cube "LegB" + { + double size = 1 + color3f[] primvars:displayColor = [(0.22, 0.23, 0.24)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (-0.5800, 0.3300, 0.3250) + double3 xformOp:scale = (0.0250, 0.0250, 0.6500) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + def Cube "LegC" + { + double size = 1 + color3f[] primvars:displayColor = [(0.22, 0.23, 0.24)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (0.5800, -0.3300, 0.3250) + double3 xformOp:scale = (0.0250, 0.0250, 0.6500) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + def Cube "LegD" + { + double size = 1 + color3f[] primvars:displayColor = [(0.22, 0.23, 0.24)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (-0.5800, -0.3300, 0.3250) + double3 xformOp:scale = (0.0250, 0.0250, 0.6500) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + } + + def Xform "Bin" + { + def Cube "BinFloor" + { + double size = 1 + color3f[] primvars:displayColor = [(0.42, 0.32, 0.20)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (0.6000, 0.0000, 0.7160) + double3 xformOp:scale = (0.2800, 0.2400, 0.0060) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + def Cube "BinWallN" + { + double size = 1 + color3f[] primvars:displayColor = [(0.42, 0.32, 0.20)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (0.6000, 0.1170, 0.7330) + double3 xformOp:scale = (0.2800, 0.0060, 0.0400) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + def Cube "BinWallS" + { + double size = 1 + color3f[] primvars:displayColor = [(0.42, 0.32, 0.20)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (0.6000, -0.1170, 0.7330) + double3 xformOp:scale = (0.2800, 0.0060, 0.0400) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + def Cube "BinWallE" + { + double size = 1 + color3f[] primvars:displayColor = [(0.42, 0.32, 0.20)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (0.7370, 0.0000, 0.7330) + double3 xformOp:scale = (0.0060, 0.2400, 0.0400) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + def Cube "BinWallW" + { + double size = 1 + color3f[] primvars:displayColor = [(0.42, 0.32, 0.20)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (0.4630, 0.0000, 0.7330) + double3 xformOp:scale = (0.0060, 0.2400, 0.0400) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + } + + def Xform "Fixture" + { + def Cube "Base" + { + double size = 1 + color3f[] primvars:displayColor = [(0.20, 0.20, 0.22)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (-0.6000, 0.0000, 0.7200) + double3 xformOp:scale = (0.1400, 0.1400, 0.0180) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + def Cylinder "PegA" + { + uniform token axis = "Z" + double height = 0.0400 + double radius = 0.0090 + color3f[] primvars:displayColor = [(0.95, 0.85, 0.20)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (-0.6500, 0.0500, 0.7490) + uniform token[] xformOpOrder = ["xformOp:translate"] + } + + def Cylinder "PegB" + { + uniform token axis = "Z" + double height = 0.0400 + double radius = 0.0090 + color3f[] primvars:displayColor = [(0.95, 0.85, 0.20)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (-0.5500, 0.0500, 0.7490) + uniform token[] xformOpOrder = ["xformOp:translate"] + } + + def Cylinder "PegC" + { + uniform token axis = "Z" + double height = 0.0400 + double radius = 0.0090 + color3f[] primvars:displayColor = [(0.95, 0.85, 0.20)] ( + interpolation = "constant" + ) + double3 xformOp:translate = (-0.6000, -0.0500, 0.7490) + uniform token[] xformOpOrder = ["xformOp:translate"] + } + } + + # Each part is an Xform carrying a single xformOp:transform, with its shape as a child. + # The OpenUSD live binding drives the parent's transform, and the viewer sink authors + # that op as a whole matrix - composing translation with a default identity rotation and + # unit scale. Rotation and scale therefore have to live somewhere the binding does not + # overwrite, which is the child. The composition is unchanged: the shape is scaled and + # rotated about its own origin, then the parent translates it. + def Xform "Parts" + { + def Xform "RedCube" + { + matrix4d xformOp:transform = ( (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 0), (0.5200, -0.0800, 0.7400, 1) ) + uniform token[] xformOpOrder = ["xformOp:transform"] + + def Cube "Shape" + { + double size = 0.0400 + color3f[] primvars:displayColor = [(0.90, 0.15, 0.15)] ( + interpolation = "constant" + ) + double xformOp:rotateZ = 20 + uniform token[] xformOpOrder = ["xformOp:rotateZ"] + } + } + + def Xform "GreenCylinder" + { + matrix4d xformOp:transform = ( (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 0), (0.6700, 0.0800, 0.7350, 1) ) + uniform token[] xformOpOrder = ["xformOp:transform"] + + def Cylinder "Shape" + { + uniform token axis = "Z" + double height = 0.0300 + double radius = 0.0200 + color3f[] primvars:displayColor = [(0.15, 0.85, 0.20)] ( + interpolation = "constant" + ) + } + } + + def Xform "BlueSphere" + { + matrix4d xformOp:transform = ( (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 0), (0.6800, -0.0800, 0.7440, 1) ) + uniform token[] xformOpOrder = ["xformOp:transform"] + + def Sphere "Shape" + { + double radius = 0.0240 + color3f[] primvars:displayColor = [(0.15, 0.30, 0.95)] ( + interpolation = "constant" + ) + } + } + + def Xform "YellowSlab" + { + matrix4d xformOp:transform = ( (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 0), (0.5200, 0.0600, 0.7290, 1) ) + uniform token[] xformOpOrder = ["xformOp:transform"] + + def Cube "Shape" + { + double size = 1 + color3f[] primvars:displayColor = [(0.95, 0.85, 0.15)] ( + interpolation = "constant" + ) + double3 xformOp:scale = (0.0640, 0.0320, 0.0180) + double xformOp:rotateZ = -15 + uniform token[] xformOpOrder = ["xformOp:rotateZ", "xformOp:scale"] + } + } + + def Xform "OrangeBrick" + { + matrix4d xformOp:transform = ( (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 0), (0.6000, 0.0000, 0.7320, 1) ) + uniform token[] xformOpOrder = ["xformOp:transform"] + + def Cube "Shape" + { + double size = 1 + color3f[] primvars:displayColor = [(0.95, 0.45, 0.10)] ( + interpolation = "constant" + ) + double3 xformOp:scale = (0.0500, 0.0280, 0.0240) + double xformOp:rotateZ = 40 + uniform token[] xformOpOrder = ["xformOp:rotateZ", "xformOp:scale"] + } + } + } +} diff --git a/samples/Robotics/BinPickingCell/Assets/palletizer-arm.usda b/samples/Robotics/BinPickingCell/Assets/palletizer-arm.usda new file mode 100644 index 0000000000..81bd9ae604 --- /dev/null +++ b/samples/Robotics/BinPickingCell/Assets/palletizer-arm.usda @@ -0,0 +1,193 @@ +#usda 1.0 +( + doc = """Four-axis palletizer arm for the OPC UA bin-picking Vision demo. + + The commanded chain is /Palletizer/Base/J1/J2/J3/Leveling/J4/Flange: + J1 rotates about Z, J2 and J3 pitch about Y, and J4 rolls the tool about + its local X approach axis. Leveling is the mechanical parallelogram + compensation driven as 90 degrees - J2 - J3, keeping the gripper + vertically tool-down without a numerical offset-wrist solve. + + Link lengths and shoulder height match BinPickingPalletizerGeometry. + Keep the prim and xform-op names stable: OPC UA OpenUSD live bindings + address them directly. + """ + defaultPrim = "Palletizer" + metersPerUnit = 1 + upAxis = "Z" +) + +def Xform "Palletizer" ( + kind = "component" +) +{ + def Xform "Base" + { + def Cylinder "Turntable" + { + uniform token axis = "Z" + double height = 0.1600 + double radius = 0.1500 + color3f[] primvars:displayColor = [(0.16, 0.17, 0.18)] + rel material:binding = + double3 xformOp:translate = (0, 0, 0.0800) + uniform token[] xformOpOrder = ["xformOp:translate"] + } + + def Cylinder "Column" + { + uniform token axis = "Z" + double height = 0.2000 + double radius = 0.1120 + color3f[] primvars:displayColor = [(0.78, 0.79, 0.78)] + rel material:binding = + double3 xformOp:translate = (0, 0, 0.1800) + uniform token[] xformOpOrder = ["xformOp:translate"] + } + + def Xform "J1" + { + double xformOp:rotateZ = 0.000 + double3 xformOp:translate = (0, 0, 0.2800) + uniform token[] xformOpOrder = ["xformOp:rotateZ", "xformOp:translate"] + + def Sphere "ShoulderHousing" + { + double radius = 0.1000 + color3f[] primvars:displayColor = [(0.78, 0.79, 0.78)] + rel material:binding = + } + + def Xform "J2" + { + double xformOp:rotateY = 31.513 + uniform token[] xformOpOrder = ["xformOp:rotateY"] + + def Cylinder "UpperArm" + { + uniform token axis = "X" + double height = 0.4800 + double radius = 0.0470 + color3f[] primvars:displayColor = [(0.90, 0.90, 0.87)] + rel material:binding = + double3 xformOp:translate = (0.2400, 0, 0) + uniform token[] xformOpOrder = ["xformOp:translate"] + } + + def Xform "J3" + { + double xformOp:rotateY = -65.890 + double3 xformOp:translate = (0.4800, 0, 0) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:rotateY"] + + def Sphere "ElbowHousing" + { + double radius = 0.0840 + color3f[] primvars:displayColor = [(0.16, 0.17, 0.18)] + rel material:binding = + } + + def Cylinder "Forearm" + { + uniform token axis = "X" + double height = 0.4800 + double radius = 0.0420 + color3f[] primvars:displayColor = [(0.90, 0.90, 0.87)] + rel material:binding = + double3 xformOp:translate = (0.2400, 0, 0) + uniform token[] xformOpOrder = ["xformOp:translate"] + } + + def Xform "Leveling" + { + double xformOp:rotateY = 124.377 + double3 xformOp:translate = (0.4800, 0, 0) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:rotateY"] + + def Sphere "WristHousing" + { + double radius = 0.0640 + color3f[] primvars:displayColor = [(0.10, 0.38, 0.68)] + rel material:binding = + } + + def Xform "J4" + { + double xformOp:rotateX = 90.000 + uniform token[] xformOpOrder = ["xformOp:rotateX"] + + def Xform "Flange" + { + def Cylinder "IsoFlange" + { + uniform token axis = "X" + double height = 0.0180 + double radius = 0.0315 + color3f[] primvars:displayColor = [(0.78, 0.79, 0.78)] + rel material:binding = + double3 xformOp:translate = (0.0090, 0, 0) + uniform token[] xformOpOrder = ["xformOp:translate"] + } + } + } + } + } + } + } + } + + def Scope "Materials" + { + def Material "BodyWhite" + { + token outputs:surface.connect = + def Shader "Surface" + { + uniform token info:id = "UsdPreviewSurface" + color3f inputs:diffuseColor = (0.90, 0.90, 0.87) + float inputs:roughness = 0.52 + float inputs:metallic = 0 + token outputs:surface + } + } + + def Material "DarkGrey" + { + token outputs:surface.connect = + def Shader "Surface" + { + uniform token info:id = "UsdPreviewSurface" + color3f inputs:diffuseColor = (0.16, 0.17, 0.18) + float inputs:roughness = 0.62 + float inputs:metallic = 0 + token outputs:surface + } + } + + def Material "Silver" + { + token outputs:surface.connect = + def Shader "Surface" + { + uniform token info:id = "UsdPreviewSurface" + color3f inputs:diffuseColor = (0.78, 0.79, 0.78) + float inputs:roughness = 0.30 + float inputs:metallic = 0.6 + token outputs:surface + } + } + + def Material "AccentBlue" + { + token outputs:surface.connect = + def Shader "Surface" + { + uniform token info:id = "UsdPreviewSurface" + color3f inputs:diffuseColor = (0.10, 0.38, 0.68) + float inputs:roughness = 0.45 + float inputs:metallic = 0 + token outputs:surface + } + } + } +} diff --git a/samples/Robotics/BinPickingCell/Assets/palletizer-gripper.usda b/samples/Robotics/BinPickingCell/Assets/palletizer-gripper.usda new file mode 100644 index 0000000000..b9514e5f72 --- /dev/null +++ b/samples/Robotics/BinPickingCell/Assets/palletizer-gripper.usda @@ -0,0 +1,130 @@ +#usda 1.0 +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +( + doc = """High-contrast palletizer gripper for the bin-picking demo. + + Geometry and live jaw-slide prims come from the shared gripper asset. This + composition overrides only the body and carrier materials so the connector + between the blue palletizer wrist and yellow jaws remains visible against + the viewer's black background. + """ + defaultPrim = "PalletizerGripper" + metersPerUnit = 1 + upAxis = "Z" +) + +def Xform "PalletizerGripper" ( + references = @gripper.usda@ +) +{ + over "AdapterPlate" + { + color3f[] primvars:displayColor = [(0.80, 0.82, 0.86)] + rel material:binding = + } + + over "Body" + { + color3f[] primvars:displayColor = [(0.92, 0.94, 0.97)] + rel material:binding = + } + + def Cube "VisibleBody" ( + prepend apiSchemas = ["MaterialBindingAPI"] + ) + { + double size = 1 + color3f[] primvars:displayColor = [(0.92, 0.94, 0.97)] + rel material:binding = + double3 xformOp:translate = (0.0750, 0, 0) + double3 xformOp:scale = (0.0650, 0.0400, 0.0400) + uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] + } + + over "GuideRodUpper" + { + color3f[] primvars:displayColor = [(0.80, 0.82, 0.86)] + rel material:binding = + } + + over "GuideRodLower" + { + color3f[] primvars:displayColor = [(0.80, 0.82, 0.86)] + rel material:binding = + } + + over "FingerLeftSlide" + { + over "Carrier" + { + color3f[] primvars:displayColor = [(0.80, 0.82, 0.86)] + rel material:binding = + } + } + + over "FingerRightSlide" + { + over "Carrier" + { + color3f[] primvars:displayColor = [(0.80, 0.82, 0.86)] + rel material:binding = + } + } + + def Scope "BinPickingMaterials" + { + def Material "BodyWhite" + { + token outputs:surface.connect = + def Shader "Surface" + { + uniform token info:id = "UsdPreviewSurface" + color3f inputs:diffuseColor = (0.92, 0.94, 0.97) + float inputs:roughness = 0.42 + float inputs:metallic = 0.05 + token outputs:surface + } + } + + def Material "ConnectorSilver" + { + token outputs:surface.connect = + def Shader "Surface" + { + uniform token info:id = "UsdPreviewSurface" + color3f inputs:diffuseColor = (0.80, 0.82, 0.86) + float inputs:roughness = 0.28 + float inputs:metallic = 0.65 + token outputs:surface + } + } + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingAgentInferenceProvider.cs b/samples/Robotics/BinPickingCell/BinPickingAgentInferenceProvider.cs new file mode 100644 index 0000000000..baac53b269 --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingAgentInferenceProvider.cs @@ -0,0 +1,792 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Server; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Vision.BinPickingCell +{ + /// + /// Off-server perception path. Combined + /// and + /// that publishes results the + /// Server itself did not compute — the "agent-as-VLM" story of + /// clause 8.2, where inference runs outside the Server and the + /// results arrive over §9 feedback (SubmitDetections, + /// SubmitCorrection). + /// + /// + /// + /// Registered on the pipeline when the run selects + /// --inferenceLocation EdgeOffServer; the on-server ground + /// truth is used otherwise. Only one path is active at a time so + /// the pipeline's advertised inference-location facet + /// (VIS-Inference-OnServer vs + /// VIS-Inference-OffServer, derived from the pipeline + /// builder's onServer flag) says honestly which path is in + /// force. + /// + /// + /// refuses with + /// in this mode: the + /// point of the off-server path is that the Server does not have + /// a local model. Results arrive through the sink methods; the + /// message that comes back with the refusal spells that out for + /// the agent so it is not silent. + /// + /// + /// Submissions are validated, not trusted: a language model can + /// hallucinate a class label that is not a part in this cell, a + /// confidence outside 0..1, a bounding box outside the image, or + /// so many detections it cannot be a plausible bin. The provider + /// refuses the submission with + /// and a + /// that says WHY — the same message + /// the agent's tool sees — rather than silently trimming the + /// bad input into something that looks acceptable. §9 says the + /// Server refuses purposes it does not permit; malformed content + /// belongs to the same "refuse-with-a-reason" surface. + /// + /// + /// Every result carries the two provenance signals the address + /// space models: + /// + /// is + /// agent-off-server-1 (or the exact model tag when a + /// real REST VLM answered), so for + /// the ground-truth path and this one are distinct enough for a + /// reader to tell them apart on that field alone. + /// is + /// , a URN that names this sink and + /// the submission's purpose so a consumer can trace back what + /// produced the value. + /// + /// A correction publishes a fresh DetectionResultType whose + /// ExplanationUri encodes the corrected result's id — this + /// is the specification's learning path (a failed pick becomes a + /// labelled sample) and it is wired even if nothing consumes it + /// yet. + /// + /// + [System.Diagnostics.CodeAnalysis.SuppressMessage( + "Performance", "CA1812", + Justification = "Instantiated by the DI container via AddSingleton.")] + internal sealed class BinPickingAgentInferenceProvider + : IVisionInferenceProvider, IVisionFeedbackSink + { + public BinPickingAgentInferenceProvider( + IBinPickingTargetProvider targetProvider, + ILogger logger) + { + m_targetProvider = targetProvider ?? throw new ArgumentNullException(nameof(targetProvider)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + /// + /// True when the provider has been bound to a pipeline. Consumed + /// by the proof hosted service to know when it is safe to run. + /// + public bool IsAttached => m_target != null; + + /// + /// The bound pipeline's node id, or + /// before . + /// + public NodeId PipelineNodeId => m_target?.PipelineNodeId ?? NodeId.Null; + + /// + /// The sensor node id the pipeline was configured against, or + /// before . + /// + public NodeId SensorNodeId => m_target?.SensorNodeId ?? NodeId.Null; + + /// + /// The deployment node id the pipeline was configured against, + /// or before . + /// + public NodeId DeploymentNodeId => m_target?.DeploymentNodeId ?? NodeId.Null; + + /// + /// Camera-in-world pose the pipeline was configured with. The + /// proof service composes a submitted camera-frame pose to + /// world through this so it does not have to walk the address + /// space. + /// + public VisionPose3DDataType CameraInWorldPose => + m_target?.CameraInWorld ?? new VisionPose3DDataType(); + + /// + /// The camera frame the pipeline is calibrated against — + /// exposed so the proof can label the frame of a synthesised + /// pose the way the on-server provider labels its own. + /// + public string CameraFrameId => m_target?.CameraFrameId ?? string.Empty; + + /// + /// Camera intrinsics width in pixels, exposed so the proof can + /// generate an in-bounds bounding box without duplicating the + /// configuration. + /// + public double ImageWidth => m_target?.ImageWidth ?? 0.0; + + /// + /// Camera intrinsics height in pixels. + /// + public double ImageHeight => m_target?.ImageHeight ?? 0.0; + + /// + /// Looks up a previously-published result by its identifier. + /// The correction path calls this to confirm the target + /// result actually exists before publishing the correction. + /// + public bool TryGetResult(string resultId, out DetectionResultState state) + { + return m_results.TryGetValue(resultId, out state!); + } + + /// + /// The identifier of the most recent detection result + /// published through . Empty + /// when no submission has been accepted yet. Corrections do + /// not update this — the proof service relies on it to find + /// the id it just published so it can inspect the addressed + /// result and drive a correction against it. + /// + public string LastPublishedResultId => m_lastPublishedResultId; + + /// + /// Called from the Vision configurator once the pipeline node + /// and its Results folder are available. + /// + /// is null. + /// + public void Attach(BinPickingInferenceTarget target) + { + if (target == null) + { + throw new ArgumentNullException(nameof(target)); + } + if (Interlocked.CompareExchange(ref m_target, target, null) != null) + { + throw new InvalidOperationException( + "BinPickingAgentInferenceProvider has already been attached to a pipeline."); + } + m_logger.AgentSinkAttached( + target.PipelineNodeId.IsNull ? string.Empty : target.PipelineNodeId.ToString()); + } + + /// + public ValueTask RunInferenceAsync( + VisionInferenceRunRequest request, + CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + // Off-server perception has no server-side computation to perform. Rather than + // silently return an empty result the agent might mistake for "nothing found", + // spell out that submissions arrive through the Feedback object. The message is + // the one an MCP tool surfaces to the calling model. + var result = new VisionInferenceRunResult( + new ServiceResult( + StatusCodes.BadNotSupported, + LocalizedText.From( + "This pipeline is configured for off-server perception (InferenceLocation=" + + "EdgeOffServer). Publish results by calling SubmitDetections on the " + + "pipeline's Feedback object.")), + string.Empty); + return ValueTask.FromResult(result); + } + + /// + public ValueTask StartContinuousAsync( + NodeId pipeline, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + // Continuous inference is driven by the agent's cadence in this mode; there is no + // Server-side clock to start. Refusing preserves the invariant "InferenceLocation + // says honestly which path is in force" — a Good response would suggest the Server + // was polling something it is not. + return ValueTask.FromResult(new ServiceResult( + StatusCodes.BadNotSupported, + LocalizedText.From( + "Continuous inference is not supported when InferenceLocation=EdgeOffServer. " + + "The off-server agent drives its own cadence and publishes results through " + + "SubmitDetections."))); + } + + /// + public ValueTask StopAsync( + NodeId pipeline, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return ValueTask.FromResult(ServiceResult.Good); + } + + /// + public async ValueTask SubmitDetectionsAsync( + VisionSubmitDetectionsRequest request, + CancellationToken cancellationToken) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + cancellationToken.ThrowIfCancellationRequested(); + BinPickingInferenceTarget target = RequireTarget(); + if (!request.Pipeline.Equals(target.PipelineNodeId)) + { + return Refuse( + "SubmitDetections", + StatusCodes.BadNodeIdUnknown, + "The pipeline node id does not match the attached off-server pipeline."); + } + // Part 9.5 pairs the array with the flag. SceneIsEmpty is how the agent + // reports an emptied bin - the terminating condition of the pick loop - + // so an empty array is accepted exactly when it is set. The dispatcher + // checks this too; the check is kept so the provider is correct when + // driven directly. + ServiceResult? refusal = ValidateDetections( + target, request.Detections, allowEmpty: request.SceneIsEmpty); + if (refusal != null) + { + return refusal; + } + if (!IsKnownPurpose(request.Purpose)) + { + return Refuse( + "SubmitDetections", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Purpose '{request.Purpose}' is not a defined VisionFeedbackPurposeEnum value.")); + } + string modelTag = ResolveModelTag(request.FrameReference); + string modelVersion = FormattableString.Invariant($"agent-off-server:{modelTag}"); + string explanation = FormattableString.Invariant( + $"{ExplanationUri}?purpose={request.Purpose}&source={Uri.EscapeDataString(modelTag)}"); + string resultId = "det-agent-" + Guid.NewGuid().ToString("N"); + DateTimeUtc timestamp = DateTimeUtc.From(DateTime.UtcNow); + try + { + m_targetProvider.PublishDetections( + resultId, + timestamp, + request.Detections, + target.CameraInWorld, + target.CameraFrameId); + await PublishAsync( + target, + resultId, + timestamp, + request.Detections, + modelVersion, + explanation, + request.FrameReference, + cancellationToken).ConfigureAwait(false); + } + catch (ServiceResultException ex) + { + // The frame maths raises BadInvalidArgument for a zero-norm quaternion rather + // than substituting identity. Surface it as a clean refusal so the agent sees + // the same shape of refusal it would for any other invalid input. + return Refuse("SubmitDetections", ex.StatusCode, ex.Message); + } + m_logger.AgentDetectionsPublished( + resultId, request.Detections.Count, request.Purpose, modelTag); + m_lastPublishedResultId = resultId; + return ServiceResult.Good; + } + + /// + public ValueTask SubmitInspectionResultAsync( + VisionSubmitInspectionResultRequest request, + CancellationToken cancellationToken) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + cancellationToken.ThrowIfCancellationRequested(); + // The bin-picking cell is a detection cell, not an inspection cell. A model + // submitting an inspection is confused about what pipeline this is, so refuse + // rather than accept-and-silently-discard. + return ValueTask.FromResult(new ServiceResult( + StatusCodes.BadNotSupported, + LocalizedText.From( + "This pipeline exposes DetectionResultType results only. Use " + + "SubmitDetections or SubmitCorrection with corrected detections."))); + } + + /// + public async ValueTask SubmitCorrectionAsync( + VisionSubmitCorrectionRequest request, + CancellationToken cancellationToken) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + cancellationToken.ThrowIfCancellationRequested(); + BinPickingInferenceTarget target = RequireTarget(); + if (!request.Pipeline.Equals(target.PipelineNodeId)) + { + return Refuse( + "SubmitCorrection", + StatusCodes.BadNodeIdUnknown, + "The pipeline node id does not match the attached off-server pipeline."); + } + if (string.IsNullOrEmpty(request.ResultId)) + { + return Refuse( + "SubmitCorrection", + StatusCodes.BadInvalidArgument, + "ResultId is required and must name the result being corrected."); + } + if (!m_results.ContainsKey(request.ResultId)) + { + return Refuse( + "SubmitCorrection", + StatusCodes.BadNodeIdUnknown, + FormattableString.Invariant( + $"ResultId '{request.ResultId}' does not name a result on this pipeline.")); + } + if (request.CorrectedCharacteristics.Count > 0) + { + return Refuse( + "SubmitCorrection", + StatusCodes.BadNotSupported, + "This pipeline publishes DetectionResultType only; corrections must carry " + + "CorrectedDetections and not CorrectedCharacteristics."); + } + // Part 9.5 asks for at most one non-empty corrected array, and both empty + // is the false-positive retraction when RetractAll says so. The dispatcher + // checks this too; the check is kept so the provider is correct when + // driven directly. + ServiceResult? refusal = ValidateDetections( + target, request.CorrectedDetections, allowEmpty: request.RetractAll); + if (refusal != null) + { + return refusal; + } + if (!IsKnownPurpose(request.Purpose)) + { + return Refuse( + "SubmitCorrection", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Purpose '{request.Purpose}' is not a defined VisionFeedbackPurposeEnum value.")); + } + string reason = request.Reason.IsNull + ? string.Empty + : request.Reason.Text ?? string.Empty; + string correctionResultId = "det-agent-correction-" + Guid.NewGuid().ToString("N"); + DateTimeUtc timestamp = DateTimeUtc.From(DateTime.UtcNow); + const string modelVersion = "agent-off-server:correction"; + string explanation = FormattableString.Invariant( + $"{ExplanationUri}?purpose={request.Purpose}&corrects={Uri.EscapeDataString(request.ResultId)}"); + try + { + await PublishAsync( + target, + correctionResultId, + timestamp, + request.CorrectedDetections, + modelVersion, + explanation, + frameReference: null, + cancellationToken).ConfigureAwait(false); + } + catch (ServiceResultException ex) + { + return Refuse("SubmitCorrection", ex.StatusCode, ex.Message); + } + m_logger.AgentCorrectionPublished( + correctionResultId, + request.ResultId, + request.CorrectedDetections.Count, + reason); + return ServiceResult.Good; + } + + /// + public ValueTask SubmitImageReferenceAsync( + VisionSubmitImageReferenceRequest request, + CancellationToken cancellationToken) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + cancellationToken.ThrowIfCancellationRequested(); + // A pure image submission without detections has no home on the pipeline's + // Results folder — the address space stores images alongside a result, never on + // their own. Accept the call so the address space stays consistent with §9.5, + // record the pointer, and let a consumer that cares walk it out of band. + m_logger.AgentImageReference( + request.Image?.Uri ?? string.Empty, + request.ResultId ?? string.Empty, + request.Purpose); + return ValueTask.FromResult(ServiceResult.Good); + } + + private BinPickingInferenceTarget RequireTarget() + { + BinPickingInferenceTarget? target = m_target; + return target ?? + throw new InvalidOperationException( + "BinPickingAgentInferenceProvider has not been attached to a pipeline."); + } + + private ServiceResult? ValidateDetections( + BinPickingInferenceTarget target, + ArrayOf detections, + bool allowEmpty) + { + if (!allowEmpty && detections.Count == 0) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + "At least one detection is required."); + } + if (detections.Count > MaxDetectionsPerSubmission) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"{detections.Count} detections exceeds the plausible ceiling of {MaxDetectionsPerSubmission} for this bin (five parts \u00d7 three).")); + } + for (int ii = 0; ii < detections.Count; ii++) + { + VisionDetectionDataType detection = detections[ii]; + if (string.IsNullOrEmpty(detection.ClassLabel)) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {ii} has no ClassLabel.")); + } + if (BinPickingPartsCatalog.TryGet(detection.ClassLabel) == null) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {ii} class '{detection.ClassLabel}' is not a part in this cell. Known classes: RedCube, GreenCylinder, BlueSphere, YellowSlab, OrangeBrick.")); + } + double confidence = detection.Confidence; + if (double.IsNaN(confidence) || + confidence < 0.0 || + confidence > 1.0) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {ii} confidence {confidence:0.###} is outside [0, 1].")); + } + if (detection.HasBoundingBox2D) + { + ServiceResult? boxRefusal = ValidateBoundingBox2D(target, ii, detection.BoundingBox2D); + if (boxRefusal != null) + { + return boxRefusal; + } + } + if (detection.HasPose) + { + ServiceResult? poseRefusal = ValidatePose(target, ii, detection.Pose); + if (poseRefusal != null) + { + return poseRefusal; + } + } + } + return null; + } + + private ServiceResult? ValidateBoundingBox2D( + BinPickingInferenceTarget target, + int index, + VisionBoundingBox2DDataType box) + { + if (double.IsNaN(box.CenterX) || + double.IsNaN(box.CenterY) || + double.IsNaN(box.Width) || + double.IsNaN(box.Height)) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {index} BoundingBox2D contains NaN.")); + } + if (box.Width <= 0.0 || box.Height <= 0.0) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {index} BoundingBox2D has non-positive extents (w={box.Width:0.##}, h={box.Height:0.##}).")); + } + double halfW = box.Width * 0.5; + double halfH = box.Height * 0.5; + double minU = box.CenterX - halfW; + double maxU = box.CenterX + halfW; + double minV = box.CenterY - halfH; + double maxV = box.CenterY + halfH; + if (maxU <= 0.0 || + minU >= target.ImageWidth || + maxV <= 0.0 || + minV >= target.ImageHeight) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {index} BoundingBox2D (cx={box.CenterX:0.#}, cy={box.CenterY:0.#}, w={box.Width:0.#}, h={box.Height:0.#}) lies entirely outside the {target.ImageWidth:0}x{target.ImageHeight:0} image.")); + } + return null; + } + + private ServiceResult? ValidatePose( + BinPickingInferenceTarget target, + int index, + VisionPose3DDataType pose) + { + if (!string.Equals(pose.FrameId, target.CameraFrameId, StringComparison.Ordinal)) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + $"Detection {index} Pose.FrameId '{pose.FrameId}' does not match " + + $"the calibrated camera frame '{target.CameraFrameId}'."); + } + System.ReadOnlySpan orientation = pose.Orientation.Span; + if (orientation.Length < 4) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {index} Pose.Orientation must carry four components (x, y, z, w) per \u00a75.12.")); + } + double normSq = (orientation[0] * orientation[0]) + + (orientation[1] * orientation[1]) + + (orientation[2] * orientation[2]) + + (orientation[3] * orientation[3]); + if (normSq <= 0.0) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {index} Pose.Orientation has zero norm and does not describe a rotation.")); + } + System.ReadOnlySpan position = pose.Position.Span; + if (position.Length < 3) + { + return Refuse( + "Validate", + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {index} Pose.Position must carry three components.")); + } + return null; + } + + private static bool IsKnownPurpose(VisionFeedbackPurposeEnum purpose) + { + return purpose switch + { + VisionFeedbackPurposeEnum.Overlay => true, + VisionFeedbackPurposeEnum.Reconciliation => true, + VisionFeedbackPurposeEnum.GroundTruthLabel => true, + VisionFeedbackPurposeEnum.Trigger => true, + _ => false + }; + } + + private static string ResolveModelTag(VisionImageReferenceDataType? frameReference) + { + // A submission's frame reference carries an optional model tag in its URI when + // an MCP tool includes one; keep the wire simple and just report the URI or a + // placeholder. Nothing here consumes it, but it flows to ModelVersionUsed so a + // consumer can distinguish a hand-driven submission from a real VLM one. + if (frameReference == null || string.IsNullOrEmpty(frameReference.Uri)) + { + return "unspecified"; + } + return frameReference.Uri; + } + + private async Task PublishAsync( + BinPickingInferenceTarget target, + string resultId, + DateTimeUtc timestamp, + ArrayOf detections, + string modelVersion, + string explanationUri, + VisionImageReferenceDataType? frameReference, + CancellationToken cancellationToken) + { + ISystemContext context = target.SystemContext; + var qualifiedName = new QualifiedName(resultId, target.InstanceNamespaceIndex); + DetectionResultState state = context.CreateInstanceOfDetectionResultType( + target.ResultsFolder, qualifiedName); + state.ReferenceTypeId = Opc.Ua.ReferenceTypeIds.Organizes; + if (state.ResultId != null) + { + state.ResultId.Value = resultId; + } + if (state.CreationTime != null) + { + state.CreationTime.Value = timestamp; + } + state.CreateOrReplaceSensor(context, null).Value = target.SensorNodeId; + state.CreateOrReplacePipeline(context, null).Value = target.PipelineNodeId; + state.CreateOrReplaceModelVersionUsed(context, null).Value = modelVersion; + state.CreateOrReplaceConfidence(context, null).Value = ComputeAggregateConfidence(detections); + state.CreateOrReplaceExplanationUri(context, null).Value = explanationUri; + BaseDataVariableState frame = + state.CreateOrReplaceFrame(context, null); + frame.Value = frameReference ?? + new VisionImageReferenceDataType + { + Uri = FormattableString.Invariant( + $"opcua-inline://binpicking-cell/frames/{resultId}"), + Digest = ByteString.Empty, + DigestAlgorithm = string.Empty, + Format = VisionClipFormatEnum.Png, + PixelFormat = target.PixelFormat, + Width = (uint)Math.Round(target.ImageWidth), + Height = (uint)Math.Round(target.ImageHeight), + SizeBytes = 0u, + Timestamp = timestamp + }; + if (state.Detections != null) + { + state.Detections.Value = detections; + } + state.AddFrameId(context, NodeId.Null); + if (state.FrameId != null) + { + state.FrameId.Value = target.CameraFrameId; + } + state.NodeId = context.RequireNodeIdFactory().New(context, state); + context.AssignInstanceChildNodeIds(state, state.NodeId); + target.ResultsFolder.AddChild(state); + await target.NodeManager.AddPredefinedNodeAsync(state, cancellationToken).ConfigureAwait(false); + m_results[resultId] = state; + } + + private static double ComputeAggregateConfidence( + ArrayOf detections) + { + if (detections.Count == 0) + { + return 0.0; + } + double sum = 0.0; + for (int ii = 0; ii < detections.Count; ii++) + { + sum += detections[ii].Confidence; + } + return sum / detections.Count; + } + + private ServiceResult Refuse(string operation, StatusCode code, string message) + { + m_logger.AgentRefused(operation, code.Code, message); + return new ServiceResult(code, LocalizedText.From(message)); + } + + private const string ExplanationUri = "urn:opcfoundation:BinPickingCell:vision:agent-off-server"; + private const int MaxDetectionsPerSubmission = 15; + + private readonly ILogger m_logger; + private readonly IBinPickingTargetProvider m_targetProvider; + + private readonly ConcurrentDictionary m_results + = new(StringComparer.Ordinal); + + private BinPickingInferenceTarget? m_target; + private string m_lastPublishedResultId = string.Empty; + } + + internal static partial class BinPickingAgentInferenceProviderLog + { + [LoggerMessage(EventId = BinPickingCellEventIds.Agent + 1, + Level = LogLevel.Information, + Message = "Bin-picking agent-driven off-server perception attached to pipeline {PipelineNodeId}.")] + public static partial void AgentSinkAttached( + this ILogger logger, + string pipelineNodeId); + + [LoggerMessage(EventId = BinPickingCellEventIds.Agent + 2, + Level = LogLevel.Information, + Message = "Agent SubmitDetections published result {ResultId} " + + "({DetectionCount} detections, purpose={Purpose}, source={ModelTag}).")] + public static partial void AgentDetectionsPublished( + this ILogger logger, + string resultId, int detectionCount, VisionFeedbackPurposeEnum purpose, string modelTag); + + [LoggerMessage(EventId = BinPickingCellEventIds.Agent + 3, + Level = LogLevel.Information, + Message = "Agent SubmitCorrection published result {CorrectionResultId} " + + "correcting {OriginalResultId} ({DetectionCount} detections, reason='{Reason}').")] + public static partial void AgentCorrectionPublished( + this ILogger logger, + string correctionResultId, string originalResultId, + int detectionCount, string reason); + + [LoggerMessage(EventId = BinPickingCellEventIds.Agent + 4, + Level = LogLevel.Information, + Message = "Agent SubmitImageReference: uri={Uri} resultId={ResultId} purpose={Purpose}.")] + public static partial void AgentImageReference( + this ILogger logger, + string uri, string resultId, VisionFeedbackPurposeEnum purpose); + + [LoggerMessage(EventId = BinPickingCellEventIds.Agent + 5, + Level = LogLevel.Warning, + Message = "Agent {Operation} refused with code 0x{StatusCode:X8}: {Reason}.")] + public static partial void AgentRefused( + this ILogger logger, + string operation, uint statusCode, string reason); + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingCaptureProof.cs b/samples/Robotics/BinPickingCell/BinPickingCaptureProof.cs new file mode 100644 index 0000000000..b2c86c9a7e --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingCaptureProof.cs @@ -0,0 +1,328 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Opc.Ua.Vision.OpenUsd; + +namespace Vision.BinPickingCell +{ + /// + /// Renders one frame of the cell stage as soon as the server has + /// started, saves it as a PNG outside the source tree, and reports + /// how many distinct colours the frame contains and what the mean + /// RGB in each of the five part regions looks like. + /// + /// + /// + /// The purpose of the sample is to prove that the scene renders in + /// colour and that the mixed parts are visually distinguishable — + /// the demo it feeds is a language model looking at the frame and + /// picking "the red part". A blank or white frame would silently + /// destroy that. This service therefore also validates the guard + /// the OpenUSD capture provider enforces on drawn-geometry counts, + /// and treats a + /// result as a soft warning (typical of a CI host with no graphics + /// device) rather than a fatal error. + /// + /// + /// The service does not stop the host on any outcome. Set the + /// captureOnStartup=false configuration key to skip the + /// diagnostic entirely. + /// + /// + internal sealed class BinPickingCaptureProof : BackgroundService + { + public BinPickingCaptureProof( + ISceneCameraCaptureProvider capture, + BinPickingCellStage stage, + ILogger logger, + bool enabled, + string? artifactDirectory) + { + m_capture = capture ?? throw new ArgumentNullException(nameof(capture)); + m_stage = stage ?? throw new ArgumentNullException(nameof(stage)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + m_enabled = enabled; + m_artifactDirectory = string.IsNullOrEmpty(artifactDirectory) + ? Path.Combine(Path.GetTempPath(), "OPCFoundation", "BinPickingCell") + : artifactDirectory; + } + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + if (!m_enabled) + { + m_logger.CaptureSkipped(); + return; + } + m_logger.CaptureProofStarted(m_capture.Backend); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = m_stage.CellStagePath, + PrimPath = BinPickingVisionCell.CameraPrimPath, + Width = ProofWidth, + Height = ProofHeight, + TimeCode = 0.0, + Format = SceneCameraImageFormat.Png, + TimestampUtc = DateTime.UtcNow + }; + SceneCameraCaptureResult result = await m_capture + .CaptureAsync(request, stoppingToken) + .ConfigureAwait(false); + if (result.Status != SceneCameraCaptureStatus.Succeeded) + { + m_logger.CaptureProofNoImage(result.Status, result.Reason ?? string.Empty); + return; + } + byte[] png = result.Image.ToArray(); + Directory.CreateDirectory(m_artifactDirectory); + string outPath = Path.Combine(m_artifactDirectory, "bin-picking-frame.png"); + await File.WriteAllBytesAsync(outPath, png, stoppingToken).ConfigureAwait(false); + + (byte[] Rgba, int W, int H) decoded; + try + { + decoded = PngDecoder.Decode(png); + } + catch (Exception ex) + { + m_logger.CaptureProofDecodeFailed(outPath, ex.Message); + return; + } + + int distinct = CountDistinctColours(decoded.Rgba); + (int mR, int mG, int mB) = MeanRgb(decoded.Rgba, decoded.W, decoded.H); + List<(string Label, int R, int G, int B)> partSamples = SamplePartRegions( + decoded.Rgba, decoded.W, decoded.H); + + m_logger.CaptureProofSaved( + outPath, decoded.W, decoded.H, png.Length, distinct, mR, mG, mB); + foreach ((string label, int r, int g, int b) in partSamples) + { + m_logger.CaptureProofPart(label, r, g, b); + } + + AppendReport(m_artifactDirectory, outPath, decoded, distinct, mR, mG, mB, partSamples, + m_capture.Backend, result.Elapsed); + } + + private static int CountDistinctColours(byte[] rgba) + { + var seen = new HashSet(); + for (int ii = 0; ii + 3 < rgba.Length; ii += 4) + { + int packed = rgba[ii] << 16 | rgba[ii + 1] << 8 | rgba[ii + 2]; + seen.Add(packed); + if (seen.Count > 65536) + { + return seen.Count; + } + } + return seen.Count; + } + + private static (int R, int G, int B) MeanRgb(byte[] rgba, int width, int height) + { + long r = 0; + long g = 0; + long b = 0; + long count = 0; + for (int y = 0; y < height; y++) + { + for (int x = 0; x < width; x++) + { + int offset = 4 * ((y * width) + x); + r += rgba[offset]; + g += rgba[offset + 1]; + b += rgba[offset + 2]; + count++; + } + } + if (count == 0) + { + return (0, 0, 0); + } + return ((int)(r / count), (int)(g / count), (int)(b / count)); + } + + private static List<(string Label, int R, int G, int B)> SamplePartRegions( + byte[] rgba, int width, int height) + { + var samples = new List<(string, int, int, int)>(); + foreach ((string label, double fx, double fy) in s_partSampleFractions) + { + int cx = (int)Math.Clamp(Math.Round(fx * width), 0.0, width - 1); + int cy = (int)Math.Clamp(Math.Round(fy * height), 0.0, height - 1); + (int r, int g, int b) = AverageWindow(rgba, width, height, cx, cy, radius: 20); + samples.Add((label, r, g, b)); + } + return samples; + } + + private static (int R, int G, int B) AverageWindow( + byte[] rgba, int width, int height, int cx, int cy, int radius) + { + long r = 0; + long g = 0; + long b = 0; + long count = 0; + int x0 = Math.Max(0, cx - radius); + int x1 = Math.Min(width - 1, cx + radius); + int y0 = Math.Max(0, cy - radius); + int y1 = Math.Min(height - 1, cy + radius); + for (int y = y0; y <= y1; y++) + { + for (int x = x0; x <= x1; x++) + { + int offset = 4 * ((y * width) + x); + r += rgba[offset]; + g += rgba[offset + 1]; + b += rgba[offset + 2]; + count++; + } + } + if (count == 0) + { + return (0, 0, 0); + } + return ((int)(r / count), (int)(g / count), (int)(b / count)); + } + + private static void AppendReport( + string directory, + string imagePath, + (byte[] Rgba, int W, int H) frame, + int distinctColours, + int meanR, int meanG, int meanB, + List<(string Label, int R, int G, int B)> partSamples, + SceneCameraCaptureBackend backend, + TimeSpan elapsed) + { + string reportPath = Path.Combine(directory, "bin-picking-frame.report.txt"); + using StreamWriter writer = File.CreateText(reportPath); + CultureInfo culture = CultureInfo.InvariantCulture; + writer.WriteLine("BinPickingCell capture proof"); + writer.WriteLine("============================"); + writer.WriteLine(string.Format(culture, "Image : {0}", imagePath)); + writer.WriteLine(string.Format(culture, "Dimensions : {0} x {1}", frame.W, frame.H)); + writer.WriteLine(string.Format(culture, "Backend : {0}", backend)); + writer.WriteLine(string.Format(culture, "Elapsed : {0:0.0} ms", elapsed.TotalMilliseconds)); + writer.WriteLine(string.Format(culture, "Distinct RGB colours : {0}", distinctColours)); + writer.WriteLine(string.Format(culture, + "Mean RGB (whole frame): ({0}, {1}, {2})", meanR, meanG, meanB)); + writer.WriteLine(); + writer.WriteLine("Part samples (mean RGB of a 41x41 patch around the centroid):"); + foreach ((string label, int r, int g, int b) in partSamples) + { + writer.WriteLine(string.Format(culture, + " {0,-15} : ({1,3}, {2,3}, {3,3})", label, r, g, b)); + } + } + + /// + /// The proof renders at the same resolution the sensor declares and the cell + /// delivers, so what it checks is the picture an agent is actually handed. + /// + private const int ProofWidth = (int)BinPickingVisionCell.SensorWidth; + private const int ProofHeight = (int)BinPickingVisionCell.SensorHeight; + + /// + /// Where each part projects to in the delivered frame, as a fraction of width and + /// height. These are the ground-truth detector's own BoundingBox2D centres divided + /// by the frame size, so the proof checks the colour at the pixel the detector + /// points an agent at rather than at an independently guessed spot. + /// + private static readonly (string Label, double Fx, double Fy)[] s_partSampleFractions = + [ + ("RedCube", 0.450, 0.690), + ("GreenCylinder", 0.546, 0.375), + ("BlueSphere", 0.660, 0.559), + ("YellowSlab", 0.406, 0.401), + ("OrangeBrick", 0.572, 0.497) + ]; + + private readonly ISceneCameraCaptureProvider m_capture; + private readonly BinPickingCellStage m_stage; + private readonly ILogger m_logger; + private readonly bool m_enabled; + private readonly string m_artifactDirectory; + } + + internal static partial class BinPickingCaptureProofLog + { + [LoggerMessage(EventId = BinPickingCellEventIds.Startup + 1, + Level = LogLevel.Information, + Message = "Capture-proof diagnostic starting; renderer backend={Backend}.")] + public static partial void CaptureProofStarted( + this ILogger logger, SceneCameraCaptureBackend backend); + + [LoggerMessage(EventId = BinPickingCellEventIds.Startup + 2, + Level = LogLevel.Information, + Message = "Capture-proof diagnostic disabled by configuration (captureOnStartup=false).")] + public static partial void CaptureSkipped(this ILogger logger); + + [LoggerMessage(EventId = BinPickingCellEventIds.Startup + 3, + Level = LogLevel.Warning, + Message = "Capture-proof diagnostic did not produce a frame: {Status} - {Reason}.")] + public static partial void CaptureProofNoImage( + this ILogger logger, + SceneCameraCaptureStatus status, + string reason); + + [LoggerMessage(EventId = BinPickingCellEventIds.Startup + 4, + Level = LogLevel.Warning, + Message = "Capture-proof frame saved to {Path} but PNG decoder failed: {Reason}.")] + public static partial void CaptureProofDecodeFailed( + this ILogger logger, string path, string reason); + + [LoggerMessage(EventId = BinPickingCellEventIds.Startup + 5, + Level = LogLevel.Information, + Message = "Capture-proof frame {Width}x{Height} ({Bytes} bytes) saved to {Path}; " + + "distinct RGB colours={Distinct}; mean RGB=({MeanR},{MeanG},{MeanB}).")] + public static partial void CaptureProofSaved( + this ILogger logger, + string path, + int width, int height, int bytes, + int distinct, int meanR, int meanG, int meanB); + + [LoggerMessage(EventId = BinPickingCellEventIds.Startup + 6, + Level = LogLevel.Information, + Message = "Part '{Label}' mean RGB = ({R},{G},{B}).")] + public static partial void CaptureProofPart( + this ILogger logger, + string label, int r, int g, int b); + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingCell.csproj b/samples/Robotics/BinPickingCell/BinPickingCell.csproj new file mode 100644 index 0000000000..4256b5f972 --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingCell.csproj @@ -0,0 +1,63 @@ + + + net10.0 + Exe + false + BinPickingCell + BinPickingCell + Bin-picking work cell that hosts both the Robot Intent and Vision node managers with an eye-in-hand camera. Reproduces the OPC UA Robotics-Vision Addendum's worked example against the OPC UA .NET Standard stack. + Vision.BinPickingCell + enable + + $(NoWarn);CA1014;CA1822;CA1812 + + false + true + win-x64 + false + + true + + + + + + + + + + + + + + + + + + + + + + + + + + + + Cell.usda + + palletizer-arm.usda + + palletizer-gripper.usda + gripper.usda + + diff --git a/samples/Robotics/BinPickingCell/BinPickingCellGeometry.cs b/samples/Robotics/BinPickingCell/BinPickingCellGeometry.cs new file mode 100644 index 0000000000..5aaccccb0b --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingCellGeometry.cs @@ -0,0 +1,120 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Opc.Ua; +using Robotics.IntentEnabledRobot.Simulation; + +namespace Vision.BinPickingCell +{ + /// + /// The cell's dimensions and furniture as solids the arm must not move through, + /// expressed in the arm's base frame. + /// + /// + /// + /// The numbers mirror Assets/Cell.usda. The arm's base frame is on a riser at + /// world z = 0.920 while the bench top is at z = 0.720, so the work surface is + /// 0.200 m below the base-frame origin. + /// + /// + /// Only the bench and the bin walls are declared. The fixture plate and its locating + /// pegs stand a few millimetres proud of the bench and are exactly where the tool has + /// to work, so treating them as obstacles would refuse the placements the cell exists + /// to perform without changing what the arm visibly does wrong. + /// + /// + internal static class BinPickingCellGeometry + { + /// + /// Height of the work surface in the world frame. + /// + public const double BenchTopMetres = 0.720; + + /// + /// Height of the robot's base-frame origin in the world frame. + /// + public const double RobotBaseHeightMetres = 0.920; + + /// + /// Centre of the fixture in the world frame. + /// + public const double FixtureCentreX = BinPickingPartsCatalog.FixtureCentreX; + + /// + /// Top face of the fixture plate in the world frame. + /// + public const double FixturePlateTopMetres = BenchTopMetres + 0.009; + + /// + /// Top face of the fixture locating pegs in the world frame. + /// + public const double FixturePegTopMetres = FixturePlateTopMetres + 0.040; + public const double FixturePegOffsetMetres = 0.050; + + /// + /// Builds the collision model the arm's solver checks its configurations against. + /// + public static SimulatedCollisionModel CreateCollisionModel() + { + return new SimulatedCollisionModel( + ArrayOf.Create(s_obstacles.AsSpan()), + ArrayOf.Create(s_segmentRadii.AsSpan())); + } + + /// + /// Point pairs are J1->J2 (same shoulder origin), shoulder->elbow, + /// elbow->wrist, and wrist->TCP. + /// + private static readonly double[] s_segmentRadii = [0.0, 0.047, 0.042, 0.018]; + + /// + /// Bench top: world z 0.650 to 0.720, which is 0.200 m below the robot base frame. + /// This is the one that stops a forearm crossing the table: the old height test only + /// sampled joint origins, which a link can straddle. Its clearance is zero because + /// it is a surface, not an obstacle standing on one - a link may come down to it, + /// and with the tool vertical this arm's wrist has to. + /// Bin walls: world z 0.713 to 0.753, 6 mm thick, around a 0.28 x 0.24 tray centred + /// on the Bin location. These are objects, so they keep the full link radius. The + /// tray's inside is deliberately left open so the tool can still descend into it. + /// + private static readonly SimulatedObstacleBox[] s_obstacles = + [ + new("Bench", 0.0, 0.0, 1.4000, 0.9000, -0.2700, -0.2000, Clearance: 0.0), + new("BinWallN", BinPickingPartsCatalog.BinCentreX, 0.1170, + 0.2800, 0.0060, -0.2070, -0.1670), + new("BinWallS", BinPickingPartsCatalog.BinCentreX, -0.1170, + 0.2800, 0.0060, -0.2070, -0.1670), + new("BinWallE", BinPickingPartsCatalog.BinCentreX + 0.1370, 0.0000, + 0.0060, 0.2400, -0.2070, -0.1670), + new("BinWallW", BinPickingPartsCatalog.BinCentreX - 0.1370, 0.0000, + 0.0060, 0.2400, -0.2070, -0.1670) + ]; + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingCellOptions.cs b/samples/Robotics/BinPickingCell/BinPickingCellOptions.cs new file mode 100644 index 0000000000..e84a75e825 --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingCellOptions.cs @@ -0,0 +1,124 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +namespace Vision.BinPickingCell +{ + /// + /// Where inference runs for the sample's pipeline. Selected once + /// at startup and pinned for the lifetime of the process — the + /// pipeline's advertised inference-location facet is derived from + /// this and cannot change afterwards, so it "says honestly" which + /// path is in force. + /// + /// + /// + /// The value maps onto the specification's InferenceLocation + /// concept: the on-server ground truth is + /// and the agent-as-VLM path is + /// . The two are mutually exclusive by + /// construction — the pipeline binds a single inference provider + /// and (optionally) a single feedback sink; wiring both at once + /// would let a submitted result and a computed result publish on + /// the same pipeline out of any known order, which is the + /// invariant the option prevents. + /// + /// + internal enum BinPickingInferenceLocation + { + /// + /// The Server computes results locally through the + /// deterministic ground-truth detector. The pipeline advertises + /// VIS-Inference-OnServer. This is the CI and offline + /// default; it needs neither a GPU nor a model. + /// + OnServer = 0, + + /// + /// Inference runs off-Server: an agent connected over MCP + /// looks at the frame, decides what it sees, and calls + /// SubmitDetections. The pipeline advertises + /// VIS-Inference-OffServer and publishes results the + /// Server itself did not compute. + /// + EdgeOffServer = 1 + } + + /// + /// Startup options for the bin-picking sample. Populated in + /// Program.cs from the host configuration and registered as + /// a DI singleton so both the vision configurator and the proof + /// services pick up the same value. + /// + [System.Diagnostics.CodeAnalysis.SuppressMessage( + "Performance", "CA1812", + Justification = "Instantiated by the DI container via AddSingleton.")] + internal sealed class BinPickingCellOptions + { + /// + /// Selected inference-location mode. See + /// . Defaults to + /// . + /// + public BinPickingInferenceLocation InferenceLocation { get; init; } + = BinPickingInferenceLocation.OnServer; + + /// + /// Attempts to parse the CLI/config value into a + /// . Accepts the + /// exact enum names case-insensitively so both + /// OnServer and on-server resolve; returns + /// for anything else, including empty. + /// + public static bool TryParseLocation( + string? value, out BinPickingInferenceLocation location) + { + if (string.IsNullOrWhiteSpace(value)) + { + location = BinPickingInferenceLocation.OnServer; + return false; + } + string normalised = value.Trim().Replace("-", string.Empty, StringComparison.Ordinal); + if (string.Equals(normalised, "OnServer", StringComparison.OrdinalIgnoreCase)) + { + location = BinPickingInferenceLocation.OnServer; + return true; + } + if (string.Equals(normalised, "EdgeOffServer", StringComparison.OrdinalIgnoreCase) || + string.Equals(normalised, "OffServer", StringComparison.OrdinalIgnoreCase)) + { + location = BinPickingInferenceLocation.EdgeOffServer; + return true; + } + location = BinPickingInferenceLocation.OnServer; + return false; + } + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingCellStage.cs b/samples/Robotics/BinPickingCell/BinPickingCellStage.cs new file mode 100644 index 0000000000..3f6958b4ee --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingCellStage.cs @@ -0,0 +1,157 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Reflection; +using System.Security.Cryptography; +using System.Text; + +namespace Vision.BinPickingCell +{ + /// + /// Materialises the embedded USD assets to a stable per-user directory so + /// the offscreen renderer can open the stage by file path. The cell layer + /// (Cell.usda) references palletizer-arm.usda and gripper.usda + /// by relative path, so both sublayers must be extracted next to the root. + /// + /// + /// The output directory sits under + /// {LocalApplicationData}/OPCFoundation/UA-.NETStandard/BinPickingCell/<hash>/stage + /// where hash is derived from the sample assembly location. That is + /// intentional: two side-by-side builds do not overwrite each other's + /// assets, and the assets survive across restarts so the renderer plug-in + /// cache can be reused. The class overwrites existing files only when + /// their content differs, so subsequent runs are cheap. + /// + internal sealed class BinPickingCellStage + { + public const string CellStageAsset = "Cell.usda"; + public const string ArmAsset = "palletizer-arm.usda"; + public const string PalletizerGripperAsset = "palletizer-gripper.usda"; + public const string GripperAsset = "gripper.usda"; + + /// + /// Gets the absolute path to the cell root layer on disk. Only valid + /// after has returned. + /// + public string CellStagePath { get; private set; } = string.Empty; + + /// + /// Gets the directory the assets were extracted to. + /// + public string StageDirectory { get; private set; } = string.Empty; + + /// + /// Extracts the embedded USD sublayers to disk and returns the cell + /// stage path. + /// + public string Extract() + { + Assembly assembly = typeof(BinPickingCellStage).Assembly; + string root = ResolveRootDirectory(assembly); + Directory.CreateDirectory(root); + StageDirectory = root; + WriteAssetIfChanged(assembly, CellStageAsset, Path.Combine(root, CellStageAsset)); + WriteAssetIfChanged(assembly, ArmAsset, Path.Combine(root, ArmAsset)); + WriteAssetIfChanged( + assembly, + PalletizerGripperAsset, + Path.Combine(root, PalletizerGripperAsset)); + WriteAssetIfChanged(assembly, GripperAsset, Path.Combine(root, GripperAsset)); + CellStagePath = Path.Combine(root, CellStageAsset); + return CellStagePath; + } + + private static void WriteAssetIfChanged(Assembly assembly, string resourceName, string outputPath) + { + using Stream? stream = assembly.GetManifestResourceStream(resourceName); + if (stream == null) + { + throw new InvalidOperationException( + $"Embedded resource '{resourceName}' is missing from the BinPickingCell assembly."); + } + using var memory = new MemoryStream(); + stream.CopyTo(memory); + byte[] bytes = memory.ToArray(); + if (File.Exists(outputPath)) + { + byte[] existing = File.ReadAllBytes(outputPath); + if (BytesEqual(existing, bytes)) + { + return; + } + } + File.WriteAllBytes(outputPath, bytes); + } + + private static bool BytesEqual(byte[] left, byte[] right) + { + if (left.Length != right.Length) + { + return false; + } + for (int ii = 0; ii < left.Length; ii++) + { + if (left[ii] != right[ii]) + { + return false; + } + } + return true; + } + + private static string ResolveRootDirectory(Assembly assembly) + { + string localAppData = Environment.GetFolderPath( + Environment.SpecialFolder.LocalApplicationData); + if (string.IsNullOrEmpty(localAppData)) + { + localAppData = Path.GetTempPath(); + } + string location = assembly.Location; + string discriminator = string.IsNullOrEmpty(location) + ? AppContext.BaseDirectory + : location; + byte[] hash = SHA256.HashData(Encoding.UTF8.GetBytes(discriminator)); + var builder = new StringBuilder(16); + for (int ii = 0; ii < 8; ii++) + { + builder.Append(hash[ii].ToString("x2", System.Globalization.CultureInfo.InvariantCulture)); + } + return Path.Combine( + localAppData, + "OPCFoundation", + "UA-.NETStandard", + "BinPickingCell", + builder.ToString(), + "stage"); + } + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingGroundTruthInferenceProvider.cs b/samples/Robotics/BinPickingCell/BinPickingGroundTruthInferenceProvider.cs new file mode 100644 index 0000000000..19a939cf45 --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingGroundTruthInferenceProvider.cs @@ -0,0 +1,685 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Server; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Vision.BinPickingCell +{ + /// + /// Deterministic on-server inference provider that derives detections + /// from the cell's ground truth: the parts' authored world positions + /// (see ) projected through the + /// same camera intrinsics the sensor publishes. + /// + /// + /// + /// The provider needs neither a model nor a GPU: it reads + /// on every tick, keeps + /// only parts that are still + /// (so a picked part disappears from the next result), and emits one + /// per remaining part with a + /// projected 2-D box, a 3-D box with grasp pose, and a full 6-DoF + /// grasp pose in the camera_eih frame the sensor is calibrated + /// against. + /// + /// + /// Convention: + /// + /// Positions are metres, orientations are unit quaternions in + /// (x, y, z, w) ordering (§5.12). + /// Reported Pose.FrameId is always + /// camera_eih; a consumer composes camera → flange → base + /// using the vision-side frame tree to obtain a base-frame grasp. + /// The demo tunes the vision-side flange transform so the + /// composed camera pose matches the USD-authored camera prim; that + /// is how BB2D and Pose stay internally consistent. + /// 2-D projection uses the classic OpenCV pinhole model with + /// the sensor's published intrinsics (, + /// , , ); lens + /// distortion is ignored — the calibration residual (0.21 pixels) + /// is well below the class-separation margin. + /// + /// + /// + /// Results are materialised as + /// nodes under Pipeline.Results, per the docstring on + /// IVisionFeedbackSink. Old results are kept up to + /// to bound the address-space + /// footprint of the continuous mode. + /// + /// + [System.Diagnostics.CodeAnalysis.SuppressMessage( + "Performance", "CA1812", + Justification = "Instantiated by the DI container via AddSingleton.")] + internal sealed class BinPickingGroundTruthInferenceProvider : IVisionInferenceProvider, IDisposable + { + public BinPickingGroundTruthInferenceProvider( + BinPickingWorldState worldState, + IBinPickingTargetProvider targetProvider, + ILogger logger) + { + m_worldState = worldState ?? throw new ArgumentNullException(nameof(worldState)); + m_targetProvider = targetProvider ?? throw new ArgumentNullException(nameof(targetProvider)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + /// + /// True when the provider has been bound to a pipeline. Consumed + /// by the proof hosted service to know when it is safe to run. + /// + public bool IsAttached => m_target != null; + + /// + /// The bound pipeline's node id, or if + /// the provider has not been attached yet. + /// + public NodeId PipelineNodeId => m_target?.PipelineNodeId ?? NodeId.Null; + + /// + /// The sensor node id the pipeline was configured against, or + /// before . + /// + public NodeId SensorNodeId => m_target?.SensorNodeId ?? NodeId.Null; + + /// + /// The deployment node id the pipeline was configured against, + /// or before . + /// + public NodeId DeploymentNodeId => m_target?.DeploymentNodeId ?? NodeId.Null; + + /// + /// Camera-in-world pose used for projection and pose-in-camera. + /// Snapshotted here so a client (or the proof hosted service) + /// can chain camera → world without walking the OPC UA frame + /// tree. + /// + public VisionPose3DDataType CameraInWorldPose => + m_target?.CameraInWorld ?? new VisionPose3DDataType(); + + /// + /// Looks up a previously-published detection result by its + /// identifier. Returns false if the id is unknown or the + /// provider has been disposed. + /// + public bool TryGetResult(string resultId, out DetectionResultState state) + { + return m_results.TryGetValue(resultId, out state!); + } + + /// + /// Called from the Vision configurator once the pipeline node + /// has been created and its Results folder is available. Stores + /// the references the provider needs to publish + /// DetectionResultType instances. + /// + /// is null. + /// + public void Attach(BinPickingInferenceTarget target) + { + if (target == null) + { + throw new ArgumentNullException(nameof(target)); + } + if (Interlocked.CompareExchange(ref m_target, target, null) != null) + { + throw new InvalidOperationException( + "BinPickingGroundTruthInferenceProvider has already been attached to a pipeline."); + } + m_logger.ProviderAttached( + target.PipelineNodeId.IsNull ? string.Empty : target.PipelineNodeId.ToString()); + } + + /// + public async ValueTask RunInferenceAsync( + VisionInferenceRunRequest request, + CancellationToken cancellationToken) + { + BinPickingInferenceTarget target = RequireTarget(); + cancellationToken.ThrowIfCancellationRequested(); + IReadOnlyList parts = m_worldState.Snapshot(); + string resultId = "det-" + Guid.NewGuid().ToString("N"); + DateTimeUtc timestamp = request.Timestamp.IsNull + ? DateTimeUtc.From(DateTime.UtcNow) + : request.Timestamp; + var detections = new List(parts.Count); + for (int ii = 0; ii < parts.Count; ii++) + { + BinPickingPartSnapshot snapshot = parts[ii]; + if (snapshot.Location != BinPickingPartLocation.InBin) + { + continue; + } + if (!TryBuildDetection(target, snapshot, out VisionDetectionDataType detection)) + { + continue; + } + detections.Add(detection); + } + ArrayOf payload = detections.ToArray().ToArrayOf(); + m_targetProvider.PublishWorldState(resultId, timestamp, parts); + await PublishDetectionAsync( + target, resultId, timestamp, request, payload, cancellationToken).ConfigureAwait(false); + m_logger.ProducedDetectionResult( + resultId, + detections.Count, + parts.Count); + return new VisionInferenceRunResult(ServiceResult.Good, resultId); + } + + /// + public ValueTask StartContinuousAsync( + NodeId pipeline, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + BinPickingInferenceTarget target = RequireTarget(); + if (pipeline.IsNull || !pipeline.Equals(target.PipelineNodeId)) + { + return ValueTask.FromResult(new ServiceResult( + StatusCodes.BadNodeIdUnknown, + LocalizedText.From( + "The pipeline node id does not match the attached bin-picking pipeline."))); + } + lock (m_continuousLock) + { + if (m_continuousCts != null) + { + return ValueTask.FromResult(ServiceResult.Good); + } + var cts = new CancellationTokenSource(); + m_continuousCts = cts; + m_continuousTask = Task.Run(() => RunContinuousAsync(cts.Token), CancellationToken.None); + } + m_logger.ContinuousStarted(); + return ValueTask.FromResult(ServiceResult.Good); + } + + /// + public async ValueTask StopAsync( + NodeId pipeline, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + BinPickingInferenceTarget target = RequireTarget(); + if (pipeline.IsNull || !pipeline.Equals(target.PipelineNodeId)) + { + return new ServiceResult( + StatusCodes.BadNodeIdUnknown, + LocalizedText.From( + "The pipeline node id does not match the attached bin-picking pipeline.")); + } + CancellationTokenSource? cts; + Task? task; + lock (m_continuousLock) + { + cts = m_continuousCts; + task = m_continuousTask; + m_continuousCts = null; + m_continuousTask = null; + } + if (cts != null) + { + await cts.CancelAsync().ConfigureAwait(false); + cts.Dispose(); + } + if (task != null) + { + try + { + await task.ConfigureAwait(false); + } + catch (OperationCanceledException) + { + } + } + m_logger.ContinuousStopped(); + return ServiceResult.Good; + } + + /// + public void Dispose() + { + CancellationTokenSource? cts; + lock (m_continuousLock) + { + cts = m_continuousCts; + m_continuousCts = null; + m_continuousTask = null; + } + cts?.Cancel(); + cts?.Dispose(); + } + + private BinPickingInferenceTarget RequireTarget() + { + BinPickingInferenceTarget? target = m_target; + return target ?? + throw new InvalidOperationException( + "BinPickingGroundTruthInferenceProvider has not been attached to a pipeline."); + } + + private async Task RunContinuousAsync(CancellationToken cancellationToken) + { + using var timer = new PeriodicTimer(ContinuousPeriod); + while (await timer.WaitForNextTickAsync(cancellationToken).ConfigureAwait(false)) + { + try + { + var request = new VisionInferenceRunRequest( + RequireTarget().PipelineNodeId, + RequireTarget().SensorNodeId, + RequireTarget().DeploymentNodeId, + DateTimeUtc.From(DateTime.UtcNow)); + _ = await RunInferenceAsync(request, cancellationToken).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + throw; + } + catch (Exception ex) + { + m_logger.ContinuousTickFailed(ex.Message); + } + } + } + + private bool TryBuildDetection( + BinPickingInferenceTarget target, + BinPickingPartSnapshot snapshot, + out VisionDetectionDataType detection) + { + (double xc, double yc, double zc) = target.WorldToCamera( + snapshot.WorldX, snapshot.WorldY, snapshot.WorldZ); + if (zc <= 0.0) + { + detection = new VisionDetectionDataType(); + return false; + } + if (!TryProjectBoundingBox2D(target, snapshot, out VisionBoundingBox2DDataType box2D)) + { + detection = new VisionDetectionDataType(); + return false; + } + VisionPose3DDataType poseInCamera = new() + { + FrameId = target.CameraFrameId, + Position = new[] { xc, yc, zc }.ToArrayOf(), + Orientation = s_identityOrientation.ToArrayOf(), + Covariance = ArrayOf.Empty + }; + var boundingBox3D = new VisionBoundingBox3DDataType + { + Center = poseInCamera, + Size = new[] + { + snapshot.Part.Size[0], + snapshot.Part.Size[1], + snapshot.Part.Size[2] + }.ToArrayOf() + }; + detection = new VisionDetectionDataType + { + DetectionId = FormattableString.Invariant( + $"det-{snapshot.Part.ClassLabel}-{snapshot.Part.ClassId}"), + ClassLabel = snapshot.Part.ClassLabel, + ClassId = snapshot.Part.ClassId, + Confidence = 0.99, + HasBoundingBox2D = true, + BoundingBox2D = box2D, + HasBoundingBox3D = true, + BoundingBox3D = boundingBox3D, + HasPose = true, + Pose = new VisionPose3DDataType + { + FrameId = target.CameraFrameId, + Position = new[] { xc, yc, zc }.ToArrayOf(), + Orientation = s_identityOrientation.ToArrayOf(), + Covariance = ArrayOf.Empty + }, + TrackId = snapshot.Part.ClassLabel + }; + return true; + } + + private static bool TryProjectBoundingBox2D( + BinPickingInferenceTarget target, + BinPickingPartSnapshot snapshot, + out VisionBoundingBox2DDataType box2D) + { + double sizeX = snapshot.Part.Size[0]; + double sizeY = snapshot.Part.Size[1]; + double sizeZ = snapshot.Part.Size[2]; + double rotZ = snapshot.RotationZDegrees * Math.PI / 180.0; + double cosZ = Math.Cos(rotZ); + double sinZ = Math.Sin(rotZ); + double minU = double.PositiveInfinity; + double maxU = double.NegativeInfinity; + double minV = double.PositiveInfinity; + double maxV = double.NegativeInfinity; + Span local = stackalloc double[3]; + for (int ii = -1; ii <= 1; ii += 2) + { + for (int jj = -1; jj <= 1; jj += 2) + { + for (int kk = -1; kk <= 1; kk += 2) + { + local[0] = ii * (sizeX * 0.5); + local[1] = jj * (sizeY * 0.5); + local[2] = kk * (sizeZ * 0.5); + double dxWorld = (local[0] * cosZ) - (local[1] * sinZ); + double dyWorld = (local[0] * sinZ) + (local[1] * cosZ); + double dzWorld = local[2]; + double x = snapshot.WorldX + dxWorld; + double y = snapshot.WorldY + dyWorld; + double z = snapshot.WorldZ + dzWorld; + (double xc, double yc, double zc) = target.WorldToCamera(x, y, z); + if (zc <= 0.0) + { + box2D = new VisionBoundingBox2DDataType(); + return false; + } + double u = (target.Fx * xc / zc) + target.Cx; + double v = (target.Fy * yc / zc) + target.Cy; + if (u < minU) + { + minU = u; + } + if (u > maxU) + { + maxU = u; + } + if (v < minV) + { + minV = v; + } + if (v > maxV) + { + maxV = v; + } + } + } + } + double clampedMinU = Math.Clamp(minU, 0.0, target.ImageWidth); + double clampedMaxU = Math.Clamp(maxU, 0.0, target.ImageWidth); + double clampedMinV = Math.Clamp(minV, 0.0, target.ImageHeight); + double clampedMaxV = Math.Clamp(maxV, 0.0, target.ImageHeight); + double width = clampedMaxU - clampedMinU; + double height = clampedMaxV - clampedMinV; + if (width <= 0.0 || height <= 0.0) + { + box2D = new VisionBoundingBox2DDataType(); + return false; + } + box2D = new VisionBoundingBox2DDataType + { + CenterX = (clampedMinU + clampedMaxU) * 0.5, + CenterY = (clampedMinV + clampedMaxV) * 0.5, + Width = width, + Height = height, + Rotation = 0.0 + }; + return true; + } + + private async Task PublishDetectionAsync( + BinPickingInferenceTarget target, + string resultId, + DateTimeUtc timestamp, + VisionInferenceRunRequest request, + ArrayOf detections, + CancellationToken cancellationToken) + { + ISystemContext context = target.SystemContext; + var qualifiedName = new QualifiedName(resultId, target.InstanceNamespaceIndex); + DetectionResultState state = context.CreateInstanceOfDetectionResultType( + target.ResultsFolder, qualifiedName); + state.ReferenceTypeId = Opc.Ua.ReferenceTypeIds.Organizes; + if (state.ResultId != null) + { + state.ResultId.Value = resultId; + } + if (state.CreationTime != null) + { + state.CreationTime.Value = timestamp; + } + state.CreateOrReplaceSensor(context, null).Value = request.Sensor; + state.CreateOrReplacePipeline(context, null).Value = request.Pipeline; + state.CreateOrReplaceModelVersionUsed(context, null).Value = ModelVersion; + state.CreateOrReplaceConfidence(context, null).Value = 0.99; + state.CreateOrReplaceExplanationUri(context, null).Value = ExplanationUri; + BaseDataVariableState frame = + state.CreateOrReplaceFrame(context, null); + frame.Value = new VisionImageReferenceDataType + { + Uri = FormattableString.Invariant( + $"opcua-inline://binpicking-cell/frames/{resultId}"), + Digest = ByteString.Empty, + DigestAlgorithm = string.Empty, + Format = VisionClipFormatEnum.Png, + PixelFormat = target.PixelFormat, + Width = (uint)Math.Round(target.ImageWidth), + Height = (uint)Math.Round(target.ImageHeight), + SizeBytes = 0u, + Timestamp = timestamp + }; + if (state.Detections != null) + { + state.Detections.Value = detections; + } + state.AddFrameId(context, NodeId.Null); + if (state.FrameId != null) + { + state.FrameId.Value = target.CameraFrameId; + } + state.NodeId = context.RequireNodeIdFactory().New(context, state); + context.AssignInstanceChildNodeIds(state, state.NodeId); + target.ResultsFolder.AddChild(state); + await target.NodeManager.AddPredefinedNodeAsync(state, cancellationToken).ConfigureAwait(false); + m_results[resultId] = state; + } + + private const string ModelVersion = "binpicking-groundtruth-1"; + private const string ExplanationUri = "urn:opcfoundation:BinPickingCell:vision:groundtruth"; + private static readonly TimeSpan ContinuousPeriod = TimeSpan.FromMilliseconds(500); + private static readonly double[] s_identityOrientation = [0.0, 0.0, 0.0, 1.0]; + + private readonly BinPickingWorldState m_worldState; + private readonly IBinPickingTargetProvider m_targetProvider; + private readonly ILogger m_logger; + private readonly ConcurrentDictionary m_results = new(StringComparer.Ordinal); + private readonly Lock m_continuousLock = new(); + private BinPickingInferenceTarget? m_target; + private CancellationTokenSource? m_continuousCts; + private Task? m_continuousTask; + } + + /// + /// Everything the provider needs to publish results into the address + /// space and to project part positions into camera pixels. Populated + /// by the Vision configurator once the pipeline node and its Results + /// folder are available. + /// + internal sealed class BinPickingInferenceTarget + { + public BinPickingInferenceTarget( + AsyncCustomNodeManager nodeManager, + ISystemContext systemContext, + ushort instanceNamespaceIndex, + NodeId pipelineNodeId, + NodeId sensorNodeId, + NodeId deploymentNodeId, + FolderState resultsFolder, + string cameraFrameId, + string pixelFormat, + double fx, double fy, double cx, double cy, + double imageWidth, double imageHeight, + VisionPose3DDataType cameraInWorld) + { + NodeManager = nodeManager ?? throw new ArgumentNullException(nameof(nodeManager)); + SystemContext = systemContext ?? throw new ArgumentNullException(nameof(systemContext)); + InstanceNamespaceIndex = instanceNamespaceIndex; + PipelineNodeId = pipelineNodeId.IsNull + ? throw new ArgumentException("Pipeline NodeId must not be null.", nameof(pipelineNodeId)) + : pipelineNodeId; + SensorNodeId = sensorNodeId.IsNull + ? throw new ArgumentException("Sensor NodeId must not be null.", nameof(sensorNodeId)) + : sensorNodeId; + DeploymentNodeId = deploymentNodeId.IsNull + ? throw new ArgumentException( + "Deployment NodeId must not be null.", nameof(deploymentNodeId)) + : deploymentNodeId; + ResultsFolder = resultsFolder ?? throw new ArgumentNullException(nameof(resultsFolder)); + CameraFrameId = cameraFrameId ?? throw new ArgumentNullException(nameof(cameraFrameId)); + PixelFormat = pixelFormat ?? throw new ArgumentNullException(nameof(pixelFormat)); + Fx = fx; + Fy = fy; + Cx = cx; + Cy = cy; + ImageWidth = imageWidth; + ImageHeight = imageHeight; + CameraInWorld = cameraInWorld ?? throw new ArgumentNullException(nameof(cameraInWorld)); + (m_cameraInvOrientation, m_cameraPositionInWorld) = InvertCameraInWorld(cameraInWorld); + } + + public AsyncCustomNodeManager NodeManager { get; } + + public ISystemContext SystemContext { get; } + + public ushort InstanceNamespaceIndex { get; } + + public NodeId PipelineNodeId { get; } + + public NodeId SensorNodeId { get; } + + public NodeId DeploymentNodeId { get; } + + public FolderState ResultsFolder { get; } + + public string CameraFrameId { get; } + + public string PixelFormat { get; } + + public double Fx { get; } + + public double Fy { get; } + + public double Cx { get; } + + public double Cy { get; } + + public double ImageWidth { get; } + + public double ImageHeight { get; } + + public VisionPose3DDataType CameraInWorld { get; } + + /// + /// Transforms a world position into the camera frame used for + /// projection and pose reporting. Uses the pre-inverted camera + /// orientation so the hot path allocates nothing. + /// + public (double X, double Y, double Z) WorldToCamera(double x, double y, double z) + { + double dx = x - m_cameraPositionInWorld.X; + double dy = y - m_cameraPositionInWorld.Y; + double dz = z - m_cameraPositionInWorld.Z; + (double qx, double qy, double qz, double qw) = m_cameraInvOrientation; + double tx = (qy * dz) - (qz * dy); + double ty = (qz * dx) - (qx * dz); + double tz = (qx * dy) - (qy * dx); + double rotatedX = dx + (2.0 * ((qw * tx) + (qy * tz) - (qz * ty))); + double rotatedY = dy + (2.0 * ((qw * ty) + (qz * tx) - (qx * tz))); + double rotatedZ = dz + (2.0 * ((qw * tz) + (qx * ty) - (qy * tx))); + return (rotatedX, rotatedY, rotatedZ); + } + + private static ((double X, double Y, double Z, double W) InverseOrientation, + (double X, double Y, double Z) CameraPositionInWorld) InvertCameraInWorld( + VisionPose3DDataType cameraInWorld) + { + ReadOnlySpan p = cameraInWorld.Position.Span; + ReadOnlySpan q = cameraInWorld.Orientation.Span; + if (p.Length < 3 || q.Length < 4) + { + throw new ArgumentException( + "Camera-in-world pose must carry a 3-vector position and 4-vector quaternion.", + nameof(cameraInWorld)); + } + return ((-q[0], -q[1], -q[2], q[3]), (p[0], p[1], p[2])); + } + + private readonly (double X, double Y, double Z, double W) m_cameraInvOrientation; + private readonly (double X, double Y, double Z) m_cameraPositionInWorld; + } + + internal static partial class BinPickingGroundTruthInferenceProviderLog + { + [LoggerMessage(EventId = BinPickingCellEventIds.Inference + 1, + Level = LogLevel.Information, + Message = "Bin-picking ground-truth inference provider attached to pipeline {PipelineNodeId}.")] + public static partial void ProviderAttached( + this ILogger logger, + string pipelineNodeId); + + [LoggerMessage(EventId = BinPickingCellEventIds.Inference + 2, + Level = LogLevel.Information, + Message = "Bin-picking ground-truth inference produced result {ResultId} " + + "with {Detections} of {TrackedParts} tracked parts visible.")] + public static partial void ProducedDetectionResult( + this ILogger logger, + string resultId, int detections, int trackedParts); + + [LoggerMessage(EventId = BinPickingCellEventIds.Inference + 3, + Level = LogLevel.Information, + Message = "Bin-picking ground-truth inference started continuous mode.")] + public static partial void ContinuousStarted( + this ILogger logger); + + [LoggerMessage(EventId = BinPickingCellEventIds.Inference + 4, + Level = LogLevel.Information, + Message = "Bin-picking ground-truth inference stopped continuous mode.")] + public static partial void ContinuousStopped( + this ILogger logger); + + [LoggerMessage(EventId = BinPickingCellEventIds.Inference + 5, + Level = LogLevel.Warning, + Message = "Bin-picking ground-truth inference continuous tick failed: {Reason}.")] + public static partial void ContinuousTickFailed( + this ILogger logger, + string reason); + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingInferenceProof.cs b/samples/Robotics/BinPickingCell/BinPickingInferenceProof.cs new file mode 100644 index 0000000000..1c0c9b06b0 --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingInferenceProof.cs @@ -0,0 +1,411 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Vision.BinPickingCell +{ + /// + /// Hosted service that exercises the on-server ground-truth + /// perception path end-to-end, once the Vision pipeline is + /// available. Runs one inference, dumps every detection to the + /// console (class label, confidence, 2-D box in pixels and 6-DoF + /// grasp pose in camera_eih), composes the RedCube pose to + /// the world frame and cross-checks it against the authored + /// USD position, then simulates a pick of RedCube and re-runs the + /// inference to prove the detector tracks the world (the picked + /// class disappears from the next result). + /// + /// + /// + /// This diagnostic is the sample-side answer to "does the loop + /// work"? It never fails the host on a mismatch — it just logs + /// what it saw. The pass/fail decision is left to the reader of + /// the console output. + /// + /// + internal sealed class BinPickingInferenceProof : BackgroundService + { + public BinPickingInferenceProof( + BinPickingGroundTruthInferenceProvider provider, + BinPickingWorldState worldState, + ILogger logger) + { + m_provider = provider ?? throw new ArgumentNullException(nameof(provider)); + m_worldState = worldState ?? throw new ArgumentNullException(nameof(worldState)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + m_logger.ProofWaiting(); + for (int ii = 0; ii < AttachAttempts; ii++) + { + if (m_provider.IsAttached) + { + break; + } + await Task.Delay(AttachPollInterval, stoppingToken).ConfigureAwait(false); + } + if (!m_provider.IsAttached) + { + m_logger.ProofProviderNotAttached(); + return; + } + + m_logger.ProofBanner("on-server ground-truth inference"); + + VisionInferenceRunRequest request = BuildRequest(); + VisionInferenceRunResult first = await m_provider + .RunInferenceAsync(request, stoppingToken) + .ConfigureAwait(false); + if (!ServiceResult.IsGood(first.ServiceResult)) + { + m_logger.ProofRunFailed(first.ServiceResult.ToString()); + return; + } + + if (!m_provider.TryGetResult(first.ResultId, out DetectionResultState resultState) || + resultState == null || + resultState.Detections == null) + { + m_logger.ProofResultUnavailable(first.ResultId); + return; + } + + ArrayOf detections = resultState.Detections.Value; + m_logger.ProofResultHeader(first.ResultId, detections.Count); + var detectionSnapshot = new VisionDetectionDataType[detections.Count]; + for (int ii = 0; ii < detections.Count; ii++) + { + detectionSnapshot[ii] = detections[ii]; + } + foreach (VisionDetectionDataType detection in detectionSnapshot) + { + LogDetection(detection); + } + + VisionDetectionDataType? redCube = TryFindDetection(detectionSnapshot, RedCubeClass); + if (redCube != null && redCube.HasPose) + { + LogComposedPose(redCube); + } + else + { + m_logger.ProofNoRedCube(); + } + + m_logger.ProofPicking(RedCubeClass); + const double gripperCarryX = 0.30; + const double gripperCarryY = 0.05; + const double gripperCarryZ = 0.95; + const double fixtureX = 0.10; + const double fixtureY = 0.20; + const double fixtureZ = 0.82; + m_worldState.MarkHeld(RedCubeClass, gripperCarryX, gripperCarryY, gripperCarryZ); + m_worldState.MarkPlaced(RedCubeClass, fixtureX, fixtureY, fixtureZ); + + VisionInferenceRunResult second = await m_provider + .RunInferenceAsync(BuildRequest(), stoppingToken) + .ConfigureAwait(false); + if (!ServiceResult.IsGood(second.ServiceResult)) + { + m_logger.ProofRunFailed(second.ServiceResult.ToString()); + return; + } + if (!m_provider.TryGetResult(second.ResultId, out DetectionResultState secondState) || + secondState == null || + secondState.Detections == null) + { + m_logger.ProofResultUnavailable(second.ResultId); + return; + } + + ArrayOf afterPick = secondState.Detections.Value; + m_logger.ProofPostPickHeader(second.ResultId, afterPick.Count); + var afterSnapshot = new VisionDetectionDataType[afterPick.Count]; + bool redCubeStillPresent = false; + for (int ii = 0; ii < afterPick.Count; ii++) + { + VisionDetectionDataType det = afterPick[ii]; + afterSnapshot[ii] = det; + LogDetection(det); + if (string.Equals(det.ClassLabel, RedCubeClass, StringComparison.Ordinal)) + { + redCubeStillPresent = true; + } + } + + if (redCubeStillPresent) + { + m_logger.ProofPickFailed(RedCubeClass); + } + else + { + m_logger.ProofPickSucceeded(RedCubeClass); + } + + // Put the bin back. This proof runs at startup, before any client connects, and + // the world it mutates is the one the paired client's demo then works against. + // Leaving RedCube picked meant the demo's default target was already gone, so it + // reported success for a part it never touched. + m_worldState.Reset(); + + m_logger.ProofCompleted(); + } + + private VisionInferenceRunRequest BuildRequest() + { + return new VisionInferenceRunRequest( + m_provider.PipelineNodeId, + m_provider.SensorNodeId, + m_provider.DeploymentNodeId, + DateTimeUtc.From(DateTime.UtcNow)); + } + + private static VisionDetectionDataType? TryFindDetection( + VisionDetectionDataType[] detections, string classLabel) + { + foreach (VisionDetectionDataType detection in detections) + { + if (string.Equals(detection.ClassLabel, classLabel, StringComparison.Ordinal)) + { + return detection; + } + } + return null; + } + + private void LogDetection(VisionDetectionDataType detection) + { + CultureInfo culture = CultureInfo.InvariantCulture; + string boxSummary = string.Empty; + if (detection.HasBoundingBox2D) + { + VisionBoundingBox2DDataType box = detection.BoundingBox2D; + boxSummary = string.Format( + culture, + "cx={0:0.0} cy={1:0.0} w={2:0.0} h={3:0.0}", + box.CenterX, box.CenterY, box.Width, box.Height); + } + string poseSummary = string.Empty; + if (detection.HasPose) + { + VisionPose3DDataType pose = detection.Pose; + (double px, double py, double pz) = ReadVec3(pose.Position); + (double qx, double qy, double qz, double qw) = ReadQuat(pose.Orientation); + poseSummary = string.Format( + culture, + "frame='{0}' pos=({1:0.000},{2:0.000},{3:0.000}) " + + "quat=({4:0.000},{5:0.000},{6:0.000},{7:0.000})", + pose.FrameId, px, py, pz, qx, qy, qz, qw); + } + m_logger.ProofDetection( + detection.ClassLabel ?? "", + detection.Confidence, + boxSummary, + poseSummary); + } + + private void LogComposedPose(VisionDetectionDataType detection) + { + VisionPose3DDataType poseCam = detection.Pose; + (double px, double py, double pz) = ReadVec3(poseCam.Position); + VisionPose3DDataType cameraInWorld = m_provider.CameraInWorldPose; + (double cx, double cy, double cz) = ReadVec3(cameraInWorld.Position); + (double qx, double qy, double qz, double qw) = ReadQuat(cameraInWorld.Orientation); + (double rx, double ry, double rz) = QuaternionRotate(qx, qy, qz, qw, px, py, pz); + double worldX = cx + rx; + double worldY = cy + ry; + double worldZ = cz + rz; + var authored = BinPickingPartsCatalog.TryGet(RedCubeClass); + if (authored != null) + { + double ax = authored.InitialWorldPosition[0]; + double ay = authored.InitialWorldPosition[1]; + double az = authored.InitialWorldPosition[2]; + double dx = worldX - ax; + double dy = worldY - ay; + double dz = worldZ - az; + double error = Math.Sqrt((dx * dx) + (dy * dy) + (dz * dz)); + m_logger.ProofComposedPose( + RedCubeClass, + worldX, worldY, worldZ, + ax, ay, az, + error); + } + } + + private static (double X, double Y, double Z) ReadVec3(ArrayOf vec) + { + System.ReadOnlySpan span = vec.Span; + if (span.Length < 3) + { + return (0.0, 0.0, 0.0); + } + return (span[0], span[1], span[2]); + } + + private static (double X, double Y, double Z, double W) ReadQuat(ArrayOf vec) + { + System.ReadOnlySpan span = vec.Span; + if (span.Length < 4) + { + return (0.0, 0.0, 0.0, 1.0); + } + return (span[0], span[1], span[2], span[3]); + } + + private static (double X, double Y, double Z) QuaternionRotate( + double qx, double qy, double qz, double qw, + double vx, double vy, double vz) + { + double tx = (qy * vz) - (qz * vy); + double ty = (qz * vx) - (qx * vz); + double tz = (qx * vy) - (qy * vx); + double rx = vx + (2.0 * ((qw * tx) + (qy * tz) - (qz * ty))); + double ry = vy + (2.0 * ((qw * ty) + (qz * tx) - (qx * tz))); + double rz = vz + (2.0 * ((qw * tz) + (qx * ty) - (qy * tx))); + return (rx, ry, rz); + } + + private const string RedCubeClass = "RedCube"; + private const int AttachAttempts = 300; + private static readonly TimeSpan AttachPollInterval = TimeSpan.FromMilliseconds(100); + + private readonly BinPickingGroundTruthInferenceProvider m_provider; + private readonly BinPickingWorldState m_worldState; + private readonly ILogger m_logger; + } + + internal static partial class BinPickingInferenceProofLog + { + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 1, + Level = LogLevel.Information, + Message = "Bin-picking on-server inference proof waiting for pipeline to attach.")] + public static partial void ProofWaiting(this ILogger logger); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 2, + Level = LogLevel.Warning, + Message = "Bin-picking on-server inference proof gave up: provider never attached.")] + public static partial void ProofProviderNotAttached( + this ILogger logger); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 3, + Level = LogLevel.Information, + Message = "=== Bin-picking demo: {Banner} ===")] + public static partial void ProofBanner( + this ILogger logger, string banner); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 4, + Level = LogLevel.Warning, + Message = "RunInferenceAsync returned non-good service result: {ServiceResult}")] + public static partial void ProofRunFailed( + this ILogger logger, string serviceResult); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 5, + Level = LogLevel.Warning, + Message = "DetectionResult '{ResultId}' was not stored on the provider — cannot read back.")] + public static partial void ProofResultUnavailable( + this ILogger logger, string resultId); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 6, + Level = LogLevel.Information, + Message = "Detection result {ResultId}: {DetectionCount} parts in view (initial scan).")] + public static partial void ProofResultHeader( + this ILogger logger, + string resultId, int detectionCount); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 7, + Level = LogLevel.Information, + Message = " {ClassLabel} (conf={Confidence:0.00}) box2D=[{BoundingBox2D}] pose=[{Pose}]")] + public static partial void ProofDetection( + this ILogger logger, + string classLabel, double confidence, string boundingBox2D, string pose); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 8, + Level = LogLevel.Information, + Message = "Composed {ClassLabel} pose camera_eih -> world = " + + "({ComposedX:0.000},{ComposedY:0.000},{ComposedZ:0.000}); " + + "authored=({AuthoredX:0.000},{AuthoredY:0.000},{AuthoredZ:0.000}); " + + "residual={ResidualMetres:0.0000} m")] + public static partial void ProofComposedPose( + this ILogger logger, + string classLabel, + double composedX, double composedY, double composedZ, + double authoredX, double authoredY, double authoredZ, + double residualMetres); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 9, + Level = LogLevel.Warning, + Message = "The RedCube detection was not present or had no pose; skipping compose step.")] + public static partial void ProofNoRedCube( + this ILogger logger); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 10, + Level = LogLevel.Information, + Message = "Simulating pick+place of {ClassLabel} to see whether the detector tracks the world.")] + public static partial void ProofPicking( + this ILogger logger, string classLabel); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 11, + Level = LogLevel.Information, + Message = "Detection result {ResultId}: {DetectionCount} parts in view (after pick).")] + public static partial void ProofPostPickHeader( + this ILogger logger, + string resultId, int detectionCount); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 12, + Level = LogLevel.Error, + Message = "{ClassLabel} still visible after the pick — detector is NOT tracking world state.")] + public static partial void ProofPickFailed( + this ILogger logger, string classLabel); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 13, + Level = LogLevel.Information, + Message = "{ClassLabel} correctly disappeared from the detection result after the pick — " + + "the ground-truth path tracks the world.")] + public static partial void ProofPickSucceeded( + this ILogger logger, string classLabel); + + [LoggerMessage(EventId = BinPickingCellEventIds.Proof + 14, + Level = LogLevel.Information, + Message = "=== Bin-picking demo: on-server inference proof completed ===")] + public static partial void ProofCompleted( + this ILogger logger); + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingMediaProvider.cs b/samples/Robotics/BinPickingCell/BinPickingMediaProvider.cs new file mode 100644 index 0000000000..4a37289521 --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingMediaProvider.cs @@ -0,0 +1,280 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Vision; +using Opc.Ua.Vision.OpenUsd; +using Opc.Ua.Vision.Server; + +namespace Vision.BinPickingCell +{ + /// + /// Bridges the Vision media surface () + /// to the OpenUSD offscreen renderer. + /// + /// + /// + /// The provider serves the eye-in-hand camera as clip frames rendered + /// from the sample cell stage; it does not serve a live RTSP stream and + /// therefore reports from + /// . That is the correct sample behaviour: + /// the sample models the mandatory RTSP stream endpoint in the address + /// space per spec §6.2, but a real RTSP server is not part of a + /// self-contained sample. + /// + /// + /// When the OpenUSD capture provider has no graphics backend + /// (), this + /// provider surfaces it as + /// so the Vision server can report the condition to callers rather than + /// fail to start. The provider itself starts and reports the condition + /// through . + /// + /// + [System.Diagnostics.CodeAnalysis.SuppressMessage( + "Performance", "CA1812", + Justification = "Instantiated by the DI container via AddSingleton.")] + internal sealed class BinPickingMediaProvider : IVisionMediaProvider + { + public BinPickingMediaProvider( + ISceneCameraCaptureProvider capture, + BinPickingSensorSpec spec, + ILogger logger) + { + m_capture = capture ?? throw new ArgumentNullException(nameof(capture)); + m_spec = spec ?? throw new ArgumentNullException(nameof(spec)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + /// + /// Gets the capture backend the provider is bound to. + /// + public SceneCameraCaptureBackend Backend => m_capture.Backend; + + /// + public ValueTask GetStreamAsync( + VisionStreamRequest request, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + var session = new VisionStreamSessionDataType + { + SessionToken = ByteString.Empty, + Uri = string.Empty, + Protocol = request.PreferredProtocol, + ExpiresAt = DateTimeUtc.MinValue + }; + return ValueTask.FromResult(new VisionStreamLease( + new ServiceResult(StatusCodes.BadNotSupported, + LocalizedText.From( + "The bin-picking sample does not host a live RTSP stream. " + + "Use GetClip to fetch one-shot rendered frames.")), + session, + request.Endpoint)); + } + + /// + public ValueTask ReleaseStreamAsync( + ByteString sessionToken, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return ValueTask.FromResult(ServiceResult.Good); + } + + /// + public ValueTask ConfigureStreamAsync( + VisionStreamConfigurationRequest request, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return ValueTask.FromResult(new ServiceResult(StatusCodes.BadNotSupported, + LocalizedText.From( + "The bin-picking sample renders single frames per GetClip; the stream endpoint " + + "is a modelling placeholder and cannot be reconfigured."))); + } + + /// + public ValueTask SelectEndpointAsync( + NodeId streamEndpoint, NodeId clipEndpoint, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return ValueTask.FromResult(ServiceResult.Good); + } + + /// + public async ValueTask GetClipAsync( + VisionClipRequest request, CancellationToken cancellationToken) + { + DateTime timestamp = request.Timestamp.IsNull + ? DateTime.UtcNow + : request.Timestamp.ToDateTime(); + var captureRequest = new SceneCameraCaptureRequest + { + StageIdentifier = m_spec.StageIdentifier, + PrimPath = m_spec.CameraPrimPath, + Width = m_spec.CaptureWidth, + Height = m_spec.CaptureHeight, + TimeCode = 0.0, + Format = SceneCameraImageFormat.Png, + TimestampUtc = timestamp + }; + SceneCameraCaptureResult result = await m_capture + .CaptureAsync(captureRequest, cancellationToken) + .ConfigureAwait(false); + ServiceResult serviceResult = MapStatus(result); + if (result.Status != SceneCameraCaptureStatus.Succeeded) + { + m_logger.CaptureFailed(result.Status, result.Reason ?? string.Empty); + var failureImage = new VisionImageReferenceDataType + { + Uri = string.Empty, + Digest = ByteString.Empty, + DigestAlgorithm = string.Empty, + Format = VisionClipFormatEnum.Jpeg, + PixelFormat = m_spec.PixelFormat, + Width = (uint)m_spec.CaptureWidth, + Height = (uint)m_spec.CaptureHeight, + SizeBytes = 0u, + Timestamp = DateTimeUtc.From(timestamp) + }; + return new VisionClipResult( + serviceResult, failureImage, request.Endpoint, ByteString.Empty); + } + ByteString png = result.Image; + byte[] digest = ComputeDigest(png); + var image = new VisionImageReferenceDataType + { + // A reference, not a container. Embedding the encoded frame here as a + // base64 data URI sent the image twice - once in this String and once in + // the inline ByteString below - and a 1.3 MB PNG becomes a 1.7 MB string + // against a 64 KB MaxStringLength, so the Server could not encode its own + // camera output and every read failed with BadEncodingLimitsExceeded. + Uri = FormattableString.Invariant( + $"opcua-inline://binpicking-cell/frames/{timestamp:yyyyMMddHHmmssfff}"), + Digest = ByteString.From(digest), + DigestAlgorithm = "SHA-256", + Format = VisionClipFormatEnum.Png, + PixelFormat = m_spec.PixelFormat, + Width = (uint)result.Width, + Height = (uint)result.Height, + SizeBytes = (uint)png.Length, + Timestamp = DateTimeUtc.From(timestamp) + }; + ByteString inline = request.RequestInline ? png : ByteString.Empty; + m_logger.CaptureSucceeded(result.Width, result.Height, png.Length); + return new VisionClipResult(ServiceResult.Good, image, request.Endpoint, inline); + } + + private static ServiceResult MapStatus(SceneCameraCaptureResult result) + { + return result.Status switch + { + SceneCameraCaptureStatus.Succeeded => ServiceResult.Good, + SceneCameraCaptureStatus.NoRenderingBackend => new ServiceResult( + StatusCodes.BadResourceUnavailable, + LocalizedText.From(result.Reason ?? "No graphics backend is available on this host.")), + SceneCameraCaptureStatus.InvalidRequest => new ServiceResult( + StatusCodes.BadInvalidArgument, + LocalizedText.From(result.Reason ?? "The capture request was rejected as invalid.")), + SceneCameraCaptureStatus.StageOpenFailed => new ServiceResult( + StatusCodes.BadResourceUnavailable, + LocalizedText.From(result.Reason ?? "The USD stage could not be opened.")), + SceneCameraCaptureStatus.CameraResolveFailed => new ServiceResult( + StatusCodes.BadNodeIdUnknown, + LocalizedText.From(result.Reason ?? "The camera prim could not be resolved on the stage.")), + SceneCameraCaptureStatus.RenderFailed => new ServiceResult( + StatusCodes.BadInternalError, + LocalizedText.From(result.Reason ?? "The scene renderer failed.")), + SceneCameraCaptureStatus.BlankFrame => new ServiceResult( + StatusCodes.BadNoDataAvailable, + LocalizedText.From(result.Reason ?? "The renderer produced a blank frame.")), + SceneCameraCaptureStatus.EncodingFailed => new ServiceResult( + StatusCodes.BadEncodingError, + LocalizedText.From(result.Reason ?? "The rendered frame could not be encoded.")), + _ => new ServiceResult(StatusCodes.BadInternalError, + LocalizedText.From(result.Reason ?? "The scene camera capture provider reported an unknown status.")) + }; + } + + private static byte[] ComputeDigest(ByteString png) + { + return System.Security.Cryptography.SHA256.HashData(png.Span); + } + + private readonly ISceneCameraCaptureProvider m_capture; + private readonly BinPickingSensorSpec m_spec; + private readonly ILogger m_logger; + } + + /// + /// Static description of the eye-in-hand sensor used by the sample. + /// + /// + /// Path or URI to the USD stage the sensor renders from. + /// + /// + /// Absolute prim path of the UsdGeomCamera that acts as the + /// camera view. + /// + /// + /// GenICam PFNC pixel-format string reported on frames the provider + /// returns. + /// + /// + /// Rendered frame width in pixels. + /// + /// + /// Rendered frame height in pixels. + /// + internal sealed record BinPickingSensorSpec( + string StageIdentifier, + string CameraPrimPath, + string PixelFormat, + int CaptureWidth, + int CaptureHeight); + + internal static partial class BinPickingMediaProviderLog + { + [LoggerMessage(EventId = BinPickingCellEventIds.MediaProvider + 1, + Level = LogLevel.Information, + Message = "Rendered eye-in-hand frame {Width}x{Height} ({Bytes} bytes).")] + public static partial void CaptureSucceeded( + this ILogger logger, int width, int height, int bytes); + + [LoggerMessage(EventId = BinPickingCellEventIds.MediaProvider + 2, + Level = LogLevel.Warning, + Message = "Eye-in-hand capture failed: {Status} - {Reason}.")] + public static partial void CaptureFailed( + this ILogger logger, + SceneCameraCaptureStatus status, + string reason); + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingOffServerProof.cs b/samples/Robotics/BinPickingCell/BinPickingOffServerProof.cs new file mode 100644 index 0000000000..d8c07f4aaf --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingOffServerProof.cs @@ -0,0 +1,697 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Vision.BinPickingCell +{ + /// + /// Hosted service that stands in for the connected MCP agent and + /// exercises the off-server perception path end-to-end. + /// + /// + /// + /// The demo's headline is that a language model over MCP does the + /// seeing — but the sample must be able to prove the server-side + /// half without an actual model in the loop. This service is that + /// proof: it plays the role of an agent that has looked at the + /// frame and decided what it sees, and calls + /// the way + /// the MCP tool would. It also drives every validation refusal so + /// the messages a real agent would see are visible in the log. + /// + /// + /// This service is only wired when the run selects + /// InferenceLocation=EdgeOffServer, so its evidence + /// complements — never conflicts with — the on-server proof. + /// + /// + internal sealed class BinPickingOffServerProof : BackgroundService + { + public BinPickingOffServerProof( + BinPickingAgentInferenceProvider provider, + ILogger logger) + { + m_provider = provider ?? throw new ArgumentNullException(nameof(provider)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + m_logger.ProofWaiting(); + for (int ii = 0; ii < AttachAttempts; ii++) + { + if (m_provider.IsAttached) + { + break; + } + await Task.Delay(AttachPollInterval, stoppingToken).ConfigureAwait(false); + } + if (!m_provider.IsAttached) + { + m_logger.ProofProviderNotAttached(); + return; + } + + m_logger.ProofBanner("off-server agent-as-VLM perception"); + + await ProveRunInferenceRefusedAsync(stoppingToken).ConfigureAwait(false); + string? initialResultId = await ProveHappyPathAsync(stoppingToken).ConfigureAwait(false); + await ProveValidationRefusalsAsync(stoppingToken).ConfigureAwait(false); + if (initialResultId != null) + { + await ProveCorrectionAsync(initialResultId, stoppingToken).ConfigureAwait(false); + await ProveCorrectionAgainstUnknownResultAsync(stoppingToken).ConfigureAwait(false); + } + + m_logger.ProofCompleted(); + } + + private async Task ProveRunInferenceRefusedAsync(CancellationToken cancellationToken) + { + var request = new VisionInferenceRunRequest( + m_provider.PipelineNodeId, + m_provider.SensorNodeId, + m_provider.DeploymentNodeId, + DateTimeUtc.From(DateTime.UtcNow)); + VisionInferenceRunResult runResult = await m_provider + .RunInferenceAsync(request, cancellationToken) + .ConfigureAwait(false); + m_logger.ProofRunInferenceRefused( + runResult.ServiceResult.StatusCode.Code, + runResult.ServiceResult.LocalizedText.Text ?? string.Empty); + } + + private async Task ProveHappyPathAsync(CancellationToken cancellationToken) + { + m_logger.ProofBanner("valid submission"); + (VisionDetectionDataType redCube, VisionPose3DDataType poseInCamera) = BuildRedCubeDetection(); + (VisionDetectionDataType greenCylinder, _) = BuildGreenCylinderDetection(); + var detections = new[] { redCube, greenCylinder }.ToArrayOf(); + var frameReference = new VisionImageReferenceDataType + { + Uri = "opcua-agent://binpicking-cell/vlm/frames/proof-happy", + Digest = ByteString.Empty, + DigestAlgorithm = string.Empty, + Format = VisionClipFormatEnum.Png, + PixelFormat = "RGB8", + Width = (uint)Math.Round(m_provider.ImageWidth), + Height = (uint)Math.Round(m_provider.ImageHeight), + SizeBytes = 0u, + Timestamp = DateTimeUtc.From(DateTime.UtcNow) + }; + ServiceResult submission = await m_provider + .SubmitDetectionsAsync( + new VisionSubmitDetectionsRequest( + m_provider.PipelineNodeId, + VisionFeedbackPurposeEnum.Reconciliation, + detections, + frameReference, + ByteString.Empty), + cancellationToken) + .ConfigureAwait(false); + if (!ServiceResult.IsGood(submission)) + { + m_logger.ProofSubmissionFailed(submission.StatusCode.Code, submission.ToString()); + return null; + } + string resultId = m_provider.LastPublishedResultId; + if (string.IsNullOrEmpty(resultId) || + !m_provider.TryGetResult(resultId, out DetectionResultState state)) + { + m_logger.ProofResultUnavailable(resultId); + return null; + } + LogResultShape(state); + LogComposedPose(RedCubeClass, poseInCamera); + return resultId; + } + + private async Task ProveValidationRefusalsAsync(CancellationToken cancellationToken) + { + m_logger.ProofBanner("validation refusals (the messages an agent would see)"); + await ExpectRefusalAsync( + "hallucinated class label", + new VisionSubmitDetectionsRequest( + m_provider.PipelineNodeId, + VisionFeedbackPurposeEnum.Reconciliation, + new[] { BuildDetectionWithClass("PurpleWidget") }.ToArrayOf(), + new VisionImageReferenceDataType(), + ByteString.Empty), + cancellationToken).ConfigureAwait(false); + await ExpectRefusalAsync( + "confidence outside [0, 1]", + new VisionSubmitDetectionsRequest( + m_provider.PipelineNodeId, + VisionFeedbackPurposeEnum.Reconciliation, + new[] { BuildDetectionWithConfidence(1.42) }.ToArrayOf(), + new VisionImageReferenceDataType(), + ByteString.Empty), + cancellationToken).ConfigureAwait(false); + await ExpectRefusalAsync( + "bounding box entirely outside image", + new VisionSubmitDetectionsRequest( + m_provider.PipelineNodeId, + VisionFeedbackPurposeEnum.Reconciliation, + new[] { BuildDetectionWithBoxOutsideImage() }.ToArrayOf(), + new VisionImageReferenceDataType(), + ByteString.Empty), + cancellationToken).ConfigureAwait(false); + await ExpectRefusalAsync( + "zero-norm quaternion", + new VisionSubmitDetectionsRequest( + m_provider.PipelineNodeId, + VisionFeedbackPurposeEnum.Reconciliation, + new[] { BuildDetectionWithZeroNormQuat() }.ToArrayOf(), + new VisionImageReferenceDataType(), + ByteString.Empty), + cancellationToken).ConfigureAwait(false); + await ExpectAcceptedAsync( + "empty detections report an empty bin", + new VisionSubmitDetectionsRequest( + m_provider.PipelineNodeId, + VisionFeedbackPurposeEnum.Reconciliation, + ArrayOf.Empty, + new VisionImageReferenceDataType(), + ByteString.Empty), + cancellationToken).ConfigureAwait(false); + await ExpectRefusalAsync( + "detection count exceeds the cell's plausible ceiling", + new VisionSubmitDetectionsRequest( + m_provider.PipelineNodeId, + VisionFeedbackPurposeEnum.Reconciliation, + BuildManyDetections(HallucinationCount), + new VisionImageReferenceDataType(), + ByteString.Empty), + cancellationToken).ConfigureAwait(false); + await ExpectRefusalAsync( + "purpose not a defined enum value", + new VisionSubmitDetectionsRequest( + m_provider.PipelineNodeId, + (VisionFeedbackPurposeEnum)77, + new[] { BuildBlueSphereDetection() }.ToArrayOf(), + new VisionImageReferenceDataType(), + ByteString.Empty), + cancellationToken).ConfigureAwait(false); + } + + private async Task ProveCorrectionAsync(string originalResultId, CancellationToken cancellationToken) + { + m_logger.ProofBanner("submitting a correction (§9 learning path)"); + VisionDetectionDataType corrected = BuildBlueSphereDetection(); + ServiceResult correctionResult = await m_provider + .SubmitCorrectionAsync( + new VisionSubmitCorrectionRequest( + m_provider.PipelineNodeId, + originalResultId, + VisionFeedbackPurposeEnum.GroundTruthLabel, + new[] { corrected }.ToArrayOf(), + ArrayOf.Empty, + LocalizedText.From("Original miscalled the class; corrected to BlueSphere."), + ByteString.Empty), + cancellationToken) + .ConfigureAwait(false); + m_logger.ProofCorrectionResult( + originalResultId, + correctionResult.StatusCode.Code, + correctionResult.LocalizedText.Text ?? string.Empty); + } + + private async Task ProveCorrectionAgainstUnknownResultAsync(CancellationToken cancellationToken) + { + m_logger.ProofBanner("correction against an unknown result-id"); + ServiceResult refusal = await m_provider + .SubmitCorrectionAsync( + new VisionSubmitCorrectionRequest( + m_provider.PipelineNodeId, + "det-agent-never-existed", + VisionFeedbackPurposeEnum.GroundTruthLabel, + new[] { BuildBlueSphereDetection() }.ToArrayOf(), + ArrayOf.Empty, + LocalizedText.From("Correction referencing a fabricated id."), + ByteString.Empty), + cancellationToken) + .ConfigureAwait(false); + m_logger.ProofCorrectionResult( + "det-agent-never-existed", + refusal.StatusCode.Code, + refusal.LocalizedText.Text ?? string.Empty); + } + + private async Task ExpectRefusalAsync( + string scenario, VisionSubmitDetectionsRequest request, CancellationToken cancellationToken) + { + ServiceResult result = await m_provider + .SubmitDetectionsAsync(request, cancellationToken) + .ConfigureAwait(false); + m_logger.ProofRefusalScenario( + scenario, result.StatusCode.Code, result.LocalizedText.Text ?? string.Empty); + } + + private async Task ExpectAcceptedAsync( + string scenario, VisionSubmitDetectionsRequest request, CancellationToken cancellationToken) + { + ServiceResult result = await m_provider + .SubmitDetectionsAsync(request, cancellationToken) + .ConfigureAwait(false); + m_logger.ProofAcceptedScenario( + scenario, result.StatusCode.Code, result.LocalizedText.Text ?? string.Empty); + } + + private (VisionDetectionDataType Detection, VisionPose3DDataType PoseInCamera) BuildRedCubeDetection() + { + BinPickingPart? part = BinPickingPartsCatalog.TryGet(RedCubeClass); + if (part == null) + { + throw new InvalidOperationException("RedCube missing from catalog."); + } + (double xc, double yc, double zc) = WorldToCamera( + part.InitialWorldPosition[0], + part.InitialWorldPosition[1], + part.InitialWorldPosition[2]); + var pose = new VisionPose3DDataType + { + FrameId = m_provider.CameraFrameId, + Position = new[] { xc, yc, zc }.ToArrayOf(), + Orientation = s_identityOrientation.ToArrayOf(), + Covariance = ArrayOf.Empty + }; + var detection = new VisionDetectionDataType + { + DetectionId = "vlm-red-cube-0", + ClassLabel = RedCubeClass, + ClassId = part.ClassId, + Confidence = 0.94, + HasBoundingBox2D = true, + BoundingBox2D = new VisionBoundingBox2DDataType + { + CenterX = m_provider.ImageWidth * 0.5, + CenterY = m_provider.ImageHeight * 0.5, + Width = m_provider.ImageWidth * 0.10, + Height = m_provider.ImageHeight * 0.10, + Rotation = 0.0 + }, + HasPose = true, + Pose = pose, + TrackId = RedCubeClass + }; + return (detection, pose); + } + + private (VisionDetectionDataType Detection, VisionPose3DDataType PoseInCamera) BuildGreenCylinderDetection() + { + BinPickingPart? part = BinPickingPartsCatalog.TryGet("GreenCylinder"); + if (part == null) + { + throw new InvalidOperationException("GreenCylinder missing from catalog."); + } + (double xc, double yc, double zc) = WorldToCamera( + part.InitialWorldPosition[0], + part.InitialWorldPosition[1], + part.InitialWorldPosition[2]); + var pose = new VisionPose3DDataType + { + FrameId = m_provider.CameraFrameId, + Position = new[] { xc, yc, zc }.ToArrayOf(), + Orientation = s_identityOrientation.ToArrayOf(), + Covariance = ArrayOf.Empty + }; + var detection = new VisionDetectionDataType + { + DetectionId = "vlm-green-cyl-1", + ClassLabel = part.ClassLabel, + ClassId = part.ClassId, + Confidence = 0.83, + HasBoundingBox2D = true, + BoundingBox2D = new VisionBoundingBox2DDataType + { + CenterX = m_provider.ImageWidth * 0.55, + CenterY = m_provider.ImageHeight * 0.50, + Width = m_provider.ImageWidth * 0.09, + Height = m_provider.ImageHeight * 0.09, + Rotation = 0.0 + }, + HasPose = true, + Pose = pose, + TrackId = part.ClassLabel + }; + return (detection, pose); + } + + private VisionDetectionDataType BuildBlueSphereDetection() + { + BinPickingPart part = BinPickingPartsCatalog.TryGet("BlueSphere") + ?? throw new InvalidOperationException("BlueSphere missing from catalog."); + return new VisionDetectionDataType + { + DetectionId = "vlm-blue-sphere", + ClassLabel = part.ClassLabel, + ClassId = part.ClassId, + Confidence = 0.88, + HasBoundingBox2D = true, + BoundingBox2D = new VisionBoundingBox2DDataType + { + CenterX = m_provider.ImageWidth * 0.60, + CenterY = m_provider.ImageHeight * 0.50, + Width = m_provider.ImageWidth * 0.10, + Height = m_provider.ImageHeight * 0.10, + Rotation = 0.0 + }, + HasPose = false, + TrackId = part.ClassLabel + }; + } + + private VisionDetectionDataType BuildDetectionWithClass(string classLabel) + { + return new VisionDetectionDataType + { + DetectionId = "vlm-invalid-class", + ClassLabel = classLabel, + ClassId = 99u, + Confidence = 0.61, + HasBoundingBox2D = true, + BoundingBox2D = InBoundsBox(), + HasPose = false, + TrackId = classLabel + }; + } + + private VisionDetectionDataType BuildDetectionWithConfidence(double confidence) + { + return new VisionDetectionDataType + { + DetectionId = "vlm-invalid-confidence", + ClassLabel = RedCubeClass, + ClassId = 1u, + Confidence = confidence, + HasBoundingBox2D = true, + BoundingBox2D = InBoundsBox(), + HasPose = false, + TrackId = RedCubeClass + }; + } + + private VisionDetectionDataType BuildDetectionWithBoxOutsideImage() + { + return new VisionDetectionDataType + { + DetectionId = "vlm-invalid-box", + ClassLabel = RedCubeClass, + ClassId = 1u, + Confidence = 0.5, + HasBoundingBox2D = true, + BoundingBox2D = new VisionBoundingBox2DDataType + { + CenterX = m_provider.ImageWidth + 500.0, + CenterY = m_provider.ImageHeight + 500.0, + Width = 40.0, + Height = 40.0, + Rotation = 0.0 + }, + HasPose = false, + TrackId = RedCubeClass + }; + } + + private VisionDetectionDataType BuildDetectionWithZeroNormQuat() + { + return new VisionDetectionDataType + { + DetectionId = "vlm-invalid-pose", + ClassLabel = RedCubeClass, + ClassId = 1u, + Confidence = 0.5, + HasBoundingBox2D = false, + HasPose = true, + Pose = new VisionPose3DDataType + { + FrameId = m_provider.CameraFrameId, + Position = s_zeroNormPosition.ToArrayOf(), + Orientation = s_zeroNormQuat.ToArrayOf(), + Covariance = ArrayOf.Empty + }, + TrackId = RedCubeClass + }; + } + + private VisionBoundingBox2DDataType InBoundsBox() + { + return new VisionBoundingBox2DDataType + { + CenterX = m_provider.ImageWidth * 0.5, + CenterY = m_provider.ImageHeight * 0.5, + Width = 40.0, + Height = 40.0, + Rotation = 0.0 + }; + } + + private ArrayOf BuildManyDetections(int count) + { + var detections = new VisionDetectionDataType[count]; + BinPickingPart part = BinPickingPartsCatalog.TryGet(RedCubeClass)!; + for (int ii = 0; ii < count; ii++) + { + detections[ii] = new VisionDetectionDataType + { + DetectionId = FormattableString.Invariant($"vlm-many-{ii}"), + ClassLabel = part.ClassLabel, + ClassId = part.ClassId, + Confidence = 0.5, + HasBoundingBox2D = true, + BoundingBox2D = InBoundsBox(), + HasPose = false, + TrackId = part.ClassLabel + }; + } + return detections.ToArrayOf(); + } + + private void LogResultShape(DetectionResultState state) + { + string? resultId = state.ResultId?.Value; + string? modelVersion = state.ModelVersionUsed?.Value; + string? explanationUri = state.ExplanationUri?.Value; + NodeId sensorId = state.Sensor?.Value ?? NodeId.Null; + NodeId pipelineId = state.Pipeline?.Value ?? NodeId.Null; + int detectionCount = state.Detections?.Value.Count ?? 0; + string? frameId = state.FrameId?.Value; + m_logger.ProofPublishedResult( + resultId ?? string.Empty, + detectionCount, + sensorId, + pipelineId, + modelVersion ?? string.Empty, + explanationUri ?? string.Empty, + frameId ?? string.Empty); + } + + private void LogComposedPose(string classLabel, VisionPose3DDataType poseInCamera) + { + System.ReadOnlySpan pos = poseInCamera.Position.Span; + if (pos.Length < 3) + { + return; + } + (double x, double y, double z) = QuaternionComposeToWorld(pos[0], pos[1], pos[2]); + BinPickingPart? authored = BinPickingPartsCatalog.TryGet(classLabel); + if (authored == null) + { + return; + } + double ax = authored.InitialWorldPosition[0]; + double ay = authored.InitialWorldPosition[1]; + double az = authored.InitialWorldPosition[2]; + double dx = x - ax; + double dy = y - ay; + double dz = z - az; + double residual = Math.Sqrt((dx * dx) + (dy * dy) + (dz * dz)); + m_logger.ProofComposedPose(classLabel, x, y, z, ax, ay, az, residual); + } + + private (double X, double Y, double Z) WorldToCamera(double x, double y, double z) + { + VisionPose3DDataType cameraInWorld = m_provider.CameraInWorldPose; + System.ReadOnlySpan pos = cameraInWorld.Position.Span; + System.ReadOnlySpan ori = cameraInWorld.Orientation.Span; + if (pos.Length < 3 || ori.Length < 4) + { + return (0.0, 0.0, 0.0); + } + double dx = x - pos[0]; + double dy = y - pos[1]; + double dz = z - pos[2]; + double qx = -ori[0]; + double qy = -ori[1]; + double qz = -ori[2]; + double qw = ori[3]; + double tx = (qy * dz) - (qz * dy); + double ty = (qz * dx) - (qx * dz); + double tz = (qx * dy) - (qy * dx); + double rx = dx + (2.0 * ((qw * tx) + (qy * tz) - (qz * ty))); + double ry = dy + (2.0 * ((qw * ty) + (qz * tx) - (qx * tz))); + double rz = dz + (2.0 * ((qw * tz) + (qx * ty) - (qy * tx))); + return (rx, ry, rz); + } + + private (double X, double Y, double Z) QuaternionComposeToWorld(double cx, double cy, double cz) + { + VisionPose3DDataType cameraInWorld = m_provider.CameraInWorldPose; + System.ReadOnlySpan pos = cameraInWorld.Position.Span; + System.ReadOnlySpan ori = cameraInWorld.Orientation.Span; + if (pos.Length < 3 || ori.Length < 4) + { + return (0.0, 0.0, 0.0); + } + double qx = ori[0]; + double qy = ori[1]; + double qz = ori[2]; + double qw = ori[3]; + double tx = (qy * cz) - (qz * cy); + double ty = (qz * cx) - (qx * cz); + double tz = (qx * cy) - (qy * cx); + double rx = cx + (2.0 * ((qw * tx) + (qy * tz) - (qz * ty))); + double ry = cy + (2.0 * ((qw * ty) + (qz * tx) - (qx * tz))); + double rz = cz + (2.0 * ((qw * tz) + (qx * ty) - (qy * tx))); + return (pos[0] + rx, pos[1] + ry, pos[2] + rz); + } + + private const int AttachAttempts = 300; + private const string RedCubeClass = "RedCube"; + private const int HallucinationCount = 20; + private static readonly TimeSpan AttachPollInterval = TimeSpan.FromMilliseconds(100); + private static readonly double[] s_identityOrientation = [0.0, 0.0, 0.0, 1.0]; + private static readonly double[] s_zeroNormPosition = [0.0, 0.0, 0.5]; + private static readonly double[] s_zeroNormQuat = [0.0, 0.0, 0.0, 0.0]; + + private readonly BinPickingAgentInferenceProvider m_provider; + private readonly ILogger m_logger; + } + + internal static partial class BinPickingOffServerProofLog + { + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 1, + Level = LogLevel.Information, + Message = "Bin-picking off-server perception proof waiting for pipeline to attach.")] + public static partial void ProofWaiting(this ILogger logger); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 2, + Level = LogLevel.Warning, + Message = "Bin-picking off-server perception proof gave up: agent provider never attached.")] + public static partial void ProofProviderNotAttached( + this ILogger logger); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 3, + Level = LogLevel.Information, + Message = "=== Bin-picking demo: {Banner} ===")] + public static partial void ProofBanner( + this ILogger logger, string banner); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 4, + Level = LogLevel.Information, + Message = "RunInference refused (as designed for InferenceLocation=EdgeOffServer): " + + "code=0x{StatusCode:X8} reason='{Reason}'")] + public static partial void ProofRunInferenceRefused( + this ILogger logger, + uint statusCode, string reason); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 5, + Level = LogLevel.Information, + Message = "Published off-server result {ResultId} ({DetectionCount} detections): " + + "Sensor={SensorId} Pipeline={PipelineId} ModelVersionUsed='{ModelVersion}' " + + "ExplanationUri='{ExplanationUri}' FrameId='{FrameId}'.")] + public static partial void ProofPublishedResult( + this ILogger logger, + string resultId, int detectionCount, NodeId sensorId, NodeId pipelineId, + string modelVersion, string explanationUri, string frameId); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 6, + Level = LogLevel.Information, + Message = "Composed submitted {ClassLabel} pose camera_eih -> world = " + + "({ComposedX:0.000},{ComposedY:0.000},{ComposedZ:0.000}); " + + "authored=({AuthoredX:0.000},{AuthoredY:0.000},{AuthoredZ:0.000}); " + + "residual={ResidualMetres:0.0000} m")] + public static partial void ProofComposedPose( + this ILogger logger, + string classLabel, + double composedX, double composedY, double composedZ, + double authoredX, double authoredY, double authoredZ, + double residualMetres); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 7, + Level = LogLevel.Warning, + Message = "Submission was refused when the proof expected it to succeed: " + + "code=0x{StatusCode:X8} reason='{Reason}'")] + public static partial void ProofSubmissionFailed( + this ILogger logger, + uint statusCode, string reason); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 8, + Level = LogLevel.Warning, + Message = "Result '{ResultId}' was not stored on the provider — cannot inspect.")] + public static partial void ProofResultUnavailable( + this ILogger logger, string resultId); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 9, + Level = LogLevel.Information, + Message = "Refusal scenario '{Scenario}' → code=0x{StatusCode:X8} reason='{Reason}'")] + public static partial void ProofRefusalScenario( + this ILogger logger, + string scenario, uint statusCode, string reason); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 12, + Level = LogLevel.Information, + Message = "Accepted scenario '{Scenario}' → code=0x{StatusCode:X8} reason='{Reason}'")] + public static partial void ProofAcceptedScenario( + this ILogger logger, + string scenario, uint statusCode, string reason); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 10, + Level = LogLevel.Information, + Message = "Correction against {OriginalResultId} → code=0x{StatusCode:X8} reason='{Reason}'")] + public static partial void ProofCorrectionResult( + this ILogger logger, + string originalResultId, uint statusCode, string reason); + + [LoggerMessage(EventId = BinPickingCellEventIds.OffServerProof + 11, + Level = LogLevel.Information, + Message = "=== Bin-picking demo: off-server perception proof completed ===")] + public static partial void ProofCompleted( + this ILogger logger); + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingPalletizerGeometry.cs b/samples/Robotics/BinPickingCell/BinPickingPalletizerGeometry.cs new file mode 100644 index 0000000000..9d28914601 --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingPalletizerGeometry.cs @@ -0,0 +1,54 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Vision.BinPickingCell +{ + /// + /// Dimensions of the branch-stable palletizer used by the bin-picking cell. + /// + internal static class BinPickingPalletizerGeometry + { + public const string RobotBaseFrameId = "robot_base"; + public const int AxisCount = 4; + public const double ShoulderHeightMetres = 0.280; + public const double UpperArmLengthMetres = 0.480; + public const double ForearmLengthMetres = 0.480; + public const double FlangeToTcpMetres = 0.185; + + public const double MaximumReachMetres = + UpperArmLengthMetres + ForearmLengthMetres; + + public const double BaseYawLimitRadians = 3.1415926535897931; + public const double ShoulderMinimumRadians = -1.3962634015954636; + public const double ShoulderMaximumRadians = 2.2689280275926285; + public const double ElbowMinimumRadians = -2.6179938779914944; + public const double ElbowMaximumRadians = 2.6179938779914944; + public const double ToolRollLimitRadians = 3.1415926535897931; + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingPalletizerKinematics.cs b/samples/Robotics/BinPickingCell/BinPickingPalletizerKinematics.cs new file mode 100644 index 0000000000..e372d36922 --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingPalletizerKinematics.cs @@ -0,0 +1,607 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; +using Opc.Ua; +using Opc.Ua.RobotIntent; +using Robotics.IntentEnabledRobot.Kinematics; +using Robotics.IntentEnabledRobot.Simulation; + +namespace Vision.BinPickingCell +{ + /// + /// Analytic kinematics for the bin-picking palletizer. + /// + internal sealed class BinPickingPalletizerKinematics : ISimulatedArmKinematics + { + public int AxisCount => BinPickingPalletizerGeometry.AxisCount; + + public double MaximumReach => BinPickingPalletizerGeometry.MaximumReachMetres; + + public ArrayOf InitialJointAngles => ArrayOf.Create(s_initialJointAngles.AsSpan()); + + public double MinimumLinkHeight { get; set; } = double.NegativeInfinity; + + public SimulatedCollisionModel? Collisions { get; set; } + + public void SetHeldObjectEnvelope(double sizeX, double sizeY, double sizeZ) + { + if (sizeX < 0.0 || sizeY < 0.0 || sizeZ < 0.0) + { + throw new ArgumentOutOfRangeException( + nameof(sizeX), + "Held-object dimensions must be non-negative."); + } + m_heldObjectSizeX = sizeX; + m_heldObjectSizeY = sizeY; + m_heldObjectSizeZ = sizeZ; + } + + public void ClearHeldObjectEnvelope() + { + m_heldObjectSizeX = 0.0; + m_heldObjectSizeY = 0.0; + m_heldObjectSizeZ = 0.0; + } + + public SimulatedArmForwardPose Forward(ReadOnlySpan jointAngles) + { + RequireJointCount(jointAngles); + + double baseYaw = jointAngles[0]; + double shoulder = jointAngles[1]; + double elbow = jointAngles[2]; + double toolRoll = jointAngles[3]; + double forearmPitch = shoulder + elbow; + + double cosBase = Math.Cos(baseYaw); + double sinBase = Math.Sin(baseYaw); + + double elbowRadius = + BinPickingPalletizerGeometry.UpperArmLengthMetres * Math.Cos(shoulder); + double elbowZ = + BinPickingPalletizerGeometry.ShoulderHeightMetres - + (BinPickingPalletizerGeometry.UpperArmLengthMetres * Math.Sin(shoulder)); + + double wristRadius = elbowRadius + + (BinPickingPalletizerGeometry.ForearmLengthMetres * Math.Cos(forearmPitch)); + double wristZ = elbowZ - + (BinPickingPalletizerGeometry.ForearmLengthMetres * Math.Sin(forearmPitch)); + + double[] shoulderPosition = + [0.0, 0.0, BinPickingPalletizerGeometry.ShoulderHeightMetres]; + double[] elbowPosition = + [elbowRadius * cosBase, elbowRadius * sinBase, elbowZ]; + double[] wristPosition = + [wristRadius * cosBase, wristRadius * sinBase, wristZ]; + + double[] baseOrientation = Orientation(baseYaw, 0.0, 0.0); + double[] shoulderOrientation = Orientation(baseYaw, shoulder, 0.0); + double[] elbowOrientation = Orientation(baseYaw, forearmPitch, 0.0); + double[] toolOrientation = Orientation(baseYaw, HalfTurn, toolRoll); + ArrayOf toolAxis = PoseMath.RotateVector(toolOrientation, s_axisX); + ReadOnlySpan axis = toolAxis.Span; + double[] toolPosition = + [ + wristPosition[0] + (axis[0] * BinPickingPalletizerGeometry.FlangeToTcpMetres), + wristPosition[1] + (axis[1] * BinPickingPalletizerGeometry.FlangeToTcpMetres), + wristPosition[2] + (axis[2] * BinPickingPalletizerGeometry.FlangeToTcpMetres) + ]; + + ArrayOf frames = ArrayOf.Create( + [ + Pose(shoulderPosition, baseOrientation), + Pose(shoulderPosition, shoulderOrientation), + Pose(elbowPosition, elbowOrientation), + Pose(wristPosition, toolOrientation) + ]); + return new SimulatedArmForwardPose( + Pose(toolPosition, toolOrientation), + frames); + } + + public SimulatedArmIkResult Inverse( + Pose3DDataType target, + ReadOnlySpan referenceJointAngles) + { + ArgumentNullException.ThrowIfNull(target); + RequireJointCount(referenceJointAngles); + if (target.Position.Count < 3 || target.Orientation.Count < 4) + { + return new SimulatedArmIkResult( + SimulatedArmKinematicFailure.Kinematics, + "The target must carry a 3D position and quaternion orientation.", + []); + } + + ArrayOf toolAxis = PoseMath.RotateVector(target.Orientation.Span, s_axisX); + ReadOnlySpan axis = toolAxis.Span; + if (Math.Abs(axis[0]) > OrientationTolerance || + Math.Abs(axis[1]) > OrientationTolerance || + Math.Abs(axis[2] + 1.0) > OrientationTolerance) + { + return new SimulatedArmIkResult( + SimulatedArmKinematicFailure.Kinematics, + "The palletizer supports tool-down targets only.", + []); + } + + ReadOnlySpan targetPosition = target.Position.Span; + double wristX = targetPosition[0] - + (axis[0] * BinPickingPalletizerGeometry.FlangeToTcpMetres); + double wristY = targetPosition[1] - + (axis[1] * BinPickingPalletizerGeometry.FlangeToTcpMetres); + double wristZ = targetPosition[2] - + (axis[2] * BinPickingPalletizerGeometry.FlangeToTcpMetres); + double radius = Math.Sqrt((wristX * wristX) + (wristY * wristY)); + double verticalDrop = + BinPickingPalletizerGeometry.ShoulderHeightMetres - wristZ; + const double upper = BinPickingPalletizerGeometry.UpperArmLengthMetres; + const double forearm = BinPickingPalletizerGeometry.ForearmLengthMetres; + double cosineElbow = + (((radius * radius) + (verticalDrop * verticalDrop)) - + (upper * upper) - + (forearm * forearm)) / + (2.0 * upper * forearm); + if (cosineElbow < -1.0 - PositionTolerance || + cosineElbow > 1.0 + PositionTolerance) + { + return new SimulatedArmIkResult( + SimulatedArmKinematicFailure.Unreachable, + "The target lies outside the palletizer workspace.", + []); + } + cosineElbow = Math.Clamp(cosineElbow, -1.0, 1.0); + + double baseYaw = NormalizeAngle(Math.Atan2(wristY, wristX)); + if (!TryExtractToolRoll( + target.Orientation.Span, + baseYaw, + out double requestedToolRoll)) + { + return new SimulatedArmIkResult( + SimulatedArmKinematicFailure.Kinematics, + "The target yaw cannot be represented by the palletizer wrist.", + []); + } + requestedToolRoll = NormalizeAngle(requestedToolRoll); + + var solutions = new List(2); + double elbowMagnitude = Math.Acos(cosineElbow); + AddBranch( + elbowMagnitude, + baseYaw, + requestedToolRoll, + radius, + verticalDrop, + referenceJointAngles, + solutions); + if (elbowMagnitude > PositionTolerance) + { + AddBranch( + -elbowMagnitude, + baseYaw, + requestedToolRoll, + radius, + verticalDrop, + referenceJointAngles, + solutions); + } + solutions.Sort(static (left, right) => + { + // Prefer elbow-up when travel is effectively tied; it keeps the elbow and + // wrist above the work instead of folding toward the table. + double delta = left.TravelCost - right.TravelCost; + if (Math.Abs(delta) > 1e-9) + { + return delta < 0.0 ? -1 : 1; + } + return left.JointAngles[2].CompareTo(right.JointAngles[2]); + }); + + return solutions.Count == 0 + ? new SimulatedArmIkResult( + SimulatedArmKinematicFailure.JointLimit, + "All palletizer branches exceed a joint limit.", + []) + : new SimulatedArmIkResult( + SimulatedArmKinematicFailure.None, + string.Empty, + ArrayOf.Create(solutions.ToArray().AsSpan())); + } + + public bool TrySelectNearest( + Pose3DDataType target, + ReadOnlySpan currentJointAngles, + [NotNullWhen(true)] out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure) + { + return TrySelectNearestCore( + target, + currentJointAngles, + requireClearPath: true, + out solution, + out failure); + } + + public bool TrySelectNearestConfiguration( + Pose3DDataType target, + ReadOnlySpan currentJointAngles, + [NotNullWhen(true)] out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure) + { + return TrySelectNearestCore( + target, + currentJointAngles, + requireClearPath: false, + out solution, + out failure); + } + + public bool IsWithinLimits(ReadOnlySpan jointAngles) + { + RequireJointCount(jointAngles); + return Math.Abs(jointAngles[0]) <= BinPickingPalletizerGeometry.BaseYawLimitRadians && + jointAngles[1] >= BinPickingPalletizerGeometry.ShoulderMinimumRadians && + jointAngles[1] <= BinPickingPalletizerGeometry.ShoulderMaximumRadians && + jointAngles[2] >= BinPickingPalletizerGeometry.ElbowMinimumRadians && + jointAngles[2] <= BinPickingPalletizerGeometry.ElbowMaximumRadians && + Math.Abs(jointAngles[3]) <= BinPickingPalletizerGeometry.ToolRollLimitRadians; + } + + public ArrayOf InterpolateJoints( + ReadOnlySpan start, + ReadOnlySpan end, + double fraction) + { + RequireJointCount(start); + RequireJointCount(end); + double t = Math.Clamp(fraction, 0.0, 1.0); + return ArrayOf.Create( + [ + Lerp(start[0], end[0], t), + Lerp(start[1], end[1], t), + Lerp(start[2], end[2], t), + Lerp(start[3], end[3], t) + ]); + } + + public Pose3DDataType InterpolateCartesian( + Pose3DDataType start, + Pose3DDataType end, + double fraction) + { + ArgumentNullException.ThrowIfNull(start); + ArgumentNullException.ThrowIfNull(end); + double t = Math.Clamp(fraction, 0.0, 1.0); + ReadOnlySpan a = start.Position.Span; + ReadOnlySpan b = end.Position.Span; + return new Pose3DDataType + { + FrameId = string.IsNullOrEmpty(end.FrameId) ? start.FrameId : end.FrameId, + Position = ArrayOf.Create( + [ + Lerp(a[0], b[0], t), + Lerp(a[1], b[1], t), + Lerp(a[2], b[2], t) + ]), + Orientation = Nlerp(start.Orientation.Span, end.Orientation.Span, t) + }; + } + + public bool ClearsPath(ReadOnlySpan start, ReadOnlySpan target) + { + RequireJointCount(start); + RequireJointCount(target); + Span configuration = stackalloc double[BinPickingPalletizerGeometry.AxisCount]; + for (int step = 0; step <= PathSampleCount; step++) + { + double fraction = (double)step / PathSampleCount; + for (int ii = 0; ii < configuration.Length; ii++) + { + configuration[ii] = Lerp(start[ii], target[ii], fraction); + } + if (!ClearsWorkSurface(configuration)) + { + return false; + } + } + return true; + } + + public bool ClearsWorkSurface(ReadOnlySpan jointAngles) + { + SimulatedArmForwardPose pose = Forward(jointAngles); + ReadOnlySpan frames = pose.JointFramePoses.Span; + if (!double.IsNegativeInfinity(MinimumLinkHeight)) + { + for (int ii = 0; ii < frames.Length; ii++) + { + if (frames[ii].Position.Span[2] < MinimumLinkHeight) + { + return false; + } + } + if (pose.ToolPose.Position.Span[2] < MinimumLinkHeight) + { + return false; + } + } + if (Collisions == null) + { + return true; + } + Span points = stackalloc double[(frames.Length + 1) * 3]; + for (int ii = 0; ii < frames.Length; ii++) + { + ReadOnlySpan position = frames[ii].Position.Span; + points[(ii * 3) + 0] = position[0]; + points[(ii * 3) + 1] = position[1]; + points[(ii * 3) + 2] = position[2]; + } + ReadOnlySpan tool = pose.ToolPose.Position.Span; + points[(frames.Length * 3) + 0] = tool[0]; + points[(frames.Length * 3) + 1] = tool[1]; + points[(frames.Length * 3) + 2] = tool[2]; + if (!Collisions.IsClear(points, out _)) + { + return false; + } + if (m_heldObjectSizeX <= 0.0 || + m_heldObjectSizeY <= 0.0 || + m_heldObjectSizeZ <= 0.0) + { + return true; + } + return Collisions.IsBoxClear( + tool[0], + tool[1], + tool[2] - + global::Robotics.IntentEnabledRobot.Simulation.SimulatedArmExecutor + .HeldPartTcpOffset, + m_heldObjectSizeX, + m_heldObjectSizeY, + m_heldObjectSizeZ, + out _); + } + + public IntentFailureEnum MapFailure(SimulatedArmKinematicFailure failure) + { + return SimulatedArmKinematics.ToIntentFailure(failure); + } + + /// + /// Creates a tool-down orientation for one base yaw and jaw roll. + /// + public static ArrayOf ToolDownOrientation(double baseYaw, double toolRoll) + { + return Orientation(baseYaw, HalfTurn, toolRoll).ToArrayOf(); + } + + private static void AddBranch( + double elbow, + double baseYaw, + double toolRoll, + double radius, + double verticalDrop, + ReadOnlySpan reference, + List solutions) + { + double shoulder = Math.Atan2(verticalDrop, radius) - + Math.Atan2( + BinPickingPalletizerGeometry.ForearmLengthMetres * Math.Sin(elbow), + BinPickingPalletizerGeometry.UpperArmLengthMetres + + (BinPickingPalletizerGeometry.ForearmLengthMetres * Math.Cos(elbow))); + double[] candidate = [baseYaw, shoulder, elbow, toolRoll]; + var kinematics = new BinPickingPalletizerKinematics(); + if (!kinematics.IsWithinLimits(candidate)) + { + return; + } + double cost = 0.0; + for (int ii = 0; ii < candidate.Length; ii++) + { + double delta = candidate[ii] - reference[ii]; + cost += delta * delta; + } + solutions.Add(new SimulatedArmIkSolution(candidate.ToArrayOf(), cost)); + } + + private bool TrySelectNearestCore( + Pose3DDataType target, + ReadOnlySpan currentJointAngles, + bool requireClearPath, + [NotNullWhen(true)] out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure) + { + SimulatedArmIkResult result = Inverse(target, currentJointAngles); + failure = result.Failure; + solution = null; + ReadOnlySpan candidates = result.Solutions.Span; + for (int ii = 0; ii < candidates.Length; ii++) + { + ReadOnlySpan angles = candidates[ii].JointAngles.Span; + if (ClearsWorkSurface(angles) && + (!requireClearPath || ClearsPath(currentJointAngles, angles))) + { + solution = candidates[ii]; + return true; + } + } + if (!result.Solutions.IsEmpty) + { + failure = SimulatedArmKinematicFailure.WorkSurface; + } + return false; + } + + private static bool TryExtractToolRoll( + ReadOnlySpan orientation, + double baseYaw, + out double toolRoll) + { + double[] baseDown = Orientation(baseYaw, HalfTurn, 0.0); + double[] relative = Multiply(Inverse(baseDown), orientation); + relative = Normalize(relative); + if (Math.Abs(relative[1]) > OrientationTolerance || + Math.Abs(relative[2]) > OrientationTolerance) + { + toolRoll = 0.0; + return false; + } + toolRoll = 2.0 * Math.Atan2(relative[0], relative[3]); + return true; + } + + private static Pose3DDataType Pose(double[] position, double[] orientation) + { + return new Pose3DDataType + { + FrameId = BinPickingPalletizerGeometry.RobotBaseFrameId, + Position = position.ToArrayOf(), + Orientation = orientation.ToArrayOf() + }; + } + + private static double[] Orientation(double yaw, double pitch, double roll) + { + return Normalize( + Multiply( + Multiply( + [0.0, 0.0, Math.Sin(yaw * 0.5), Math.Cos(yaw * 0.5)], + [0.0, Math.Sin(pitch * 0.5), 0.0, Math.Cos(pitch * 0.5)]), + [Math.Sin(roll * 0.5), 0.0, 0.0, Math.Cos(roll * 0.5)])); + } + + private static double[] Multiply(ReadOnlySpan left, ReadOnlySpan right) + { + double lx = left[0]; + double ly = left[1]; + double lz = left[2]; + double lw = left[3]; + double rx = right[0]; + double ry = right[1]; + double rz = right[2]; + double rw = right[3]; + return + [ + (lw * rx) + (lx * rw) + (ly * rz) - (lz * ry), + (lw * ry) - (lx * rz) + (ly * rw) + (lz * rx), + (lw * rz) + (lx * ry) - (ly * rx) + (lz * rw), + (lw * rw) - (lx * rx) - (ly * ry) - (lz * rz) + ]; + } + + private static double[] Inverse(ReadOnlySpan value) + { + return [-value[0], -value[1], -value[2], value[3]]; + } + + private static double[] Normalize(ReadOnlySpan value) + { + double norm = Math.Sqrt( + (value[0] * value[0]) + + (value[1] * value[1]) + + (value[2] * value[2]) + + (value[3] * value[3])); + return + [ + value[0] / norm, + value[1] / norm, + value[2] / norm, + value[3] / norm + ]; + } + + private static ArrayOf Nlerp( + ReadOnlySpan start, + ReadOnlySpan end, + double fraction) + { + double dot = + (start[0] * end[0]) + + (start[1] * end[1]) + + (start[2] * end[2]) + + (start[3] * end[3]); + double sign = dot < 0.0 ? -1.0 : 1.0; + double[] value = + [ + Lerp(start[0], end[0] * sign, fraction), + Lerp(start[1], end[1] * sign, fraction), + Lerp(start[2], end[2] * sign, fraction), + Lerp(start[3], end[3] * sign, fraction) + ]; + return Normalize(value).ToArrayOf(); + } + + private static double NormalizeAngle(double value) + { + while (value > Math.PI) + { + value -= TwoPi; + } + while (value < -Math.PI) + { + value += TwoPi; + } + return value; + } + + private static double Lerp(double start, double end, double fraction) + { + return start + ((end - start) * fraction); + } + + private static void RequireJointCount(ReadOnlySpan jointAngles) + { + if (jointAngles.Length != BinPickingPalletizerGeometry.AxisCount) + { + throw new ArgumentException( + $"Expected {BinPickingPalletizerGeometry.AxisCount} joint angles.", + nameof(jointAngles)); + } + } + + private static readonly double[] s_axisX = [1.0, 0.0, 0.0]; + + private static readonly double[] s_initialJointAngles = + [0.0, 0.4811790135369469, -1.6798993676150382, HalfTurn]; + + private const double HalfTurn = Math.PI / 2.0; + private const double TwoPi = Math.PI * 2.0; + private const double PositionTolerance = 1e-7; + private const double OrientationTolerance = 1e-5; + private const int PathSampleCount = 32; + private double m_heldObjectSizeX; + private double m_heldObjectSizeY; + private double m_heldObjectSizeZ; + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingParts.cs b/samples/Robotics/BinPickingCell/BinPickingParts.cs new file mode 100644 index 0000000000..0df8696d3b --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingParts.cs @@ -0,0 +1,423 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; + +namespace Vision.BinPickingCell +{ + /// + /// Where a part currently lives in the cell. + /// + internal enum BinPickingPartLocation + { + /// + /// The part is in the bin at its authored position. + /// + InBin = 0, + + /// + /// The part is currently held by the gripper. + /// + Held = 1, + + /// + /// The part has been placed on the fixture (or somewhere else + /// outside the bin). + /// + Placed = 2 + } + + /// + /// Immutable description of a part in the cell. Values mirror the + /// authored transforms in Assets/Cell.usda: same class labels, + /// same initial world positions, same colours and same axis-aligned + /// bounding-box sizes. The detector reads this catalog rather than + /// re-parsing the USD file so the ground truth is deterministic and + /// available even when the OpenUSD render backend is not. + /// + /// + /// Human-readable class name (matches Cell.usda prim name). + /// + /// + /// Small integer id, monotonic and stable across restarts. A client + /// can key on the id when the label is not convenient. + /// + /// + /// Coarse shape hint: cube, cylinder, sphere, + /// slab or brick. Not part of the OPC UA payload but + /// used to compute the 3-D size vector below. + /// + /// + /// Approximate authored displayColor (RGB in [0,1]). Kept for the + /// demo's traceability — a client that reads the rendered frame can + /// cross-reference this against the mean colour of the reported + /// bounding box. + /// + /// + /// Position (metres) in the world frame, matching the + /// authored USD translate. + /// + /// + /// Rotation about the world Z axis, degrees, matching the authored + /// xformOp:rotateZ. + /// + /// + /// Axis-aligned size (width, depth, height) in metres before Z + /// rotation — the extents the detector reports as + /// BoundingBox3D.Size. + /// + internal sealed record BinPickingPart( + string ClassLabel, + uint ClassId, + string Shape, + double[] Colour, + double[] InitialWorldPosition, + double RotationZDegrees, + double[] Size); + + /// + /// Mutable per-part runtime state. Copied out under a lock into a + /// snapshot so the detector can iterate lock-free. + /// + internal sealed class BinPickingPartRuntime + { + public BinPickingPartRuntime(BinPickingPart part) + { + Part = part ?? throw new ArgumentNullException(nameof(part)); + WorldX = part.InitialWorldPosition[0]; + WorldY = part.InitialWorldPosition[1]; + WorldZ = part.InitialWorldPosition[2]; + RotationZDegrees = part.RotationZDegrees; + Location = BinPickingPartLocation.InBin; + } + + public BinPickingPart Part { get; } + + public double WorldX { get; set; } + + public double WorldY { get; set; } + + public double WorldZ { get; set; } + + public double RotationZDegrees { get; set; } + + public BinPickingPartLocation Location { get; set; } + } + + /// + /// Immutable snapshot of one part's live state, safe to hand out to + /// the detector without any lock. + /// + internal sealed record BinPickingPartSnapshot( + BinPickingPart Part, + double WorldX, + double WorldY, + double WorldZ, + double RotationZDegrees, + BinPickingPartLocation Location); + + /// + /// Aggregate live state for the five parts in the cell. Registered + /// as a DI singleton and mutated from either the arm-executor's + /// pick/place events or from the demo hosted service; the detector + /// reads a lock-free snapshot on every inference tick. + /// + /// + /// + /// The class deliberately does not depend on the OPC UA server + /// address space or on the OpenUSD stage: this is where the demo + /// declares "what parts exist and where they are right now". The + /// values are seeded from + /// which mirrors the authored transforms in Assets/Cell.usda. + /// + /// + /// Mutation uses because these + /// operations are short and synchronous; the detector never blocks + /// on the lock — it copies the state into an + /// and + /// walks that copy. + /// + /// + [System.Diagnostics.CodeAnalysis.SuppressMessage( + "Performance", "CA1812", + Justification = "Instantiated by the DI container via AddSingleton.")] + internal sealed class BinPickingWorldState + { + public BinPickingWorldState() + { + IReadOnlyList catalog = BinPickingPartsCatalog.Parts; + m_parts = new BinPickingPartRuntime[catalog.Count]; + for (int ii = 0; ii < catalog.Count; ii++) + { + m_parts[ii] = new BinPickingPartRuntime(catalog[ii]); + } + } + + /// + /// Returns a lock-free snapshot of every part's current state. + /// + public IReadOnlyList Snapshot() + { + lock (m_lock) + { + var copy = new BinPickingPartSnapshot[m_parts.Length]; + for (int ii = 0; ii < m_parts.Length; ii++) + { + BinPickingPartRuntime runtime = m_parts[ii]; + copy[ii] = new BinPickingPartSnapshot( + runtime.Part, + runtime.WorldX, + runtime.WorldY, + runtime.WorldZ, + runtime.RotationZDegrees, + runtime.Location); + } + return copy; + } + } + + /// + /// Marks the part with as + /// and moves its + /// world position to , + /// , . + /// + /// + /// true when the class was recognised. + /// + /// is null. + public bool MarkHeld(string classLabel, double worldX, double worldY, double worldZ) + { + if (classLabel == null) + { + throw new ArgumentNullException(nameof(classLabel)); + } + lock (m_lock) + { + for (int ii = 0; ii < m_parts.Length; ii++) + { + BinPickingPartRuntime runtime = m_parts[ii]; + if (string.Equals(runtime.Part.ClassLabel, classLabel, StringComparison.Ordinal)) + { + runtime.Location = BinPickingPartLocation.Held; + runtime.WorldX = worldX; + runtime.WorldY = worldY; + runtime.WorldZ = worldZ; + return true; + } + } + } + return false; + } + + /// + /// Marks the part with as released at + /// , , + /// , and records whether that spot is inside the bin. + /// + /// + /// The location label follows the coordinates rather than the operation: a part + /// put back inside the bin's footprint is + /// again. It used to become whatever + /// the coordinates said, and since the detector only reports parts that are InBin, + /// a part the robot had returned to the bin stayed invisible to the camera - the + /// world model claiming one thing while its own coordinates said another. + /// + /// + /// true when the class was recognised. + /// + /// is null. + public bool MarkPlaced(string classLabel, double worldX, double worldY, double worldZ) + { + if (classLabel == null) + { + throw new ArgumentNullException(nameof(classLabel)); + } + lock (m_lock) + { + for (int ii = 0; ii < m_parts.Length; ii++) + { + BinPickingPartRuntime runtime = m_parts[ii]; + if (string.Equals(runtime.Part.ClassLabel, classLabel, StringComparison.Ordinal)) + { + runtime.Location = BinPickingPartsCatalog.IsInsideBin(worldX, worldY) + ? BinPickingPartLocation.InBin + : BinPickingPartLocation.Placed; + runtime.WorldX = worldX; + runtime.WorldY = worldY; + runtime.WorldZ = worldZ; + return true; + } + } + } + return false; + } + + /// + /// Resets every part to its authored position and marks it as + /// . Used by the proof + /// service so a second run of the demo starts from a known + /// state. + /// + public void Reset() + { + lock (m_lock) + { + for (int ii = 0; ii < m_parts.Length; ii++) + { + BinPickingPartRuntime runtime = m_parts[ii]; + runtime.WorldX = runtime.Part.InitialWorldPosition[0]; + runtime.WorldY = runtime.Part.InitialWorldPosition[1]; + runtime.WorldZ = runtime.Part.InitialWorldPosition[2]; + runtime.RotationZDegrees = runtime.Part.RotationZDegrees; + runtime.Location = BinPickingPartLocation.InBin; + } + } + } + + private readonly BinPickingPartRuntime[] m_parts; + private readonly Lock m_lock = new(); + } + + /// + /// Static catalog seeded from Assets/Cell.usda. The numbers + /// mirror the authored transforms exactly; changing one without the + /// other silently splits the ground truth from the rendered image. + /// + internal static class BinPickingPartsCatalog + { + /// + /// Returns the five parts of the reference bin, in the order + /// they are declared in the USD stage. + /// + public static IReadOnlyList Parts => s_parts; + + /// + /// Gets whether a world position is inside the bin's footprint. + /// + /// + /// The bin is where parts are picked from and returned to, so "is it in the bin" + /// is what decides whether the camera should still be reporting a part. Keeping + /// the footprint here, next to the authored part positions, keeps one answer to + /// where the bin is: the cell's Bin Location is built from these same numbers. + /// + public static bool IsInsideBin(double worldX, double worldY) + { + return Math.Abs(worldX - BinCentreX) <= BinHalfExtent && + Math.Abs(worldY - BinCentreY) <= BinHalfExtent; + } + + /// + /// Looks up a part by its class label. Returns null when + /// the label is unknown — used by the demo hosted service to + /// cross-check a composed pose against the authored world + /// position. + /// + public static BinPickingPart? TryGet(string classLabel) + { + if (classLabel == null) + { + return null; + } + for (int ii = 0; ii < s_parts.Length; ii++) + { + if (string.Equals(s_parts[ii].ClassLabel, classLabel, StringComparison.Ordinal)) + { + return s_parts[ii]; + } + } + return null; + } + + /// + /// Centre of the bin in the world frame, matching Assets/Cell.usda. + /// + public const double BinCentreX = 0.60; + + /// + /// Centre of the bin in the world frame, matching Assets/Cell.usda. + /// + public const double BinCentreY = 0.0; + + /// + /// Centre of the fixture in the world frame, matching Assets/Cell.usda. + /// + public const double FixtureCentreX = -0.600; + + /// + /// Half the bin's inner span; a part within this of the centre is in the bin. + /// + public const double BinHalfExtent = 0.12; + + private static readonly BinPickingPart[] s_parts = [ + new BinPickingPart( + ClassLabel: "RedCube", + ClassId: 1u, + Shape: "cube", + Colour: [0.90, 0.15, 0.15], + InitialWorldPosition: [0.5200, -0.0800, 0.7400], + RotationZDegrees: 20.0, + Size: [0.0400, 0.0400, 0.0400]), + new BinPickingPart( + ClassLabel: "GreenCylinder", + ClassId: 2u, + Shape: "cylinder", + Colour: [0.15, 0.85, 0.20], + InitialWorldPosition: [0.6700, 0.0800, 0.7350], + RotationZDegrees: 0.0, + Size: [0.0400, 0.0400, 0.0300]), + new BinPickingPart( + ClassLabel: "BlueSphere", + ClassId: 3u, + Shape: "sphere", + Colour: [0.15, 0.30, 0.95], + InitialWorldPosition: [0.6800, -0.0800, 0.7440], + RotationZDegrees: 0.0, + Size: [0.0480, 0.0480, 0.0480]), + new BinPickingPart( + ClassLabel: "YellowSlab", + ClassId: 4u, + Shape: "slab", + Colour: [0.95, 0.85, 0.15], + InitialWorldPosition: [0.5200, 0.0600, 0.7290], + RotationZDegrees: -15.0, + Size: [0.0640, 0.0320, 0.0180]), + new BinPickingPart( + ClassLabel: "OrangeBrick", + ClassId: 5u, + Shape: "brick", + Colour: [0.95, 0.45, 0.10], + InitialWorldPosition: [0.6000, 0.0000, 0.7320], + RotationZDegrees: 40.0, + Size: [0.0500, 0.0280, 0.0240]) + ]; + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingRobotCell.cs b/samples/Robotics/BinPickingCell/BinPickingRobotCell.cs new file mode 100644 index 0000000000..3a858f539f --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingRobotCell.cs @@ -0,0 +1,1086 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Robotics.Server; +using Opc.Ua.Robotics.Server.Builders; +using Opc.Ua.RobotIntent; +using Opc.Ua.Server; +using Robotics.IntentEnabledRobot.Kinematics; +using Robotics.IntentEnabledRobot.Simulation; +using ThreeDCartesianCoordinates = Opc.Ua.ThreeDCartesianCoordinates; +using ThreeDFrame = Opc.Ua.ThreeDFrame; +using ThreeDOrientation = Opc.Ua.ThreeDOrientation; + +namespace Vision.BinPickingCell +{ + /// + /// Robot Intent side of the bin-picking cell. Reuses the simulated arm + /// executor from the IntentEnabledRobot sample and exposes the + /// controller with the frame identifiers from the OPC UA Robotics-Vision + /// Addendum (§4 worked example) so the Vision node manager can name its + /// coordinate frames the same way. + /// + /// + /// + /// The frame names world, robot_base, flange and + /// gripper_tcp match the addendum's frame tree and the + /// FrameId values used by the vision-side calibrations. Publishing + /// the same names from both node managers lets a client cross-reference + /// the intent side ("go to gripper_tcp") with the vision side + /// ("HandEye calibrates camera_eih to flange") + /// without any translation step. + /// + /// + /// The controller carries only the two locations the demo actually uses + /// (Bin and Fixture); the extra "Inspect" and "Handoff" + /// stops from IntentEnabledRobot are omitted here because the + /// bin-picking demo picks from the bin and places on the fixture and + /// nothing else. + /// + /// + internal sealed partial class BinPickingRobotCell : IDisposable + { + public BinPickingRobotCell( + ILogger logger, + SimulatedArmExecutor executor, + BinPickingPalletizerKinematics kinematics, + BinPickingWorldState worldState, + IBinPickingTargetProvider targetProvider) + { + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + m_executor = executor ?? throw new ArgumentNullException(nameof(executor)); + m_kinematics = kinematics ?? throw new ArgumentNullException(nameof(kinematics)); + m_worldState = worldState ?? throw new ArgumentNullException(nameof(worldState)); + m_targetProvider = targetProvider ?? throw new ArgumentNullException(nameof(targetProvider)); + m_toolDownOrientation = executor.CurrentSnapshot.ToolPose.Orientation; + m_executor.SnapshotChanged += OnSnapshotChanged; + m_executor.ResolveLocationPosition = TryResolveLocationPosition; + m_executor.ResolveLocationPose = TryResolveLocationPose; + m_executor.ResolvePickPose = TryResolvePickPose; + m_executor.PickAttemptFinished = OnPickAttemptFinished; + m_executor.PreferCartesianDescent = PreferCartesianDescent; + m_executor.Diagnostic = message => m_logger.ArmTravel(message); + UpdateMovingObstacles(); + } + + internal ServerSystemContext SystemContext => m_systemContext ?? + throw new InvalidOperationException( + "BinPickingRobotCell has not been attached to a Robot Intent build context."); + + internal AsyncCustomNodeManager Manager => m_manager ?? + throw new InvalidOperationException( + "BinPickingRobotCell has not been attached to a Robot Intent build context."); + + internal IIntentControllerBuilder Controller => m_controller ?? + throw new InvalidOperationException( + "The bin-picking intent controller has not been materialised."); + + internal IEnumerable Axes => m_axes; + + internal ushort InstanceNamespaceIndex => m_instanceNamespaceIndex; + + internal IReadOnlyDictionary LocationNodes => m_locationNodes; + + /// + /// Configures the Robot Intent controller and OpenUSD nodes for the cell. + /// + /// is null. + public async ValueTask ConfigureAsync( + IRobotIntentBuildContext context, CancellationToken cancellationToken) + { + if (context == null) + { + throw new ArgumentNullException(nameof(context)); + } + m_manager = context.Manager; + m_systemContext = context.Manager.SystemContext; + m_instanceNamespaceIndex = context.InstanceNamespaceIndex; + await MaterialiseOpenUsdFacilityAsync(cancellationToken).ConfigureAwait(false); + m_controller = await context.AddIntentControllerAsync( + "BinPickingController", + ConfigureController, + cancellationToken).ConfigureAwait(false); + await MaterialisePartStateAsync(cancellationToken).ConfigureAwait(false); + await MaterialiseRepresentationsAsync(cancellationToken).ConfigureAwait(false); + PublishSnapshot(m_executor.CurrentSnapshot); + ArrayOf facets = m_controller.ComputeFacets(); + m_logger.RobotCellReady(m_axes.Count, m_locations.Count, facets); + } + + /// + public void Dispose() + { + m_executor.SnapshotChanged -= OnSnapshotChanged; + } + + private void ConfigureController(IIntentControllerBuilder controller) + { + controller + .WithOperationalMode(OperationalModeEnum.AutomaticExternal) + .WithReady(true) + .WithMaxQueueDepth(16) + .Accepts(cancelSupported: true) + .Accepts(cancelSupported: true, pauseSupported: true) + .Accepts(cancelSupported: false) + .Accepts(cancelSupported: true) + .Accepts(cancelSupported: false) + .Accepts(cancelSupported: true) + .Accepts(cancelSupported: true, pauseSupported: true); + + controller.State.Capabilities!.MissionsSupported!.Value = true; + controller.State.Capabilities.BlendingSupported!.Value = false; + controller.State.Capabilities.ForceControlSupported!.Value = false; + controller.State.Capabilities.MaxTrajectoryPoints!.Value = 64u; + + IIntentFrameBuilder world = controller.AddFrame( + "World", + WorldFrameId, + FrameRoleEnum.World, + Pose(WorldFrameId, 0.0, 0.0, 0.0)); + IIntentFrameBuilder @base = controller.AddFrame( + "RobotBase", + RobotBaseFrameId, + FrameRoleEnum.Base, + Pose(WorldFrameId, 0.0, 0.0, RobotBaseHeightMetres), + frame => frame.WithParent(world)); + IIntentFrameBuilder flange = controller.AddFrame( + "Flange", + FlangeFrameId, + FrameRoleEnum.MechanicalInterface, + m_kinematics.Forward(m_kinematics.InitialJointAngles.Span).JointFramePoses[3], + frame => frame.WithParent(@base)); + IIntentFrameBuilder tool = controller.AddFrame( + "GripperTcp", + ToolFrameId, + FrameRoleEnum.Tool, + Pose( + FlangeFrameId, + BinPickingPalletizerGeometry.FlangeToTcpMetres, + 0.0, + 0.0), + frame => frame.WithParent(flange)); + controller.AddTool("ParallelGripper", tool, fitted: true); + + IIntentOutputSignalBuilder gripperLeft = controller.AddOutput( + "GripperLeftSlide", + Opc.Ua.DataTypeIds.ThreeDCartesianCoordinates, + ToVariant(GripperSlide(m_executor.CurrentSnapshot.GripperOpening, 1.0))); + IIntentOutputSignalBuilder gripperRight = controller.AddOutput( + "GripperRightSlide", + Opc.Ua.DataTypeIds.ThreeDCartesianCoordinates, + ToVariant(GripperSlide(m_executor.CurrentSnapshot.GripperOpening, -1.0))); + m_gripperLeftSlideValue = gripperLeft.State.Value; + m_gripperRightSlideValue = gripperRight.State.Value; + IIntentOutputSignalBuilder leveling = controller.AddOutput( + "PalletizerLeveling", + Opc.Ua.DataTypeIds.Double, + new Variant(LevelingDegrees(m_executor.CurrentSnapshot.JointAngles.Span))); + m_levelingValue = leveling.State.Value; + + for (uint index = 0; index < s_axes.Length; index++) + { + IIntentAxisBuilder axis = controller.AddAxis(s_axes[index], index, AxisKindEnum.Revolute); + (double minimum, double maximum) = AxisLimits(index); + ConfigureAxis(axis.State, minimum, maximum); + m_axes.Add(axis.State); + } + + // Publish where the arm actually is. Leaving Position unset reports 0 for every + // axis, which this arm is never in: at all-zeros the elbow and forearm hang + // straight down through the bench. A client - and the OpenUSD live binding that + // renders from these very nodes - would faithfully show a pose the robot never + // held, so seed them from the simulator's own starting configuration. + PublishSnapshot(m_executor.CurrentSnapshot); + + foreach ((string name, double x, double y, double z, double rz) in s_locations) + { + bool isBin = string.Equals(name, BinLocationName, StringComparison.Ordinal); + + // A home slot starts occupied because its part starts there, and the bin + // starts occupied because all of them do. Occupied is kept true to the + // world from here on by UpdateLocationOccupancy: a slot that stayed + // "occupied" after the robot emptied it would be a node reporting + // something the cell can see is no longer the case. + uint capacity = isBin ? PayloadSlotCount : 1u; + bool occupied = isBin || name.StartsWith(HomeLocationPrefix, StringComparison.Ordinal); + IIntentLocationBuilder location = controller.AddLocation( + name, + Pose(WorldFrameId, x, y, z, rz), + builder => builder.WithOccupancy(occupied, capacity)); + m_locations.Add(location.State); + m_locationNodes[name] = location.State.NodeId; + m_locationStates[name] = location.State; + } + + controller.WithDescription(description => description + .WithKinematicChain(CreateKinematicChain()) + .WithLimits( + m_kinematics.MaximumReach, + payloadLimit: 2.0, + maxCartesianSpeed: 0.25, + maxCartesianAcceleration: 0.7)); + } + + private void ConfigureAxis(global::Opc.Ua.RobotIntent.AxisState axis, double min, double max) + { + axis.CreateOrReplaceMinPosition(SystemContext, null).Value = min; + axis.CreateOrReplaceMaxPosition(SystemContext, null).Value = max; + axis.CreateOrReplaceMaxSpeed(SystemContext, null).Value = MaxAxisSpeedDegreesPerSecond; + } + + private ArrayOf CreateKinematicChain() + { + return ArrayOf.Create( + [ + Joint( + s_axes[0], + Pose( + RobotBaseFrameId, + 0.0, + 0.0, + BinPickingPalletizerGeometry.ShoulderHeightMetres), + s_axisZ), + Joint(s_axes[1], Pose(s_axes[0], 0.0, 0.0, 0.0), s_axisY), + Joint( + s_axes[2], + Pose( + s_axes[1], + BinPickingPalletizerGeometry.UpperArmLengthMetres, + 0.0, + 0.0), + s_axisY), + Joint( + s_axes[3], + Pose( + s_axes[2], + BinPickingPalletizerGeometry.ForearmLengthMetres, + 0.0, + 0.0), + s_axisX) + ]); + } + + private static KinematicJointDataType Joint( + string axisId, + Pose3DDataType origin, + double[] axisVector) + { + return new KinematicJointDataType + { + AxisId = axisId, + Kind = AxisKindEnum.Revolute, + OriginTransform = origin, + AxisVector = axisVector.ToArrayOf() + }; + } + + private static (double Minimum, double Maximum) AxisLimits(uint index) + { + const double radiansToDegrees = 180.0 / Math.PI; + return index switch + { + 0 => ( + -BinPickingPalletizerGeometry.BaseYawLimitRadians * radiansToDegrees, + BinPickingPalletizerGeometry.BaseYawLimitRadians * radiansToDegrees), + 1 => ( + BinPickingPalletizerGeometry.ShoulderMinimumRadians * radiansToDegrees, + BinPickingPalletizerGeometry.ShoulderMaximumRadians * radiansToDegrees), + 2 => ( + BinPickingPalletizerGeometry.ElbowMinimumRadians * radiansToDegrees, + BinPickingPalletizerGeometry.ElbowMaximumRadians * radiansToDegrees), + _ => ( + -BinPickingPalletizerGeometry.ToolRollLimitRadians * radiansToDegrees, + BinPickingPalletizerGeometry.ToolRollLimitRadians * radiansToDegrees) + }; + } + + private void OnSnapshotChanged(object? sender, SimulatedArmSnapshot snapshot) + { + PublishSnapshot(snapshot); + } + + /// + /// Resolves one of this cell's Locations to an approach position in the arm's base + /// frame, so a Pick or a Place travels to the bin or the fixture instead of + /// actuating the gripper where it stands. + /// + /// + /// The Locations are authored in the world frame and the kinematics work in the + /// robot base frame, so the base origin is subtracted. The approach height lifts + /// the target off the bench: a target on the surface asks the solver for a pose + /// with the tool exactly at table height, which is both harder to reach and not + /// what an approach looks like. + /// + private bool TryResolveLocationPosition(NodeId location, out ArrayOf position) + { + foreach ((string name, double x, double y, double z, double _) in s_locations) + { + if (m_locationNodes.TryGetValue(name, out NodeId nodeId) && nodeId == location) + { + // Carrying something means this is the move before a Place, because a + // Pick travels with the gripper empty and closes on arrival while a + // Place travels holding the part and opens. So the tool can descend to + // just above the height that leaves the part on its support. The small + // clearance keeps the wrist and jaws out of an accumulated stack; the + // support model settles the released part the remaining 17 mm instead + // of leaving it floating or driving the tool into what is already there. + double toolWorldZ = m_carriedClass.Length > 0 + ? RestingCentreHeight(m_carriedClass, x, y) + + SimulatedArmExecutor.HeldPartTcpOffset + + PlaceReleaseClearanceMetres + : z + ApproachHeightMetres; + position = new[] { x, y, toolWorldZ - RobotBaseHeightMetres }.ToArrayOf(); + return true; + } + } + position = ArrayOf.Empty; + return false; + } + + /// + /// Resolves a Location to a deliberate tool-down pose rather than inheriting the + /// yaw left behind by the previous grasp. + /// + private bool TryResolveLocationPose(NodeId location, out Pose3DDataType pose) + { + if (TryResolveLocationPosition(location, out ArrayOf position)) + { + foreach ((string name, double _, double _, double _, double _) in s_locations) + { + if (m_locationNodes.TryGetValue(name, out NodeId nodeId) && nodeId == location) + { + pose = new Pose3DDataType + { + FrameId = RobotBaseFrameId, + Position = position, + // The Location's Rz describes how a part or fixture is authored, + // not a mandatory wrist yaw. Applying Fixture's 25 degrees here + // made the first place succeed and left no retract path for the + // next pick. Hold one solved tool-down orientation everywhere; + // the executor's deterministic yaw search still turns it when + // the standard pose itself has no clear solution. + Orientation = m_toolDownOrientation + }; + return true; + } + } + } + pose = new Pose3DDataType(); + return false; + } + + private bool TryResolvePickPose( + NodeId location, + string objectClass, + out Pose3DDataType pose) + { + if (!m_targetProvider.TryResolve(objectClass, out BinPickingTarget target)) + { + pose = new Pose3DDataType(); + return false; + } + foreach ((string name, double x, double y, double _, double _) in s_locations) + { + if (!m_locationNodes.TryGetValue(name, out NodeId nodeId) || nodeId != location) + { + continue; + } + double radius = string.Equals(name, BinLocationName, StringComparison.Ordinal) + ? BinSlotRadiusMetres + : GraspReachRadiusMetres; + if (Math.Abs(target.WorldX - x) > radius || + Math.Abs(target.WorldY - y) > radius) + { + break; + } + double baseYaw = Math.Atan2(target.WorldY, target.WorldX); + pose = new Pose3DDataType + { + FrameId = RobotBaseFrameId, + Position = new[] + { + target.WorldX, + target.WorldY, + target.WorldZ + + SimulatedArmExecutor.HeldPartTcpOffset - + RobotBaseHeightMetres + }.ToArrayOf(), + Orientation = BinPickingPalletizerKinematics.ToolDownOrientation( + baseYaw, + PalletizerWorkToolRollRadians) + }; + m_pickTargetClass = objectClass; + UpdateMovingObstacles(); + return true; + } + pose = new Pose3DDataType(); + return false; + } + + private void OnPickAttemptFinished(string objectClass) + { + if (string.Equals(m_pickTargetClass, objectClass, StringComparison.Ordinal)) + { + m_pickTargetClass = string.Empty; + UpdateMovingObstacles(); + } + } + + /// + /// Gets whether a Location is inside the open bin, where the final approach should + /// be vertical rather than a joint interpolation that can sweep through a wall. + /// + private bool PreferCartesianDescent(NodeId location) + { + foreach ((string name, NodeId nodeId) in m_locationNodes) + { + if (nodeId == location) + { + // Empty-gripper picks are made Cartesian by the executor regardless of + // this value. Loaded placements into a home slot are Cartesian too, and + // the executor records every joint sample so the next command can replay + // the exact approach in reverse. The fixture keeps the short local joint + // approach because its final Cartesian branch is less reliable. + return string.Equals(name, BinLocationName, StringComparison.Ordinal) || + name.StartsWith(HomeLocationPrefix, StringComparison.Ordinal); + } + } + return false; + } + + /// + /// Gets whether the named part is lying under the tool, which is the only place a + /// grasp can pick it up from. + /// + /// + /// A Pick names an object class and a source Location. Without this check the cell + /// hands over whatever class the intent names no matter where that part actually + /// is, so picking from an empty bin still produces a part in the gripper - and a + /// loop that keeps picking and placing walks every part onto one spot and stacks + /// them into the air, each Place resting the part on the pile the last one left. + /// The test is against the tool's own position rather than the Location the intent + /// named: the tool has already travelled there by the time it closes, and a field + /// remembering the last resolved Location goes stale as soon as intents queue + /// back to back - which made every grasp fail and the robot stop moving at all. + /// + private bool CanGrasp(string classLabel, double toolX, double toolY) + { + IReadOnlyList parts = m_worldState.Snapshot(); + for (int ii = 0; ii < parts.Count; ii++) + { + BinPickingPartSnapshot part = parts[ii]; + if (!string.Equals(part.Part.ClassLabel, classLabel, StringComparison.Ordinal)) + { + continue; + } + bool within = Math.Abs(part.WorldX - toolX) <= GraspReachRadiusMetres && + Math.Abs(part.WorldY - toolY) <= GraspReachRadiusMetres; + if (!within) + { + m_logger.GraspFoundNothing(classLabel, FormatPosition(toolX, toolY)); + } + return within; + } + return false; + } + + /// + /// Formats a position for a log message. + /// + private static string FormatPosition(double x, double y) + { + return string.Create( + System.Globalization.CultureInfo.InvariantCulture, $"({x:F3}, {y:F3})"); + } + + /// + /// Gets the height a part's centre comes to rest at over a spot on the bench, given + /// everything else that is standing there. + /// + /// + /// This is what makes a second part placed on the same spot end up on top of the + /// first rather than inside it, and it is why a placed part stops floating: the + /// part used to be left wherever the tool centre point was, which for a Place at the + /// approach height was 165 mm above the bench. + /// + private double RestingCentreHeight(string classLabel, double x, double y) + { + BinPickingPart? part = BinPickingPartsCatalog.TryGet(classLabel); + if (part == null) + { + return BenchTopMetres; + } + return m_support.RestingCentreHeight( + x, y, part.Size[0], part.Size[1], part.Size[2], SupportingParts(classLabel)); + } + + /// + /// Gets the other parts as solids that can hold something up, leaving out the one + /// being placed and anything currently in the gripper. + /// + private ArrayOf SupportingParts(string exclude) + { + IReadOnlyList parts = m_worldState.Snapshot(); + var solids = new List(parts.Count); + for (int ii = 0; ii < parts.Count; ii++) + { + BinPickingPartSnapshot snapshot = parts[ii]; + if (string.Equals(snapshot.Part.ClassLabel, exclude, StringComparison.Ordinal) || + snapshot.Location == BinPickingPartLocation.Held) + { + continue; + } + solids.Add(new SimulatedSupportSolid( + snapshot.Part.ClassLabel, + snapshot.WorldX, + snapshot.WorldY, + snapshot.Part.Size[0], + snapshot.Part.Size[1], + snapshot.WorldZ + (snapshot.Part.Size[2] * 0.5))); + } + return ArrayOf.Create(solids.ToArray().AsSpan()); + } + + /// + /// Moves the carried part in the cell's world model so the world changes when the + /// robot changes it, rather than only when a proof service says so. + /// + /// + /// The arm reports the carried position in its own base frame; the world model and + /// the ground-truth detector work in the world frame, so the base height is added + /// back. Running on every snapshot is what makes the part travel with the tool + /// instead of teleporting when the grasp opens. + /// + private void TrackHeldPart(SimulatedArmSnapshot snapshot) + { + ReadOnlySpan carried = snapshot.HeldPartPosition.Span; + if (carried.Length < 3) + { + return; + } + double worldX = carried[0]; + double worldY = carried[1]; + double worldZ = carried[2] + RobotBaseHeightMetres; + + if (snapshot.HasObject && snapshot.HeldObjectClass.Length > 0) + { + if (m_carriedClass.Length == 0 && + !CanGrasp(snapshot.HeldObjectClass, worldX, worldY)) + { + // The gripper closed where the part is not. Attaching it anyway is + // what let a Pick teleport a part out of a stack on the far side of + // the bench into the tool: the cell would then carry it off and set it + // down somewhere it was never taken from. Closing on nothing is the + // honest outcome, and it leaves the part where it lies. + return; + } + _ = m_worldState.MarkHeld(snapshot.HeldObjectClass, worldX, worldY, worldZ); + m_carriedClass = snapshot.HeldObjectClass; + m_pickTargetClass = string.Empty; + SetHeldObjectEnvelope(snapshot.HeldObjectClass); + PublishPartPosition(snapshot.HeldObjectClass, worldX, worldY, worldZ); + UpdateLocationOccupancy(); + UpdateMovingObstacles(); + return; + } + if (!snapshot.HasObject && m_carriedClass.Length > 0) + { + // The gripper opened. The part settles onto whatever is under it rather + // than staying at the tool centre point: released at the approach height it + // would hang in the air, and released over another part it would stand + // inside it. The tool has normally already descended to the resting height + // by this point, so this is a few millimetres of settle, not a fall. + double restingZ = m_support.ClampAboveSupport( + worldX, + worldY, + PartSize(m_carriedClass, 0), + PartSize(m_carriedClass, 1), + PartSize(m_carriedClass, 2), + worldZ, + SupportingParts(m_carriedClass)); + _ = m_worldState.MarkPlaced(m_carriedClass, worldX, worldY, restingZ); + PublishPartPosition(m_carriedClass, worldX, worldY, restingZ); + m_carriedClass = string.Empty; + m_pickTargetClass = string.Empty; + m_kinematics.ClearHeldObjectEnvelope(); + UpdateLocationOccupancy(); + UpdateMovingObstacles(); + } + } + + /// + /// Republishes moving workpieces as solids the arm has to keep out of. + /// + /// + /// The bench and bin walls are fixed. The fixture, pegs, non-target workpieces and + /// accumulated stack are republished as moving solids. Whatever the gripper carries + /// is omitted from that list and represented by the palletizer's held-object swept + /// envelope instead. + /// + private void UpdateMovingObstacles() + { + SimulatedCollisionModel? collisions = m_kinematics?.Collisions; + if (collisions == null) + { + return; + } + IReadOnlyList parts = m_worldState.Snapshot(); + var solids = new List(parts.Count + 4) + { + new( + "FixturePlate", + BinPickingCellGeometry.FixtureCentreX, + 0.0, + 0.140, + 0.140, + FixturePlateTopMetres - RobotBaseHeightMetres - 0.018, + FixturePlateTopMetres - RobotBaseHeightMetres), + new( + "FixturePegA", + BinPickingCellGeometry.FixtureCentreX - + BinPickingCellGeometry.FixturePegOffsetMetres, + BinPickingCellGeometry.FixturePegOffsetMetres, + 0.018, + 0.018, + FixturePlateTopMetres - RobotBaseHeightMetres, + FixturePegTopMetres - RobotBaseHeightMetres), + new( + "FixturePegB", + BinPickingCellGeometry.FixtureCentreX + + BinPickingCellGeometry.FixturePegOffsetMetres, + BinPickingCellGeometry.FixturePegOffsetMetres, + 0.018, + 0.018, + FixturePlateTopMetres - RobotBaseHeightMetres, + FixturePegTopMetres - RobotBaseHeightMetres), + new( + "FixturePegC", + BinPickingCellGeometry.FixtureCentreX, + -BinPickingCellGeometry.FixturePegOffsetMetres, + 0.018, + 0.018, + FixturePlateTopMetres - RobotBaseHeightMetres, + FixturePegTopMetres - RobotBaseHeightMetres) + }; + + for (int ii = 0; ii < parts.Count; ii++) + { + BinPickingPartSnapshot part = parts[ii]; + if (part.Location == BinPickingPartLocation.Held || + string.Equals(part.Part.ClassLabel, m_carriedClass, StringComparison.Ordinal) || + string.Equals(part.Part.ClassLabel, m_pickTargetClass, StringComparison.Ordinal)) + { + continue; + } + double halfHeight = part.Part.Size[2] * 0.5; + solids.Add(new SimulatedObstacleBox( + part.Part.ClassLabel, + part.WorldX, + part.WorldY, + part.Part.Size[0], + part.Part.Size[1], + part.WorldZ - halfHeight - RobotBaseHeightMetres, + part.WorldZ + halfHeight - RobotBaseHeightMetres)); + } + collisions.MovingObstacles = ArrayOf.Create(solids.ToArray().AsSpan()); + } + + private void SetHeldObjectEnvelope(string classLabel) + { + BinPickingPart? part = BinPickingPartsCatalog.TryGet(classLabel); + if (part == null) + { + m_kinematics.ClearHeldObjectEnvelope(); + return; + } + m_kinematics.SetHeldObjectEnvelope( + part.Size[0], + part.Size[1], + part.Size[2]); + } + + /// + /// Republishes each Location's Occupied flag from where the parts actually are. + /// + /// + /// Occupancy is declarative in the Robot Intent model - nothing enforces it - which + /// makes it easy to author once and leave wrong. A client asking "is the bin empty + /// yet" would then be told "no" forever. A Location counts as occupied when a part + /// that is not in the gripper is standing within its slot radius. + /// + private void UpdateLocationOccupancy() + { + if (m_systemContext == null || m_locationStates.Count == 0) + { + return; + } + IReadOnlyList parts = m_worldState.Snapshot(); + foreach ((string name, double x, double y, double _, double _) in s_locations) + { + if (!m_locationStates.TryGetValue(name, out global::Opc.Ua.RobotIntent.LocationState? state) || + state.Occupied == null) + { + continue; + } + double radius = string.Equals(name, BinLocationName, StringComparison.Ordinal) + ? BinSlotRadiusMetres + : HomeSlotRadiusMetres; + bool occupied = false; + for (int ii = 0; ii < parts.Count && !occupied; ii++) + { + BinPickingPartSnapshot part = parts[ii]; + occupied = part.Location != BinPickingPartLocation.Held && + Math.Abs(part.WorldX - x) <= radius && + Math.Abs(part.WorldY - y) <= radius; + } + if (state.Occupied.Value == occupied) + { + continue; + } + state.Occupied.Value = occupied; + state.Occupied.ClearChangeMasks(m_systemContext, false); + } + } + + /// + /// Gets one extent of a part, or zero when the catalogue does not know it. + /// + private static double PartSize(string classLabel, int axis) + { + BinPickingPart? part = BinPickingPartsCatalog.TryGet(classLabel); + return (part?.Size[axis]) ?? 0.0; + } + + /// + /// Pushes a part's world position onto its variable so the OpenUSD live binding, and + /// any other subscriber, follows it. + /// + /// + /// The value is a rather than a bare + /// double[3] on purpose. §5.8 of the OpenUSD companion specification defines + /// a translation source as a structured 3D coordinate, and the connector's + /// translation profile fails closed on anything else - so a plain array left every + /// part's translation unresolved and the parts never moved in the viewport, however + /// faithfully the server tracked them. + /// + private void PublishPartPosition(string classLabel, double worldX, double worldY, double worldZ) + { + if (m_systemContext == null || + !m_partPositionNodes.TryGetValue(classLabel, out BaseDataVariableState? node)) + { + return; + } + node.Value = new Variant(new ExtensionObject(new ThreeDCartesianCoordinates + { + X = worldX, + Y = worldY, + Z = worldZ + })); + node.ClearChangeMasks(m_systemContext, false); + } + + private void PublishSnapshot(SimulatedArmSnapshot snapshot) + { + TrackHeldPart(snapshot); + if (m_systemContext == null) + { + return; + } + for (int ii = 0; ii < m_axes.Count && ii < snapshot.JointAngles.Count; ii++) + { + global::Opc.Ua.RobotIntent.AxisState axis = m_axes[ii]; + if (axis.Position != null) + { + axis.Position.Value = snapshot.JointAngles[ii] * 180.0 / Math.PI; + axis.Position.ClearChangeMasks(m_systemContext, true); + } + } + if (m_gripperLeftSlideValue != null) + { + m_gripperLeftSlideValue.Value = ToVariant( + GripperSlide(snapshot.GripperOpening, 1.0)); + m_gripperLeftSlideValue.ClearChangeMasks(m_systemContext, true); + } + if (m_gripperRightSlideValue != null) + { + m_gripperRightSlideValue.Value = ToVariant( + GripperSlide(snapshot.GripperOpening, -1.0)); + m_gripperRightSlideValue.ClearChangeMasks(m_systemContext, true); + } + if (m_levelingValue != null && snapshot.JointAngles.Count >= 3) + { + m_levelingValue.Value = LevelingDegrees(snapshot.JointAngles.Span); + m_levelingValue.ClearChangeMasks(m_systemContext, true); + } + } + + private static double LevelingDegrees(ReadOnlySpan jointAngles) + { + return ((Math.PI / 2.0) - jointAngles[1] - jointAngles[2]) * 180.0 / Math.PI; + } + + /// + /// Converts the jaw opening into one slide's USD translation. + /// + private static ThreeDCartesianCoordinates GripperSlide(double opening, double direction) + { + double centre = Math.Clamp(opening * 0.5, GripperClosedHalfGap, GripperOpenHalfGap); + return new ThreeDCartesianCoordinates + { + X = 0.0, + Y = direction * centre, + Z = 0.0 + }; + } + + private static Variant ToVariant(ThreeDCartesianCoordinates value) + { + return new Variant(new ExtensionObject(value)); + } + + private static Pose3DDataType Pose(string frameId, double x, double y, double z, double rzDegrees = 0.0) + { + return PoseMath.FromThreeDFrame( + new ThreeDFrame + { + CartesianCoordinates = new ThreeDCartesianCoordinates + { + X = x, + Y = y, + Z = z + }, + Orientation = new ThreeDOrientation + { + C = rzDegrees * Math.PI / 180.0 + } + }, + frameId); + } + + /// + /// Gets the name of the Location that holds a part's own starting spot in the bin. + /// + internal static string HomeLocationName(string classLabel) + { + return HomeLocationPrefix + classLabel; + } + + /// + /// Builds this cell's Locations: the bin, the fixture, and one home slot per part. + /// + /// + /// A Place intent names a LocationType and carries no pose, so "the bin" is the + /// only way back unless each part has a Location of its own - and placing every + /// part at the bin puts them all on the one spot, which the support model then + /// stacks in the middle of it. The home slots are the authored scattered + /// positions, so a Place can return a part to where the cell first had it. + /// The bin and fixture coordinates are where those two actually stand in + /// Cell.usda; they used to be somewhere else entirely - the Bin at y = -0.28 when + /// the bin spans +/-0.12, the Fixture at (0.48, 0.26) when the fixture stood at + /// (-0.32, 0) - so "place it on the fixture" put the part down on bare bench a long + /// way from the fixture, and the render disagreed with the model about where the + /// cell's own furniture was. The Z is the surface a part stands on there. + /// + private static (string Name, double X, double Y, double Z, double Rz)[] BuildLocations() + { + IReadOnlyList parts = BinPickingPartsCatalog.Parts; + var locations = new List<(string Name, double X, double Y, double Z, double Rz)>(parts.Count + 2) + { + (BinLocationName, BinPickingPartsCatalog.BinCentreX, BinPickingPartsCatalog.BinCentreY, + BenchTopMetres, 0.0), + (FixtureLocationName, BinPickingCellGeometry.FixtureCentreX, 0.0, + FixturePlateTopMetres, 25.0) + }; + for (int ii = 0; ii < parts.Count; ii++) + { + BinPickingPart part = parts[ii]; + locations.Add(( + HomeLocationName(part.ClassLabel), + part.InitialWorldPosition[0], + part.InitialWorldPosition[1], + BenchTopMetres, + part.RotationZDegrees)); + } + return [.. locations]; + } + + internal const string WorldFrameId = "world"; + internal const string RobotBaseFrameId = "robot_base"; + internal const string FlangeFrameId = "flange"; + internal const string ToolFrameId = "gripper_tcp"; + internal const string CameraFrameId = "camera_eih"; + + /// + /// The Location a part is picked from and returned to as a group. + /// + internal const string BinLocationName = "Bin"; + + /// + /// The Location parts are stacked on. + /// + internal const string FixtureLocationName = "Fixture"; + + private const string HomeLocationPrefix = "Home"; + + /// + /// The robot stands on a 200 mm riser above the lowered work surface. Keeping these + /// in the shared geometry class makes the USD scene, frame tree, collision model and + /// support model describe one cell. + /// + internal const double RobotBaseHeightMetres = BinPickingCellGeometry.RobotBaseHeightMetres; + internal const double BenchTopMetres = BinPickingCellGeometry.BenchTopMetres; + private const double FixturePlateTopMetres = BinPickingCellGeometry.FixturePlateTopMetres; + private const double FixturePegTopMetres = BinPickingCellGeometry.FixturePegTopMetres; + + /// + /// How far above a Location the tool travels to when it is going to pick something + /// up. Far enough to read as an approach rather than a collision. A Place does not + /// use this: it descends to the height that leaves the part resting on whatever is + /// under it, so releasing does not drop the part from the approach height. + /// + private const double ApproachHeightMetres = 0.20; + private const double PlaceReleaseClearanceMetres = 0.017; + private const double GripperClosedHalfGap = 0.009; + private const double GripperOpenHalfGap = 0.040; + private const double PalletizerWorkToolRollRadians = Math.PI / 2.0; + + /// + /// How close a part has to be to a Location to count as standing in it. The bin is a + /// tray parts are scattered across, so it reuses the catalogue's footprint; a home + /// slot is one part's own spot, so its radius only has to cover the millimetre-scale + /// settle a release leaves behind. + /// + private const double BinSlotRadiusMetres = BinPickingPartsCatalog.BinHalfExtent; + private const double HomeSlotRadiusMetres = 0.02; + + /// + /// How far a part may be from the tool and still be grasped by it. Wide enough to + /// cover a Location's footprint, since a Pick travels to the Location rather than to + /// the part, and far narrower than the 0.70 m between the bin and the fixture, so a + /// grasp can never reach across the bench for something. + /// + private const double GraspReachRadiusMetres = 0.12; + + /// + /// The simulator's DefaultJointSpeed is 0.9 rad/s; Position and the limits are + /// published in degrees, so the speed limit is too. + /// + private const double MaxAxisSpeedDegreesPerSecond = 0.9 * 180.0 / Math.PI; + private const uint PayloadSlotCount = 8u; + + private static readonly (string Name, double X, double Y, double Z, double Rz)[] s_locations = + BuildLocations(); + + /// + /// The solids in this cell that never move and that a part can come to rest on, in + /// the world frame. Sizes are full extents, matching Cell.usda. + /// + private static readonly SimulatedSupportSolid[] s_supportFixtures = + [ + new("Bench", 0.0, 0.0, 1.4, 0.9, BenchTopMetres), + new("FixturePlate", BinPickingCellGeometry.FixtureCentreX, 0.0, + 0.14, 0.14, FixturePlateTopMetres), + new("FixturePegA", BinPickingCellGeometry.FixtureCentreX - + BinPickingCellGeometry.FixturePegOffsetMetres, + BinPickingCellGeometry.FixturePegOffsetMetres, + 0.018, 0.018, FixturePegTopMetres), + new("FixturePegB", BinPickingCellGeometry.FixtureCentreX + + BinPickingCellGeometry.FixturePegOffsetMetres, + BinPickingCellGeometry.FixturePegOffsetMetres, + 0.018, 0.018, FixturePegTopMetres), + new("FixturePegC", BinPickingCellGeometry.FixtureCentreX, + -BinPickingCellGeometry.FixturePegOffsetMetres, + 0.018, 0.018, FixturePegTopMetres) + ]; + + private static readonly string[] s_axes = ["J1", "J2", "J3", "J4"]; + private static readonly double[] s_axisX = [1.0, 0.0, 0.0]; + private static readonly double[] s_axisZ = [0.0, 0.0, 1.0]; + private static readonly double[] s_axisY = [0.0, 1.0, 0.0]; + + private readonly ILogger m_logger; + private readonly SimulatedArmExecutor m_executor; + private readonly BinPickingPalletizerKinematics m_kinematics; + private readonly BinPickingWorldState m_worldState; + private readonly IBinPickingTargetProvider m_targetProvider; + private readonly ArrayOf m_toolDownOrientation; + + private readonly SimulatedSupportModel m_support = + new(ArrayOf.Create(s_supportFixtures.AsSpan()), BenchTopMetres); + + private string m_carriedClass = string.Empty; + private string m_pickTargetClass = string.Empty; + private readonly List m_axes = []; + private readonly List m_locations = []; + private readonly Dictionary m_locationNodes = new(StringComparer.Ordinal); + + private readonly Dictionary m_locationStates = + new(StringComparer.Ordinal); + + private BaseVariableState? m_gripperLeftSlideValue; + private BaseVariableState? m_gripperRightSlideValue; + private BaseVariableState? m_levelingValue; + private AsyncCustomNodeManager? m_manager; + private IIntentControllerBuilder? m_controller; + private ServerSystemContext? m_systemContext; + private ushort m_instanceNamespaceIndex; + } + + internal static partial class BinPickingRobotCellLog + { + [LoggerMessage(EventId = BinPickingCellEventIds.Configurator + 1, + Level = LogLevel.Information, + Message = "Robot Intent side of BinPickingCell ready " + + "({AxisCount} axes, {LocationCount} locations, facets {Facets}).")] + public static partial void RobotCellReady( + this ILogger logger, + int axisCount, int locationCount, ArrayOf facets); + + [LoggerMessage(EventId = BinPickingCellEventIds.Configurator + 2, + Level = LogLevel.Warning, + Message = "Grasp at {ToolPosition} found no {ClassLabel} under the tool; " + + "the gripper closed on nothing.")] + public static partial void GraspFoundNothing( + this ILogger logger, + string classLabel, string toolPosition); + + [LoggerMessage(EventId = BinPickingCellEventIds.Configurator + 3, + Level = LogLevel.Information, + Message = "Arm travel: {Message}.")] + public static partial void ArmTravel( + this ILogger logger, + string message); + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingTargetProvider.cs b/samples/Robotics/BinPickingCell/BinPickingTargetProvider.cs new file mode 100644 index 0000000000..0d53472e3c --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingTargetProvider.cs @@ -0,0 +1,264 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Globalization; +using Opc.Ua; +using Opc.Ua.RobotIntent; +using Opc.Ua.Vision; + +namespace Vision.BinPickingCell +{ + internal readonly record struct BinPickingTarget( + string ClassLabel, + double WorldX, + double WorldY, + double WorldZ, + DateTime TimestampUtc, + string ResultId, + string SourceFrameId); + + internal interface IBinPickingTargetProvider + { + void PublishWorldState( + string resultId, + DateTimeUtc timestamp, + IReadOnlyList parts); + + void PublishDetections( + string resultId, + DateTimeUtc timestamp, + ArrayOf detections, + VisionPose3DDataType cameraInWorld, + string cameraFrameId); + + bool TryResolve(string classLabel, out BinPickingTarget target); + } + + /// + /// Provides the pose a Pick should target for the class selected by Vision. + /// + internal sealed class BinPickingTargetProvider : IBinPickingTargetProvider + { + public BinPickingTargetProvider( + BinPickingWorldState worldState, + BinPickingCellOptions options) + { + m_worldState = worldState ?? throw new ArgumentNullException(nameof(worldState)); + m_options = options ?? throw new ArgumentNullException(nameof(options)); + } + + public void PublishWorldState( + string resultId, + DateTimeUtc timestamp, + IReadOnlyList parts) + { + DateTime created = timestamp.IsNull ? DateTime.UtcNow : timestamp.ToDateTime(); + for (int ii = 0; ii < parts.Count; ii++) + { + BinPickingPartSnapshot part = parts[ii]; + if (part.Location != BinPickingPartLocation.InBin) + { + continue; + } + m_targets[part.Part.ClassLabel] = new BinPickingTarget( + part.Part.ClassLabel, + part.WorldX, + part.WorldY, + part.WorldZ, + created, + resultId, + WorldFrameId); + } + } + + public void PublishDetections( + string resultId, + DateTimeUtc timestamp, + ArrayOf detections, + VisionPose3DDataType cameraInWorld, + string cameraFrameId) + { + if (string.IsNullOrWhiteSpace(cameraFrameId)) + { + throw new ArgumentException("A camera frame id is required.", nameof(cameraFrameId)); + } + ReadOnlySpan cameraPosition = cameraInWorld.Position.Span; + ReadOnlySpan cameraOrientation = cameraInWorld.Orientation.Span; + if (cameraPosition.Length < 3 || cameraOrientation.Length < 4) + { + throw new ServiceResultException( + StatusCodes.BadInvalidArgument, + "The camera-in-world calibration must carry a 3D position and quaternion."); + } + DateTime created = timestamp.IsNull ? DateTime.UtcNow : timestamp.ToDateTime(); + IReadOnlyList parts = m_worldState.Snapshot(); + ReadOnlySpan values = detections.Span; + var acceptedTargets = new List(values.Length); + for (int ii = 0; ii < values.Length; ii++) + { + VisionDetectionDataType detection = values[ii]; + if (!detection.HasPose || string.IsNullOrWhiteSpace(detection.ClassLabel)) + { + continue; + } + if (!string.Equals( + detection.Pose.FrameId, + cameraFrameId, + StringComparison.Ordinal)) + { + throw new ServiceResultException( + StatusCodes.BadInvalidArgument, + $"Detection {ii} Pose.FrameId '{detection.Pose.FrameId}' does not match " + + $"the calibrated camera frame '{cameraFrameId}'."); + } + ReadOnlySpan local = detection.Pose.Position.Span; + if (local.Length < 3) + { + continue; + } + ArrayOf rotated = PoseMath.RotateVector(cameraOrientation, local); + ReadOnlySpan offset = rotated.Span; + double worldX = cameraPosition[0] + offset[0]; + double worldY = cameraPosition[1] + offset[1]; + double worldZ = cameraPosition[2] + offset[2]; + BinPickingPartSnapshot? part = FindPart(parts, detection.ClassLabel); + if (part == null || part.Location == BinPickingPartLocation.Held) + { + throw new ServiceResultException( + StatusCodes.BadInvalidArgument, + FormattableString.Invariant( + $"Detection {ii} class '{detection.ClassLabel}' is not available to pick.")); + } + double residual = Distance(part, worldX, worldY, worldZ); + if (residual > MaximumWorldResidualMetres) + { + throw new ServiceResultException( + StatusCodes.BadInvalidArgument, + string.Format( + CultureInfo.InvariantCulture, + "Detection {0} class '{1}' is {2:F3} m from the simulated part, " + + "above the {3:F3} m limit.", + ii, + detection.ClassLabel, + residual, + MaximumWorldResidualMetres)); + } + acceptedTargets.Add(new BinPickingTarget( + detection.ClassLabel, + worldX, + worldY, + worldZ, + created, + resultId, + cameraFrameId)); + } + for (int ii = 0; ii < acceptedTargets.Count; ii++) + { + BinPickingTarget target = acceptedTargets[ii]; + m_targets[target.ClassLabel] = target; + } + } + + public bool TryResolve(string classLabel, out BinPickingTarget target) + { + if (m_targets.TryGetValue(classLabel, out target) && + DateTime.UtcNow - target.TimestampUtc <= TargetLifetime) + { + return true; + } + _ = m_targets.TryRemove(classLabel, out _); + if (m_options.InferenceLocation != BinPickingInferenceLocation.OnServer) + { + target = default; + return false; + } + + IReadOnlyList parts = m_worldState.Snapshot(); + for (int ii = 0; ii < parts.Count; ii++) + { + BinPickingPartSnapshot part = parts[ii]; + if (string.Equals(part.Part.ClassLabel, classLabel, StringComparison.Ordinal)) + { + target = new BinPickingTarget( + classLabel, + part.WorldX, + part.WorldY, + part.WorldZ, + DateTime.UtcNow, + "simulation-world-state", + WorldFrameId); + return true; + } + } + target = default; + return false; + } + + private static BinPickingPartSnapshot? FindPart( + IReadOnlyList parts, + string classLabel) + { + for (int ii = 0; ii < parts.Count; ii++) + { + if (string.Equals( + parts[ii].Part.ClassLabel, + classLabel, + StringComparison.Ordinal)) + { + return parts[ii]; + } + } + return null; + } + + private static double Distance( + BinPickingPartSnapshot part, + double worldX, + double worldY, + double worldZ) + { + double x = part.WorldX - worldX; + double y = part.WorldY - worldY; + double z = part.WorldZ - worldZ; + return Math.Sqrt((x * x) + (y * y) + (z * z)); + } + + private const string WorldFrameId = "world"; + private const double MaximumWorldResidualMetres = 0.08; + private static readonly TimeSpan TargetLifetime = TimeSpan.FromSeconds(10); + private readonly BinPickingWorldState m_worldState; + private readonly BinPickingCellOptions m_options; + + private readonly ConcurrentDictionary m_targets = + new(StringComparer.Ordinal); + } +} diff --git a/samples/Robotics/BinPickingCell/BinPickingVisionCell.cs b/samples/Robotics/BinPickingCell/BinPickingVisionCell.cs new file mode 100644 index 0000000000..f8e68f7630 --- /dev/null +++ b/samples/Robotics/BinPickingCell/BinPickingVisionCell.cs @@ -0,0 +1,531 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Server; +using Opc.Ua.Vision; +using Opc.Ua.Vision.OpenUsd; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Builders; + +namespace Vision.BinPickingCell +{ + /// + /// Vision side of the bin-picking cell. Materialises the frame tree, + /// the eye-in-hand camera sensor twin, the intrinsic and hand-eye + /// calibrations, and the media endpoints from the OPC UA + /// Robotics-Vision Addendum's worked example. + /// + /// + /// + /// The concrete values (focal lengths, distortion coefficients, + /// hand-eye pose, residual errors) are the ones the addendum ships; + /// they identify this cell as the reference example rather than an + /// arbitrary rig. The sensor is registered as a + /// twin: it renders + /// from a USD stage via the OpenUSD offscreen capture provider, and + /// carries an IVisionSimulatedType interface pointing at the + /// stage and camera prim so a client can see the twin metadata. + /// + /// + /// The frame identifiers world, robot_base, + /// flange, gripper_tcp and camera_eih match the + /// addendum and the frame names published by + /// — a client can walk from the + /// vision-side calibration to the robot-side frame without any + /// translation table. + /// + /// + /// The vision-side flange transform is authored at the "scan + /// pose" the arm would hold to point the eye-in-hand camera at the + /// bin. In a live cell the flange frame is dynamic and reflects the + /// current joint state; for this static-sample demo it is pinned to + /// the scan pose so a consumer composing + /// camera → flange → robot_base → world lands on the parts' + /// authored world positions — which is exactly what the + /// reports for + /// each detection's Pose. + /// + /// + [System.Diagnostics.CodeAnalysis.SuppressMessage( + "Performance", "CA1812", + Justification = "Instantiated by the DI container via AddSingleton.")] + internal sealed class BinPickingVisionCell + { + public BinPickingVisionCell( + ILogger logger, + BinPickingMediaProvider mediaProvider, + BinPickingCellStage stage, + BinPickingGroundTruthInferenceProvider inferenceProvider, + BinPickingAgentInferenceProvider agentProvider, + BinPickingCellOptions options) + { + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + m_mediaProvider = mediaProvider ?? throw new ArgumentNullException(nameof(mediaProvider)); + m_stage = stage ?? throw new ArgumentNullException(nameof(stage)); + m_inferenceProvider = inferenceProvider + ?? throw new ArgumentNullException(nameof(inferenceProvider)); + m_agentProvider = agentProvider ?? throw new ArgumentNullException(nameof(agentProvider)); + m_options = options ?? throw new ArgumentNullException(nameof(options)); + } + + /// + /// Configures the Vision node manager for this cell. + /// + /// is null. + public async ValueTask ConfigureAsync(IVisionBuildContext context, CancellationToken cancellationToken) + { + if (context == null) + { + throw new ArgumentNullException(nameof(context)); + } + AddFrames(context); + AddSensor(context); + AddPipeline(context); + await AttachSimulatedTwinAsync(context, cancellationToken).ConfigureAwait(false); + await FinalizePipelineAsync(context, cancellationToken).ConfigureAwait(false); + m_logger.VisionCellReady( + m_frames.Count, + m_stage.CellStagePath, + m_mediaProvider.Backend, + m_options.InferenceLocation); + } + + private void AddFrames(IVisionBuildContext context) + { + IVisionNodeBuilder nodes = context.Nodes; + nodes.AddFrame("World", frame => frame + .WithFrameId(BinPickingRobotCell.WorldFrameId) + .WithRole(VisionFrameRoleEnum.World) + .WithTransform(Pose(BinPickingRobotCell.WorldFrameId, 0.0, 0.0, 0.0))); + m_frames.Add(BinPickingRobotCell.WorldFrameId); + + nodes.AddFrame("RobotBase", frame => frame + .WithFrameId(BinPickingRobotCell.RobotBaseFrameId) + .WithRole(VisionFrameRoleEnum.Base) + .WithParent(BinPickingRobotCell.WorldFrameId) + .WithTransform(Pose( + BinPickingRobotCell.WorldFrameId, + 0.0, + 0.0, + BinPickingRobotCell.RobotBaseHeightMetres))); + m_frames.Add(BinPickingRobotCell.RobotBaseFrameId); + + nodes.AddFrame("Flange", frame => frame + .WithFrameId(BinPickingRobotCell.FlangeFrameId) + .WithRole(VisionFrameRoleEnum.MechanicalInterface) + .WithParent(BinPickingRobotCell.RobotBaseFrameId) + .WithTransform(FlangeScanPose())); + m_frames.Add(BinPickingRobotCell.FlangeFrameId); + + nodes.AddFrame("GripperTcp", frame => frame + .WithFrameId(BinPickingRobotCell.ToolFrameId) + .WithRole(VisionFrameRoleEnum.Tool) + .WithParent(BinPickingRobotCell.FlangeFrameId) + .WithTransform(Pose( + BinPickingRobotCell.FlangeFrameId, + BinPickingPalletizerGeometry.FlangeToTcpMetres, + 0.0, + 0.0))); + m_frames.Add(BinPickingRobotCell.ToolFrameId); + + nodes.AddFrame("CameraEih", frame => frame + .WithFrameId(BinPickingRobotCell.CameraFrameId) + .WithRole(VisionFrameRoleEnum.Camera) + .WithParent(BinPickingRobotCell.FlangeFrameId) + .WithTransform(HandEyeTransform())); + m_frames.Add(BinPickingRobotCell.CameraFrameId); + } + + private void AddPipeline(IVisionBuildContext context) + { + NodeId deployment = new NodeId(DeploymentBrowseName, context.InstanceNamespaceIndex); + bool offServer = m_options.InferenceLocation == BinPickingInferenceLocation.EdgeOffServer; + context.Nodes.AddPipeline(PipelineBrowseName, pipe => + { + pipe.WithPipelineId(PipelineId) + .WithSensor(FindSensor(context, SensorTwinBrowseName)?.NodeId ?? NodeId.Null) + .WithDeployment(deployment); + if (offServer) + { + // Off-server perception: publish the OffServer facet, and register the same + // agent object as both provider (so RunInference explains the mode with + // BadNotSupported) and feedback sink (so SubmitDetections/Correction arrive + // at a single owner). The ground-truth provider is not wired — the two + // paths never run at the same time. + pipe.UseInferenceProvider(m_agentProvider, onServer: false) + .UseFeedbackSink(m_agentProvider); + } + else + { + // On-server ground truth: publish the OnServer facet and register the + // deterministic detector. No feedback sink — a client cannot submit + // detections when nothing on the Server side is designed to consume them. + pipe.UseInferenceProvider(m_inferenceProvider, onServer: true); + } + }); + } + + private async ValueTask FinalizePipelineAsync( + IVisionBuildContext context, CancellationToken cancellationToken) + { + InferencePipelineState pipeline = FindPipeline(context, PipelineBrowseName) + ?? throw new InvalidOperationException( + "Pipeline '" + PipelineBrowseName + "' was not registered on the Vision node manager."); + ImageSensorState sensor = FindSensor(context, SensorTwinBrowseName) + ?? throw new InvalidOperationException( + "Sensor '" + SensorTwinBrowseName + "' was not registered on the Vision node manager."); + // The Vision builder creates and registers the Results folder for any + // pipeline that has an inference provider, so the cell only looks it up. + FolderState results = pipeline.Results + ?? throw new InvalidOperationException( + "The Vision builder must create the pipeline's Results folder."); + await ValueTask.CompletedTask.ConfigureAwait(false); + VisionIntrinsicsDataType intrinsics = BuildIntrinsics(); + NodeId deployment = new NodeId(DeploymentBrowseName, context.InstanceNamespaceIndex); + var target = new BinPickingInferenceTarget( + context.Manager, + context.Context, + context.InstanceNamespaceIndex, + pipeline.NodeId, + sensor.NodeId, + deployment, + results, + BinPickingRobotCell.CameraFrameId, + PixelFormat, + intrinsics.Fx, + intrinsics.Fy, + intrinsics.Cx, + intrinsics.Cy, + intrinsics.Width, + intrinsics.Height, + CameraInWorldPose()); + if (m_options.InferenceLocation == BinPickingInferenceLocation.EdgeOffServer) + { + m_agentProvider.Attach(target); + } + else + { + m_inferenceProvider.Attach(target); + } + } + + private static InferencePipelineState? FindPipeline(IVisionBuildContext context, string browseName) + { + FolderState? pipelines = context.Root.Pipelines; + if (pipelines == null) + { + return null; + } + var children = new List(); + pipelines.GetChildren(context.Context, children); + var qualified = new QualifiedName(browseName, context.InstanceNamespaceIndex); + foreach (BaseInstanceState child in children) + { + if (child is InferencePipelineState pipeline && pipeline.BrowseName == qualified) + { + return pipeline; + } + } + return null; + } + + private void AddSensor(IVisionBuildContext context) + { + IVisionNodeBuilder nodes = context.Nodes; + VisionIntrinsicsDataType intrinsics = BuildIntrinsics(); + nodes.AddImageSensor(SensorTwinBrowseName, sensor => sensor + .WithSensorId("cam-eih-01") + .WithModality(VisionSensorModalityEnum.Area2D) + .WithRealityKind(VisionRealityKindEnum.Simulated) + .WithManufacturer("OPC Foundation") + .WithModel("Simulated Eye-in-Hand Camera") + .WithSerialNumber("SIM-EIH-2448-2048-0001") + .WithDeviceUri("opcua-openusd://binpicking-cell/cameras/camera_eih") + .WithFrameId(BinPickingRobotCell.CameraFrameId) + .WithResolution(SensorWidth, SensorHeight) + .WithPixelFormat(PixelFormat) + .WithIntrinsics(intrinsics) + .WithOptics(optics => optics + .WithFocalLength(0.01224) + .WithAperture(2.8) + .WithWorkingDistance(0.35) + .WithLensType("Fixed C-mount") + .WithMountType("C")) + .AddIntrinsicCalibration(IntrinsicCalibrationBrowseName, calibration => calibration + .WithCalibrationId("intr-cam-eih-01-612") + .WithMethod("Zhang") + .WithResidualError(0.21) + .WithIntrinsics(intrinsics)) + .AddExtrinsicCalibration(HandEyeCalibrationBrowseName, calibration => calibration + .WithCalibrationId("hand-eye-cam-eih-01") + .WithResidualError(0.0008) + .WithMount(VisionCalibrationMountEnum.EyeInHand) + .WithFrames(BinPickingRobotCell.CameraFrameId, BinPickingRobotCell.FlangeFrameId) + .WithTransform(HandEyeTransform())) + .AddStreamEndpoint(StreamEndpointBrowseName, endpoint => endpoint + .WithEndpointId("stream-live") + .WithEndpointUri("rtsp://simulated-eih.local:554/main") + .WithProtocol(VisionStreamProtocolEnum.Rtsp) + .WithCodec(VisionVideoCodecEnum.H264) + .WithResolution(SensorWidth, SensorHeight) + .WithFrameRate(15.0) + .WithBitrate(24_000_000u) + .WithDefaultProfileName("main")) + .AddClipEndpoint(ClipEndpointBrowseName, endpoint => endpoint + .WithEndpointId("clip-pick-frames") + .WithEndpointUri("opcua-inline://binpicking-cell/clips") + .WithClipFormat(VisionClipFormatEnum.Png) + .WithQuality(90u) + .WithResolution(SensorWidth, SensorHeight) + + // The PNG this cell renders is well under a megabyte, but allow + // headroom for a busier scene rather than have the Server refuse its + // own frames. Note this ceiling was never what refused GetClip - the + // frame was already under it - see MaxInlineClipBytes. + .WithInlineDelivery(enabled: true, maxInlineClipSize: MaxInlineClipBytes) + .WithDefaultProfileName("PickFrames")) + .UseMediaProvider(m_mediaProvider)); + } + + private async ValueTask AttachSimulatedTwinAsync( + IVisionBuildContext context, CancellationToken cancellationToken) + { + ImageSensorState? sensor = FindSensor(context, SensorTwinBrowseName); + if (sensor == null) + { + throw new InvalidOperationException( + "Sensor '" + SensorTwinBrowseName + "' was not registered on the Vision node manager."); + } + IVisionSimulatedState simulated = context.Context.CreateInstanceOfIVisionSimulatedType( + sensor, + new QualifiedName("Simulated", context.VisionNamespaceIndex)); + simulated.ReferenceTypeId = Opc.Ua.ReferenceTypeIds.HasInterface; + simulated.NodeId = context.Context.RequireNodeIdFactory().New(context.Context, simulated); + simulated.CreateOrReplaceSimulatorUri(context.Context, null).Value = + "opcua-openusd://binpicking-cell"; + simulated.CreateOrReplaceStageIdentifier(context.Context, null).Value = m_stage.CellStagePath; + simulated.CreateOrReplacePrimPath(context.Context, null).Value = CameraPrimPath; + simulated.CreateOrReplaceGroundTruthAvailable(context.Context, null).Value = true; + sensor.AddChild(simulated); + await context.Manager.AddPredefinedNodeAsync(simulated, cancellationToken).ConfigureAwait(false); + } + + private static ImageSensorState? FindSensor(IVisionBuildContext context, string browseName) + { + FolderState? sensors = context.Root.Sensors; + if (sensors == null) + { + return null; + } + var children = new List(); + sensors.GetChildren(context.Context, children); + var qualified = new QualifiedName(browseName, context.InstanceNamespaceIndex); + foreach (BaseInstanceState child in children) + { + if (child is ImageSensorState imageSensor && imageSensor.BrowseName == qualified) + { + return imageSensor; + } + } + return null; + } + + private static VisionIntrinsicsDataType BuildIntrinsics() + { + // Calibrated on the native 2448x2048 grid, then scaled to the binned grid the + // camera actually delivers. Binning is an exact integer decimation, so the + // focal lengths and principal point divide by the bin factor and the + // Brown-Conrady coefficients - which are expressed in normalised image + // coordinates - carry over unchanged. + const double bin = SensorBinning; + return new VisionIntrinsicsDataType + { + Fx = 2140.5 / bin, + Fy = 2139.8 / bin, + Cx = 1223.1 / bin, + Cy = 1021.7 / bin, + Skew = 0.0, + DistortionModel = VisionDistortionModelEnum.BrownConrady, + DistortionCoefficients = new[] + { + -0.1721, + 0.0934, + 0.0002, + -0.0001, + -0.0188 + }.ToArrayOf(), + Width = SensorWidth, + Height = SensorHeight + }; + } + + private static VisionPose3DDataType Pose(string frameId, double x, double y, double z) + { + return new VisionPose3DDataType + { + FrameId = frameId, + Position = new[] { x, y, z }.ToArrayOf(), + Orientation = s_identityOrientation.ToArrayOf(), + Covariance = ArrayOf.Empty + }; + } + + private static VisionPose3DDataType HandEyeTransform() + { + return new VisionPose3DDataType + { + FrameId = BinPickingRobotCell.FlangeFrameId, + Position = s_handEyePosition.ToArrayOf(), + Orientation = s_handEyeOrientation.ToArrayOf(), + Covariance = ArrayOf.Empty + }; + } + + private static VisionPose3DDataType FlangeScanPose() + { + return new VisionPose3DDataType + { + FrameId = BinPickingRobotCell.RobotBaseFrameId, + Position = s_flangeScanPosition.ToArrayOf(), + Orientation = s_flangeScanOrientation.ToArrayOf(), + Covariance = ArrayOf.Empty + }; + } + + private static VisionPose3DDataType CameraInWorldPose() + { + return new VisionPose3DDataType + { + FrameId = BinPickingRobotCell.WorldFrameId, + Position = s_cameraInWorldPosition.ToArrayOf(), + Orientation = s_cameraInWorldOrientation.ToArrayOf(), + Covariance = ArrayOf.Empty + }; + } + + private static readonly double[] s_identityOrientation = [0.0, 0.0, 0.0, 1.0]; + private static readonly double[] s_handEyePosition = [0.020, 0.0, 0.160]; + private static readonly double[] s_handEyeOrientation = [RootHalf, 0.0, RootHalf, 0.0]; + + private static readonly double[] s_cameraInWorldPosition = + [BinPickingPartsCatalog.BinCentreX, -0.160, 1.405]; + + /// + /// The palletizer flange is tool-down and rolled 90 degrees at the scan pose. + /// Composing the authored hand-eye transform places the camera 160 mm toward -Y + /// and 20 mm below the flange, looking straight down with a -90 degree image roll. + /// + private static readonly double[] s_cameraInWorldOrientation = + [RootHalf, -RootHalf, 0.0, 0.0]; + + /// + /// Analytic palletizer home: TCP (0.60, 0, 0.32), wrist/flange 185 mm above it. + /// + private static readonly double[] s_flangeScanPosition = [0.600, 0.0, 0.505]; + + private static readonly double[] s_flangeScanOrientation = + [0.5, 0.5, -0.5, 0.5]; + + /// + /// A 90-degree rotation, to full double precision. Writing it as 0.7071 leaves the + /// quaternion with a norm of 0.99999041, which is 9.6e-6 off unit - ten times the + /// 1e-6 tolerance the pose validator enforces - so composing a detection through + /// these frames fails with BadOutOfRange. + /// + private const double RootHalf = 0.70710678118654752; + + internal const string SensorTwinBrowseName = "BinPickingCameraTwin"; + + /// + /// + /// The simulated device is a 2448x2048 area-scan camera, but it is operated with + /// 4x4 binning, which is what an industrial camera does when a bin-picking cycle + /// needs frame rate more than it needs pixels. Every resolution the model reports - + /// sensor, stream endpoint, clip endpoint, intrinsics - is the binned one, because + /// that is what the device delivers. The native size survives only in the model and + /// serial number, where it identifies the hardware rather than the image. + /// + /// + /// These used to disagree: the sensor declared 2448x2048, the clip endpoint + /// 1280x1024, and the renderer produced 640x512. The detector projects through the + /// declared intrinsics, so its boxes landed in 2448x2048 space while an agent was + /// handed a 640x512 picture - the coordinates pointed off the image it could see. + /// 640x512 was not even the same aspect ratio as the camera it claimed to be. + /// + /// + internal const uint NativeSensorWidth = 2448u; + internal const uint NativeSensorHeight = 2048u; + internal const uint SensorBinning = 4u; + internal const uint SensorWidth = NativeSensorWidth / SensorBinning; + internal const uint SensorHeight = NativeSensorHeight / SensorBinning; + + /// + /// The clip endpoint serves 612x512 PNGs. Allow headroom for a busier scene + /// rather than have the Server refuse its own frames. + /// + internal const uint MaxInlineClipBytes = 32u * 1024u * 1024u; + internal const string IntrinsicCalibrationBrowseName = "Intrinsics612x512"; + internal const string HandEyeCalibrationBrowseName = "HandEye"; + internal const string StreamEndpointBrowseName = "LiveRtsp"; + internal const string ClipEndpointBrowseName = "PickFrames"; + internal const string PipelineBrowseName = "BinPickingPipeline"; + internal const string PipelineId = "pipe-onserver-groundtruth"; + internal const string DeploymentBrowseName = "OnServerDeployment"; + internal const string PixelFormat = "BayerRG8"; + + internal const string CameraPrimPath = + "/World/Robot/Palletizer/Base/J1/J2/J3/Leveling/J4/Flange/Camera"; + + private readonly ILogger m_logger; + private readonly BinPickingMediaProvider m_mediaProvider; + private readonly BinPickingCellStage m_stage; + private readonly BinPickingGroundTruthInferenceProvider m_inferenceProvider; + private readonly BinPickingAgentInferenceProvider m_agentProvider; + private readonly BinPickingCellOptions m_options; + private readonly List m_frames = []; + } + + internal static partial class BinPickingVisionCellLog + { + [LoggerMessage(EventId = BinPickingCellEventIds.Configurator + 10, + Level = LogLevel.Information, + Message = "Vision side of BinPickingCell ready ({FrameCount} frames, " + + "stage {StageIdentifier}, backend {Backend}, InferenceLocation={InferenceLocation}).")] + public static partial void VisionCellReady( + this ILogger logger, + int frameCount, string stageIdentifier, SceneCameraCaptureBackend backend, + BinPickingInferenceLocation inferenceLocation); + } +} diff --git a/samples/Robotics/BinPickingCell/EventIds.cs b/samples/Robotics/BinPickingCell/EventIds.cs new file mode 100644 index 0000000000..3fbfd7f89d --- /dev/null +++ b/samples/Robotics/BinPickingCell/EventIds.cs @@ -0,0 +1,48 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Vision.BinPickingCell +{ + /// + /// Centrally managed event-id offsets for source-generated log messages + /// in this assembly. + /// + internal static class BinPickingCellEventIds + { + public const int Configurator = 0; + public const int MediaProvider = 20; + public const int Stage = 40; + public const int Startup = 60; + public const int Inference = 80; + public const int Proof = 100; + public const int Agent = 140; + public const int OffServerProof = 180; + public const int OpenUsdRepresentation = 220; + } +} diff --git a/samples/Robotics/BinPickingCell/OpenUsdRepresentation.cs b/samples/Robotics/BinPickingCell/OpenUsdRepresentation.cs new file mode 100644 index 0000000000..863230aa6b --- /dev/null +++ b/samples/Robotics/BinPickingCell/OpenUsdRepresentation.cs @@ -0,0 +1,455 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.OpenUsd; +using Opc.Ua.OpenUsd.Server; +using ReferenceTypeIds = Opc.Ua.ReferenceTypeIds; + +namespace Vision.BinPickingCell +{ + /// + /// Publishes the OpenUSD representation for the bin-picking cell. + /// + internal sealed partial class BinPickingRobotCell + { + private async ValueTask MaterialiseOpenUsdFacilityAsync(CancellationToken cancellationToken) + { + try + { + ushort ns = Manager.Server.NamespaceUris.GetIndexOrAppend(Opc.Ua.OpenUsd.Namespaces.OpenUSD); + OpenUsdRootState root = SystemContext.CreateInstanceOfOpenUsdRootType( + null!, new QualifiedName("OpenUSD", ns)); + root.NodeId = new NodeId("OpenUSD", InstanceNamespaceIndex); + + FolderState stages = root.Stages ?? root.CreateOrReplaceStages(SystemContext, null); + _ = root.Representations ?? root.CreateOrReplaceRepresentations(SystemContext, null); + + m_cellStage = SystemContext.CreateInstanceOfOpenUsdStageType( + stages, new QualifiedName("BinPickingCellStage", ns)); + stages.AddChild(m_cellStage); + m_cellStage.CreateOrReplaceRootLayerIdentifier(SystemContext, null).Value = RootLayerIdentifier; + + List servedAssets = LoadServedAssets(); + byte[] rootLayerBytes = servedAssets.Find(a => a.Kind == OpenUsdAssetKindEnum.RootLayer)!.Bytes; + byte[] digest; +#pragma warning disable CA1850 // Prefer static HashData (net48/netstandard2.0 compatibility) + using (var sha = System.Security.Cryptography.SHA256.Create()) + { + digest = sha.ComputeHash(rootLayerBytes); + } +#pragma warning restore CA1850 + if (m_cellStage.RootLayerDigest != null) + { + m_cellStage.RootLayerDigest.Value = (ByteString)digest; + } + if (m_cellStage.RootLayerDigestAlgorithm != null) + { + m_cellStage.RootLayerDigestAlgorithm.Value = OpenUsdDigestAlgorithmEnum.Sha256; + } + + root.AddReference(ReferenceTypeIds.HasComponent, true, Opc.Ua.ObjectIds.Server); + UsdAssetDelivery.AttachStageAssets(SystemContext, m_cellStage, ns, servedAssets); + SystemContext.AssignInstanceChildNodeIds(root); + _ = await Manager.AddNodeAsync( + SystemContext, + NodeId.Null, + root, + cancellationToken).ConfigureAwait(false); + + m_openUsdRoot = root; + await LinkOpenUsdRootToServerAsync(cancellationToken).ConfigureAwait(false); + m_logger.MaterialisedOpenUsdFacility(root.NodeId, m_cellStage.NodeId); + } + catch (Exception ex) + { + m_cellStage = null; + m_openUsdRoot = null; + m_logger.OpenUsdFacilityFailed(ex); + } + } + + /// + /// Publishes one world position variable per part, so the parts have somewhere to be + /// read from and something for the OpenUSD live bindings to follow. + /// + /// + /// This is the cell's simulation ground truth, not a standard OPC UA concept: a part + /// lying in a bin is not modelled by Robot Intent or by Vision, which describe the + /// robot and what a sensor concluded rather than the scenery. It is published because + /// the scene has to be drivable from the address space to be watchable, and because a + /// client comparing what the detector claims against where the part actually is needs + /// both halves. + /// + private ValueTask MaterialisePartStateAsync(CancellationToken cancellationToken) + { + var folder = new FolderState(null) + { + NodeId = new NodeId("WorldState", InstanceNamespaceIndex), + SymbolicName = "WorldState", + BrowseName = new QualifiedName("WorldState", InstanceNamespaceIndex), + DisplayName = new LocalizedText("WorldState"), + Description = new LocalizedText( + "Simulation ground truth: where each part actually is, independent of what " + + "the vision pipeline reports."), + TypeDefinitionId = Opc.Ua.ObjectTypeIds.FolderType, + EventNotifier = EventNotifiers.None + }; + + // Inverse reference on this node, forward reference on the Server object below: + // two directions of the same edge, mirroring the OpenUSD root above. + folder.AddReference(ReferenceTypeIds.HasComponent, true, Opc.Ua.ObjectIds.Server); + + foreach (BinPickingPart part in BinPickingPartsCatalog.Parts) + { + var position = new BaseDataVariableState(folder) + { + NodeId = new NodeId("WorldState_" + part.ClassLabel, InstanceNamespaceIndex), + SymbolicName = part.ClassLabel, + BrowseName = new QualifiedName(part.ClassLabel, InstanceNamespaceIndex), + DisplayName = new LocalizedText(part.ClassLabel), + Description = new LocalizedText("Position of " + part.ClassLabel + " in the world frame, in metres."), + TypeDefinitionId = VariableTypeIds.BaseDataVariableType, + ReferenceTypeId = ReferenceTypeIds.HasComponent, + + // A structured coordinate, not a double[3]: the OpenUSD companion + // specification defines a translation source as a structured 3D value + // and the connector's translation profile fails closed on anything + // else, so an array left every part unresolved and the viewport never + // moved a part however faithfully the server tracked it. + DataType = Opc.Ua.DataTypeIds.ThreeDCartesianCoordinates, + ValueRank = ValueRanks.Scalar, + AccessLevel = AccessLevels.CurrentRead, + UserAccessLevel = AccessLevels.CurrentRead, + Value = new Variant(new ExtensionObject(new ThreeDCartesianCoordinates + { + X = part.InitialWorldPosition[0], + Y = part.InitialWorldPosition[1], + Z = part.InitialWorldPosition[2] + })) + }; + folder.AddChild(position); + m_partPositionNodes[part.ClassLabel] = position; + } + + // Give the folder and its children their NodeIds before anything hangs a + // representation off them: AttachRepresentation derives the representation's + // NodeId from its owner, so unassigned owners produce five identical ids. + SystemContext.AssignInstanceChildNodeIds(folder); + m_partStateFolder = folder; + return ValueTask.CompletedTask; + } + + /// + /// Registers the part-state subtree once its representations are attached. Adding it + /// earlier would register the position variables, and then register them again as the + /// parents of representations grafted on afterwards. + /// + private async ValueTask AddPartStateAsync(CancellationToken cancellationToken) + { + if (m_partStateFolder == null) + { + return; + } + _ = await Manager.AddNodeAsync( + SystemContext, + NodeId.Null, + m_partStateFolder, + cancellationToken).ConfigureAwait(false); + + // The Server object belongs to the core node manager, so the forward reference is + // added afterwards rather than by passing it as the parent: this manager cannot + // resolve i=2253 while adding its own node. + await Manager.Server.NodeManager.AddReferencesAsync( + Opc.Ua.ObjectIds.Server, + [new NodeStateReference(ReferenceTypeIds.HasComponent, false, m_partStateFolder.NodeId)], + cancellationToken).ConfigureAwait(false); + } + + private async ValueTask MaterialiseRepresentationsAsync(CancellationToken cancellationToken) + { + if (m_cellStage == null || m_openUsdRoot == null) + { + return; + } + + ushort usdNs = Manager.Server.NamespaceUris.GetIndexOrAppend(Opc.Ua.OpenUsd.Namespaces.OpenUSD); + List representations = []; + List partRepresentations = []; + + OpenUsdRepresentationState controllerRep = AttachRepresentation(Controller.State, "/World", usdNs); + representations.Add(controllerRep); + + CreateBinding( + controllerRep, + usdNs, + "GripperLeftSlide", + GuidFor("gripper:left"), + m_gripperLeftSlideValue?.NodeId ?? NodeId.Null, + "/World/Robot/Palletizer/Base/J1/J2/J3/Leveling/J4/Flange/Gripper/FingerLeftSlide", + "xformOp:translate", + "double3", + OpenUsdRenderTargetKindEnum.Translation, + 1.0); + CreateBinding( + controllerRep, + usdNs, + "PalletizerLeveling", + GuidFor("palletizer:leveling"), + m_levelingValue?.NodeId ?? NodeId.Null, + "/World/Robot/Palletizer/Base/J1/J2/J3/Leveling", + "xformOp:rotateY", + "double", + OpenUsdRenderTargetKindEnum.Rotation, + 1.0); + CreateBinding( + controllerRep, + usdNs, + "GripperRightSlide", + GuidFor("gripper:right"), + m_gripperRightSlideValue?.NodeId ?? NodeId.Null, + "/World/Robot/Palletizer/Base/J1/J2/J3/Leveling/J4/Flange/Gripper/FingerRightSlide", + "xformOp:translate", + "double3", + OpenUsdRenderTargetKindEnum.Translation, + 1.0); + + int axisIndex = 0; + foreach (global::Opc.Ua.RobotIntent.AxisState axis in Axes) + { + OpenUsdRepresentationState axisRep = AttachRepresentation(axis, s_axisUsd[axisIndex].PrimPath, usdNs); + CreateBinding( + axisRep, usdNs, $"{s_axisUsd[axisIndex].Name}Rotation", GuidFor(s_axisUsd[axisIndex].Name), + axis.Position?.NodeId ?? NodeId.Null, s_axisUsd[axisIndex].PrimPath, s_axisUsd[axisIndex].RotateOp, + "double", OpenUsdRenderTargetKindEnum.Rotation, 1.0); + representations.Add(axisRep); + axisIndex++; + } + + foreach (BinPickingPart part in BinPickingPartsCatalog.Parts) + { + if (!m_partPositionNodes.TryGetValue(part.ClassLabel, out BaseDataVariableState? position)) + { + continue; + } + string primPath = PartPrimPath(part.ClassLabel); + OpenUsdRepresentationState partRep = AttachRepresentation(position, primPath, usdNs); + CreateBinding( + partRep, usdNs, $"{part.ClassLabel}Translation", GuidFor("part:" + part.ClassLabel), + position.NodeId, primPath, "xformOp:translate", + "double3", OpenUsdRenderTargetKindEnum.Translation, 1.0); + representations.Add(partRep); + partRepresentations.Add(partRep); + } + + foreach (OpenUsdRepresentationState representation in representations) + { + OrganiseRepresentation(representation); + if (partRepresentations.Contains(representation)) + { + // Rides into the address space inside the part-state folder below; + // registering it here as well would add its parent a second time. + continue; + } + _ = await Manager.AddNodeAsync( + SystemContext, + representation.Parent!.NodeId, + representation, + cancellationToken).ConfigureAwait(false); + } + await AddPartStateAsync(cancellationToken).ConfigureAwait(false); + m_logger.MaterialisedRepresentations(representations.Count); + } + + private static List LoadServedAssets() + { + return + [ + new("stage.usda", OpenUsdAssetKindEnum.RootLayer, ReadEmbeddedAsset("Cell.usda")), + new( + "palletizer-arm.usda", + OpenUsdAssetKindEnum.Reference, + ReadEmbeddedAsset("palletizer-arm.usda")), + new( + "palletizer-gripper.usda", + OpenUsdAssetKindEnum.Reference, + ReadEmbeddedAsset("palletizer-gripper.usda")), + new("gripper.usda", OpenUsdAssetKindEnum.Reference, ReadEmbeddedAsset("gripper.usda")) + ]; + } + + private static byte[] ReadEmbeddedAsset(string resourceName) + { + using Stream? stream = typeof(BinPickingRobotCell).Assembly.GetManifestResourceStream(resourceName); + if (stream == null) + { + return []; + } + using var memory = new MemoryStream(); + stream.CopyTo(memory); + return memory.ToArray(); + } + + private async ValueTask LinkOpenUsdRootToServerAsync(CancellationToken cancellationToken) + { + if (m_openUsdRoot == null) + { + return; + } + IReference[] references = + [ + new NodeStateReference(ReferenceTypeIds.HasComponent, false, m_openUsdRoot.NodeId) + ]; + await Manager.Server.NodeManager.AddReferencesAsync( + Opc.Ua.ObjectIds.Server, + references, + cancellationToken).ConfigureAwait(false); + } + + private OpenUsdRepresentationState AttachRepresentation(NodeState owner, string primPath, ushort ns) + { + OpenUsdRepresentationState rep = SystemContext.CreateInstanceOfOpenUsdRepresentationType( + owner, new QualifiedName("OpenUsdRepresentation", ns)); + rep.ReferenceTypeId = ReferenceTypeIds.HasComponent; + owner.AddChild(rep); + AssignInstanceSubtree(rep, owner); + rep.CreateOrReplaceStage(SystemContext, null).Value = m_cellStage!.NodeId; + rep.CreateOrReplacePrimPath(SystemContext, null).Value = primPath; + return rep; + } + + private void OrganiseRepresentation(OpenUsdRepresentationState rep) + { + FolderState? registry = m_openUsdRoot?.Representations; + if (registry == null) + { + return; + } + registry.AddReference(ReferenceTypeIds.Organizes, false, rep.NodeId); + rep.AddReference(ReferenceTypeIds.Organizes, true, registry.NodeId); + } + + private OpenUsdLiveBindingState CreateBinding( + OpenUsdRepresentationState rep, + ushort ns, + string name, + Guid bindingDefinitionId, + NodeId sourceNodeId, + string targetPrimPath, + string targetPropertyName, + string targetUsdTypeName, + OpenUsdRenderTargetKindEnum? kind, + double scale, + string? sourceSemanticId = null) + { + OpenUsdLiveBindingState binding = rep.AddLiveBinding( + SystemContext, + ns, + m_cellStage!.NodeId, + name, + bindingDefinitionId, + sourceNodeId, + targetPrimPath, + targetPropertyName, + targetUsdTypeName, + kind, + scale, + sourceSemanticId: sourceSemanticId); + AssignInstanceSubtree(binding, rep); + return binding; + } + + private void AssignInstanceSubtree(BaseInstanceState node, NodeState referenceRoot) + { + NodeId previousNodeId = SystemContext.AssignInstanceNodeId(node); + SystemContext.AssignInstanceChildNodeIds(node, previousNodeId, referenceRoot); + } + + private static Guid GuidFor(string token) + { + byte[] hash; +#pragma warning disable CA1850 // Prefer static HashData (net48/netstandard2.0 compatibility) + using (var sha = System.Security.Cryptography.SHA256.Create()) + { + hash = sha.ComputeHash(System.Text.Encoding.UTF8.GetBytes("bin-picking-cell:" + token)); + } +#pragma warning restore CA1850 + byte[] guidBytes = new byte[16]; + Array.Copy(hash, guidBytes, guidBytes.Length); + return new Guid(guidBytes); + } + + private const string RootLayerIdentifier = "stage.usda"; + + /// + /// The prim a part's world position drives. Mirrors the Parts scope in Cell.usda. + /// + private static string PartPrimPath(string classLabel) + { + return "/World/Parts/" + classLabel; + } + + private static readonly (string Name, string PrimPath, string RotateOp)[] s_axisUsd = + [ + ("J1", "/World/Robot/Palletizer/Base/J1", "xformOp:rotateZ"), + ("J2", "/World/Robot/Palletizer/Base/J1/J2", "xformOp:rotateY"), + ("J3", "/World/Robot/Palletizer/Base/J1/J2/J3", "xformOp:rotateY"), + ("J4", "/World/Robot/Palletizer/Base/J1/J2/J3/Leveling/J4", "xformOp:rotateX") + ]; + + private OpenUsdRootState? m_openUsdRoot; + private OpenUsdStageState? m_cellStage; + private FolderState? m_partStateFolder; + private readonly Dictionary m_partPositionNodes = []; + } + + internal static partial class OpenUsdRepresentationLog + { + [LoggerMessage(EventId = BinPickingCellEventIds.OpenUsdRepresentation + 1, + Level = LogLevel.Information, + Message = "Materialised OpenUSD facility (root {RootId}, stage {StageId}).")] + public static partial void MaterialisedOpenUsdFacility(this ILogger logger, NodeId rootId, NodeId stageId); + + [LoggerMessage(EventId = BinPickingCellEventIds.OpenUsdRepresentation + 2, + Level = LogLevel.Error, + Message = "Failed to materialise the OpenUSD facility.")] + public static partial void OpenUsdFacilityFailed(this ILogger logger, Exception exception); + + [LoggerMessage(EventId = BinPickingCellEventIds.OpenUsdRepresentation + 3, + Level = LogLevel.Information, + Message = "Materialised {RepresentationCount} OpenUSD representations.")] + public static partial void MaterialisedRepresentations(this ILogger logger, int representationCount); + } +} diff --git a/samples/Robotics/BinPickingCell/PngDecoder.cs b/samples/Robotics/BinPickingCell/PngDecoder.cs new file mode 100644 index 0000000000..8b245ece50 --- /dev/null +++ b/samples/Robotics/BinPickingCell/PngDecoder.cs @@ -0,0 +1,192 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.IO.Compression; + +namespace Vision.BinPickingCell +{ + /// + /// Minimal PNG decoder for 8-bit RGBA images. Handles the output of + /// the in-repo PngEncoder used by the OpenUSD capture provider + /// (single IHDR + single or few IDATs + IEND, colour-type 6, bit-depth + /// 8, filter type 0 or 1 per row) and any similarly-conformant PNG. + /// Adds no NuGet dependency; the DEFLATE step goes through the BCL's + /// . + /// + /// + /// The decoder is deliberately restricted to what the sample diagnostic + /// needs. It rejects interlaced streams, palette images, and 16-bit + /// depths with ; the OpenUSD + /// provider only ever emits the RGBA8 non-interlaced flavour. + /// + internal static class PngDecoder + { + public static (byte[] Rgba, int Width, int Height) Decode(byte[] png) + { + if (png == null) + { + throw new ArgumentNullException(nameof(png)); + } + if (png.Length < 8 || + png[0] != 0x89 || + png[1] != (byte)'P' || + png[2] != (byte)'N' || + png[3] != (byte)'G' || + png[4] != 0x0D || + png[5] != 0x0A || + png[6] != 0x1A || + png[7] != 0x0A) + { + throw new InvalidDataException("Not a PNG stream."); + } + int width = 0; + int height = 0; + byte bitDepth = 0; + byte colourType = 0; + byte interlace = 0; + using var idat = new MemoryStream(); + int offset = 8; + while (offset + 8 <= png.Length) + { + int length = ReadUInt32BE(png, offset); + string chunkType = System.Text.Encoding.ASCII.GetString(png, offset + 4, 4); + int dataStart = offset + 8; + if (chunkType == "IHDR") + { + width = ReadUInt32BE(png, dataStart); + height = ReadUInt32BE(png, dataStart + 4); + bitDepth = png[dataStart + 8]; + colourType = png[dataStart + 9]; + interlace = png[dataStart + 12]; + } + else if (chunkType == "IDAT") + { + idat.Write(png, dataStart, length); + } + else if (chunkType == "IEND") + { + break; + } + offset = dataStart + length + 4; + } + if (bitDepth != 8 || colourType != 6) + { + throw new NotSupportedException( + $"PNG must be 8-bit RGBA (colour type 6, bit depth 8); got type={colourType}, depth={bitDepth}."); + } + if (interlace != 0) + { + throw new NotSupportedException("Interlaced PNGs are not supported by the sample decoder."); + } + if (width <= 0 || height <= 0) + { + throw new InvalidDataException("PNG dimensions are invalid."); + } + byte[] zlibBytes = idat.ToArray(); + if (zlibBytes.Length < 2) + { + throw new InvalidDataException("IDAT chunk is too short."); + } + byte[] rawFiltered = InflateZlib(zlibBytes); + int rowBytes = width * 4; + int expected = height * (rowBytes + 1); + if (rawFiltered.Length != expected) + { + throw new InvalidDataException( + $"Decoded filtered size {rawFiltered.Length} does not match {expected}."); + } + byte[] rgba = new byte[height * rowBytes]; + Unfilter(rawFiltered, rgba, width, height); + return (rgba, width, height); + } + + private static byte[] InflateZlib(byte[] zlib) + { + using var input = new MemoryStream(zlib, index: 2, count: zlib.Length - 6, writable: false); + using var deflate = new DeflateStream(input, CompressionMode.Decompress, leaveOpen: false); + using var output = new MemoryStream(); + deflate.CopyTo(output); + return output.ToArray(); + } + + private static void Unfilter(byte[] filtered, byte[] rgba, int width, int height) + { + int rowBytes = width * 4; + for (int y = 0; y < height; y++) + { + int srcRow = y * (rowBytes + 1); + int dstRow = y * rowBytes; + byte type = filtered[srcRow]; + for (int x = 0; x < rowBytes; x++) + { + byte value = filtered[srcRow + 1 + x]; + byte left = x >= 4 ? rgba[dstRow + x - 4] : (byte)0; + byte up = y > 0 ? rgba[((y - 1) * rowBytes) + x] : (byte)0; + byte upLeft = y > 0 && x >= 4 ? rgba[((y - 1) * rowBytes) + x - 4] : (byte)0; + rgba[dstRow + x] = type switch + { + 0 => value, + 1 => (byte)(value + left), + 2 => (byte)(value + up), + 3 => (byte)(value + ((left + up) / 2)), + 4 => (byte)(value + Paeth(left, up, upLeft)), + _ => throw new NotSupportedException($"Unknown PNG row filter {type}.") + }; + } + } + } + + private static byte Paeth(byte left, byte up, byte upLeft) + { + int p = left + up - upLeft; + int pa = Math.Abs(p - left); + int pb = Math.Abs(p - up); + int pc = Math.Abs(p - upLeft); + if (pa <= pb && pa <= pc) + { + return left; + } + if (pb <= pc) + { + return up; + } + return upLeft; + } + + private static int ReadUInt32BE(byte[] source, int offset) + { + return (source[offset] << 24) | + (source[offset + 1] << 16) | + (source[offset + 2] << 8) | + source[offset + 3]; + } + } +} diff --git a/samples/Robotics/BinPickingCell/Program.cs b/samples/Robotics/BinPickingCell/Program.cs new file mode 100644 index 0000000000..c7e573c538 --- /dev/null +++ b/samples/Robotics/BinPickingCell/Program.cs @@ -0,0 +1,188 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Robotics.Server; +using Opc.Ua.Server; +using Opc.Ua.Vision.OpenUsd; +using Robotics.IntentEnabledRobot.Kinematics; +using Robotics.IntentEnabledRobot.Simulation; +using Vision.BinPickingCell; + +BinPickingCellStage stage = new(); +string stagePath = stage.Extract(); + +HostApplicationBuilder builder = Host.CreateApplicationBuilder(args); + +builder.Logging.ClearProviders(); +builder.Logging.AddConsole(); + +int port = int.TryParse(builder.Configuration["port"], out int configuredPort) + ? configuredPort + : 62855; +string host = builder.Configuration["host"] is { Length: > 0 } configuredHost + ? configuredHost + : "localhost"; +bool captureOnStartup = !string.Equals(builder.Configuration["captureOnStartup"], "false", + StringComparison.OrdinalIgnoreCase); +string? artifactDirectory = builder.Configuration["artifactDirectory"]; +BinPickingCellOptions cellOptions = BuildCellOptions(builder.Configuration); +bool offServer = cellOptions.InferenceLocation == BinPickingInferenceLocation.EdgeOffServer; + +var sensorSpec = new BinPickingSensorSpec( + StageIdentifier: stagePath, + CameraPrimPath: BinPickingVisionCell.CameraPrimPath, + PixelFormat: BinPickingVisionCell.PixelFormat, + CaptureWidth: (int)BinPickingVisionCell.SensorWidth, + CaptureHeight: (int)BinPickingVisionCell.SensorHeight); + +builder.Services.AddSingleton(stage); +builder.Services.AddSingleton(sensorSpec); +builder.Services.AddSingleton(cellOptions); + +// The arm is bolted to the bench, so the bench is z = 0 in its own base frame. Telling +// the solver that stops it handing back configurations that reach through the work +// surface - several inverse-kinematic solutions for a target near the bench do exactly +// that, and taking the nearest one regardless renders an arm passing through its table. +// +// The height alone is not enough, though: it is a plane sampled at the joint origins, so a +// link can span it, and it says nothing about the bin or the fixture. +// BinPickingCellGeometry.CreateCollisionModel() describes the cell's furniture as solids +// and SimulatedArmKinematics.Collisions enforces it along every link, so a configuration +// that puts part of the arm inside the bench or through a bin wall is refused rather than +// rendered. The palletizer kinematics tests pin the remaining work envelope at every cell +// work position. +builder.Services.AddSingleton(_ => new BinPickingPalletizerKinematics +{ + MinimumLinkHeight = + BinPickingCellGeometry.BenchTopMetres - BinPickingCellGeometry.RobotBaseHeightMetres, + Collisions = BinPickingCellGeometry.CreateCollisionModel() +}); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddOpenUsdSceneCameraCaptureProvider(); +builder.Services.AddHostedService(services => + new BinPickingCaptureProof( + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService>(), + enabled: captureOnStartup, + artifactDirectory: artifactDirectory)); +if (offServer) +{ + builder.Services.AddHostedService(services => + new BinPickingOffServerProof( + services.GetRequiredService(), + services.GetRequiredService>())); +} +else +{ + builder.Services.AddHostedService(services => + new BinPickingInferenceProof( + services.GetRequiredService(), + services.GetRequiredService(), + services.GetRequiredService>())); +} + +builder.Services + .AddOpcUa() + .AddServer(options => + { + options.ApplicationName = "BinPickingCell"; + options.ApplicationUri = "urn:localhost:OPCFoundation:BinPickingCell"; + options.ProductUri = "uri:opcfoundation.org:BinPickingCell"; + options.AutoAcceptUntrustedCertificates = true; + + // A rendered frame is a few hundred kilobytes and the response that carries it + // larger still, so leave the transport room for a busier scene rather than have + // GetClip refuse the cell's own camera output. Note the failure this originally + // chased was not a quota at all - the provider was embedding the encoded image in + // a String field - so none of these were ever the fix; see BinPickingMediaProvider. + options.MaxByteStringLength = 32 * 1024 * 1024; + options.MaxArrayLength = 32 * 1024 * 1024; + options.MaxMessageSize = 64 * 1024 * 1024; + options.EndpointUrls.Add($"opc.tcp://{host}:{port}/BinPickingCell"); + }) + .ConfigureRoles(options => options.Roles.Add(new RoleDefinitionOptions + { + Name = BrowseNames.WellKnownRole_Operator, + Identities = + { + new RoleIdentityMappingOptions + { + CriteriaType = IdentityCriteriaType.Anonymous + } + } + })) + .AddRobotIntent() + .AddRobotIntentExecutor() + .ConfigureRobotIntent(async (context, cancellationToken) => + await context.GetRequiredService() + .ConfigureAsync(context, cancellationToken).ConfigureAwait(false)) + .AddVision(options => options.InstanceNamespaceUri = "urn:opcfoundation:BinPickingCell:vision:instances") + .AddVisionMediaProvider( + BinPickingVisionCell.SensorTwinBrowseName) + .ConfigureVision(async (context, cancellationToken) => + { + BinPickingVisionCell cell = context.GetRequiredService(); + await cell.ConfigureAsync(context, cancellationToken).ConfigureAwait(false); + }); + +// AddRobotIntentExecutor registers the concrete executor so its IIntentExecutor and any +// observers share one instance. Replace that default-constructed UR executor after the +// fluent registration with the palletizer-backed instance for this cell only. +builder.Services.Replace(ServiceDescriptor.Singleton( + services => new SimulatedArmExecutor( + services.GetRequiredService()))); + +using IHost app = builder.Build(); +await app.RunAsync().ConfigureAwait(false); + +static BinPickingCellOptions BuildCellOptions(Microsoft.Extensions.Configuration.IConfiguration configuration) +{ + string? raw = configuration["inferenceLocation"]; + // A parse failure just means "use the OnServer default"; the out value is set for us + // and the caller does not need a separate error path for an unknown key. + _ = BinPickingCellOptions.TryParseLocation(raw, out BinPickingInferenceLocation location); + return new BinPickingCellOptions + { + InferenceLocation = location + }; +} diff --git a/samples/Robotics/BinPickingCell/README.md b/samples/Robotics/BinPickingCell/README.md new file mode 100644 index 0000000000..c446aeecf2 --- /dev/null +++ b/samples/Robotics/BinPickingCell/README.md @@ -0,0 +1,374 @@ + + +# Bin Picking Cell + +Reference server for the vision-guided bin-picking sample. Hosts a +branch-stable four-axis palletizer as a Robot Intent controller, a +Vision companion with an eye-in-hand camera parented to the flange, an +in-address-space OpenUSD scene, and the frame tree the Robotics-Vision +Addendum's example uses: + +```text +world → robot_base → flange → gripper_tcp + ↳ camera_eih +``` + +```mermaid +flowchart TD + World["world
/World"] --> Base["robot_base
/World/Robot/Palletizer/Base"] + Base --> Flange["flange
/World/Robot/Palletizer/.../Flange"] + Flange --> Tcp["gripper_tcp
/World/Robot/Palletizer/.../Flange/Gripper/Tcp"] + Flange --> Camera["camera_eih
/World/Robot/Palletizer/.../Flange/Camera"] +``` + +Five parts (`RedCube`, `GreenCylinder`, `BlueSphere`, `YellowSlab`, +`OrangeBrick`) sit in a bin on the workbench; a fixture is placed next +to it. The sample's paired client at +[`samples/Robotics/BinPickingClient`](../BinPickingClient) runs the +perception-to-action loop against this server, either as a scripted +demo or under the control of an MCP-connected language model. + +The cell is the example that motivates the [Vision developer +guide](../../../docs/Vision.md) — every §5.12 convention (quaternion +`(x, y, z, w)`, metres, corner-datum principal point, empty-covariance +sentinel), every §6.4 media state and every facet the guide lists +appears in the cell's address space. + +## Running + +Prerequisites: .NET 10 SDK. + +```powershell +dotnet run --project samples\Robotics\BinPickingCell\BinPickingCell.csproj -- --insecure +``` + +The server listens on `opc.tcp://localhost:62855/BinPickingCell` by +default. The endpoint URL and port are configurable through +`--host ` and `--port `; the anonymous operator role is +mapped in code so the demo client can connect without user credentials. + +`--insecure` is a demo convenience: it accepts any client certificate +and does not enforce trust. Do not use it in production. + +## Inference-location option + +The cell selects one perception path once at startup and pins it for +the lifetime of the process. The pipeline's advertised inference- +location facet is derived from this and cannot change afterwards, so +the cell always tells the truth about which path is in force. + +- `--inferenceLocation OnServer` (default) — the deterministic + `BinPickingGroundTruthInferenceProvider` computes `DetectionResultType` + results locally. The pipeline advertises `VIS-Inference-OnServer`, + needs no GPU, no model and no network, and is what CI runs. +- `--inferenceLocation EdgeOffServer` — the pipeline exposes a + `VisionFeedbackType` bound to `BinPickingAgentInferenceProvider`; an + agent connected over MCP looks at the frame and calls + `SubmitDetections`. The pipeline advertises `VIS-Inference-OffServer` + and publishes results the Server itself did not compute. `RunInference` + and `StartContinuous` are refused with `Bad_NotSupported` — a client + requesting an on-Server compute path in this mode gets an actionable + refusal, not a silent stub. + +The two paths are exclusive by construction: a pipeline binds either an +inference provider or a feedback sink, never both at once, because +mixing them would let a computed and a submitted result publish on the +same pipeline out of any known order. + +## Startup options + +| Option | Meaning | +|---|---| +| `--host ` | Endpoint host name. Default `localhost`. | +| `--port ` | Endpoint port. Default `62855`. | +| `--inferenceLocation OnServer\|EdgeOffServer` | Selects the perception path. `OnServer` is the default. | +| `--captureOnStartup true\|false` | Whether the capture-proof hosted service captures a still on startup and writes it to disk (see below). Default `true`. | +| `--artifactDirectory ` | Where the capture-proof hosted service writes its still. Defaults to a temp path chosen by the host. | +| `--insecure` | Demo-only, per the note above. | + +The parser accepts `OnServer`, `EdgeOffServer`, `OffServer`, +`on-server`, `off-server` and their case-insensitive variants for +`--inferenceLocation`. Unknown values silently fall back to `OnServer` +rather than failing to start. + +## What the cell publishes + +- Under `Server/RobotIntent`, one Robot Intent controller + (`BinPickingController`) with four axes (`J1` base yaw, `J2` shoulder, + `J3` elbow and `J4` tool roll), lookup tables for `Bin`, `Fixture`, + one home location per part and `ParallelGripper`. The controller + ships `Pick` and `Place` intents targeting the sample parts. +- Under `Server/Vision`: + - The frame tree above. Every frame carries a unit `(x, y, z, w)` + quaternion and a metres-based position; `camera_eih` is authored + with the `EyeInHand` hand-eye pose the addendum specifies. + - `BinPickingCameraTwin`, an `ImageSensorType` with + `RealityKind = Simulated`, `Modality = Area2D`, an + `IntrinsicCalibrationType` (Zhang method, residual 0.21 pixels), a + `HandEye` `ExtrinsicCalibrationType` (`EyeInHand` mount, residual + 0.0008), an `IVisionSimulatedType` interface pointing at the USD + stage and `Camera` prim, a live RTSP `StreamEndpointType` and a + `PickFrames` `ClipEndpointType` with inline delivery enabled up to + 32 MiB. + - `BinPickingPipeline` (`InferencePipelineType`) bound to the sensor + twin, with the deployment reference `OnServerDeployment` and either + the ground-truth inference provider or the agent inference provider + depending on `--inferenceLocation`. +- Under `Server/OpenUSD`, the composed `Cell.usda` stage the sensor + renders from. +- Under `Server/WorldState`, one position variable per part, in the world + frame. This is the cell's **simulation ground truth**, not a standard OPC + UA concept: a part lying in a bin is not something Robot Intent or Vision + models, and the scene has to be drivable from the address space to be + watchable. The OpenUSD live bindings follow these variables, so a picked + part moves in the viewport, and a client comparing what the detector + claims against where the part actually is has both halves. + +## The USD stage + +The scene is authored in `Assets/Cell.usda` and is extracted at +startup by `BinPickingCellStage.Extract()` into a working directory the +`BinPickingMediaProvider` and `OpenUsdSceneCameraCaptureProvider` share. +It references the cell-specific `palletizer-arm.usda` and +`palletizer-gripper.usda` assets, plus the shared gripper geometry, adds +the bin, fixture and parts, and parents the eye-in-hand `UsdGeomCamera` +to the flange so the view moves with the arm. + +### Scene lighting is load-bearing + +The stage is lit by a single `DomeLight` at intensity 1000. +**Do not reintroduce a `DistantLight`** — at any intensity that shows +geometry, a `DistantLight` blows every surface to pure white regardless +of material, and the point of this cell is that a vision-language model +can pick "the red cube" by looking at the frame. Under a `DomeLight` +the five parts measure `red (220, 37, 37)`, `green (37, 208, 49)`, +`blue (37, 73, 233)` — distinct enough for an LLM to reason about. +The `Cell.usda` header records this contract explicitly so a future +scene-authoring change does not silently break the perception path. + +### Frame tree contract + +- `/World` is the `world` frame origin. +- `/World/Robot/Palletizer/Base` is the `robot_base` frame. +- `/World/Robot/Palletizer/Base/J1/J2/J3/Leveling/J4/Flange` is the + `flange` (mechanical interface). +- `/World/Robot/Palletizer/.../Flange/Camera` is the `camera_eih` + `UsdGeomCamera` the Vision sensor renders from. +- `/World/Robot/Palletizer/.../Flange/Gripper/Tcp` is `gripper_tcp`. + +The vision-side `flange` frame is authored at the scan pose the arm +would hold to point the eye-in-hand camera at the bin. In a live cell +the flange frame is dynamic and reflects the current joint state; for +this static-sample demo it is pinned to the scan pose so a consumer +composing `camera_eih → flange → robot_base → world` lands on the parts' +authored world positions — which is exactly what the ground-truth +detector reports for each detection's `Pose`. This is why the vision- +side frame ids match the robot-side frame ids exactly: the client can +compose a detection's pose from the camera into the world frame using +`VisionFrameGraph` and get a value it can hand straight to the robot- +side `Pick` intent. + +The scan pose is one analytic palletizer configuration: TCP +`(0.60, 0, 0.32)` and flange `(0.60, 0, 0.505)` in `robot_base`. +The camera is 20 mm along flange `+X` and 160 mm along flange `+Z`, +which composes to world position `(0.60, -0.16, 1.405)`. The extra +180-degree camera convention transform accounts for USD cameras +looking down local `-Z` while Vision projection uses camera-forward +`+Z`. + +The visual and numeric chain use the same geometry: + +- shoulder height 0.280 m; +- upper arm and forearm 0.480 m each; +- flange-to-jaw-centre TCP 0.185 m along flange `+X`; +- mechanical leveling `90° - J2 - J3`, so the tool remains vertical; +- `J4` rolls around the down-facing tool axis, keeping jaw aperture + visible without introducing offset-wrist IK branches. + +The analytic IK emits at most the two real elbow branches, rejects +unsupported pitch or roll, and chooses the nearest collision-clear +branch. It does not generate periodic duplicates or run a numerical +seed search, so a Cartesian leg cannot silently switch to an identical- +looking but discontinuous wrist configuration. + +### Detected-pose picking and collision + +`Pick` targets the selected detection pose, not the containing +Location's centre. The target provider stores result id, timestamp, +source frame and world pose by class label. Off-server detections must +use the calibrated `camera_eih` frame and remain within 80 mm of the +simulated part; targets expire after ten seconds. Only the built-in +on-server ground-truth path may fall back to current simulation state. + +Motion retracts vertically, crosses at a clear height, descends, closes +or opens the jaws, and reverses the exact local approach before the +operation completes. A following Pick at the same fixture XY skips the +cross-cell traverse. The collision model uses per-link radii and tests +the bench, bin walls, fixture plate and pegs, every non-target part, the +accumulated stack, and an explicit box swept by the held part. + +## What an agent actually receives + +`vision_get_frame` (MCP) and `GetClip` (OPC UA) return a **612 × 512 +PNG**, delivered as inline bytes in the method's `ByteString` output and +base64-encoded into the MCP `ImageContentBlock`. The +`VisionImageReferenceDataType` alongside it carries a +`opcua-inline://…` **reference** and the frame's dimensions — it does not +carry the image, which would ship the payload twice. + +612 × 512 is what the sensor declares, what the clip endpoint declares, +what the intrinsics describe and what the renderer produces. The +simulated device is a 2448 × 2048 area-scan camera operated with 4 × 4 +binning, so the calibrated intrinsics are divided by four and the +Brown-Conrady coefficients — expressed in normalised image coordinates — +carry over unchanged. The native size survives only in the model and +serial number, where it identifies the hardware rather than the image. + +This matters because the ground-truth detector projects through the +declared intrinsics: the `BoundingBox2D` on every detection is in the +same pixel frame as the PNG an agent was handed. Previously the sensor +said 2448 × 2048, the clip endpoint said 1280 × 1024 and the renderer +produced 640 × 512 — not even the same aspect ratio — so a model asked +to "pick the red cube you can see" got a picture and a set of +coordinates that pointed off it. + +After a successful `GetClip`, the Server publishes the frame on the clip +endpoint's `LatestClip` and its descriptor on `LatestClipMetadata`, so a +consumer that follows the model — read the published frame, call the +method only if there is none — gets a frame rather than a permanent +`Bad_NoDataAvailable`. + +## Rendering behaviour on CI + +The `Opc.Ua.Vision.OpenUsd` capture provider needs a native OpenUSD +renderer payload plus a usable graphics device to produce pixels. On CI +neither is present — the provider reports +`SceneCameraCaptureBackend.NoRenderingBackend` on `UnavailableReason`, +and every `LatestClip` read against `PickFrames` reports +`Bad_NoDataAvailable`. The sensor still exists in the address space, +every browse still works, every calibration is still readable and the +inference proof hosted service still updates the world state. + +This is by design: the point of the CI leg is to exercise the address +space, the fluent builder, the provider abstractions and the client +plumbing, not to prove that the machine has a GPU. `--demo` on the +paired client skips its "compose frame → world" step gracefully when +the frame is unavailable rather than falsely reporting a rendering +failure. + +On a workstation with the renderer payload installed and a usable +graphics device, the same sensor renders normally and returns encoded +PNG bytes through the inline `LatestClip` or the by-reference +`GetClip` path. + +## Ground-truth vs agent inference + +- **Ground-truth path** (`--inferenceLocation OnServer`): + `BinPickingGroundTruthInferenceProvider` reads the current world + state from `BinPickingWorldState`, projects each part into the + camera through the authored `HandEye` transform and intrinsics, and + publishes a `DetectionResultType` with the corresponding class label, + bounding box and pose. Because this reads real geometry, the poses + it publishes are the ones a correct agent would submit — which is + what makes the ground-truth path useful as a check on an agent's + answer. +- **Agent path** (`--inferenceLocation EdgeOffServer`): + `BinPickingAgentInferenceProvider` is the sample's off-Server + feedback sink. `SubmitDetections`, `SubmitCorrection` and + `SubmitImageReference` are accepted; `RunInference`, + `StartContinuous`, `Stop` and `SubmitInspectionResult` are refused + with `Bad_NotSupported` because this pipeline exposes + `DetectionResultType` results only and has no on-Server compute + path. The agent's submissions are validated before they land — see + below. + +Two hosted services publish loop-proof output regardless of MCP being +attached: + +- `BinPickingCaptureProof` captures a still on startup (skipped with + `--captureOnStartup false`), writes it to + `--artifactDirectory` when set, and logs the resulting size or the + `NoRenderingBackend` reason. Useful for verifying the render path + from a CI leg. +- `BinPickingInferenceProof` (on-server) drives one round of the + deterministic detector and mutates `BinPickingWorldState`, or + `BinPickingOffServerProof` (off-server) drives one round of agent + submissions with valid detections. Both prove the end-to-end write + path independently of the MCP tool surface. The on-server proof + **restores the bin afterwards**: it runs before any client connects, + and leaving a part picked would mean the paired client's demo started + against a world it had not changed itself. + +The robot moves the parts too. `Pick` travels to its `Source`, closes the +gripper around the current pose of the part named by the intent's +`ObjectClass`, marks it held only after the jaws finish closing, then +retracts. The part's world position follows the tool until `Place` +opens the gripper, settles it on the highest support, and retracts. The +ground-truth detector projects from those same positions, so it stops +reporting a part that has been moved out of the bin — which is what +makes the paired client's `--demo` verification real rather than a +formality. + +## Feedback validation + +When an agent submits detections through +`vision_submit_detections` (or `vision_submit_correction`), the sample's +feedback sink validates every field and refuses malformed submissions +with `Bad_InvalidArgument` and a message the agent can act on: + +- **Unknown class label** — refused with the exact list of parts that + do exist: `Detection 0 class 'PurplePyramid' is not a part in this + cell. Known classes: RedCube, GreenCylinder, BlueSphere, YellowSlab, + OrangeBrick.` +- **Confidence outside `[0, 1]`** — refused with the observed value. +- **Bounding box outside the image** — refused with the box centre, + extents and the image dimensions. +- **Non-positive box extents** — refused with the extents. +- **`NaN` in a box or pose** — refused with the detection index. +- **Zero-norm quaternion or `Orientation` with fewer than four + components** — refused with an explanation that references §5.12. +- **`Position` with fewer than three components** — refused likewise. +- **Pose frame other than `camera_eih`** — refused with both the + submitted and calibrated frame ids. +- **Pose more than 80 mm from the matching simulated part** — refused + with the measured residual. +- **Purpose not in `Overlay | Reconciliation | GroundTruthLabel | + Trigger`** — refused with the offending value. +- **More than 15 detections in a single submission** — refused as + implausible for the five-part cell. + +An **empty** detection set is refused too. §9.5 lists "`Detections` +empty" as `Bad_InvalidArgument`, and requires `SubmitCorrection` to +carry *exactly one* non-empty corrected array. The cell conforms, which +has a consequence worth knowing: an agent that has emptied the bin +cannot report "I looked and there is nothing there", and a false +positive cannot be retracted by correcting a result down to nothing. +Both are raised against the draft rather than worked around here — see +the [Vision developer guide](../../../docs/Vision.md#limitations). + +`SubmitInspectionResult` is refused with `Bad_NotSupported` regardless +of the arguments: this pipeline is a detection pipeline, not an +inspection pipeline, and an agent submitting an inspection result is +confused about which pipeline this is. + +## Related samples and docs + +- [BinPickingClient](../BinPickingClient) — the paired client, with + `--demo` for the scripted loop and `--mcp` for an agent-driven one. +- [Vision developer guide](../../../docs/Vision.md) — the + companion documentation for `Opc.Ua.Vision*` and + `Opc.Ua.Mcp.Vision`. +- [Robotics developer guide](../../../docs/Robotics.md) — the sibling + companion; the Robot Intent controller in this cell is the topic of + that guide. +- [OpenUSD guide](../../../docs/OpenUsd.md) — the connector and scene + materialisation used to bind the address space to the USD stage. +- [MCP Server guide](../../../docs/McpServer.md) — profile composition + (`--profile vision,robotics`), tool tables and connection semantics. diff --git a/samples/Robotics/BinPickingClient/AssemblyInfo.cs b/samples/Robotics/BinPickingClient/AssemblyInfo.cs new file mode 100644 index 0000000000..10062f988f --- /dev/null +++ b/samples/Robotics/BinPickingClient/AssemblyInfo.cs @@ -0,0 +1,32 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +[assembly: CLSCompliant(true)] diff --git a/samples/Robotics/BinPickingClient/BinPickingClient.csproj b/samples/Robotics/BinPickingClient/BinPickingClient.csproj new file mode 100644 index 0000000000..6b16ce3e53 --- /dev/null +++ b/samples/Robotics/BinPickingClient/BinPickingClient.csproj @@ -0,0 +1,43 @@ + + + $(AppTargetFrameworks) + Exe + false + BinPickingClient + BinPickingClient + Vision-guided bin-picking client that hosts the composed Vision + Robotics MCP catalogue over an OPC UA session to the BinPickingCell server, optionally opens the OpenUSD viewer for a human observer, and can execute a scripted pick-and-place demonstration that closes the perception-to-action loop without an agent attached. + BinPickingClient + enable + app.manifest + + false + true + + + $(DefineConstants);BINPICKING_CLIENT_MCP + true + + + + + + + + + + + + + + + + + + + + + diff --git a/samples/Robotics/BinPickingClient/BinPickingClientEventIds.cs b/samples/Robotics/BinPickingClient/BinPickingClientEventIds.cs new file mode 100644 index 0000000000..84b85704d3 --- /dev/null +++ b/samples/Robotics/BinPickingClient/BinPickingClientEventIds.cs @@ -0,0 +1,54 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace BinPickingClient +{ + internal static class BinPickingClientEventIds + { + public const int Connected = 7700; + public const int McpCatalogueSize = 7701; + public const int McpHostStarted = 7702; + public const int DemoStageStarted = 7703; + public const int DemoDetections = 7704; + public const int DemoPoseComposed = 7705; + public const int DemoPickSubmitted = 7706; + public const int DemoPickCompleted = 7707; + public const int DemoPickRefused = 7708; + public const int DemoPlaceSubmitted = 7709; + public const int DemoPlaceCompleted = 7710; + public const int DemoPlaceRefused = 7711; + public const int DemoPostPickDetections = 7712; + public const int DemoLoopComplete = 7713; + public const int DemoUnknownClass = 7714; + public const int DemoAuthorityNotGranted = 7715; + public const int DemoWorldStateUnchanged = 7716; + public const int DemoWorldStateChanged = 7717; + public const int DemoPipelineUnavailable = 7718; + } +} diff --git a/samples/Robotics/BinPickingClient/BinPickingClientOptions.cs b/samples/Robotics/BinPickingClient/BinPickingClientOptions.cs new file mode 100644 index 0000000000..15bb0d79f7 --- /dev/null +++ b/samples/Robotics/BinPickingClient/BinPickingClientOptions.cs @@ -0,0 +1,242 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Globalization; +using System.Linq; + +namespace BinPickingClient +{ + /// + /// Command-line options for the bin-picking client. Mirrors the switches accepted + /// by IntentViewerClient where the intent is the same, and adds the two + /// bin-picking specific switches (--demo and --part) that drive the + /// scripted end-to-end demonstration. + /// + internal sealed record BinPickingClientOptions + { + public string ServerUrl { get; init; } = "opc.tcp://localhost:62855/BinPickingCell"; + + public bool Insecure { get; init; } + + public bool View { get; init; } + + public string? Renderer { get; init; } + + /// + /// Prim path of the camera the viewport opens on. Defaults to the fixed observer + /// camera authored in the stage, which shows the cell working. Pass + /// --camera auto to let the viewer frame the scene itself, or another prim + /// path to pin a different view; note that the eye-in-hand sensor on the flange is + /// also a camera prim, so pointing this at it shows what the tool sees rather than + /// the cell. + /// + public string? CameraPath { get; init; } = DefaultObserverCameraPath; + + /// + /// Raises the log level to Debug, which surfaces every live OpenUSD binding update + /// and every target the connector had to leave unresolved. + /// + public bool Verbose { get; init; } + + /// + /// Picks every part the detector reports and places them all on the destination, + /// so they end up stacked, rather than running a single pick-and-place cycle. + /// + public bool StackAll { get; init; } + + /// + /// The stage's fixed observer camera. + /// + public const string DefaultObserverCameraPath = "/World/ObserverCamera"; + + public string? FetchAssetsDirectory { get; init; } + + public int Seconds { get; init; } + + public bool Mcp { get; init; } + + public bool Demo { get; init; } + + public string PartClassLabel { get; init; } = "RedCube"; + + public string SourceLocationName { get; init; } = "Bin"; + + public string DestinationLocationName { get; init; } = "Fixture"; + + public string ToolBrowseName { get; init; } = "ParallelGripper"; + + public string? Transport { get; init; } + + public int Port { get; init; } = 5170; + + public static BinPickingClientOptions Parse(string[] args) + { + return new BinPickingClientOptions + { + ServerUrl = GetOption(args, "--server") ?? "opc.tcp://localhost:62855/BinPickingCell", + Insecure = HasFlag(args, "--insecure"), + View = HasFlag(args, "--view"), + Renderer = GetOption(args, "--renderer"), + CameraPath = ResolveCameraPath(GetOption(args, "--camera")), + Verbose = HasFlag(args, "--verbose"), + StackAll = HasFlag(args, "--stack-all"), + FetchAssetsDirectory = GetOption(args, "--fetch-assets"), + Seconds = int.TryParse( + GetOption(args, "--seconds"), NumberStyles.Integer, CultureInfo.InvariantCulture, out int seconds) + ? seconds + : 0, + Mcp = HasFlag(args, "--mcp"), + Demo = HasFlag(args, "--demo"), + PartClassLabel = GetOption(args, "--part") ?? "RedCube", + SourceLocationName = GetOption(args, "--source") ?? "Bin", + DestinationLocationName = GetOption(args, "--destination") ?? "Fixture", + ToolBrowseName = GetOption(args, "--tool") ?? "ParallelGripper", + Transport = GetOption(args, "--transport"), + Port = int.TryParse( + GetOption(args, "--port"), NumberStyles.Integer, CultureInfo.InvariantCulture, out int port) + ? port + : 5170 + }; + } + + private static string? GetOption(string[] args, string name) + { + for (int ii = 0; ii < args.Length - 1; ii++) + { + if (string.Equals(args[ii], name, StringComparison.OrdinalIgnoreCase)) + { + return args[ii + 1]; + } + } + return null; + } + + private static bool HasFlag(string[] args, string name) + { + return args.Any(a => string.Equals(a, name, StringComparison.OrdinalIgnoreCase)); + } + + /// + /// Maps the --camera argument onto a prim path, treating auto as + /// "let the viewer frame the scene" and no argument at all as the stage's observer + /// camera. + /// + private static string? ResolveCameraPath(string? requested) + { + if (requested is null) + { + return DefaultObserverCameraPath; + } + return string.Equals(requested, "auto", StringComparison.OrdinalIgnoreCase) + ? null + : requested; + } + + internal BinPickingClientMcpTransportSelection SelectMcpTransport() + { + if (Transport is not null && + TryParseMcpTransport(Transport, out BinPickingClientMcpTransport requestedTransport)) + { + if (View && requestedTransport == BinPickingClientMcpTransport.Stdio) + { + return new BinPickingClientMcpTransportSelection( + requestedTransport, + true, + "WARNING: --transport stdio was explicitly requested with --view. " + + "MCP stdio uses stdout for protocol frames and the in-process viewer may share that stream; " + + "protocol corruption is possible."); + } + + return new BinPickingClientMcpTransportSelection( + requestedTransport, + true, + $"Using explicitly requested MCP transport '{requestedTransport.ToOptionValue()}'."); + } + + if (View) + { + return new BinPickingClientMcpTransportSelection( + BinPickingClientMcpTransport.Http, + false, + "Using MCP transport 'http' because --view is enabled. MCP stdio frames use stdout, " + + "which cannot safely coexist with the in-process OpenUSD viewer."); + } + + return new BinPickingClientMcpTransportSelection( + BinPickingClientMcpTransport.Stdio, + false, + "Using default MCP transport 'stdio'."); + } + + internal static bool TryParseMcpTransport(string? value, out BinPickingClientMcpTransport transport) + { + if (string.Equals(value, "stdio", StringComparison.OrdinalIgnoreCase)) + { + transport = BinPickingClientMcpTransport.Stdio; + return true; + } + + if (string.Equals(value, "http", StringComparison.OrdinalIgnoreCase) || + string.Equals(value, "sse", StringComparison.OrdinalIgnoreCase)) + { + transport = BinPickingClientMcpTransport.Http; + return true; + } + + transport = BinPickingClientMcpTransport.Stdio; + return false; + } + } + + internal enum BinPickingClientMcpTransport + { + Stdio, + + Http + } + + internal sealed record BinPickingClientMcpTransportSelection( + BinPickingClientMcpTransport Transport, + bool Explicit, + string Message); + + internal static class BinPickingClientMcpTransportExtensions + { + public static string ToOptionValue(this BinPickingClientMcpTransport transport) + { + return transport switch + { + BinPickingClientMcpTransport.Stdio => "stdio", + BinPickingClientMcpTransport.Http => "http", + _ => throw new ArgumentOutOfRangeException(nameof(transport), transport, "Unknown MCP transport.") + }; + } + } +} diff --git a/samples/Robotics/BinPickingClient/BinPickingDemoRunner.cs b/samples/Robotics/BinPickingClient/BinPickingDemoRunner.cs new file mode 100644 index 0000000000..853dde1052 --- /dev/null +++ b/samples/Robotics/BinPickingClient/BinPickingDemoRunner.cs @@ -0,0 +1,702 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Server or Client OR OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Client; +using Opc.Ua.Robotics.Client.Intent; +using Opc.Ua.RobotIntent; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace BinPickingClient +{ + /// + /// Runs the scripted end-to-end bin-picking demonstration. Captures a frame's + /// worth of detections via RunInference, composes the chosen part's pose + /// from the camera frame into the world frame using the Vision frame graph, + /// submits Pick and Place intents through the Robot Intent controller, and + /// re-runs inference to prove the detected world state changed after the pick. + /// + /// + /// + /// The runner does not interpret the pixels itself. That is deliberate: the + /// bin-picking cell server ships a deterministic ground-truth detector so a + /// scripted CI run never depends on an external model or GPU, and the same + /// tools are exposed over MCP so an agent can drive the loop instead. The + /// runner exists to show that the loop composes end-to-end with the tools an + /// agent would use. + /// + /// + /// Pick and Place resolve their Source, Destination and + /// Tool NodeIds against the controller's lookup tables ( and ). + /// The name defaults (Bin, Fixture, ParallelGripper) match the + /// configurator in BinPickingRobotCell; passing --source, --destination + /// or --tool lets the operator retarget the demo without a rebuild. + /// + /// + internal sealed partial class BinPickingDemoRunner + { + public BinPickingDemoRunner( + BinPickingSampleSession sample, + ITelemetryContext telemetry, + ILogger logger, + BinPickingClientOptions options) + { + m_sample = sample ?? throw new ArgumentNullException(nameof(sample)); + m_telemetry = telemetry ?? throw new ArgumentNullException(nameof(telemetry)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + m_options = options ?? throw new ArgumentNullException(nameof(options)); + } + + public async Task RunAsync( + RobotIntentControllerClient controller, + RobotIntentControllerInfo controllerInfo, + bool commandAuthorityGranted, + CancellationToken cancellationToken) + { + if (controller == null) + { + throw new ArgumentNullException(nameof(controller)); + } + if (controllerInfo == null) + { + throw new ArgumentNullException(nameof(controllerInfo)); + } + + LogStageStarted(m_logger, m_options.PartClassLabel); + + VisionClient vision = m_sample.Session.Vision(m_telemetry); + NodeId pipelineNodeId = await DiscoverSinglePipelineAsync(vision, cancellationToken) + .ConfigureAwait(false); + + VisionDetectionResultSnapshot? initialDetections = null; + VisionDetectionDataType? part = null; + if (!pipelineNodeId.IsNull) + { + VisionPipelineClient pipeline = vision.Pipeline(pipelineNodeId); + _ = await pipeline.ReadAsync(cancellationToken).ConfigureAwait(false); + initialDetections = await CaptureAndReadDetectionsAsync( + vision, pipeline, cancellationToken).ConfigureAwait(false); + LogInitialDetectionCount( + m_logger, initialDetections.ResultId ?? string.Empty, initialDetections.Detections.Count); + LogDetections(initialDetections); + + part = FindDetection(initialDetections, m_options.PartClassLabel); + if (part is null) + { + Console.Error.WriteLine( + "The chosen part label '" + + m_options.PartClassLabel + + "' was not present in the initial " + + "inference result. Known class labels: " + + FormatKnownClasses(initialDetections) + + "."); + LogUnknownClass(m_logger, m_options.PartClassLabel); + } + else if (part.HasPose) + { + await ComposeAndLogPoseAsync(vision, pipeline, part, cancellationToken) + .ConfigureAwait(false); + } + } + else + { + LogPipelineUnavailable(m_logger); + Console.Error.WriteLine( + "The bin-picking cell's Vision pipeline is not browsable through this client session. " + + "The scripted loop will still exercise the Robot Intent Pick+Place cycle so the world " + + "state can be observed, and the composed Vision + Robotics MCP catalogue is unaffected."); + } + + NodeId sourceLocation = ResolveLookup( + controllerInfo.Lookups.Locations, m_options.SourceLocationName, "source location"); + NodeId destinationLocation = ResolveLookup( + controllerInfo.Lookups.Locations, m_options.DestinationLocationName, "destination location"); + NodeId tool = ResolveLookup( + controllerInfo.Lookups.Tools, m_options.ToolBrowseName, "tool"); + if (sourceLocation.IsNull || destinationLocation.IsNull || tool.IsNull) + { + return 5; + } + + if (!commandAuthorityGranted) + { + Console.Error.WriteLine( + "Command authority was not granted for this session. The scripted demo will inspect the " + + "detection results but skip the Pick and Place intents."); + LogAuthorityNotGranted(m_logger); + return 6; + } + + string intentIdPrefix = "binpickclient-" + + DateTime.UtcNow.ToString("yyyyMMddHHmmssfff", CultureInfo.InvariantCulture); + + if (m_options.StackAll) + { + return await StackEveryPartAsync( + controller, sourceLocation, destinationLocation, tool, intentIdPrefix, + initialDetections, cancellationToken).ConfigureAwait(false); + } + + string pickIntentId = intentIdPrefix + "-pick"; + string placeIntentId = intentIdPrefix + "-place"; + + bool pickAccepted = await SubmitPickAsync( + controller, sourceLocation, tool, pickIntentId, cancellationToken).ConfigureAwait(false); + if (!pickAccepted) + { + return 7; + } + + bool placeAccepted = await SubmitPlaceAsync( + controller, destinationLocation, tool, placeIntentId, cancellationToken).ConfigureAwait(false); + if (!placeAccepted) + { + return 8; + } + + if (initialDetections is not null && !pipelineNodeId.IsNull) + { + VisionPipelineClient pipeline = vision.Pipeline(pipelineNodeId); + VisionDetectionResultSnapshot postDetections = await CaptureAndReadDetectionsAsync( + vision, pipeline, cancellationToken).ConfigureAwait(false); + LogPostDetections( + m_logger, postDetections.ResultId ?? string.Empty, postDetections.Detections.Count); + LogDetections(postDetections); + + VisionDetectionDataType? stillPresent = FindDetection(postDetections, m_options.PartClassLabel); + bool worldChanged = + stillPresent is null || + (part is not null && part.HasPose && stillPresent.HasPose && !PoseIsEqual(part.Pose, stillPresent.Pose)); + if (part is null) + { + // Nothing was there to pick, so "it is gone now" proves nothing. Say so + // rather than report a pass the run did not earn. + Console.Error.WriteLine( + "Scripted loop INCONCLUSIVE: '" + + m_options.PartClassLabel + + "' was not detected " + + "before the Pick, so its absence afterwards says nothing about the world changing."); + } + else if (worldChanged) + { + LogWorldStateChanged(m_logger, m_options.PartClassLabel); + Console.Error.WriteLine( + "Scripted loop passed: after Pick+Place the detector no longer reports '" + + m_options.PartClassLabel + + "' at its original position."); + } + else + { + LogWorldStateUnchanged(m_logger, m_options.PartClassLabel); + Console.Error.WriteLine( + "Scripted loop FAILED: the pick-and-place cycle reported success over the OPC UA " + + "controller, but the on-server ground-truth detector still reports '" + + m_options.PartClassLabel + + "' at its original position. The robot moved and the " + + "intents succeeded, so the world state did not follow the arm."); + } + } + else + { + Console.Error.WriteLine( + "Scripted loop completed the Pick+Place cycle over the OPC UA Robot Intent controller. " + + "The Vision inference verification step was skipped because the pipeline was not reachable " + + "from this client session."); + } + + LogLoopComplete(m_logger, m_options.PartClassLabel); + return 0; + } + + /// + /// Returns the NodeId of the sole inference pipeline advertised by the bin-picking cell, + /// or a null NodeId when the pipeline is not reachable from this client session. + /// + /// + /// + /// The cell's Vision node manager materialises the pipeline through the fluent builder, + /// which grafts the folder and pipeline NodeStates under the Vision root but does not + /// register them with the CustomNodeManager after the root has already been added. + /// The reference from the Vision root to the Pipelines folder is therefore visible from + /// a browse of the root, but browsing from the folder itself yields BadNodeIdUnknown. + /// The scripted demo tolerates this by treating an unreachable pipeline as a soft + /// failure and still exercising the Robot Intent Pick+Place cycle so the loop's world + /// state effect can be observed. + /// + /// + private static async Task DiscoverSinglePipelineAsync( + VisionClient vision, CancellationToken cancellationToken) + { + ArrayOf pipelines = await vision.DiscoverPipelinesAsync(cancellationToken) + .ConfigureAwait(false); + for (int ii = 0; ii < pipelines.Count; ii++) + { + NodeId candidate = pipelines[ii]; + if (!candidate.IsNull) + { + return candidate; + } + } + await foreach (VisionNodeEntry entry in vision + .EnumeratePipelinesAsync(cancellationToken) + .ConfigureAwait(false)) + { + if (!entry.NodeId.IsNull) + { + return entry.NodeId; + } + } + return NodeId.Null; + } + + private async Task CaptureAndReadDetectionsAsync( + VisionClient vision, + VisionPipelineClient pipeline, + CancellationToken cancellationToken) + { + string resultId = await pipeline.RunInferenceAsync(default, cancellationToken) + .ConfigureAwait(false); + NodeId resultNodeId = await pipeline.ResolveResultNodeIdAsync(resultId, cancellationToken) + .ConfigureAwait(false); + if (resultNodeId.IsNull) + { + throw ServiceResultException.Create( + StatusCodes.BadUnexpectedError, + "RunInference returned ResultId '{0}' but the pipeline did not publish a matching result node.", + resultId); + } + VisionResultReader reader = vision.Result(resultNodeId); + return await reader.ReadDetectionAsync(cancellationToken).ConfigureAwait(false); + } + + private void LogDetections(VisionDetectionResultSnapshot snapshot) + { + var detections = new List(snapshot.Detections.Count); + for (int ii = 0; ii < snapshot.Detections.Count; ii++) + { + detections.Add(snapshot.Detections[ii]); + } + foreach (VisionDetectionDataType detection in detections) + { + string box = detection.HasBoundingBox2D + ? FormattableString.Invariant( + $"cx={detection.BoundingBox2D.CenterX:0.0} cy={detection.BoundingBox2D.CenterY:0.0} w={detection.BoundingBox2D.Width:0.0} h={detection.BoundingBox2D.Height:0.0}") + : ""; + string pose = detection.HasPose + ? FormatPose(detection.Pose) + : ""; + Console.Error.WriteLine(FormattableString.Invariant( + $" {detection.ClassLabel} (conf={detection.Confidence:0.00}) box2D=[{box}] pose=[{pose}]")); + } + } + + private async Task ComposeAndLogPoseAsync( + VisionClient vision, + VisionPipelineClient pipeline, + VisionDetectionDataType detection, + CancellationToken cancellationToken) + { + string cameraFrameId = detection.Pose.FrameId ?? string.Empty; + if (string.IsNullOrEmpty(cameraFrameId)) + { + Console.Error.WriteLine( + "Detection published a pose without a FrameId; skipping compose step."); + return; + } + NodeId cameraFrameNode = await ResolveFrameByFrameIdAsync(vision, cameraFrameId, cancellationToken) + .ConfigureAwait(false); + NodeId worldFrameNode = await ResolveFrameByFrameIdAsync(vision, "world", cancellationToken) + .ConfigureAwait(false); + if (cameraFrameNode.IsNull || worldFrameNode.IsNull) + { + Console.Error.WriteLine( + "Vision frame graph does not expose the frames the compose step needs " + + "(source '" + + cameraFrameId + + "' or target 'world'); skipping compose step."); + return; + } + VisionFrameGraph frames = vision.Frames(); + VisionPose3DDataType composed = await frames.ComposeAsync( + detection.Pose, cameraFrameNode, worldFrameNode, cancellationToken).ConfigureAwait(false); + (double x, double y, double z) = ReadVec3(composed.Position); + LogComposedPose(m_logger, m_options.PartClassLabel, x, y, z); + _ = pipeline; + } + + /// + /// Picks every part the detector reported and places them all on the same + /// destination, which leaves them stacked because the cell rests each released part + /// on whatever is already there. + /// + /// + /// The order is the order the detector reported, so the stack is built out of what + /// the camera actually saw rather than a hard-coded list. Each cycle waits for its + /// intent to reach a terminal state before the next is submitted: overlapping them + /// puts the next Pick in the queue while the arm is still carrying the last part, + /// and the parts end up wherever the arm happened to be. + /// + private async Task StackEveryPartAsync( + RobotIntentControllerClient controller, + NodeId source, + NodeId destination, + NodeId tool, + string intentIdPrefix, + VisionDetectionResultSnapshot? detections, + CancellationToken cancellationToken) + { + var labels = new List(); + if (detections is not null) + { + foreach (VisionDetectionDataType detection in detections.Detections) + { + string label = detection.ClassLabel ?? string.Empty; + if (label.Length > 0 && !labels.Contains(label, StringComparer.Ordinal)) + { + labels.Add(label); + } + } + } + if (labels.Count == 0) + { + Console.Error.WriteLine( + "Stack-all found no detected parts to stack; nothing to do."); + return 6; + } + + Console.Error.WriteLine( + "Stacking " + labels.Count + " part(s) the camera reported: " + string.Join(", ", labels)); + int placed = 0; + foreach (string label in labels) + { + cancellationToken.ThrowIfCancellationRequested(); + string cycle = intentIdPrefix + "-" + placed; + Console.Error.WriteLine( + "--- " + label + " (" + (placed + 1) + " of " + labels.Count + ") ---"); + if (!await SubmitPickAsync( + controller, source, tool, cycle + "-pick", label, cancellationToken) + .ConfigureAwait(false)) + { + return 7; + } + if (!await SubmitPlaceAsync( + controller, destination, tool, cycle + "-place", cancellationToken) + .ConfigureAwait(false)) + { + return 8; + } + placed++; + } + + Console.Error.WriteLine( + "Stacked " + placed + " part(s) on the destination. Each one rests on the one below it."); + return 0; + } + + private Task SubmitPickAsync( + RobotIntentControllerClient controller, + NodeId source, + NodeId tool, + string intentId, + CancellationToken cancellationToken) + { + return SubmitPickAsync( + controller, source, tool, intentId, m_options.PartClassLabel, cancellationToken); + } + + private async Task SubmitPickAsync( + RobotIntentControllerClient controller, + NodeId source, + NodeId tool, + string intentId, + string objectClass, + CancellationToken cancellationToken) + { + PickIntentDataType intent = RobotIntentBuilder.Pick(source, tool, objectClass) + .WithIntentId(intentId) + .Build(); + IntentSubmissionResult submission = await controller.TrySubmitIntentAsync(intent, cancellationToken) + .ConfigureAwait(false); + if (!submission.Accepted) + { + Console.Error.WriteLine( + "Pick refused: " + submission.Failure + " - " + submission.Message.Text); + LogPickRefused(m_logger, intentId, submission.Failure); + return false; + } + Console.Error.WriteLine( + "Pick admitted: intent " + submission.IntentId + " operation " + submission.Operation + "."); + LogPickSubmitted(m_logger, submission.IntentId); + IntentOperationHandle handle = await controller.TrackOperationAsync( + submission.IntentId, submission.Operation, cancellationToken).ConfigureAwait(false); + bool succeeded; + await using (handle.ConfigureAwait(false)) + { + IntentResultDataType result = await handle.Completion.WaitAsync(cancellationToken) + .ConfigureAwait(false); + succeeded = handle.Current.ExecutionState == ExecutionStateEnum.Succeeded && + result.Failure == IntentFailureEnum.None; + Console.Error.WriteLine( + "Pick operation terminal state: " + + handle.Current.ExecutionState + + " failure=" + + result.Failure); + LogPickCompleted(m_logger, submission.IntentId, handle.Current.ExecutionState); + } + return succeeded; + } + + private async Task SubmitPlaceAsync( + RobotIntentControllerClient controller, + NodeId destination, + NodeId tool, + string intentId, + CancellationToken cancellationToken) + { + PlaceIntentDataType intent = RobotIntentBuilder.Place(destination, tool) + .WithIntentId(intentId) + .Build(); + IntentSubmissionResult submission = await controller.TrySubmitIntentAsync(intent, cancellationToken) + .ConfigureAwait(false); + if (!submission.Accepted) + { + Console.Error.WriteLine( + "Place refused: " + submission.Failure + " - " + submission.Message.Text); + LogPlaceRefused(m_logger, intentId, submission.Failure); + return false; + } + Console.Error.WriteLine( + "Place admitted: intent " + submission.IntentId + " operation " + submission.Operation + "."); + LogPlaceSubmitted(m_logger, submission.IntentId); + IntentOperationHandle handle = await controller.TrackOperationAsync( + submission.IntentId, submission.Operation, cancellationToken).ConfigureAwait(false); + bool succeeded; + await using (handle.ConfigureAwait(false)) + { + IntentResultDataType result = await handle.Completion.WaitAsync(cancellationToken) + .ConfigureAwait(false); + succeeded = handle.Current.ExecutionState == ExecutionStateEnum.Succeeded && + result.Failure == IntentFailureEnum.None; + Console.Error.WriteLine( + "Place operation terminal state: " + + handle.Current.ExecutionState + + " failure=" + + result.Failure); + LogPlaceCompleted(m_logger, submission.IntentId, handle.Current.ExecutionState); + } + return succeeded; + } + + private static async Task ResolveFrameByFrameIdAsync( + VisionClient vision, + string frameId, + CancellationToken cancellationToken) + { + ArrayOf frameNodes = await vision.DiscoverFramesAsync(cancellationToken) + .ConfigureAwait(false); + VisionFrameGraph graph = vision.Frames(); + for (int ii = 0; ii < frameNodes.Count; ii++) + { + NodeId candidate = frameNodes[ii]; + if (candidate.IsNull) + { + continue; + } + VisionFrameSnapshot snapshot = await graph.ReadAsync(candidate, cancellationToken) + .ConfigureAwait(false); + if (string.Equals(snapshot.FrameId, frameId, StringComparison.Ordinal)) + { + return candidate; + } + } + return NodeId.Null; + } + + private static NodeId ResolveLookup( + ArrayOf lookup, string name, string kind) + { + for (int ii = 0; ii < lookup.Count; ii++) + { + RobotIntentNodeLookupEntry entry = lookup[ii]; + if (string.Equals(entry.Name, name, StringComparison.Ordinal) || + string.Equals(entry.BrowseName.Name, name, StringComparison.Ordinal)) + { + return entry.NodeId; + } + } + Console.Error.WriteLine( + "The controller did not publish a " + + kind + + " named '" + + name + + "'. Known values: " + + FormatLookupNames(lookup) + + "."); + return NodeId.Null; + } + + private static string FormatLookupNames(ArrayOf lookup) + { + if (lookup.Count == 0) + { + return ""; + } + var names = new List(lookup.Count); + for (int ii = 0; ii < lookup.Count; ii++) + { + names.Add(lookup[ii].Name); + } + return string.Join(", ", names); + } + + private static VisionDetectionDataType? FindDetection( + VisionDetectionResultSnapshot snapshot, string classLabel) + { + ArrayOf detections = snapshot.Detections; + for (int ii = 0; ii < detections.Count; ii++) + { + VisionDetectionDataType candidate = detections[ii]; + if (string.Equals(candidate.ClassLabel, classLabel, StringComparison.Ordinal)) + { + return candidate; + } + } + return null; + } + + private static string FormatKnownClasses(VisionDetectionResultSnapshot snapshot) + { + ArrayOf detections = snapshot.Detections; + if (detections.Count == 0) + { + return ""; + } + var names = new List(detections.Count); + for (int ii = 0; ii < detections.Count; ii++) + { + names.Add(detections[ii].ClassLabel ?? string.Empty); + } + return string.Join(", ", names); + } + + private static bool PoseIsEqual(VisionPose3DDataType a, VisionPose3DDataType b) + { + (double ax, double ay, double az) = ReadVec3(a.Position); + (double bx, double by, double bz) = ReadVec3(b.Position); + double dx = ax - bx; + double dy = ay - by; + double dz = az - bz; + return Math.Sqrt((dx * dx) + (dy * dy) + (dz * dz)) < 1e-4; + } + + private static (double X, double Y, double Z) ReadVec3(ArrayOf vec) + { + System.ReadOnlySpan span = vec.Span; + if (span.Length < 3) + { + return (0.0, 0.0, 0.0); + } + return (span[0], span[1], span[2]); + } + + private static string FormatPose(VisionPose3DDataType pose) + { + (double x, double y, double z) = ReadVec3(pose.Position); + return FormattableString.Invariant( + $"frame='{pose.FrameId ?? ""}' pos=({x:0.000},{y:0.000},{z:0.000})"); + } + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoStageStarted, Level = LogLevel.Information, + Message = "=== Bin-picking client scripted demo — pick {ClassLabel} ===")] + private static partial void LogStageStarted(ILogger logger, string classLabel); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoDetections, Level = LogLevel.Information, + Message = "Initial inference result {ResultId} reported {DetectionCount} detections.")] + private static partial void LogInitialDetectionCount(ILogger logger, string resultId, int detectionCount); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoPoseComposed, Level = LogLevel.Information, + Message = "Composed {ClassLabel} pose to world = ({X:0.000},{Y:0.000},{Z:0.000}) m.")] + private static partial void LogComposedPose(ILogger logger, string classLabel, double x, double y, double z); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoPickSubmitted, Level = LogLevel.Information, + Message = "Submitted Pick intent {IntentId}.")] + private static partial void LogPickSubmitted(ILogger logger, string intentId); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoPickCompleted, Level = LogLevel.Information, + Message = "Pick intent {IntentId} completed in state {ExecutionState}.")] + private static partial void LogPickCompleted(ILogger logger, string intentId, ExecutionStateEnum executionState); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoPickRefused, Level = LogLevel.Warning, + Message = "Pick intent {IntentId} was refused with {Failure}.")] + private static partial void LogPickRefused(ILogger logger, string intentId, IntentFailureEnum failure); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoPlaceSubmitted, Level = LogLevel.Information, + Message = "Submitted Place intent {IntentId}.")] + private static partial void LogPlaceSubmitted(ILogger logger, string intentId); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoPlaceCompleted, Level = LogLevel.Information, + Message = "Place intent {IntentId} completed in state {ExecutionState}.")] + private static partial void LogPlaceCompleted(ILogger logger, string intentId, ExecutionStateEnum executionState); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoPlaceRefused, Level = LogLevel.Warning, + Message = "Place intent {IntentId} was refused with {Failure}.")] + private static partial void LogPlaceRefused(ILogger logger, string intentId, IntentFailureEnum failure); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoPostPickDetections, Level = LogLevel.Information, + Message = "Post-pick inference result {ResultId} reported {DetectionCount} detections.")] + private static partial void LogPostDetections(ILogger logger, string resultId, int detectionCount); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoLoopComplete, Level = LogLevel.Information, + Message = "=== Bin-picking client scripted demo for {ClassLabel} complete ===")] + private static partial void LogLoopComplete(ILogger logger, string classLabel); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoUnknownClass, Level = LogLevel.Warning, + Message = "Requested part class '{ClassLabel}' was not visible in the initial inference result.")] + private static partial void LogUnknownClass(ILogger logger, string classLabel); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoAuthorityNotGranted, Level = LogLevel.Warning, + Message = "Skipping Pick/Place because command authority was not granted for this session.")] + private static partial void LogAuthorityNotGranted(ILogger logger); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoWorldStateUnchanged, Level = LogLevel.Warning, + Message = "After Pick+Place the detector still reports '{ClassLabel}' at its original position.")] + private static partial void LogWorldStateUnchanged(ILogger logger, string classLabel); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoWorldStateChanged, Level = LogLevel.Information, + Message = "After Pick+Place the world state for '{ClassLabel}' changed as expected.")] + private static partial void LogWorldStateChanged(ILogger logger, string classLabel); + + [LoggerMessage(EventId = BinPickingClientEventIds.DemoPipelineUnavailable, Level = LogLevel.Warning, + Message = "The Vision inference pipeline is not reachable from this client session; " + + "the scripted loop will exercise the Robot Intent Pick+Place cycle without a pre-inference step.")] + private static partial void LogPipelineUnavailable(ILogger logger); + + private readonly BinPickingSampleSession m_sample; + private readonly ITelemetryContext m_telemetry; + private readonly ILogger m_logger; + private readonly BinPickingClientOptions m_options; + } +} diff --git a/samples/Robotics/BinPickingClient/BinPickingSampleSession.cs b/samples/Robotics/BinPickingClient/BinPickingSampleSession.cs new file mode 100644 index 0000000000..2e692aca6b --- /dev/null +++ b/samples/Robotics/BinPickingClient/BinPickingSampleSession.cs @@ -0,0 +1,175 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua; +using Opc.Ua.Client; +using Opc.Ua.Client.Subscriptions.Streaming; +using Opc.Ua.Configuration; + +namespace BinPickingClient +{ + /// + /// Connected OPC UA session to the bin-picking cell server, together with the + /// application configuration and default streaming subscription the sample uses. + /// + internal sealed class BinPickingSampleSession : IAsyncDisposable + { + private BinPickingSampleSession( + ISession session, + IStreamingSubscription streaming, + ApplicationConfiguration configuration) + { + Session = session; + Streaming = streaming; + m_configuration = configuration; + } + + public ISession Session { get; } + + public IStreamingSubscription Streaming { get; } + + public static async Task ConnectAsync( + BinPickingClientOptions options, + ITelemetryContext telemetry, + CancellationToken cancellationToken) + { + string pkiRoot = GetPrivateStateRoot(); + var configuration = new ApplicationConfiguration(telemetry) + { + ApplicationName = "BinPickingClient", + ApplicationUri = "urn:localhost:OPCFoundation:BinPickingClient", + ApplicationType = ApplicationType.Client, + SecurityConfiguration = new SecurityConfiguration + { + ApplicationCertificate = new CertificateIdentifier + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(pkiRoot, "own"), + SubjectName = "CN=BinPickingClient, O=OPC Foundation" + }, + TrustedIssuerCertificates = new CertificateTrustList + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(pkiRoot, "issuer") + }, + TrustedPeerCertificates = new CertificateTrustList + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(pkiRoot, "trusted") + }, + RejectedCertificateStore = new CertificateTrustList + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(pkiRoot, "rejected") + }, + AutoAcceptUntrustedCertificates = options.Insecure + }, + // Camera frames are multi-megabyte ByteStrings. The default ByteString + // ceiling is about 1 MB, so a frame this cell happily serves would be + // refused on decode with BadEncodingLimitsExceeded. + TransportQuotas = new TransportQuotas + { + MaxMessageSize = 32 * 1024 * 1024, + MaxByteStringLength = 32 * 1024 * 1024, + MaxArrayLength = 32 * 1024 * 1024 + }, + ClientConfiguration = new ClientConfiguration(), + ServerConfiguration = new ServerConfiguration() + }; + await configuration.ValidateAsync(ApplicationType.Client, cancellationToken).ConfigureAwait(false); + + var appInstance = new ApplicationInstance(configuration, telemetry); + await appInstance + .CheckApplicationInstanceCertificatesAsync(true, ct: cancellationToken) + .ConfigureAwait(false); + await appInstance.DisposeAsync().ConfigureAwait(false); + configuration.CertificateManager ??= CertificateManagerFactory.Create( + configuration.SecurityConfiguration, telemetry); + if (options.Insecure) + { + configuration.CertificateManager.AcceptError = static (_, _) => true; + Console.Error.WriteLine("WARNING: --insecure is demo-only: any server certificate is accepted."); + } + + EndpointDescription? endpointDescription = await CoreClientUtils.SelectEndpointAsync( + configuration, + options.ServerUrl, + useSecurity: true, + discoverTimeout: 15000, + telemetry, + cancellationToken).ConfigureAwait(false); + if (endpointDescription is null) + { + throw ServiceResultException.Create(StatusCodes.BadTimeout, "Could not reach {0}.", options.ServerUrl); + } + + var endpoint = new ConfiguredEndpoint( + null, + endpointDescription, + EndpointConfiguration.Create(configuration)); + ManagedSession session = await new ManagedSessionBuilder(configuration, telemetry) + .UseEndpoint(endpoint) + .WithSessionName("BinPickingClient") + // Command authority is held per Session and is only handed back when the + // Session ends, so a client that is killed keeps the robot until its + // Session lapses. Sixty seconds of a locked cell is a long time in a + // demo; fifteen still comfortably survives a transient reconnect. + .WithSessionTimeout(TimeSpan.FromSeconds(15)) + .WithUserIdentity(new UserIdentity(new AnonymousIdentityToken())) + .ConnectAsync(cancellationToken).ConfigureAwait(false); + return new BinPickingSampleSession(session, session.DefaultStreaming, configuration); + } + + public async ValueTask DisposeAsync() + { + await Streaming.DisposeAsync().ConfigureAwait(false); + await Session.CloseAsync(CancellationToken.None).ConfigureAwait(false); + await Session.DisposeAsync().ConfigureAwait(false); + (m_configuration.CertificateManager as IDisposable)?.Dispose(); + } + + private static string GetPrivateStateRoot() + { + string baseDirectory = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData); + if (string.IsNullOrEmpty(baseDirectory)) + { + baseDirectory = AppContext.BaseDirectory; + } + string root = Path.Combine(baseDirectory, "OPC Foundation", "BinPickingClient", "pki"); + Directory.CreateDirectory(root); + return root; + } + + private readonly ApplicationConfiguration m_configuration; + } +} diff --git a/samples/Robotics/BinPickingClient/Program.cs b/samples/Robotics/BinPickingClient/Program.cs new file mode 100644 index 0000000000..67a876a4c0 --- /dev/null +++ b/samples/Robotics/BinPickingClient/Program.cs @@ -0,0 +1,653 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Linq; +using System.Runtime.InteropServices; +using System.Threading; +using System.Threading.Tasks; +#if BINPICKING_CLIENT_MCP +using Microsoft.AspNetCore.Builder; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +#endif +using Microsoft.Extensions.Logging; +#if BINPICKING_CLIENT_MCP +using Microsoft.Extensions.Logging.Console; +using ModelContextProtocol.Server; +#endif +using Opc.Ua; +using Opc.Ua.Client; +#if BINPICKING_CLIENT_MCP +using Opc.Ua.Mcp; +#endif +using Opc.Ua.OpenUsd.Client; +using Opc.Ua.Robotics.Client; +using Opc.Ua.Robotics.Client.Intent; + +namespace BinPickingClient +{ + /// + /// Bin-picking client entry point. Connects to the bin-picking cell server, exposes + /// the composed Vision + Robotics MCP catalogue for an external agent, optionally + /// opens the OpenUSD viewport so a human can watch the arm move, and can run the + /// scripted pick-and-place demonstration end to end. + /// + internal static partial class Program + { + [STAThread] + public static async Task Main(string[] args) + { + BinPickingClientOptions options = BinPickingClientOptions.Parse(args); + if (options.Transport is not null && + !BinPickingClientOptions.TryParseMcpTransport(options.Transport, out _)) + { + Console.Error.WriteLine( + $"Unknown MCP transport '{options.Transport}'. Valid transports: stdio, http, sse."); + return 2; + } + + BinPickingClientMcpTransportSelection mcpTransport = options.SelectMcpTransport(); + if (options.Mcp) + { +#if BINPICKING_CLIENT_MCP + Console.Error.WriteLine(mcpTransport.Message); +#else + Console.Error.WriteLine( + "MCP hosting is unavailable for this target framework. Run the sample without --mcp, " + + "or use the net8.0, net9.0, or net10.0 target framework for MCP hosting."); + return 2; +#endif + } + + // A console provider, or the sample's own logging goes nowhere: the OpenUSD + // connector reports what it bound and any target it had to leave unresolved, + // and a live stream that silently binds nothing looks exactly like one that + // works. Errors go to stderr so stdout stays clean for MCP stdio transport. + LogLevel minimumLevel = options.Verbose ? LogLevel.Debug : LogLevel.Information; + using ILoggerFactory loggerFactory = LoggerFactory.Create(builder => builder + .SetMinimumLevel(minimumLevel) + .AddFilter(level => level >= minimumLevel) + .AddConsole(console => console.LogToStandardErrorThreshold = LogLevel.Trace)); + ILogger logger = loggerFactory.CreateLogger("BinPickingClient"); + ITelemetryContext telemetry = DefaultTelemetry.Create(builder => builder + .SetMinimumLevel(minimumLevel) + .AddFilter(level => level >= minimumLevel) + .AddConsole(console => console.LogToStandardErrorThreshold = LogLevel.Trace)); + using CancellationTokenSource lifetime = options.Seconds > 0 + ? new CancellationTokenSource(TimeSpan.FromSeconds(options.Seconds)) + : new CancellationTokenSource(); + + BinPickingSampleSession sample = await BinPickingSampleSession.ConnectAsync( + options, telemetry, lifetime.Token).ConfigureAwait(false); + await using (sample.ConfigureAwait(false)) + { + LogConnected(logger, options.ServerUrl); + + RobotIntentClient intentClient = sample.Session.RobotIntent(telemetry, sample.Streaming); + ArrayOf controllers = + await intentClient.DiscoverControllersAsync(lifetime.Token).ConfigureAwait(false); + if (controllers.Count == 0) + { + Console.Error.WriteLine( + "No Robot Intent controllers were advertised at the conformant Server/RobotIntent/Controllers path."); + return 2; + } + RobotIntentControllerClient controller = intentClient.Controller(controllers[0].NodeId); + RobotIntentControllerInfo controllerInfo = await controller + .ReadAsync(lifetime.Token).ConfigureAwait(false); + string controllerName = string.IsNullOrEmpty(controllerInfo.BrowseName.Name) + ? controllers[0].BrowseName.Name ?? "(unnamed)" + : controllerInfo.BrowseName.Name; + Console.Error.WriteLine( + $"Controller: {controllerName} ({controllerInfo.NodeId})"); + + bool commandGranted = false; + CommandAuthorityLease? authority = null; +#if BINPICKING_CLIENT_MCP + IHost? mcpHost = null; +#endif + + // Only take command authority when this process is going to command the + // robot: the scripted demo, or an MCP host driving the cell through this + // session. A viewer is an observer, and taking an exclusive lease just to + // watch locks out the very agent the sample exists to serve - an agent on + // its own MCP session gets every intent refused while a window is open. + bool willCommand = options.Demo || options.Mcp; + if (willCommand) + { + try + { + authority = await controller.RequestAuthorityAsync(lifetime.Token).ConfigureAwait(false); + if (authority.Granted) + { + Console.Error.WriteLine("Command authority: granted for this session."); + commandGranted = true; + } + else + { + Console.Error.WriteLine( + $"Command authority: held by {authority.CurrentOwner}; submissions may be refused."); + } + } + catch (ServiceResultException exception) + when (exception.StatusCode == StatusCodes.BadUserAccessDenied) + { + Console.Error.WriteLine( + "Command authority request was denied: the connecting identity lacks the Operator role. " + + "Continuing in read-only mode so vision inference and MCP tool discovery remain visible."); + } + } + else + { + Console.Error.WriteLine( + "Command authority: not requested - this session only observes, so another client " + + "or an agent can command the cell while the viewport is open."); + } + + int exitCode = 0; + try + { +#if BINPICKING_CLIENT_MCP + if (options.Mcp) + { + mcpHost = await StartMcpHostAsync( + mcpTransport, options, sample.Session, logger, lifetime.Token).ConfigureAwait(false); + } +#endif + + if (options.Demo && options.View) + { + // Order matters: the scripted loop is over in seconds, so running it + // before the viewport opens leaves nothing to watch - the arm has already + // parked by the time the window appears. Open the viewport, wait for the + // live stream to be subscribed, and only then command the robot. + var streamReady = new TaskCompletionSource( + TaskCreationOptions.RunContinuationsAsynchronously); + Task viewport = RunViewportIfAvailableAsync( + sample, options, telemetry, streamReady, lifetime.Token); + await WaitForLiveStreamAsync(streamReady, viewport, lifetime.Token).ConfigureAwait(false); + var runner = new BinPickingDemoRunner(sample, telemetry, logger, options); + exitCode = await runner.RunAsync( + controller, controllerInfo, commandGranted, lifetime.Token).ConfigureAwait(false); + Console.Error.WriteLine( + "Scripted loop finished; the viewport stays open so the cell can still be " + + "inspected. Close the window to exit."); + _ = await viewport.ConfigureAwait(false); + } + else if (options.Demo) + { + var runner = new BinPickingDemoRunner(sample, telemetry, logger, options); + exitCode = await runner.RunAsync( + controller, controllerInfo, commandGranted, lifetime.Token).ConfigureAwait(false); + } + else if (options.View) + { + bool closedByUser = await RunViewportIfAvailableAsync( + sample, options, telemetry, null, lifetime.Token).ConfigureAwait(false); +#if BINPICKING_CLIENT_MCP + if (!closedByUser && options.Mcp && !options.Demo) + { + // The viewer never opened, or it failed. An agent is still driving the + // cell over MCP, so keep serving instead of exiting underneath it. + Console.Error.WriteLine( + "MCP server still running without a viewport; connect an MCP client to drive " + + "the cell. Press Ctrl+C to exit."); + await WaitForMcpServerAsync(mcpHost!, lifetime.Token).ConfigureAwait(false); + } +#endif + } + else if (options.Mcp && !options.Demo) + { +#if BINPICKING_CLIENT_MCP + Console.Error.WriteLine( + "MCP server running; connect an MCP client to drive the cell. Press Ctrl+C to exit."); + await WaitForMcpServerAsync(mcpHost!, lifetime.Token).ConfigureAwait(false); +#endif + } + else if (!options.Mcp && !options.Demo && !options.View) + { + Console.Error.WriteLine( + "Nothing to do: no --mcp, --demo, or --view supplied. Connected and read the controller " + + "capabilities to prove the session is healthy; exiting."); + } + } + finally + { +#if BINPICKING_CLIENT_MCP + if (mcpHost is not null) + { + await StopMcpHostAsync(mcpHost).ConfigureAwait(false); + } +#endif + if (authority is not null) + { + await authority.DisposeAsync().ConfigureAwait(false); + } + } + + return exitCode; + } + } + + /// + /// Waits until the live OpenUSD stream is subscribed, so a caller can command motion + /// that the viewport will actually show. Gives up if the viewport ends first (it is + /// optional and may be unavailable) or after a short grace period, because a demo that + /// cannot be watched is still better than one that never runs. + /// + private static async Task WaitForLiveStreamAsync( + TaskCompletionSource streamReady, + Task viewport, + CancellationToken cancellationToken) + { + Task completed = await Task.WhenAny( + streamReady.Task, + viewport, + Task.Delay(TimeSpan.FromSeconds(30), cancellationToken)).ConfigureAwait(false); + if (completed == streamReady.Task) + { + // Let the first subscription values land before commanding, so the opening + // frame shows the cell at rest rather than mid-motion. + await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken).ConfigureAwait(false); + } + } + + private static async Task RunViewportIfAvailableAsync( + BinPickingSampleSession sample, + BinPickingClientOptions options, + ITelemetryContext telemetry, + TaskCompletionSource? streamReady, + CancellationToken cancellationToken) + { + if (!UsdViewHostLoader.TryLoad(out IUsdViewHost? viewHost, out string reason)) + { + Console.Error.WriteLine( + "Viewport unavailable; the sample continues without a viewer. " + reason); + streamReady?.TrySetResult(false); + return false; + } + + string liveLayerPath = PrepareLiveLayerPath(options); + string cacheDir = options.FetchAssetsDirectory + ?? Path.GetDirectoryName(liveLayerPath) + ?? AppContext.BaseDirectory; + + string stagePath = Path.Combine(cacheDir, "stage.usda"); + + // The viewport needs the served geometry: without it only the live override layer + // composes, which carries transforms but no geometry and renders as an empty scene. + await FetchAssetsAsync(sample.Session, cacheDir, cancellationToken).ConfigureAwait(false); + + if (!File.Exists(stagePath)) + { + Console.Error.WriteLine( + "No fetched stage.usda exists; viewport will open the live override layer only."); + stagePath = liveLayerPath; + } + + var viewOptions = new UsdViewOptions + { + StagePath = stagePath, + Renderer = options.Renderer, + + // Defaults to the stage's fixed observer camera, which shows the cell + // working; --camera auto hands framing back to the viewer. Do not point + // this at /World/Robot/.../Camera by default - that is the eye-in-hand + // sensor on the flange, so it shows what the tool sees, not the cell. + CameraPath = options.CameraPath, + Title = "OPC UA Bin-picking Viewer", + Telemetry = telemetry + }; + Console.Error.WriteLine("Opening OpenUSD viewport for the bin-picking cell."); + Console.Error.WriteLine(options.CameraPath is { Length: > 0 } cameraPath + ? $"Opening on the stage camera {cameraPath}." + : "No stage camera requested; the viewport frames the scene itself."); + try + { + await RunViewportOnStaThreadAsync( + viewHost!, viewOptions, sample.Session, streamReady, cancellationToken) + .ConfigureAwait(false); + return true; + } +#pragma warning disable CA1031 // The viewer is a third-party UI; no exception from it should end the session. + catch (Exception exception) when (!cancellationToken.IsCancellationRequested) +#pragma warning restore CA1031 + { + // The viewport is optional. A failure inside the renderer or its window code must + // not take down an MCP session an agent is driving, so report it and carry on. + Console.Error.WriteLine( + "The OpenUSD viewport ended with an error; the sample continues without a viewer. " + + exception.Message); + return false; + } + } + + private static Task RunViewportOnStaThreadAsync( + IUsdViewHost viewHost, + UsdViewOptions viewOptions, + ISession session, + TaskCompletionSource? streamReady, + CancellationToken cancellationToken) + { + var completion = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var uiThread = new Thread(() => + { + try + { + viewHost.RunViewport( + viewOptions, + async (sink, ct) => await StreamOpenUsdAsync( + session, sink, viewOptions.Telemetry, streamReady, ct) + .ConfigureAwait(false), + cancellationToken); + completion.TrySetResult(true); + } +#pragma warning disable CA1031 + catch (Exception exception) +#pragma warning restore CA1031 + { + completion.TrySetException(exception); + } + }) + { + IsBackground = false, + Name = "Bin-picking OpenUSD viewport" + }; + if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) + { + uiThread.SetApartmentState(ApartmentState.STA); + } + uiThread.Start(); + return completion.Task; + } + + private static async Task StreamOpenUsdAsync( + ISession session, + IUsdSink sink, + ITelemetryContext? telemetry, + TaskCompletionSource? streamReady, + CancellationToken cancellationToken) + { + // Thread telemetry in: without it the connector logs to NullLogger, so a live + // stream that binds nothing, or that leaves every target unresolved, looks + // exactly like one that is working. + var connector = new OpenUsdConnector( + session, sink, new OpenUsdConnectorOptions { EnableCommands = false }, telemetry); + await using (connector.ConfigureAwait(false)) + { + await connector.StartAsync(cancellationToken).ConfigureAwait(false); + Console.Error.WriteLine("Live OpenUSD stream started; the viewport now follows the cell."); + streamReady?.TrySetResult(true); + try + { + await PumpWhileViewportIsOpenAsync(session, cancellationToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + // Viewport shutdown ends the live stream. + } + finally + { + Console.Error.WriteLine("Live OpenUSD stream stopping."); + await connector.StopAsync(CancellationToken.None).ConfigureAwait(false); + } + } + } + + /// + /// Keeps the client doing work for as long as the viewport is open. + /// + /// + /// This used to be a single infinite delay, which is the natural thing to write: + /// the connector's subscriptions push updates, so the host has nothing else to do. + /// It also stops the viewport updating. Measured against this cell, an idle host + /// applies no stage updates at all while a busy one applies them normally + /// (openusd-dotnet issue 17), so a viewer that only watches shows a frozen scene + /// while the robot is demonstrably moving - which is exactly the case when an + /// agent, rather than this client, is driving the cell. Reading the controller on + /// a timer keeps the host doing something between updates. The read is a real + /// one so this is a poll rather than a spin, and it is cheap next to rendering. + /// + private static async Task PumpWhileViewportIsOpenAsync( + ISession session, + CancellationToken cancellationToken) + { + var serverStatus = new ReadValueId + { + NodeId = global::Opc.Ua.VariableIds.Server_ServerStatus_CurrentTime, + AttributeId = Attributes.Value + }; + ArrayOf nodesToRead = [serverStatus]; + while (!cancellationToken.IsCancellationRequested) + { + try + { + _ = await session.ReadAsync( + null, 0, TimestampsToReturn.Neither, nodesToRead, cancellationToken) + .ConfigureAwait(false); + } +#pragma warning disable CA1031 // A read failure must not close a viewport the user is watching. + catch (Exception) when (!cancellationToken.IsCancellationRequested) +#pragma warning restore CA1031 + { + // A dropped read is the reconnect handler's business, not the viewer's. + } + await Task.Delay(ViewportPumpInterval, cancellationToken).ConfigureAwait(false); + } + } + + private static async Task FetchAssetsAsync( + ISession session, string cacheDir, CancellationToken cancellationToken) + { + Directory.CreateDirectory(cacheDir); + var fetcher = new OpenUsdConnector(session, new MockUsdSink(), enableCommands: false); + await using (fetcher.ConfigureAwait(false)) + { + System.Collections.Generic.List fetched = + await fetcher.FetchServedAssetsAsync(cacheDir, cancellationToken).ConfigureAwait(false); + if (fetched.Count == 0) + { + Console.Error.WriteLine( + "Server did not advertise served OpenUSD assets; viewport will use the live override layer only."); + return; + } + Console.Error.WriteLine( + $"Fetched {fetched.Count} OpenUSD asset(s) into {cacheDir}."); + } + } + + private static string PrepareLiveLayerPath(BinPickingClientOptions options) + { + string root = options.FetchAssetsDirectory ?? Path.Combine(GetPrivateStateRoot(), "bin-picking"); + Directory.CreateDirectory(root); + string path = Path.Combine(root, "live.usda"); + if (!File.Exists(path)) + { + File.WriteAllText(path, "#usda 1.0\n(\n doc = \"OPC UA -> OpenUSD live override layer\"\n)\n"); + } + return path; + } + + private static string GetPrivateStateRoot() + { + string baseDirectory = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData); + if (string.IsNullOrEmpty(baseDirectory)) + { + baseDirectory = AppContext.BaseDirectory; + } + string root = Path.Combine(baseDirectory, "OPC Foundation", "BinPickingClient"); + Directory.CreateDirectory(root); + return root; + } + +#if BINPICKING_CLIENT_MCP + private static async Task StartMcpHostAsync( + BinPickingClientMcpTransportSelection transport, + BinPickingClientOptions options, + ISession session, + ILogger logger, + CancellationToken cancellationToken) + { + IHost host = transport.Transport == BinPickingClientMcpTransport.Stdio + ? BuildStdioMcpHost() + : BuildHttpMcpHost(options.Port); + + OpcUaSessionManager sessionManager = host.Services.GetRequiredService(); + await sessionManager.RegisterExistingSessionAsync( + "bin-picking", session, "Anonymous", cancellationToken).ConfigureAwait(false); + + int toolCount = host.Services.GetServices().Count(); + LogMcpCatalogueSize(logger, toolCount); + Console.Error.WriteLine( + $"MCP catalogue exposes {toolCount} tools (Vision + Robotics + Connection)."); + + await host.StartAsync(cancellationToken).ConfigureAwait(false); + string transportName = transport.Transport.ToOptionValue(); + LogMcpHostStarted(logger, transportName); + if (transport.Transport == BinPickingClientMcpTransport.Http) + { + Console.Error.WriteLine( + $"MCP server is listening on http://localhost:{options.Port}/mcp with Vision + Robotics tools."); + } + else + { + Console.Error.WriteLine("MCP server is listening on stdio with Vision + Robotics tools."); + } + return host; + } + + private static IHost BuildStdioMcpHost() + { + HostApplicationBuilder builder = Host.CreateApplicationBuilder(); + ConfigureMcpLogging(builder.Logging, useStdioTransport: true); + ConfigureMcpServices(builder.Services); + + IMcpServerBuilder mcpServerBuilder = builder.Services + .AddMcpServer() + .WithStdioServerTransport(); + ConfigureVisionAndRoboticsTools(mcpServerBuilder); + return builder.Build(); + } + + private static WebApplication BuildHttpMcpHost(int port) + { + WebApplicationBuilder builder = WebApplication.CreateBuilder(); + ConfigureMcpLogging(builder.Logging, useStdioTransport: false); + ConfigureMcpServices(builder.Services); + + IMcpServerBuilder mcpServerBuilder = builder.Services + .AddMcpServer() + .WithHttpTransport(); + ConfigureVisionAndRoboticsTools(mcpServerBuilder); + + WebApplication app = builder.Build(); + app.MapMcp("/mcp"); + app.Urls.Add($"http://localhost:{port}"); + return app; + } + + private static void ConfigureMcpServices(IServiceCollection services) + { + McpToolProfileSet profiles = new McpToolProfileSet(McpToolProfile.Vision).With(McpToolProfile.Robotics); + services.AddOpcUaMcpCore(new OpcUaMcpOptions { ToolProfiles = profiles }); + services.AddOpcUaMcpVision(); + services.AddOpcUaMcpRobotics(); + } + + private static void ConfigureVisionAndRoboticsTools(IMcpServerBuilder mcpServerBuilder) + { + McpToolProfileSet profiles = new McpToolProfileSet(McpToolProfile.Vision).With(McpToolProfile.Robotics); + mcpServerBuilder + .WithOpcUaMcpFilters() + .WithOpcUaCoreTools(profiles) + .WithOpcUaVisionTools(profiles) + .WithOpcUaRoboticsTools(profiles); + } + + private static void ConfigureMcpLogging(ILoggingBuilder logging, bool useStdioTransport) + { + logging.ClearProviders(); + logging.SetMinimumLevel(LogLevel.Information); + logging.AddSimpleConsole(options => + { + options.UseUtcTimestamp = true; + options.TimestampFormat = "yyyy-MM-dd HH:mm:ss "; + }); + logging.Services.Configure(o => + o.LogToStandardErrorThreshold = useStdioTransport ? LogLevel.Trace : LogLevel.Error); + } + + private static async Task WaitForMcpServerAsync(IHost host, CancellationToken cancellationToken) + { + try + { + await host.WaitForShutdownAsync(cancellationToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + // Ctrl+C or --seconds is the expected way for the host to stop. + } + } + + private static async Task StopMcpHostAsync(IHost host) + { + try + { + await host.StopAsync(CancellationToken.None).ConfigureAwait(false); + } + finally + { + host.Dispose(); + } + } +#endif + + [LoggerMessage(EventId = BinPickingClientEventIds.Connected, Level = LogLevel.Information, + Message = "Connected to bin-picking cell at {ServerUrl}.")] + private static partial void LogConnected(ILogger logger, string serverUrl); + +#if BINPICKING_CLIENT_MCP + [LoggerMessage(EventId = BinPickingClientEventIds.McpCatalogueSize, Level = LogLevel.Information, + Message = "MCP catalogue registered with {ToolCount} tools.")] + private static partial void LogMcpCatalogueSize(ILogger logger, int toolCount); + + [LoggerMessage(EventId = BinPickingClientEventIds.McpHostStarted, Level = LogLevel.Information, + Message = "MCP host started with transport {Transport}.")] + private static partial void LogMcpHostStarted(ILogger logger, string transport); +#endif + + /// + /// How often a watching client reads the server while the viewport is open. Fast + /// enough that the host is never idle for long, slow enough to stay a poll. + /// + private static readonly TimeSpan ViewportPumpInterval = TimeSpan.FromMilliseconds(50); + } +} diff --git a/samples/Robotics/BinPickingClient/README.md b/samples/Robotics/BinPickingClient/README.md new file mode 100644 index 0000000000..46a447b9a8 --- /dev/null +++ b/samples/Robotics/BinPickingClient/README.md @@ -0,0 +1,378 @@ + + +# Bin Picking Client + +This sample closes the perception-to-action loop against the `BinPickingCell` server: +it connects, discovers the Robot Intent controller and its lookup tables +(`Bin`, `Fixture`, `ParallelGripper`), requests command authority, and — with +`--mcp` — hosts the composed **Vision + Robotics** MCP catalogue so a language +model can drive the cell while a human watches. With `--view`, it can also open +the in-process OpenUSD viewport so the same session is observable in 3-D while +the agent runs. + +The MCP catalogue exposed to the agent contains **64 tools** measured from the +running host: `26` Vision tools + `42` Robotics tools − `4` shared connection +tools. This includes `vision_get_frame`, which returns the eye-in-hand camera +image as an MCP `ImageContentBlock` so the model genuinely sees pixels; the +one-call `robotics_vision_pick`; typed intent/mission tools; and the state, +authority, paging and bounded-wait tools needed to plan against refusals. + +## MCP hosting + +With `--mcp`, the client exposes the composed catalogue so an LLM can discover +controllers, read state, request authority, capture frames from the eye-in-hand +camera, submit `Pick` and `Place` intents, wait on operations, pause / resume / +cancel / retry, and re-inspect the world — all through the same `SampleSession` +the sample opened. The Vision and Robotics tools share the connection, so an +agent that captures a frame and submits a `Pick` is observing and commanding the +same cell. + +MCP hosting is compiled for `net8.0`, `net9.0`, and `net10.0`. It is unavailable +on `net48`; running that leg with `--mcp` reports the limitation on stderr. +Without `--mcp`, the sample continues to run on every framework it builds for. + +MCP options: + +- `--mcp` enables MCP hosting. +- `--transport stdio|http|sse` selects the MCP transport. `sse` is accepted as + an alias for `http`. +- `--port ` selects the Streamable HTTP port. The default is `5170`, + chosen so it does not collide with `IntentViewerClient`'s default of `5100`. + +Transport selection is announced on stderr: + +- No `--transport` and no `--view`: **stdio** is selected by default. +- No `--transport` with `--view`: **HTTP** is selected automatically and the + client explains that MCP stdio uses stdout for protocol frames, which cannot + safely coexist with the in-process OpenUSD viewer. +- `--transport http` with or without `--view`: **HTTP** is used. +- `--transport stdio` without `--view`: **stdio** is used. +- `--transport stdio --view`: the explicit request is honored, but the client + warns plainly that the viewer may share stdout and corrupt the MCP stdio + protocol. + +Example MCP client configuration for viewport mode: + +```json +{ + "servers": { + "bin-picking": { + "url": "http://localhost:5170/mcp" + } + } +} +``` + +Run the sample with HTTP MCP and the viewport: + +```powershell +dotnet run --project samples\Robotics\BinPickingClient\BinPickingClient.csproj --framework net10.0 -- --server opc.tcp://localhost:62855/BinPickingCell --insecure --view --mcp --transport http --port 5170 +``` + +For stdio MCP, point the MCP client at the command instead: + +```json +{ + "servers": { + "bin-picking": { + "command": "dotnet", + "args": [ + "run", + "--project", + "samples\\Robotics\\BinPickingClient\\BinPickingClient.csproj", + "--", + "--server", + "opc.tcp://localhost:62855/BinPickingCell", + "--insecure", + "--mcp" + ] + } + } +} +``` + +Agents can submit only what the server advertises and accepts. Refusals are +returned to the agent as tool results and are never retried on the agent's +behalf; the agent must decide whether to re-plan, ask for operator action, or +stop. + +## Scripted demonstration mode + +Use `--demo` to run the whole loop without an agent attached. This is what makes +the sample runnable in CI and by someone without an MCP client, and it is how +the loop is proven: + +```powershell +dotnet run --project samples\Robotics\BinPickingCell\BinPickingCell.csproj -- --insecure +dotnet run --project samples\Robotics\BinPickingClient\BinPickingClient.csproj --framework net10.0 -- --server opc.tcp://localhost:62855/BinPickingCell --insecure --demo +``` + +The demo runner: + +1. Discovers the sole Robot Intent controller advertised by the cell and prints + its browse name and NodeId. +2. Requests command authority for this session. +3. Attempts to reach the sole Vision inference pipeline advertised by the cell, + captures a frame's worth of detections, composes the target part's pose from + the `camera_eih` frame into the `world` frame using the Vision frame graph, + and logs the composed pose. +4. Resolves the source location (`--source`, default `Bin`), destination + location (`--destination`, default `Fixture`) and tool (`--tool`, default + `ParallelGripper`) against the controller's `Locations` and `Tools` lookup + tables. +5. Builds and submits a `Pick` intent for the chosen part with a unique + `IntentId`, waits for the returned operation to reach a terminal state, and + logs the result. +6. Builds and submits a `Place` intent for the same part with a unique + `IntentId`, waits for the returned operation to reach a terminal state, and + logs the result. +7. Re-runs inference and compares detections before and after: if the target + part disappears or its pose changed, the world state was observed to change + as expected; otherwise the runner reports the mismatch plainly. + +Use `--part ` to pick a different part (`RedCube`, +`GreenCylinder`, `BlueSphere`, `YellowSlab`, `OrangeBrick`) and +`--source` / `--destination` / `--tool` to retarget without a rebuild. + +Transcript from an actual run against the current cell: + +```text +Controller: BinPickingController (ns=3;s=7001_Controllers_BinPickingController) +Command authority: granted for this session. +Pick admitted: intent binpickclient-20260810191928321-pick operation ns=3;s=7001_Controllers_BinPickingController_Intents_binpickclient-20260810191928321-pick-... +Pick operation terminal state: Succeeded failure=None +Place admitted: intent binpickclient-20260810191928321-place operation ns=3;s=7001_Controllers_BinPickingController_Intents_binpickclient-20260810191928321-place-... +Place operation terminal state: Succeeded failure=None +``` + +### Notes on Vision inference in the current cell + +The client resolves the cell's Vision nodes by browse path from the Vision root, +so a Server that materialises Vision as instances in its own namespace — which +is what the fluent builder produces, and what this cell is — is discovered +correctly. `--demo` verifies the world change through the client-side detection +loop: it captures the detections before the Pick, submits Pick and Place, +re-runs inference, and reports whether the target part is still where it was. + +The check is honest in both directions. If the part was never detected before +the Pick, the run is reported as **inconclusive** rather than as a pass, because +"it is gone now" proves nothing about a part that was never there. + +## Viewport mode + +The viewer is deliberately optional: `BinPickingClient` does not reference it, +and loads `Opc.Ua.OpenUsd.Connector.Viewer.dll` by reflection from its own +output directory. Nothing puts it there for you, so `--view` on a fresh clone +degrades to headless with a message on stderr. Publish the viewer once and copy +its output beside the client: + +```powershell +dotnet publish tools\Opc.Ua.OpenUsd.Connector.Viewer\Opc.Ua.OpenUsd.Connector.Viewer.csproj ` + -c Release -f net10.0 -r win-x64 --self-contained false -o $env:TEMP\viewer-publish + +Copy-Item "$env:TEMP\viewer-publish\*" ` + samples\Robotics\BinPickingClient\bin\Release\net10.0 ` + -Recurse -Force -Exclude "*.deps.json","*.runtimeconfig.json" +``` + +Publish rather than copying the viewer's `bin` directory: the viewport pulls in +the Avalonia UI stack and the per-RID native OpenUSD renderer on top of its own +assembly, and only a publish resolves that whole closure. Excluding the two JSON +files matters as well - they describe the viewer as an application, and letting +them overwrite the client's own leaves the client unable to start. + +Then run, from the client's output directory so the native payload resolves: + +```powershell +samples\Robotics\BinPickingClient\bin\Release\net10.0\BinPickingClient.exe ` + --server opc.tcp://localhost:62855/BinPickingCell --insecure --view +``` + +Add `--demo` to run the scripted pick-and-place while the viewport is open, +which is the quickest way to watch the loop close in 3-D. The two are sequenced +deliberately: the viewport opens first, the client waits for the live OpenUSD +stream to be subscribed, and only then commands the robot, so the motion happens +with something watching. The window stays open when the loop finishes so the +cell can still be inspected. + +Both the arm and the parts move. The six joints follow `AxisState.Position`, and +each part follows its own world position variable under `Server/WorldState`, so a +part that is picked travels with the gripper and stays where it is placed. + +### Where a released part ends up + +A placed part comes to rest on whatever is underneath it — the bench, the fixture +plate, a locating peg, or another part. It is a resting model rather than a +physics engine: no toppling, no friction, no sliding, because none of those change +the answer a pick-and-place cell needs, which is that a part released over a bench +ends up on the bench and a part released over another part ends up on top of it +rather than inside it. + +Two consequences worth knowing: + +- **Stacking is automatic.** Place a second part at the same location and its base + lands exactly on the first one's top. Three parts placed on the fixture measure + bases 0.8380 / 0.8780 / 0.9080 against tops 0.8780 / 0.9080 / 0.9320 — no gaps, + no intersections. +- **A Place descends before it releases.** The cell knows it is carrying something, + because a Pick travels with the gripper empty and closes on arrival while a Place + travels loaded and opens, so it moves to the height that leaves the part on its + support. Releasing is a release, not a drop from the approach height. + +The arm will also refuse to reach through its own bench. Several inverse-kinematic +solutions for a target near the surface pass a link through it, and the solver now +takes the nearest one that does not, reporting `WorkSurface` when every solution +would. + +### Which camera the viewport opens on + +The viewport opens on `/World/ObserverCamera`, a fixed camera authored in the +stage that frames the whole cell from the front and slightly above: the bench +centred, the fixture on the left, the bin on the right, and enough room for the +arm to reach up without leaving the frame. It is a fixed observer, so the arm +moves within a steady view rather than the view chasing the arm. + +- `--camera auto` hands framing back to the viewer, which fits the scene bounds. +- `--camera ` opens on any other camera in the stage. + +The stage has a second camera, +`/World/Robot/Palletizer/.../Flange/Camera`, which is the +eye-in-hand sensor the Vision model renders from. Opening the viewport on it +shows what the tool sees rather than the cell, which is occasionally useful for +debugging the perception path but is not a view of the robot working. + +The observer camera's numbers were fitted to a reference framing and then +corrected against what the viewer actually rendered, because the analytic +placement and the rendered result disagreed. Treat them as measured rather than +derived: change one and re-check the framing against a capture. + +### Diagnosing the live stream + +Pass `--verbose` to raise the log level to Debug. The OpenUSD connector then +reports what it bound at start-up: + +``` +OpenUSD live stream bound 12 binding(s) across 10 representation(s) and is +monitoring 12 item(s). +``` + +and one line per live update, plus a warning for any target it had to leave +*unresolved*. That last one matters: the §5.8 profiles fail closed, so a source +value in a shape a profile does not accept produces a prim that silently never +moves while every subscription counter says the data is flowing. It is worth +knowing which of the two you are looking at. + +### Viewer-only sessions receive external motion + +A client started with `--view` alone can observe motion commanded by a +different OPC UA session. The OpenUSD connector creates a classic +`Subscription`; `ManagedSession` uses the V2 subscription engine by default. +The V2 publish manager therefore includes created classic session subscriptions +when sizing its Publish-worker pool, even when it owns no V2 subscriptions of +its own. This is the normal observer shape: two Publish workers stay active and +dispatch the classic subscription's notifications to the live USD sink. + +The viewer host also runs its long-lived `StageReadyAsync` callback outside +Avalonia's UI synchronization context, per +[openusd-dotnet#17](https://github.com/marcschier/openusd-dotnet/issues/17). +These are separate requirements: the callback must remain runnable, and the OPC +UA session must issue Publish requests for every subscription API it supports. +Pass `--verbose` to verify both `PUBLISH Worker #... - STARTED` and +`OpenUSD live update: ...` messages while another client drives the cell. + +The client fetches the cell's served OpenUSD assets automatically whenever the +viewport opens, into a per-user cache directory. Pass `--fetch-assets ` only +when you want the fetched stage written somewhere you choose, for example to +inspect `stage.usda` by hand. + +Where the viewer assembly or renderer payload is missing, the client says so +plainly on stderr and continues without opening the viewport. The renderer +payload supports `win-x64`, `linux-x64` and `osx-arm64`; substitute the matching +`-r` value above. + +## Agent workflow for vision-guided bin picking + +Use `--mcp --view` when an LLM agent should drive the same cell a human +watches. The perception-to-grasp loop is deliberately short: observe the cell, +request authority explicitly, then let `robotics_vision_pick` run inference, +select one detection and submit the Pick/Place mission on the same OPC UA +session. + +```mermaid +sequenceDiagram + participant Agent as LLM agent + participant Robotics as Robotics tools + participant Cell as BinPickingCell server + + Agent->>Robotics: robotics_list_controllers + Robotics->>Cell: browse Server/RobotIntent/Controllers + Cell-->>Agent: BinPickingController + Agent->>Robotics: robotics_request_control + Robotics->>Cell: RequestControl + Cell-->>Agent: granted + Agent->>Robotics: robotics_vision_pick + Robotics->>Cell: RunInference + Cell-->>Robotics: DetectionResult + provenance + Robotics->>Cell: SubmitMission(Pick, Place) + Cell-->>Agent: selected detection + Mission handle + Agent->>Robotics: robotics_wait_mission + Robotics->>Cell: observe Mission ExecutionState + Cell-->>Agent: terminal Mission result +``` + +The agent's tool sequence to pick and place one part: + +```text +agent -> robotics_list_controllers() +server -> [{ name: "BinPickingController", nodeId: "ns=3;s=7001_Controllers_BinPickingController" }] + +agent -> robotics_read_controller(controller="BinPickingController") +server -> SupportedIntents includes Pick and Place; locations include Bin, + Fixture and per-part staging; tools include ParallelGripper. + +agent -> robotics_read_state(controller="BinPickingController") +server -> Ready=true, OperationalMode=AutomaticExternal, ControlOwner= + +agent -> robotics_request_control(controller="BinPickingController") +server -> { granted: true } + +agent -> robotics_vision_pick(request={ + controller: "BinPickingController", + pipeline: "BinPickingPipeline", + source: "Bin", + tool: "ParallelGripper", + destination: "Fixture", + classLabel: "RedCube", + minimumConfidence: 0.9, + missionId: "place-red-cube" +}) +server -> { + provenance: { resultId: "run-...", selectedDetection: { classLabel: "RedCube", ... } }, + missionSubmission: { accepted: true, missionId: "place-red-cube", operation: "ns=..."} +} + +agent -> robotics_wait_mission( + controller="BinPickingController", + missionId="place-red-cube", + missionNodeId="ns=...", + timeoutMs=30000) +server -> { completed: true, terminalState: "Succeeded", ... } +``` + +If safety refuses the intent, the agent must observe the failure, re-read state, +and either re-plan (for example, retry with a different tool or target +location), ask for operator action, or stop. + +The lower-level `vision_run_inference`, `vision_read_detection_result`, +`robotics_submit_pick` and `robotics_submit_place` tools remain available when +an agent needs to inspect or control each stage separately. + +`--insecure` is for localhost demos only. It accepts any server certificate for +localhost demos; do not use it for production systems. diff --git a/samples/Robotics/BinPickingClient/UsdViewHostLoader.cs b/samples/Robotics/BinPickingClient/UsdViewHostLoader.cs new file mode 100644 index 0000000000..4eb7741ffc --- /dev/null +++ b/samples/Robotics/BinPickingClient/UsdViewHostLoader.cs @@ -0,0 +1,134 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Reflection; +#if NET8_0_OR_GREATER +using System.Runtime.InteropServices; +using System.Runtime.Loader; +#endif +using Opc.Ua.OpenUsd.Client; + +namespace BinPickingClient +{ + /// + /// Reflection-based loader for the optional OpenUSD viewport assembly. Mirrors the + /// shape used by IntentViewerClient so a machine without the native renderer + /// payload gracefully falls back to headless operation with a clear reason. + /// + internal static class UsdViewHostLoader + { + public static bool TryLoad(out IUsdViewHost? host, out string reason) + { + host = null; +#if !NET8_0_OR_GREATER + reason = "Rendering requires a .NET 8 or later target."; + return false; +#else + Assembly assembly; + try + { + assembly = LoadViewerAssembly(); + } + catch (Exception exception) when ( + exception is FileNotFoundException or FileLoadException or BadImageFormatException) + { + reason = + $"The optional '{ViewerAssemblyName}' assembly or its native payload was not found " + + "next to the sample. " + + "Run without --view for headless mode, or publish/run with the matching viewport package."; + return false; + } + + Type? type = assembly.GetType(ViewerTypeName, throwOnError: false); + if (type is null) + { + reason = $"'{ViewerAssemblyName}' does not contain '{ViewerTypeName}'."; + return false; + } + + try + { + if (Activator.CreateInstance(type) is IUsdViewHost loaded) + { + host = loaded; + reason = string.Empty; + return true; + } + } + catch (Exception exception) when (exception is MissingMethodException or TargetInvocationException) + { + reason = $"The viewport could not be created: {exception.Message}"; + return false; + } + + reason = $"'{ViewerTypeName}' does not implement IUsdViewHost."; + return false; +#endif + } + +#if NET8_0_OR_GREATER + private static Assembly LoadViewerAssembly() + { + string path = Path.Combine(AppContext.BaseDirectory, ViewerAssemblyName + ".dll"); + if (!File.Exists(path)) + { + throw new FileNotFoundException("The optional viewport assembly is not installed.", path); + } + + var resolver = new AssemblyDependencyResolver(path); + AssemblyLoadContext.Default.Resolving += (context, name) => + { + string? resolved = resolver.ResolveAssemblyToPath(name); + if (resolved is null || !File.Exists(resolved)) + { + resolved = Path.Combine(AppContext.BaseDirectory, name.Name + ".dll"); + if (!File.Exists(resolved)) + { + return null; + } + } + return context.LoadFromAssemblyPath(resolved); + }; + AssemblyLoadContext.Default.ResolvingUnmanagedDll += (_, unmanaged) => + { + string? resolved = resolver.ResolveUnmanagedDllToPath(unmanaged); + return resolved is not null && File.Exists(resolved) ? NativeLibrary.Load(resolved) : IntPtr.Zero; + }; + return AssemblyLoadContext.Default.LoadFromAssemblyPath(path); + } +#endif + +#if NET8_0_OR_GREATER + private const string ViewerAssemblyName = "Opc.Ua.OpenUsd.Connector.Viewer"; + private const string ViewerTypeName = "Opc.Ua.OpenUsd.Connector.Viewer.OpenUsdViewHost"; +#endif + } +} diff --git a/samples/Robotics/BinPickingClient/app.manifest b/samples/Robotics/BinPickingClient/app.manifest new file mode 100644 index 0000000000..3f657142ca --- /dev/null +++ b/samples/Robotics/BinPickingClient/app.manifest @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/samples/Robotics/IntentEnabledRobot/Assets/Bench.usda b/samples/Robotics/IntentEnabledRobot/Assets/Bench.usda index 5feffcbcc8..7828e58785 100644 --- a/samples/Robotics/IntentEnabledRobot/Assets/Bench.usda +++ b/samples/Robotics/IntentEnabledRobot/Assets/Bench.usda @@ -80,8 +80,11 @@ def Xform "World" double size = 1 color3f[] primvars:displayColor = [(0.54, 0.55, 0.55)] rel material:binding = - double3 xformOp:translate = (0, 0, 0.7800) - double3 xformOp:scale = (1.4000, 0.9000, 0.0400) + # The work surface is z = 0.820, where the arm is referenced: this bench had a + # 0.800 top, so the arm floated 20 mm above it. The underside stays at 0.760 + # where the legs meet, so the thickness carries the correction. + double3 xformOp:translate = (0, 0, 0.7900) + double3 xformOp:scale = (1.4000, 0.9000, 0.0600) uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] } diff --git a/samples/Robotics/IntentEnabledRobot/Assets/arm.usda b/samples/Robotics/IntentEnabledRobot/Assets/arm.usda index 67ec910cef..edc57bf297 100644 --- a/samples/Robotics/IntentEnabledRobot/Assets/arm.usda +++ b/samples/Robotics/IntentEnabledRobot/Assets/arm.usda @@ -195,8 +195,13 @@ def Xform "Arm" ( uniform token axis = "Y" double height = 0.0900 double radius = 0.0520 - color3f[] primvars:displayColor = [(0.16, 0.17, 0.18)] - rel material:binding = + # This is the mechanical bridge from the blue wrist to + # the silver ISO flange. DarkGrey disappears against the + # viewport's black background and makes the attached + # gripper read as a floating object, so keep the bridge + # visible with the same silver used by the flange. + color3f[] primvars:displayColor = [(0.78, 0.79, 0.78)] + rel material:binding = } def Xform "Flange" @@ -219,6 +224,7 @@ def Xform "Arm" ( { double3 xformOp:translate = (0.1200, 0, 0) uniform token[] xformOpOrder = ["xformOp:translate"] + token visibility = "invisible" def Sphere "Marker" ( prepend apiSchemas = ["MaterialBindingAPI"] diff --git a/samples/Robotics/IntentEnabledRobot/Assets/gripper.usda b/samples/Robotics/IntentEnabledRobot/Assets/gripper.usda index 4e9b0f95af..ecb05b6fdf 100644 --- a/samples/Robotics/IntentEnabledRobot/Assets/gripper.usda +++ b/samples/Robotics/IntentEnabledRobot/Assets/gripper.usda @@ -4,7 +4,7 @@ The root prim mounts on the arm flange with +X as the tool approach direction. FingerLeftSlide and FingerRightSlide are the prismatic Xforms a live layer can drive to open and close the jaws; their authored translations show the gripper open at a readable default. /Gripper/Tcp lies between the fingertips and is the tool centre point used by Robot Intent targets. - Every gprim carries both a material binding and a matching primvars:displayColor, so the asset reads correctly on renderers that ignore the material graph. Meshes are doubleSided so face winding cannot cull a surface. Keep the slide prim names and their translate ops stable if the gripper geometry is refined. + Every gprim carries both a material binding and a matching primvars:displayColor, so the asset reads correctly on renderers that ignore the material graph. Meshes are doubleSided so face winding cannot cull a surface. Keep the slide prim names and their translate ops stable if the gripper geometry is refined. The TCP marker is a debugging aid and is invisible by default; applications drive the two slide translates to show the jaw aperture. """ defaultPrim = "Gripper" metersPerUnit = 1 @@ -33,8 +33,8 @@ def Xform "Gripper" ( ) { double size = 1 - color3f[] primvars:displayColor = [(0.14, 0.15, 0.17)] - rel material:binding = + color3f[] primvars:displayColor = [(0.10, 0.38, 0.68)] + rel material:binding = double3 xformOp:translate = (0.0550, 0, 0) double3 xformOp:scale = (0.0450, 0.0550, 0.0320) uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] @@ -76,8 +76,8 @@ def Xform "Gripper" ( ) { double size = 1 - color3f[] primvars:displayColor = [(0.18, 0.19, 0.21)] - rel material:binding = + color3f[] primvars:displayColor = [(0.60, 0.61, 0.64)] + rel material:binding = double3 xformOp:translate = (0.1040, 0, 0) double3 xformOp:scale = (0.0200, 0.0120, 0.0300) uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] @@ -88,10 +88,10 @@ def Xform "Gripper" ( ) { double size = 1 - color3f[] primvars:displayColor = [(0.05, 0.05, 0.06)] - rel material:binding = - double3 xformOp:translate = (0.1550, 0, 0) - double3 xformOp:scale = (0.0520, 0.0100, 0.0220) + color3f[] primvars:displayColor = [(0.95, 0.75, 0.05)] + rel material:binding = + double3 xformOp:translate = (0.1700, 0, 0) + double3 xformOp:scale = (0.0750, 0.0160, 0.0350) uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] } } @@ -106,8 +106,8 @@ def Xform "Gripper" ( ) { double size = 1 - color3f[] primvars:displayColor = [(0.18, 0.19, 0.21)] - rel material:binding = + color3f[] primvars:displayColor = [(0.60, 0.61, 0.64)] + rel material:binding = double3 xformOp:translate = (0.1040, 0, 0) double3 xformOp:scale = (0.0200, 0.0120, 0.0300) uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] @@ -118,10 +118,10 @@ def Xform "Gripper" ( ) { double size = 1 - color3f[] primvars:displayColor = [(0.05, 0.05, 0.06)] - rel material:binding = - double3 xformOp:translate = (0.1550, 0, 0) - double3 xformOp:scale = (0.0520, 0.0100, 0.0220) + color3f[] primvars:displayColor = [(0.95, 0.75, 0.05)] + rel material:binding = + double3 xformOp:translate = (0.1700, 0, 0) + double3 xformOp:scale = (0.0750, 0.0160, 0.0350) uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:scale"] } } @@ -130,6 +130,7 @@ def Xform "Gripper" ( { double3 xformOp:translate = (0.1850, 0, 0) uniform token[] xformOpOrder = ["xformOp:translate"] + token visibility = "invisible" def Sphere "Marker" ( prepend apiSchemas = ["MaterialBindingAPI"] @@ -180,6 +181,19 @@ def Xform "Gripper" ( float inputs:metallic = 0 token outputs:surface } + + def Material "JawYellow" + { + token outputs:surface.connect = + def Shader "Surface" + { + uniform token info:id = "UsdPreviewSurface" + color3f inputs:diffuseColor = (0.95, 0.75, 0.05) + float inputs:roughness = 0.50 + float inputs:metallic = 0 + token outputs:surface + } + } } def Material "TcpBlue" diff --git a/samples/Robotics/IntentEnabledRobot/Kinematics/ISimulatedArmKinematics.cs b/samples/Robotics/IntentEnabledRobot/Kinematics/ISimulatedArmKinematics.cs new file mode 100644 index 0000000000..b3b0c5b848 --- /dev/null +++ b/samples/Robotics/IntentEnabledRobot/Kinematics/ISimulatedArmKinematics.cs @@ -0,0 +1,111 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Diagnostics.CodeAnalysis; +using Opc.Ua; +using Opc.Ua.RobotIntent; + +namespace Robotics.IntentEnabledRobot.Kinematics +{ + /// + /// Kinematics operations consumed by SimulatedArmExecutor. + /// + public interface ISimulatedArmKinematics + { + /// + /// Gets the number of commanded axes. + /// + int AxisCount { get; } + + /// + /// Gets the maximum advertised Cartesian reach in metres. + /// + double MaximumReach { get; } + + /// + /// Gets the configuration the simulated arm starts in. + /// + ArrayOf InitialJointAngles { get; } + + /// + /// Computes the tool and joint-frame poses for one configuration. + /// + SimulatedArmForwardPose Forward(ReadOnlySpan jointAngles); + + /// + /// Gets whether all axes are within their configured limits. + /// + bool IsWithinLimits(ReadOnlySpan jointAngles); + + /// + /// Selects the nearest clear solution, including its joint-space path. + /// + bool TrySelectNearest( + Pose3DDataType target, + ReadOnlySpan currentJointAngles, + [NotNullWhen(true)] out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure); + + /// + /// Selects the nearest clear configuration when the caller samples the path. + /// + bool TrySelectNearestConfiguration( + Pose3DDataType target, + ReadOnlySpan currentJointAngles, + [NotNullWhen(true)] out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure); + + /// + /// Interpolates two joint configurations. + /// + ArrayOf InterpolateJoints( + ReadOnlySpan start, + ReadOnlySpan end, + double fraction); + + /// + /// Interpolates two Cartesian poses. + /// + Pose3DDataType InterpolateCartesian( + Pose3DDataType start, + Pose3DDataType end, + double fraction); + + /// + /// Gets whether every sampled configuration along a joint-space path is clear. + /// + bool ClearsPath(ReadOnlySpan start, ReadOnlySpan target); + + /// + /// Maps a kinematic refusal to the Robot Intent failure model. + /// + IntentFailureEnum MapFailure(SimulatedArmKinematicFailure failure); + } +} diff --git a/samples/Robotics/IntentEnabledRobot/Kinematics/SimulatedArmKinematics.cs b/samples/Robotics/IntentEnabledRobot/Kinematics/SimulatedArmKinematics.cs index 79ab2c42d0..271684f35f 100644 --- a/samples/Robotics/IntentEnabledRobot/Kinematics/SimulatedArmKinematics.cs +++ b/samples/Robotics/IntentEnabledRobot/Kinematics/SimulatedArmKinematics.cs @@ -28,11 +28,12 @@ * ======================================================================*/ using System; -using System.Diagnostics.CodeAnalysis; using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; using System.Globalization; using Opc.Ua; using Opc.Ua.RobotIntent; +using Robotics.IntentEnabledRobot.Simulation; namespace Robotics.IntentEnabledRobot.Kinematics { @@ -59,9 +60,39 @@ public enum SimulatedArmKinematicFailure /// /// Every geometric solution violates at least one joint limit. /// - JointLimit + JointLimit, + + /// + /// Every geometric solution drives a link through the work surface the arm is + /// mounted on. + /// + WorkSurface } + /// + /// Resolves a Location NodeId to a position in the arm's base frame. + /// + /// The Location NodeId carried by a Pick or Place intent. + /// The resolved position (x, y, z) in metres. + /// true when the location is known to the host. + public delegate bool LocationPositionResolver(NodeId location, out ArrayOf position); + + /// + /// Resolves a Location NodeId to a complete tool pose in the arm's base frame. + /// + /// The Location NodeId carried by a Pick or Place intent. + /// The resolved tool pose. + /// true when the location is known to the host. + public delegate bool LocationPoseResolver(NodeId location, out Pose3DDataType pose); + + /// + /// Resolves a named workpiece at a Pick source to a tool pose. + /// + public delegate bool PickPoseResolver( + NodeId source, + string objectClass, + out Pose3DDataType pose); + /// /// One inverse-kinematic solution for the simulated arm. /// @@ -258,8 +289,17 @@ public double VelocityAt(double elapsedSeconds) /// /// Kinematics and path helpers for the UR5e-style sample arm. /// - public sealed class SimulatedArmKinematics + public sealed class SimulatedArmKinematics : ISimulatedArmKinematics { + /// + public int AxisCount => JointCount; + + /// + public double MaximumReach => Reach; + + /// + public ArrayOf InitialJointAngles => ArrayOf.Create(s_initialJointAngles.AsSpan()); + /// /// Initializes kinematics with the sample joint limits. /// @@ -316,6 +356,7 @@ public SimulatedArmForwardPose Forward(ArrayOf jointAngles) /// /// Computes all valid inverse-kinematic solutions found from the eight UR-style branches. /// + /// public SimulatedArmIkResult Inverse(Pose3DDataType target, ReadOnlySpan referenceJointAngles) { if (target is null) @@ -393,16 +434,250 @@ public bool TrySelectNearest( ReadOnlySpan currentJointAngles, [NotNullWhen(true)] out SimulatedArmIkSolution? solution, out SimulatedArmKinematicFailure failure) + { + return TrySelectNearestCore( + target, + currentJointAngles, + requireClearJointPath: true, + out solution, + out failure); + } + + /// + /// Selects the nearest clear configuration without testing a joint-space path to + /// it. + /// + /// + /// Used while following an explicitly sampled Cartesian path. Each sample is checked + /// for collision, and selecting the nearest solution to the previous sample keeps + /// the branch continuous. Testing a second, joint-interpolated path between those + /// samples rejects valid Cartesian motion and is the wrong path to validate. + /// + public bool TrySelectNearestConfiguration( + Pose3DDataType target, + ReadOnlySpan currentJointAngles, + [NotNullWhen(true)] out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure) + { + return TrySelectNearestCore( + target, + currentJointAngles, + requireClearJointPath: false, + out solution, + out failure); + } + + private bool TrySelectNearestCore( + Pose3DDataType target, + ReadOnlySpan currentJointAngles, + bool requireClearJointPath, + [NotNullWhen(true)] out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure) { SimulatedArmIkResult result = Inverse(target, currentJointAngles); failure = result.Failure; - solution = result.Solutions.IsEmpty ? null : result.Solutions[0]; - return solution is not null; + solution = null; + if (result.Solutions.IsEmpty) + { + return false; + } + + // Solutions come back nearest-first. Take the nearest one that neither reaches + // through the surface the arm stands on nor into the cell's furniture, and that + // can be reached without sweeping a link through either when the caller asks us + // to validate a joint-space path. + // + // Ordering these by WristInversionPenalty first - so a shape that holds the + // wrist the right way up wins over a closer one that doubles it back - was + // tried and measured worse: the loop failed on its third operation against nine + // of ten without it. Choosing a different solution changes where the arm starts + // the next move from, and the tidier shape led into dead ends. The penalty is + // kept because it names what is wrong with the posture in the close-ups, but + // preferring it needs the approach and retract poses solved for properly first, + // so that a tidy choice does not strand the next one. + ReadOnlySpan candidates = result.Solutions.Span; + for (int ii = 0; ii < candidates.Length; ii++) + { + if (ClearsWorkSurface(candidates[ii].JointAngles.Span) && + (!requireClearJointPath || + ClearsPath(currentJointAngles, candidates[ii].JointAngles.Span))) + { + solution = candidates[ii]; + return true; + } + } + + // Refusing is the honest answer. Returning the first solution anyway would put + // a link through the bench, and a move that cannot be made without doing that + // is one the arm should decline rather than mime. + failure = SimulatedArmKinematicFailure.WorkSurface; + return false; + } + + /// + /// Gets how far a configuration has the wrist the wrong way up. + /// + /// + /// With the tool pointing down the chain should descend from the wrist to the part: + /// J4 above J5 above J6 above the tool centre point. A configuration that climbs + /// instead has doubled the wrist back over itself, which reaches the same point and + /// looks like a fault. Counting the steps that climb gives an order to prefer + /// between candidates that are otherwise all legal, and zero means the wrist hangs + /// the way a person would expect. + /// + /// + /// The configuration to score, in radians. + /// + public int WristInversionPenalty(ReadOnlySpan jointAngles) + { + SimulatedArmForwardPose pose = Forward(jointAngles); + ReadOnlySpan frames = pose.JointFramePoses.Span; + if (frames.Length < JointCount) + { + return 0; + } + double toolZ = pose.ToolPose.Position.Span[2]; + int penalty = 0; + for (int ii = 3; ii < JointCount - 1; ii++) + { + if (frames[ii + 1].Position.Span[2] > frames[ii].Position.Span[2]) + { + penalty++; + } + } + if (toolZ > frames[JointCount - 1].Position.Span[2]) + { + penalty++; + } + return penalty; + } + + /// + /// Gets or sets the lowest height, in the arm's own base frame, that any joint + /// origin may occupy. Defaults to no constraint. + /// + /// + /// An arm bolted to a bench has the bench at zero in its base frame, so a host that + /// mounts it that way sets this to zero and the solver stops handing back poses + /// that pass through the work surface. + /// + public double MinimumLinkHeight { get; set; } = double.NegativeInfinity; + + /// + /// Gets or sets the solids the arm must not move through. Defaults to none. + /// + /// + /// only knows about a horizontal plane and only + /// samples joint origins, so it cannot see a link crossing the middle of a bench, + /// and it does not know the bin or the fixture are there at all. A host that + /// describes its furniture here gets configurations refused for reaching into any + /// of it. + /// + public SimulatedCollisionModel? Collisions { get; set; } + + /// + /// Gets a value indicating whether a configuration keeps the whole arm out of the + /// work surface and out of every declared obstacle. + /// + /// + /// The configuration to test, in radians. + /// + /// + /// true when no part of the arm reaches below the work surface or inside a + /// solid. + /// + public bool ClearsWorkSurface(ReadOnlySpan jointAngles) + { + if (double.IsNegativeInfinity(MinimumLinkHeight) && Collisions == null) + { + return true; + } + SimulatedArmForwardPose pose = Forward(jointAngles); + ReadOnlySpan frames = pose.JointFramePoses.Span; + if (!double.IsNegativeInfinity(MinimumLinkHeight)) + { + for (int ii = 0; ii < frames.Length; ii++) + { + if (frames[ii].Position.Span[2] < MinimumLinkHeight) + { + return false; + } + } + if (pose.ToolPose.Position.Span[2] < MinimumLinkHeight) + { + return false; + } + } + if (Collisions == null) + { + return true; + } + + // The chain starts at the first joint origin, not at the base: the arm is + // bolted to the bench, so a capsule around the pedestal is inside the bench by + // construction and would refuse every configuration there is. The tool point + // closes the chain past the flange - a wrist that clears everything while the + // gripper is buried in the bin is not a configuration the arm can hold. + Span points = stackalloc double[(frames.Length + 1) * 3]; + for (int ii = 0; ii < frames.Length; ii++) + { + ReadOnlySpan position = frames[ii].Position.Span; + points[(ii * 3) + 0] = position[0]; + points[(ii * 3) + 1] = position[1]; + points[(ii * 3) + 2] = position[2]; + } + ReadOnlySpan tool = pose.ToolPose.Position.Span; + points[(frames.Length * 3) + 0] = tool[0]; + points[(frames.Length * 3) + 1] = tool[1]; + points[(frames.Length * 3) + 2] = tool[2]; + return Collisions.IsClear(points, out _); + } + + /// + /// Gets a value indicating whether every configuration along a joint-space move + /// stays clear. + /// + /// + /// Filtering the goal alone is not enough: the arm travels by interpolating from + /// where it is to where it is going, so a start and a goal that both clear the + /// bench can still be joined by a path that sweeps a link straight through it. + /// + /// + /// The configuration the move starts from, in radians. + /// + /// + /// The configuration the move ends at, in radians. + /// + public bool ClearsPath(ReadOnlySpan start, ReadOnlySpan target) + { + // Only checked when a host has described its furniture. The height plane alone + // is too blunt for a path: a swing from one side of the cell to the other dips + // a link below the plane part-way round almost every time, so enforcing it here + // refuses ordinary moves and the arm stops rather than travels. + if (Collisions == null) + { + return true; + } + Span configuration = stackalloc double[start.Length]; + for (int step = 0; step <= PathSampleCount; step++) + { + double fraction = (double)step / PathSampleCount; + for (int ii = 0; ii < start.Length && ii < target.Length; ii++) + { + configuration[ii] = start[ii] + ((target[ii] - start[ii]) * fraction); + } + if (!ClearsWorkSurface(configuration)) + { + return false; + } + } + return true; } /// /// Interpolates between poses with straight-line position and spherical-linear orientation. /// + /// public Pose3DDataType InterpolateCartesian(Pose3DDataType start, Pose3DDataType end, double fraction) { if (start is null) @@ -463,6 +738,12 @@ public static IntentFailureEnum ToIntentFailure(SimulatedArmKinematicFailure fai }; } + /// + public IntentFailureEnum MapFailure(SimulatedArmKinematicFailure failure) + { + return ToIntentFailure(failure); + } + /// /// Gets a value indicating whether the joint vector is within configured limits. /// @@ -574,10 +855,10 @@ private bool TryRefine(Pose3DDataType target, ReadOnlySpan seed, out dou for (int iteration = 0; iteration < 80; iteration++) { ComputeError(target, solution, error); - if (Math.Sqrt((error[0] * error[0]) + (error[1] * error[1]) + (error[2] * error[2])) - < PositionTolerance && - Math.Sqrt((error[3] * error[3]) + (error[4] * error[4]) + (error[5] * error[5])) - < OrientationTolerance) + if (Math.Sqrt((error[0] * error[0]) + (error[1] * error[1]) + (error[2] * error[2])) < + PositionTolerance && + Math.Sqrt((error[3] * error[3]) + (error[4] * error[4]) + (error[5] * error[5])) < + OrientationTolerance) { return !IsWristSingular(solution); } @@ -785,16 +1066,6 @@ private static double Norm(double x, double y, double z) return Math.Sqrt((x * x) + (y * y) + (z * z)); } - private static double SquaredNorm(ReadOnlySpan values) - { - double sum = 0.0; - for (int i = 0; i < values.Length; i++) - { - sum += values[i] * values[i]; - } - return sum; - } - private static double MaxAbsoluteDifference(ReadOnlySpan left, ReadOnlySpan right) { double max = 0.0; @@ -832,6 +1103,7 @@ private static void RequireJointCount(ReadOnlySpan jointAngles) private const double D5 = 0.0997; private const double D6 = 0.0996; private const double FlangeToTcp = 0.165; + private const int PathSampleCount = 24; private const double PositionTolerance = 1e-5; private const double OrientationTolerance = 1e-5; private const double SingularityTolerance = 1e-4; @@ -857,6 +1129,15 @@ private static void RequireJointCount(ReadOnlySpan jointAngles) 2.0 * Math.PI ]; + /// + /// Eight UR-style branches: elbow up and down, wrist flipped, shoulder forward and + /// back. Sixteen were tried - the extra eight starting J2 and J4 in other basins to + /// look for a less contorted shape near the base - and measured: they raised the + /// distinct-posture count but every shape they added was refused by clearance, so + /// the number of *usable* postures at the home slots did not move. They were dropped + /// again because a seed costs a full Newton refinement on every solve, and this + /// solver runs per step of a Cartesian move. + /// private static readonly double[][] s_seedTemplates = [ [0.0, -1.2, 1.4, -1.7, 0.8, 0.0], @@ -869,6 +1150,9 @@ private static void RequireJointCount(ReadOnlySpan jointAngles) [Math.PI, 0.4, -1.4, 1.0, -0.8, Math.PI] ]; + private static readonly double[] s_initialJointAngles = + [-3.0484844, 0.3128706, 0.8261335, 2.0025887, -2.7856466, -1.5707963]; + private readonly double[] m_minimumLimits; private readonly double[] m_maximumLimits; diff --git a/samples/Robotics/IntentEnabledRobot/README.md b/samples/Robotics/IntentEnabledRobot/README.md index ffa1b2ccf0..aa489e1750 100644 --- a/samples/Robotics/IntentEnabledRobot/README.md +++ b/samples/Robotics/IntentEnabledRobot/README.md @@ -4,6 +4,11 @@ This sample is a small OPC UA Robot Intent server for one stationary UR5e-style `Server/RobotIntent/Controllers/UR5eIntentController`, the controller's frames, tools, locations, axes, outputs, programs, description, safety state and OpenUSD representation. +This sample intentionally retains the UR-style six-axis arm and numerical IK. The +[`BinPickingCell`](../BinPickingCell) sample injects a separate analytic four-axis +palletizer through the shared `ISimulatedArmKinematics` executor seam; changing the +bin-picking robot does not change this standalone sample's model or behavior. + ## Run ```powershell diff --git a/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedArmExecutor.cs b/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedArmExecutor.cs index 127b7c4567..1779dbdfaa 100644 --- a/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedArmExecutor.cs +++ b/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedArmExecutor.cs @@ -28,7 +28,9 @@ * ======================================================================*/ using System; +using System.Collections.Generic; using System.Diagnostics.CodeAnalysis; +using System.Globalization; using System.Threading; using System.Threading.Tasks; using Opc.Ua; @@ -53,7 +55,8 @@ public SimulatedArmSnapshot( bool hasObject, string toolName, ArrayOf heldPartPosition, - ArrayOf stackSlotsFilled) + ArrayOf stackSlotsFilled, + string heldObjectClass = "") { JointAngles = jointAngles; ToolPose = toolPose; @@ -63,6 +66,7 @@ public SimulatedArmSnapshot( ToolName = toolName; HeldPartPosition = heldPartPosition; StackSlotsFilled = stackSlotsFilled; + HeldObjectClass = heldObjectClass; } /// @@ -100,6 +104,13 @@ public SimulatedArmSnapshot( /// public ArrayOf HeldPartPosition { get; } + /// + /// Gets the class label of the object the gripper carries, empty when it carries + /// nothing. says that something is held; this says what, + /// which is what a host needs to move the right item in its own world model. + /// + public string HeldObjectClass { get; } + /// /// Gets a value for each pallet slot indicating whether that slot has been filled. /// @@ -155,13 +166,36 @@ public SimulatedArmExecutor(SimulatedArmKinematics kinematics) { } + /// + /// Initializes a simulated executor over a kinematics provider. + /// + public SimulatedArmExecutor(ISimulatedArmKinematics kinematics) + : this(kinematics, RealTimeSimulatedArmClock.Shared) + { + } + /// /// Initializes a simulated executor. /// public SimulatedArmExecutor(SimulatedArmKinematics kinematics, ISimulatedArmClock clock) + : this((ISimulatedArmKinematics)kinematics, clock) + { + } + + /// + /// Initializes a simulated executor over a kinematics provider. + /// + public SimulatedArmExecutor(ISimulatedArmKinematics kinematics, ISimulatedArmClock clock) { m_kinematics = kinematics ?? throw new ArgumentNullException(nameof(kinematics)); m_clock = clock ?? throw new ArgumentNullException(nameof(clock)); + m_jointAngles = m_kinematics.InitialJointAngles.Span.ToArray(); + if (m_jointAngles.Length != m_kinematics.AxisCount) + { + throw new ArgumentException( + "The initial joint configuration does not match the kinematics axis count.", + nameof(kinematics)); + } PublishCurrentPoseLocked(); } @@ -170,6 +204,62 @@ public SimulatedArmExecutor(SimulatedArmKinematics kinematics, ISimulatedArmCloc /// public event EventHandler? SnapshotChanged; + /// + /// Gets how far below the tool centre point a grasped part hangs. + /// + /// + /// A host that decides where a part should come to rest needs this to work back + /// from the part's resting height to the tool pose that leaves it there. + /// + public const double HeldPartTcpOffset = 0.035; + + /// + /// Resolves a Location NodeId to the position, in this arm's base frame, that a + /// Pick or Place should travel to before actuating the gripper. Optional: leave it + /// unset and both intents actuate the gripper where the arm already stands. + /// + public LocationPositionResolver? ResolveLocationPosition { get; set; } + + /// + /// Resolves a Location NodeId to a complete tool pose. When set, this takes + /// precedence over . + /// + /// + /// A position-only resolver makes every move inherit whatever orientation the + /// previous one left behind. A cell that knows how its tool should approach a bin + /// or fixture supplies this instead, so one grasp cannot strand the next move at a + /// yaw the arm cannot retract from. + /// + public LocationPoseResolver? ResolveLocationPose { get; set; } + + /// + /// Resolves a Pick to the current pose of the selected workpiece. + /// + public PickPoseResolver? ResolvePickPose { get; set; } + + /// + /// Notifies the host after a Pick attempt finishes, whether it succeeded or failed. + /// + public Action? PickAttemptFinished { get; set; } + + /// + /// Gets or sets a host decision on whether the final approach to a Location should + /// be a straight Cartesian descent. Optional: when unset, the executor uses a + /// collision-checked joint path. + /// + /// + /// A cell knows the difference between descending inside an open bin and descending + /// onto a fixture. The shared executor does not. Letting the host state that + /// difference keeps cell coordinates out of this type while allowing a vertical + /// descent where crossing a bin wall on a joint interpolation would be wrong. + /// + public Func? PreferCartesianDescent { get; set; } + + /// + /// Gets or sets an optional diagnostic sink for travel planning decisions. + /// + internal Action? Diagnostic { get; set; } + /// /// Gets the latest observable state without exposing synchronization primitives. /// @@ -185,6 +275,16 @@ public async ValueTask ExecuteAsync( throw new ArgumentNullException(nameof(execution)); } + if (execution.Intent is JointMoveIntentDataType + or LinearMoveIntentDataType + or CircularMoveIntentDataType + or TrajectoryIntentDataType + or CartesianPathIntentDataType + or ForceIntentDataType) + { + InvalidateRecordedApproach(); + } + return execution.Intent switch { JointMoveIntentDataType joint => await ExecuteJointMoveAsync(joint, execution, cancellationToken) @@ -240,7 +340,7 @@ private async ValueTask ExecuteJointMoveAsync( ArrayOf target; if (intent.HasJointTargets) { - if (intent.JointTargets.Count != SimulatedArmKinematics.JointCount || + if (intent.JointTargets.Count != m_kinematics.AxisCount || !m_kinematics.IsWithinLimits(intent.JointTargets.Span)) { return IntentOutcome.Fail( @@ -259,7 +359,7 @@ private async ValueTask ExecuteJointMoveAsync( else { return IntentOutcome.Fail( - SimulatedArmKinematics.ToIntentFailure(failure), "The target pose cannot be reached."); + m_kinematics.MapFailure(failure), "The target pose cannot be reached."); } double distance = JointDistance(start, target.Span); @@ -308,7 +408,7 @@ private async ValueTask ExecuteCircularMoveAsync( fraction => { Pose3DDataType pose = InterpolateArc(start, via, intent.Target, fraction); - if (m_kinematics.TrySelectNearest( + if (m_kinematics.TrySelectNearestConfiguration( pose, CurrentSnapshot.JointAngles.Span, out SimulatedArmIkSolution? solution, @@ -327,7 +427,7 @@ private async ValueTask ExecuteCircularMoveAsync( if (moveFailure != SimulatedArmKinematicFailure.None) { return IntentOutcome.Fail( - SimulatedArmKinematics.ToIntentFailure(moveFailure), "The circular path is not feasible."); + m_kinematics.MapFailure(moveFailure), "The circular path is not feasible."); } return outcome.State == ExecutionStateEnum.Succeeded ? IntentOutcome.SucceededAt(CurrentSnapshot.ToolPose) @@ -425,7 +525,7 @@ private async ValueTask ExecuteForceAsync( double distance = fraction * intent.MaxDistance; Pose3DDataType pose = TranslatePose(start, directionX, directionY, directionZ, distance); contacted = IsContact(pose.Position.Span); - if (m_kinematics.TrySelectNearest( + if (m_kinematics.TrySelectNearestConfiguration( pose, CurrentSnapshot.JointAngles.Span, out SimulatedArmIkSolution? solution, @@ -448,7 +548,7 @@ private async ValueTask ExecuteForceAsync( if (moveFailure != SimulatedArmKinematicFailure.None) { return IntentOutcome.Fail( - SimulatedArmKinematics.ToIntentFailure(moveFailure), "The force path is not feasible."); + m_kinematics.MapFailure(moveFailure), "The force path is not feasible."); } return contacted ? IntentOutcome.SucceededAt(CurrentSnapshot.ToolPose) @@ -459,7 +559,8 @@ private async ValueTask ExecuteForceAsync( private async ValueTask ExecuteGraspAsync( GraspIntentDataType intent, IntentExecution execution, - CancellationToken cancellationToken) + CancellationToken cancellationToken, + string objectClass = "") { SetNonCancellable(execution.IntentId); try @@ -471,6 +572,7 @@ private async ValueTask ExecuteGraspAsync( lock (m_lock) { m_hasObject = true; + m_heldObjectClass = objectClass ?? string.Empty; PublishCurrentPoseLocked(); } SnapshotChanged?.Invoke(this, CurrentSnapshot); @@ -507,6 +609,7 @@ private async ValueTask ExecuteReleaseAsync( FillNextStackSlotLocked(); } m_hasObject = false; + m_heldObjectClass = string.Empty; PublishCurrentPoseLocked(); } SnapshotChanged?.Invoke(this, CurrentSnapshot); @@ -518,11 +621,36 @@ private async ValueTask ExecutePickAsync( IntentExecution execution, CancellationToken cancellationToken) { - await m_clock.DelayAsync(TimeSpan.FromMilliseconds(80), cancellationToken).ConfigureAwait(false); - return await ExecuteGraspAsync( - new GraspIntentDataType { Force = intent.Force, Width = GripperClosed, Tool = intent.Tool }, - execution, - cancellationToken).ConfigureAwait(false); + string objectClass = intent.ObjectClass ?? string.Empty; + try + { + await m_clock.DelayAsync(TimeSpan.FromMilliseconds(80), cancellationToken).ConfigureAwait(false); + if (!await MoveToLocationAsync( + intent.Source, + execution, + cancellationToken, + objectClass).ConfigureAwait(false)) + { + return Unreachable("Pick"); + } + IntentOutcome grasp = await ExecuteGraspAsync( + new GraspIntentDataType { Force = intent.Force, Width = GripperClosed, Tool = intent.Tool }, + execution, + cancellationToken, + objectClass).ConfigureAwait(false); + if (grasp.State != ExecutionStateEnum.Succeeded) + { + return grasp; + } + return await RetractAfterActionAsync( + "Pick", + execution, + cancellationToken).ConfigureAwait(false); + } + finally + { + PickAttemptFinished?.Invoke(objectClass); + } } private async ValueTask ExecutePlaceAsync( @@ -531,8 +659,663 @@ private async ValueTask ExecutePlaceAsync( CancellationToken cancellationToken) { await m_clock.DelayAsync(TimeSpan.FromMilliseconds(80), cancellationToken).ConfigureAwait(false); - return await ExecuteReleaseAsync( + if (!await MoveToLocationAsync(intent.Destination, execution, cancellationToken) + .ConfigureAwait(false)) + { + return Unreachable("Place"); + } + IntentOutcome release = await ExecuteReleaseAsync( new ReleaseIntentDataType(), execution, cancellationToken).ConfigureAwait(false); + if (release.State != ExecutionStateEnum.Succeeded) + { + return release; + } + return await RetractAfterActionAsync( + "Place", + execution, + cancellationToken).ConfigureAwait(false); + } + + private async ValueTask RetractAfterActionAsync( + string action, + IntentExecution execution, + CancellationToken cancellationToken) + { + if (await RetractFromLastApproachAsync(execution, cancellationToken).ConfigureAwait(false)) + { + return IntentOutcome.Success; + } + return IntentOutcome.Fail( + IntentFailureEnum.Unreachable, + action + " completed its tool action but could not reverse the local approach."); + } + + /// + /// Reports an intent that could not be carried out because the arm could not get to + /// the Location it named. + /// + private IntentOutcome Unreachable(string what) + { + return IntentOutcome.Fail( + IntentFailureEnum.Unreachable, + what + " could not reach the Location: " + LastTravelFailure); + } + + /// + /// Travels to the approach position a host resolved for a Location, so a Pick or a + /// Place is a move followed by a gripper action rather than a gripper action alone. + /// + /// + /// A Location arrives as a NodeId, which this executor cannot resolve on its own - + /// it has no address space - so a host that knows its own cell supplies + /// . The tool keeps its current orientation and + /// only the position changes, because the current orientation belongs to a + /// configuration the arm is already in and so keeps the inverse-kinematic solve + /// well conditioned. + /// + /// A host with no resolver gets the gripper action where the arm stands: it has not + /// told the executor where anything is, so there is nothing to travel to. But a + /// Location that is resolved and cannot be reached fails the intent. It used + /// to be best effort - the arm would stay where it was and close the gripper anyway, + /// reporting success - which reads as "picked from the fixture" while the tool is + /// still over the bin. A pick that never went anywhere is not a pick, and saying so + /// is what lets a caller notice. + /// + /// + /// + /// true when the arm reached the Location, or when no resolver is configured. + /// + private async ValueTask MoveToLocationAsync( + NodeId location, + IntentExecution execution, + CancellationToken cancellationToken, + string objectClass = "") + { + if (location.IsNull) + { + return true; + } + if (!await RetractFromLastApproachAsync(execution, cancellationToken).ConfigureAwait(false)) + { + LastTravelFailure = "could not reverse the last local approach"; + Diagnostic?.Invoke(LastTravelFailure); + return false; + } + Pose3DDataType current = CurrentSnapshot.ToolPose; + ArrayOf position; + ArrayOf orientation; + string frameId; + if (objectClass.Length > 0 && + ResolvePickPose != null && + ResolvePickPose(location, objectClass, out Pose3DDataType pickPose)) + { + position = pickPose.Position; + orientation = pickPose.Orientation; + frameId = pickPose.FrameId ?? current.FrameId ?? string.Empty; + } + else if (ResolveLocationPose != null && + ResolveLocationPose(location, out Pose3DDataType resolvedPose)) + { + position = resolvedPose.Position; + orientation = resolvedPose.Orientation; + frameId = resolvedPose.FrameId ?? current.FrameId ?? string.Empty; + } + else if (ResolveLocationPosition != null && + ResolveLocationPosition(location, out position)) + { + orientation = current.Orientation; + frameId = current.FrameId ?? string.Empty; + } + else + { + return true; + } + if (position.Count < 3 || orientation.Count < 4) + { + return true; + } + ReadOnlySpan targetPosition = position.Span; + ReadOnlySpan currentPosition = current.Position.Span; + ArrayOf currentOrientation = current.Orientation; + bool sameWorkPosition = + Distance2D(currentPosition, targetPosition) <= SameLocationToleranceMetres; + + // Travel over the cell rather than straight at the target. A single joint-space + // move interpolates between two configurations, and the straight line between + // "over the bin" and "over the fixture" dips: the arm sweeps a link through the + // bench on the way, which is what makes it look like it is passing through the + // table even when both ends of the move are clear. Lifting to a transit height, + // crossing, and descending is both how a real cell moves and a set of legs whose + // straight-line paths stay clear. + double transitZ = Math.Max( + Math.Max(currentPosition[2], targetPosition[2]), TransitHeightMetres); + double[] lift = [currentPosition[0], currentPosition[1], transitZ]; + double[] cross = [targetPosition[0], targetPosition[1], transitZ]; + double[] descend = [targetPosition[0], targetPosition[1], targetPosition[2]]; + // An empty gripper picking from any work area should come straight down on the + // object. A loaded gripper placing onto the fixture may need the short, + // collision-checked joint approach instead. The host preference still marks + // bin/home locations as vertical for both directions. + bool preferCartesianDescent = !CurrentSnapshot.HasObject || + PreferCartesianDescent?.Invoke(location) == true; + Diagnostic?.Invoke(string.Create( + CultureInfo.InvariantCulture, + $"start=({currentPosition[0]:F3},{currentPosition[1]:F3},{currentPosition[2]:F3}) " + + $"target=({targetPosition[0]:F3},{targetPosition[1]:F3},{targetPosition[2]:F3}) " + + $"cartesianDescent={preferCartesianDescent}")); + + if (sameWorkPosition) + { + Diagnostic?.Invoke("target is at the current work position; skipping cross-cell traverse"); + bool localDescent = await MovePlannedCartesianAsync( + descend, + frameId, + orientation, + execution, + recordRetractPath: true, + cancellationToken).ConfigureAwait(false); + if (!localDescent) + { + LastTravelFailure = "could not descend at the current work position"; + Diagnostic?.Invoke(LastTravelFailure); + } + return localDescent; + } + + // Lift in Cartesian space so the tool moves vertically away from the work. + // Cross in joint space: a straight Cartesian line between the bin and fixture + // passes over the base, where the tool sits on the shoulder axis and the inverse + // kinematics are singular. Descend on a collision-checked joint path too: the + // final work pose is reachable, but re-solving every point on the vertical line + // can switch branches and reject the leg before it gets there. Raising the + // pedestal, lowering the bench and moving both work areas out leaves 9-13 clear + // candidates at the transit poses, so the arm can retract and traverse instead + // of making one sweep through the table. + bool retracted = await MoveToolToAsync( + lift, frameId, currentOrientation, execution, cancellationToken).ConfigureAwait(false); + if (!retracted) + { + // A low fixture pose can be reachable while the numerical solver has no + // continuous Cartesian branch straight above it. The destination is still + // a clear pose, so try the collision-checked joint path before refusing the + // Pick or Place. + retracted = await SwingToAsync( + lift, frameId, currentOrientation, execution, cancellationToken).ConfigureAwait(false); + } + if (!retracted) + { + LastTravelFailure = "could not retract vertically from the work"; + Diagnostic?.Invoke(LastTravelFailure); + return false; + } + Diagnostic?.Invoke("retracted to the clear height"); + if (!await SwingToAsync(cross, frameId, orientation, execution, cancellationToken) + .ConfigureAwait(false)) + { + Diagnostic?.Invoke("direct clear-height traverse unavailable; trying bypass"); + // The direct interpolation between work areas on opposite sides can sweep + // through the shoulder axis even though both ends are clear. Route around + // it at the same safe height, the way a motion planner would choose a + // waypoint around a keep-out cylinder. + double[] nearBypass = [-TransitBypassXMetres, TransitBypassYMetres, transitZ]; + double[] farBypass = [TransitBypassXMetres, TransitBypassYMetres, transitZ]; + bool bypassed = await SwingToAsync( + nearBypass, frameId, orientation, execution, cancellationToken).ConfigureAwait(false); + if (!bypassed) + { + bypassed = await MoveToolToAsync( + nearBypass, frameId, orientation, execution, cancellationToken).ConfigureAwait(false); + } + Diagnostic?.Invoke(bypassed + ? "reached the near clear-height bypass" + : "could not reach the near clear-height bypass"); + if (bypassed) + { + bypassed = await SwingToAsync( + farBypass, frameId, orientation, execution, cancellationToken).ConfigureAwait(false); + if (!bypassed) + { + bypassed = await MoveToolToAsync( + farBypass, frameId, orientation, execution, cancellationToken).ConfigureAwait(false); + } + Diagnostic?.Invoke(bypassed + ? "reached the far clear-height bypass" + : "could not reach the far clear-height bypass"); + } + if (bypassed) + { + bypassed = await SwingToAsync( + cross, frameId, orientation, execution, cancellationToken).ConfigureAwait(false); + if (!bypassed) + { + bypassed = await MoveToolToAsync( + cross, frameId, orientation, execution, cancellationToken).ConfigureAwait(false); + } + Diagnostic?.Invoke(bypassed + ? "reached the far side from the bypass arc" + : "could not reach the far side from the bypass arc"); + } + if (!bypassed) + { + LastTravelFailure = "could not traverse the cell around the base at the clear height"; + Diagnostic?.Invoke(LastTravelFailure); + return false; + } + } + Diagnostic?.Invoke("traversed the cell at the clear height"); + bool descended; + if (preferCartesianDescent) + { + descended = await MovePlannedCartesianAsync( + descend, + frameId, + orientation, + execution, + recordRetractPath: true, + cancellationToken).ConfigureAwait(false); + } + else + { + // First establish a configuration locally above the fixture. Solving the + // final pose from the far-side transit configuration can leave no connected + // joint interpolation even though the pose itself has several clear + // solutions. From 40 mm above, the short descent has the nearby branch a + // real approach motion needs without standing so high above a tall stack + // that the final branch disconnects again. + double[] preApproach = + [descend[0], descend[1], descend[2] + LocalApproachHeightMetres]; + descended = await MoveToolToAsync( + preApproach, frameId, orientation, execution, cancellationToken) + .ConfigureAwait(false); + if (descended) + { + double[] retractJointAngles = GetJoints(); + descended = await MovePlannedCartesianAsync( + descend, + frameId, + orientation, + execution, + recordRetractPath: true, + cancellationToken) + .ConfigureAwait(false); + if (!descended) + { + descended = await SwingToAsync( + descend, frameId, orientation, execution, cancellationToken) + .ConfigureAwait(false); + if (descended) + { + m_retractJointAngles = retractJointAngles; + m_retractJointEndpoint = GetJoints(); + } + } + } + } + if (!descended) + { + // The chosen path can still be unavailable because the numerical IK solver + // switches branches along a Cartesian line, or because the direct joint + // interpolation sweeps a link through an obstacle. Try the other path from + // wherever the first attempt stopped before refusing the intent. + descended = preferCartesianDescent + ? await SwingToAsync(descend, frameId, orientation, execution, cancellationToken) + .ConfigureAwait(false) + : await MoveToolToAsync( + descend, + frameId, + orientation, + execution, + cancellationToken, + recordRetractPath: true) + .ConfigureAwait(false); + } + if (!descended) + { + LastTravelFailure = "could not descend onto the work from the clear height"; + Diagnostic?.Invoke(LastTravelFailure); + return false; + } + Diagnostic?.Invoke("descended onto the work"); + return true; + } + + /// + /// Replays the reverse of the last short fixture approach. + /// + /// + /// The final fixture pose may require the yaw search to choose an orientation from + /// which IK cannot independently solve a vertical retract. The path into that pose + /// was already collision checked, so retaining its start configuration gives an + /// exact, deterministic path back out instead of asking the numerical solver to + /// rediscover one after the gripper action. + /// + private async ValueTask RetractFromLastApproachAsync( + IntentExecution execution, + CancellationToken cancellationToken) + { + List? sampledPath = m_retractCartesianPath; + if (sampledPath != null) + { + m_retractCartesianPath = null; + double[] current = GetJoints(); + if (!SameJointConfiguration(current, sampledPath[^1])) + { + Diagnostic?.Invoke("discarded a stale Cartesian retract path"); + return true; + } + double[] previous = current; + for (int ii = sampledPath.Count - 2; ii >= 0; ii--) + { + cancellationToken.ThrowIfCancellationRequested(); + double[] next = sampledPath[ii]; + if (CurrentSnapshot.HasObject && + !m_kinematics.ClearsPath(previous, next)) + { + return false; + } + SetJoints(next); + previous = next; + if (!await DelayTickAsync(cancellationToken).ConfigureAwait(false)) + { + return false; + } + } + Diagnostic?.Invoke("replayed the last Cartesian approach in reverse"); + return true; + } + + double[]? target = m_retractJointAngles; + if (target == null) + { + return true; + } + double[]? endpoint = m_retractJointEndpoint; + m_retractJointAngles = null; + m_retractJointEndpoint = null; + double[] start = GetJoints(); + if (endpoint == null || !SameJointConfiguration(start, endpoint)) + { + Diagnostic?.Invoke("discarded a stale joint retract path"); + return true; + } + if (!m_kinematics.ClearsPath(start, target)) + { + return false; + } + double distance = JointDistance(start, target); + var profile = new TrapezoidalVelocityProfile( + distance, JointSpeed(new MotionConstraintsDataType()), DefaultJointAcceleration); + IntentOutcome outcome = await FollowProfileAsync( + profile, + execution, + fraction => SetJoints( + m_kinematics.InterpolateJoints(start, target, fraction).Span), + DefaultJointAcceleration, + cancellationToken).ConfigureAwait(false); + if (outcome.State == ExecutionStateEnum.Succeeded) + { + Diagnostic?.Invoke("reversed the last local approach"); + return true; + } + return false; + } + + /// + /// Invalidates a recorded local approach when another motion changes the arm pose. + /// + private void InvalidateRecordedApproach() + { + m_retractCartesianPath = null; + m_retractJointAngles = null; + m_retractJointEndpoint = null; + } + + /// + /// Gets whether two joint configurations agree closely enough to replay a recorded + /// path from the current pose. + /// + private static bool SameJointConfiguration(double[] left, double[] right) + { + if (left.Length != right.Length) + { + return false; + } + for (int ii = 0; ii < left.Length; ii++) + { + if (Math.Abs(left[ii] - right[ii]) > RetractEndpointToleranceRadians) + { + return false; + } + } + return true; + } + + /// + /// Swings the tool to a position in joint space, so the arm rotates around its base + /// rather than trying to carry the tool across the axis it stands on. + /// + private async ValueTask SwingToAsync( + double[] position, + string frameId, + ArrayOf orientation, + IntentExecution execution, + CancellationToken cancellationToken) + { + double[] start = GetJoints(); + if (!TrySolveWithYawSearch(position, frameId, orientation, start, out SimulatedArmIkSolution? solution)) + { + return false; + } + double distance = JointDistance(start, solution.JointAngles.Span); + var profile = new TrapezoidalVelocityProfile( + distance, JointSpeed(new MotionConstraintsDataType()), DefaultJointAcceleration); + _ = await FollowProfileAsync( + profile, + execution, + fraction => SetJoints( + m_kinematics.InterpolateJoints(start, solution.JointAngles.Span, fraction).Span), + DefaultJointAcceleration, + cancellationToken).ConfigureAwait(false); + return true; + } + + /// + /// Solves for a tool position, turning the tool about the vertical when the + /// orientation it is holding does not work out. + /// + /// + /// A parallel gripper coming straight down is free to choose its rotation about the + /// tool axis - the jaws close on a part the same way whichever way round they are - + /// but the cell never used that freedom: it reused whatever orientation the arm was + /// left holding, so each Location got exactly one pose and one chance. That is fine + /// until the solver has to satisfy clearance as well, at which point a single pose + /// often has no answer while the same position a few degrees round has several. + /// The requested orientation is tried first so nothing changes when it works, and + /// the offsets are a fixed sequence so the same target always resolves the same way. + /// + private bool TrySolveWithYawSearch( + double[] position, + string frameId, + ArrayOf orientation, + double[] start, + [NotNullWhen(true)] out SimulatedArmIkSolution? solution) + { + ReadOnlySpan requested = orientation.Span; + foreach (double degrees in s_yawOffsetsDegrees) + { + var target = new Pose3DDataType + { + FrameId = frameId, + Position = position.ToArrayOf(), + Orientation = degrees == 0.0 + ? orientation + : TurnAboutVertical(requested, degrees).ToArrayOf() + }; + if (m_kinematics.TrySelectNearest( + target, start, out solution, out SimulatedArmKinematicFailure _)) + { + return true; + } + } + solution = null; + return false; + } + + /// + /// Turns an orientation about the world vertical. + /// + private static double[] TurnAboutVertical(ReadOnlySpan orientation, double degrees) + { + double half = degrees * Math.PI / 360.0; + double sin = Math.Sin(half); + double cos = Math.Cos(half); + return + [ + (cos * orientation[0]) - (sin * orientation[1]), + (cos * orientation[1]) + (sin * orientation[0]), + (cos * orientation[2]) + (sin * orientation[3]), + (cos * orientation[3]) - (sin * orientation[2]) + ]; + } + + /// + /// Gets why the last travel to a Location was refused, for the failure message. + /// + private string LastTravelFailure { get; set; } = string.Empty; + + /// + /// Moves the tool centre point to one position along a straight line, keeping its + /// orientation. + /// + /// + /// The line is followed in Cartesian space and re-solved at each step rather than + /// solved once and interpolated in joint space. Interpolating in joint space takes + /// whatever route the two configurations happen to describe, and when the solver + /// picks a different elbow branch for the far end that route swings a link through + /// the bench - so with clearance enforced, every candidate gets refused and the arm + /// simply stops. Re-solving along a straight line keeps each step next to the last + /// one, which is also what the move looks like on a real cell. + /// + private async ValueTask MoveToolToAsync( + double[] position, + string frameId, + ArrayOf orientation, + IntentExecution execution, + CancellationToken cancellationToken, + bool recordRetractPath = false) + { + var target = new Pose3DDataType + { + FrameId = frameId, + Position = position.ToArrayOf(), + Orientation = orientation + }; + List? jointPath = recordRetractPath ? [GetJoints()] : null; + IntentOutcome outcome = await MoveCartesianAsync( + target, + new MotionConstraintsDataType(), + execution, + cancellationToken, + jointPath) + .ConfigureAwait(false); + bool succeeded = outcome.State == ExecutionStateEnum.Succeeded; + if (recordRetractPath) + { + m_retractCartesianPath = succeeded && jointPath is { Count: > 1 } + ? jointPath + : null; + } + return succeeded; + } + + /// + /// Plans a complete, locally sampled Cartesian move before executing any of it. + /// + /// + /// Solving while moving can switch into a branch that has no continuation near the + /// target, even when another yaw has a clear path all the way down. This method + /// evaluates every sample first, tries the same deterministic yaw spread used by + /// direct moves, and executes only a sequence that reached the target. The samples + /// are close enough that checking every configuration is the swept-path + /// approximation; no unrelated joint interpolation is substituted for it. + /// + private async ValueTask MovePlannedCartesianAsync( + double[] position, + string frameId, + ArrayOf orientation, + IntentExecution execution, + bool recordRetractPath, + CancellationToken cancellationToken) + { + Pose3DDataType startPose = CurrentSnapshot.ToolPose; + double[] startingJoints = GetJoints(); + double[] requested = orientation.Span.ToArray(); + foreach (double degrees in s_yawOffsetsDegrees) + { + ArrayOf candidateOrientation = degrees == 0.0 + ? orientation + : TurnAboutVertical(requested, degrees).ToArrayOf(); + var target = new Pose3DDataType + { + FrameId = frameId, + Position = position.ToArrayOf(), + Orientation = candidateOrientation + }; + var path = new List(CartesianPlanningSamples + 1) + { + (double[])startingJoints.Clone() + }; + double[] reference = (double[])startingJoints.Clone(); + bool complete = true; + for (int step = 1; step <= CartesianPlanningSamples; step++) + { + double fraction = (double)step / CartesianPlanningSamples; + Pose3DDataType pose = m_kinematics.InterpolateCartesian(startPose, target, fraction); + if (!m_kinematics.TrySelectNearestConfiguration( + pose, + reference, + out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure _)) + { + complete = false; + break; + } + reference = solution.JointAngles.Span.ToArray(); + path.Add(reference); + } + if (!complete) + { + continue; + } + + double distance = Distance(startPose.Position.Span, target.Position.Span); + var profile = new TrapezoidalVelocityProfile( + distance, DefaultCartesianSpeed, DefaultCartesianAcceleration); + IntentOutcome outcome = await FollowProfileAsync( + profile, + execution, + fraction => + { + int index = Math.Clamp( + (int)Math.Round(fraction * (path.Count - 1)), + 0, + path.Count - 1); + SetPose(path[index]); + }, + DefaultCartesianAcceleration, + cancellationToken).ConfigureAwait(false); + bool succeeded = outcome.State == ExecutionStateEnum.Succeeded; + if (recordRetractPath) + { + m_retractCartesianPath = succeeded ? path : null; + } + return succeeded; + } + if (recordRetractPath) + { + m_retractCartesianPath = null; + } + return false; } private async ValueTask ExecuteToolChangeAsync( @@ -583,7 +1366,8 @@ private async ValueTask MoveCartesianAsync( Pose3DDataType target, MotionConstraintsDataType constraints, IntentExecution execution, - CancellationToken cancellationToken) + CancellationToken cancellationToken, + List? jointPath = null) { Pose3DDataType start = CurrentSnapshot.ToolPose; double distance = Distance(start.Position.Span, target.Position.Span); @@ -596,13 +1380,14 @@ private async ValueTask MoveCartesianAsync( fraction => { Pose3DDataType pose = m_kinematics.InterpolateCartesian(start, target, fraction); - if (m_kinematics.TrySelectNearest( + if (m_kinematics.TrySelectNearestConfiguration( pose, CurrentSnapshot.JointAngles.Span, out SimulatedArmIkSolution? solution, out SimulatedArmKinematicFailure failure)) { SetPose(solution.JointAngles.Span); + jointPath?.Add(solution.JointAngles.Span.ToArray()); } else { @@ -615,7 +1400,7 @@ private async ValueTask MoveCartesianAsync( if (moveFailure != SimulatedArmKinematicFailure.None) { return IntentOutcome.Fail( - SimulatedArmKinematics.ToIntentFailure(moveFailure), "The Cartesian path is not feasible."); + m_kinematics.MapFailure(moveFailure), "The Cartesian path is not feasible."); } return outcome.State == ExecutionStateEnum.Succeeded ? IntentOutcome.SucceededAt(CurrentSnapshot.ToolPose) @@ -825,7 +1610,8 @@ private void SetPose(ReadOnlySpan jointAngles) m_hasObject, m_toolName, HeldPartPosition(forward.ToolPose), - ArrayOf.Create(m_stackSlotsFilled.AsSpan())); + ArrayOf.Create(m_stackSlotsFilled.AsSpan()), + m_heldObjectClass); } SnapshotChanged?.Invoke(this, CurrentSnapshot); } @@ -842,7 +1628,8 @@ private void PublishCurrentPoseLocked() m_hasObject, m_toolName, HeldPartPosition(pose.ToolPose), - ArrayOf.Create(m_stackSlotsFilled.AsSpan())); + ArrayOf.Create(m_stackSlotsFilled.AsSpan()), + m_heldObjectClass); } private void FillNextStackSlotLocked() @@ -1039,10 +1826,10 @@ private static double StopAcceleration(double acceleration, StopModeEnum stopMod }; } - private static double JointDistance(ReadOnlySpan a, ReadOnlySpan b) + private double JointDistance(ReadOnlySpan a, ReadOnlySpan b) { double sum = 0.0; - for (int i = 0; i < SimulatedArmKinematics.JointCount; i++) + for (int i = 0; i < m_kinematics.AxisCount; i++) { double delta = a[i] - b[i]; sum += delta * delta; @@ -1058,6 +1845,13 @@ private static double Distance(ReadOnlySpan a, ReadOnlySpan b) return Math.Sqrt((dx * dx) + (dy * dy) + (dz * dz)); } + private static double Distance2D(ReadOnlySpan a, ReadOnlySpan b) + { + double x = a[0] - b[0]; + double y = a[1] - b[1]; + return Math.Sqrt((x * x) + (y * y)); + } + private static ArrayOf HeldPartPosition(Pose3DDataType toolPose) { ReadOnlySpan position = toolPose.Position.Span; @@ -1069,6 +1863,32 @@ private static ArrayOf HeldPartPosition(Pose3DDataType toolPose) } private const double DefaultCartesianSpeed = 0.25; + + /// + /// How high the tool lifts to before crossing the cell, in the arm's base frame. + /// Above the bin walls and above a full stack on the fixture, so a straight + /// joint-space leg at this height clears the furniture between two work positions. + /// + private const double TransitHeightMetres = 0.32; + private const double TransitBypassXMetres = 0.20; + private const double TransitBypassYMetres = -0.35; + private const double LocalApproachHeightMetres = 0.04; + private const int CartesianPlanningSamples = 32; + private const double RetractEndpointToleranceRadians = 1e-5; + private const double SameLocationToleranceMetres = 0.01; + + /// + /// The rotations about the vertical a Pick or Place may use when the orientation the + /// arm is holding has no clear solution. Zero first, so a target that already works + /// resolves exactly as before, then outwards in both directions. + /// + private static readonly double[] s_yawOffsetsDegrees = + [ + 0.0, 15.0, -15.0, 30.0, -30.0, 45.0, -45.0, 60.0, -60.0, 75.0, -75.0, + 90.0, -90.0, 105.0, -105.0, 120.0, -120.0, 135.0, -135.0, 150.0, -150.0, + 165.0, -165.0, 180.0 + ]; + private const double DefaultCartesianAcceleration = 0.7; private const double DefaultJointSpeed = 0.9; private const double DefaultJointAcceleration = 2.0; @@ -1076,15 +1896,43 @@ private static ArrayOf HeldPartPosition(Pose3DDataType toolPose) private const double BenchTopZ = 0.829; private const double GripperOpen = 0.08; private const double GripperClosed = 0.018; - private const double HeldPartTcpOffset = 0.035; private const int StackSlotCount = 8; private readonly System.Threading.Lock m_lock = new(); - private readonly SimulatedArmKinematics m_kinematics; + private readonly ISimulatedArmKinematics m_kinematics; private readonly ISimulatedArmClock m_clock; - private readonly double[] m_jointAngles = [-0.45, -0.95, 1.55, -0.9, 0.75, 0.0]; + + /// + /// + /// Home configuration, radians. This arm is mounted on a bench in both samples that + /// use it, so the pose has to keep every joint above the work surface, and in the + /// bin-picking cell it also has to aim the eye-in-hand camera: it is solved so the + /// camera prim lands at the world position the Vision model declares for it + /// (0.38, 0, 1.35) looking straight down, which puts the bin 0.50 m away and 1.8 + /// degrees off the optical axis - matching the standoff the detections report. + /// + /// + /// Two constraints on the solution are easy to miss and both were violated by + /// earlier attempts: + /// + /// + /// - It is the elbow-back branch. The elbow-forward solutions reach the same + /// camera pose but park a link directly under the camera, and the frame comes + /// back showing the arm's own upper arm instead of the bin. + /// - The wrist stays 25 degrees clear of J4 and J6 lining up. Aiming a + /// straight-down camera from a point on the base's own X-Z plane lands exactly + /// on that singularity, so the camera roll is tilted 15 degrees to get off it. + /// A singular home pose is not a cosmetic problem: the first IK solve of any + /// motion away from home fails, so every intent returns Kinematics. + /// + /// + private readonly double[] m_jointAngles; + private double[]? m_retractJointAngles; + private double[]? m_retractJointEndpoint; + private List? m_retractCartesianPath; private double m_gripperOpening = GripperOpen; private bool m_hasObject; + private string m_heldObjectClass = string.Empty; private readonly bool[] m_stackSlotsFilled = new bool[StackSlotCount]; private string m_toolName = "parallel-gripper"; private string m_nonCancellableIntentId = string.Empty; diff --git a/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedCollisionModel.cs b/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedCollisionModel.cs new file mode 100644 index 0000000000..c35b3937a4 --- /dev/null +++ b/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedCollisionModel.cs @@ -0,0 +1,383 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Opc.Ua; + +namespace Robotics.IntentEnabledRobot.Simulation +{ + /// + /// An axis-aligned solid the arm must not pass through, in the arm's base frame. + /// + /// + /// What the box represents, for diagnostics. + /// + /// + /// Centre of the box on X. + /// + /// + /// Centre of the box on Y. + /// + /// + /// Full extent on X. + /// + /// + /// Full extent on Y. + /// + /// + /// Underside of the box. + /// + /// + /// Top of the box. + /// + /// + /// How much of the link's own thickness has to stay outside this solid. A wall or a + /// post is an object the arm must not intersect, so it takes the full link radius. A + /// work surface is different: it is a half-space the arm must not go *below*, and a + /// link that comes right down to the table is doing its job - a gripper has to reach a + /// part lying on it. Inflating the bench by a link radius refuses exactly the poses the + /// cell exists to make, because with the tool vertical this arm's wrist stacks along + /// the tool axis and J4 ends up 22 mm above the surface when the tool is 56 mm above + /// it. Zero here means "not below", which is what a work surface actually asks for. + /// + public readonly record struct SimulatedObstacleBox( + string Name, + double CentreX, + double CentreY, + double SizeX, + double SizeY, + double MinZ, + double MaxZ, + double Clearance = double.NaN); + + /// + /// Decides whether an arm configuration puts any part of the arm inside the cell's + /// furniture. + /// + /// + /// + /// This replaces a test that compared each joint origin against a single + /// horizontal plane. That test passes a configuration whose links pass clean through + /// the bench, because the point where a link ends can sit above the plane while the + /// length of the link between two joints dips below it - which is exactly what "the + /// arm moves through the table" looks like. It also had nothing to say about the bin + /// or the fixture, so the wrist could come to rest inside the bin and no part of the + /// simulation objected. + /// + /// + /// Each link is treated as a capsule between consecutive joint origins and tested + /// against axis-aligned boxes. That is a coarse model - it is not a physics engine, + /// there is no contact response and nothing here pushes back - but it answers the one + /// question the arm needs answered before it moves: would this configuration put a + /// link inside something solid. + /// + /// + public sealed class SimulatedCollisionModel + { + /// + /// Creates a collision model over a set of obstacles. + /// + /// + /// The solids the arm must stay out of, in the arm's base frame. + /// + /// + /// How thick the arm's links are treated as being, in metres. + /// + /// + /// How thick the tool beyond the flange is treated as being, in metres. It is + /// slender next to a link, and unlike a link it is meant to approach surfaces + /// closely - a gripper that cannot come within a link's thickness of the bench + /// cannot pick anything up off it. + /// + public SimulatedCollisionModel( + ArrayOf obstacles, + double linkRadius, + double toolRadius) + { + Obstacles = obstacles; + LinkRadius = linkRadius; + ToolRadius = toolRadius; + } + + /// + /// Creates a collision model with one radius per chain segment. + /// + /// Fixed solids in the arm base frame. + /// + /// Radius for each consecutive point pair supplied to . + /// + public SimulatedCollisionModel( + ArrayOf obstacles, + ArrayOf segmentRadii) + { + if (segmentRadii.IsEmpty) + { + throw new ArgumentException( + "At least one segment radius is required.", + nameof(segmentRadii)); + } + Obstacles = obstacles; + SegmentRadii = segmentRadii; + LinkRadius = segmentRadii[0]; + ToolRadius = segmentRadii[^1]; + } + + /// + /// Gets the solids the arm must stay out of. + /// + public ArrayOf Obstacles { get; } + + /// + /// Gets the radius the arm's links are treated as having. + /// + public double LinkRadius { get; } + + /// + /// Gets the radius the tool beyond the flange is treated as having. + /// + public double ToolRadius { get; } + + /// + /// Gets per-segment radii when the arm provides them. + /// + public ArrayOf SegmentRadii { get; } + + /// + /// Gets or sets the solids that move, tested alongside the fixed ones. + /// + /// + /// The furniture is fixed, but the workpieces are not: a stack built on the fixture + /// is as solid as the fixture under it, and an arm that reaches through it looks + /// exactly as wrong as one reaching through the bench. The host owns where the parts + /// are, so it republishes them here when it moves one - and leaves out whatever the + /// gripper is carrying, since a part travelling with the tool cannot be an obstacle + /// to it. + /// + public ArrayOf MovingObstacles { get; set; } + + /// + /// Gets whether a chain of joint origins, plus the tool point beyond the last one, + /// stays clear of every obstacle. + /// + /// + /// Successive points along the arm, each as three coordinates in the arm's base + /// frame: the joint origins followed by the tool centre point. + /// + /// + /// The first obstacle the arm intersects, when the result is false. + /// + public bool IsClear(ReadOnlySpan points, out string hit) + { + hit = string.Empty; + if (points.Length < 6 || (Obstacles.Count == 0 && MovingObstacles.Count == 0)) + { + return true; + } + ReadOnlySpan boxes = Obstacles.Span; + ReadOnlySpan moving = MovingObstacles.Span; + int lastSegment = points.Length - 6; + for (int segment = 0; segment <= lastSegment; segment += 3) + { + int segmentIndex = segment / 3; + double radius = segmentIndex < SegmentRadii.Count + ? SegmentRadii[segmentIndex] + : segment == lastSegment + ? ToolRadius + : LinkRadius; + double ax = points[segment]; + double ay = points[segment + 1]; + double az = points[segment + 2]; + double bx = points[segment + 3]; + double by = points[segment + 4]; + double bz = points[segment + 5]; + if (!IsSegmentClear(boxes, ax, ay, az, bx, by, bz, radius, out hit) || + !IsSegmentClear(moving, ax, ay, az, bx, by, bz, radius, out hit)) + { + return false; + } + } + return true; + } + + /// + /// Gets whether an axis-aligned workpiece stays clear of every obstacle. + /// + /// + public bool IsBoxClear( + double centreX, + double centreY, + double centreZ, + double sizeX, + double sizeY, + double sizeZ, + out string hit) + { + if (sizeX < 0.0 || sizeY < 0.0 || sizeZ < 0.0) + { + throw new ArgumentOutOfRangeException( + nameof(sizeX), + "Workpiece dimensions must be non-negative."); + } + if (!IsBoxClear( + Obstacles.Span, + centreX, + centreY, + centreZ, + sizeX, + sizeY, + sizeZ, + out hit)) + { + return false; + } + return IsBoxClear( + MovingObstacles.Span, + centreX, + centreY, + centreZ, + sizeX, + sizeY, + sizeZ, + out hit); + } + + private static bool IsBoxClear( + ReadOnlySpan boxes, + double centreX, + double centreY, + double centreZ, + double sizeX, + double sizeY, + double sizeZ, + out string hit) + { + hit = string.Empty; + double minX = centreX - (sizeX * 0.5); + double maxX = centreX + (sizeX * 0.5); + double minY = centreY - (sizeY * 0.5); + double maxY = centreY + (sizeY * 0.5); + double minZ = centreZ - (sizeZ * 0.5); + double maxZ = centreZ + (sizeZ * 0.5); + for (int ii = 0; ii < boxes.Length; ii++) + { + SimulatedObstacleBox box = boxes[ii]; + double boxMinX = box.CentreX - (box.SizeX * 0.5); + double boxMaxX = box.CentreX + (box.SizeX * 0.5); + double boxMinY = box.CentreY - (box.SizeY * 0.5); + double boxMaxY = box.CentreY + (box.SizeY * 0.5); + if (maxX > boxMinX + ContactToleranceMetres && + minX < boxMaxX - ContactToleranceMetres && + maxY > boxMinY + ContactToleranceMetres && + minY < boxMaxY - ContactToleranceMetres && + maxZ > box.MinZ + ContactToleranceMetres && + minZ < box.MaxZ - ContactToleranceMetres) + { + hit = box.Name; + return false; + } + } + return true; + } + + /// + /// Gets whether one link stays clear of a set of solids. + /// + private static bool IsSegmentClear( + ReadOnlySpan boxes, + double ax, double ay, double az, + double bx, double by, double bz, + double radius, + out string hit) + { + hit = string.Empty; + for (int ii = 0; ii < boxes.Length; ii++) + { + double clearance = double.IsNaN(boxes[ii].Clearance) ? radius : boxes[ii].Clearance; + if (IntersectsSegment(boxes[ii], ax, ay, az, bx, by, bz, clearance)) + { + hit = boxes[ii].Name; + return false; + } + } + return true; + } + + /// + /// Gets whether a capsule of around the segment from A to + /// B reaches inside a box. + /// + /// + /// The segment is sampled rather than solved analytically. Sampling is enough here + /// because the boxes are large next to the sample spacing, and it keeps the test + /// short enough to run for every candidate solution and every step of a path. + /// + private static bool IntersectsSegment( + SimulatedObstacleBox box, + double ax, double ay, double az, + double bx, double by, double bz, + double radius) + { + double minX = box.CentreX - (box.SizeX * 0.5) - radius; + double maxX = box.CentreX + (box.SizeX * 0.5) + radius; + double minY = box.CentreY - (box.SizeY * 0.5) - radius; + double maxY = box.CentreY + (box.SizeY * 0.5) + radius; + double minZ = box.MinZ - radius; + double maxZ = box.MaxZ + radius; + + // A segment can cross a box while both ends sit outside it, so walk the + // segment: testing only the endpoints is the mistake this class exists to fix. + double length = Norm(bx - ax, by - ay, bz - az); + int samples = Math.Max(2, (int)Math.Ceiling(length / SampleSpacingMetres) + 1); + for (int ii = 0; ii <= samples; ii++) + { + double t = (double)ii / samples; + double x = ax + ((bx - ax) * t); + double y = ay + ((by - ay) * t); + double z = az + ((bz - az) * t); + if (x >= minX && x <= maxX && y >= minY && y <= maxY && z >= minZ && z <= maxZ) + { + return true; + } + } + return false; + } + + private static double Norm(double x, double y, double z) + { + return Math.Sqrt((x * x) + (y * y) + (z * z)); + } + + /// + /// Fine enough to catch a link clipping the corner of a bin wall, coarse enough that + /// a whole configuration costs a few hundred comparisons. + /// + private const double SampleSpacingMetres = 0.02; + private const double ContactToleranceMetres = 1e-6; + } +} diff --git a/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedSupportModel.cs b/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedSupportModel.cs new file mode 100644 index 0000000000..84ceb19341 --- /dev/null +++ b/samples/Robotics/IntentEnabledRobot/Simulation/SimulatedSupportModel.cs @@ -0,0 +1,282 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Opc.Ua; + +namespace Robotics.IntentEnabledRobot.Simulation +{ + /// + /// An axis-aligned solid a part can come to rest on: the bench, a fixture plate, a + /// locating peg, or another part. + /// + /// + /// Identifies the solid, so a caller can tell what a part ended up standing on. + /// + /// + /// Centre of the footprint along X, in the world frame. + /// + /// + /// Centre of the footprint along Y, in the world frame. + /// + /// + /// Full extent of the footprint along X. + /// + /// + /// Full extent of the footprint along Y. + /// + /// + /// World height of the upper face, which is what something resting on it stands on. + /// + public readonly record struct SimulatedSupportSolid( + string Name, + double CentreX, + double CentreY, + double SizeX, + double SizeY, + double Top); + + /// + /// Works out what a part comes to rest on, so a simulated cell can put a released part + /// where gravity would leave it instead of where the tool happened to let go. + /// + /// + /// + /// This is a resting model, not a physics engine: it answers "what is the highest + /// solid under this footprint" and nothing else. There is no toppling, no friction and + /// no sliding, because none of those change the answer a pick-and-place cell needs - + /// which is that a part released over a bench ends up on the bench, a part released + /// over another part ends up on top of it, and neither ends up inside the other. + /// + /// + /// A part released without such a model stays wherever the tool centre point was, which + /// in this cell left it a measured 165 mm in the air. + /// + /// + public sealed class SimulatedSupportModel + { + /// + /// Initializes the model with the solids that never move. + /// + /// + /// Benches, plates, pegs and anything else a part can stand on but that a robot + /// does not carry. + /// + /// + /// The height a part falls to when nothing at all is under it, which stops a + /// footprint off the edge of every solid returning negative infinity. + /// + /// + /// is null. + /// + public SimulatedSupportModel(ArrayOf fixtures, double groundLevel) + { + m_fixtures = fixtures; + GroundLevel = groundLevel; + } + + /// + /// Gets the height a part falls to when nothing is under it. + /// + public double GroundLevel { get; } + + /// + /// Gets the height of the highest solid under a footprint, which is the surface + /// something placed there comes to rest on. + /// + /// + /// Centre of the footprint along X, in the world frame. + /// + /// + /// Centre of the footprint along Y, in the world frame. + /// + /// + /// Full extent of the footprint along X. + /// + /// + /// Full extent of the footprint along Y. + /// + /// + /// Solids that move, typically the other parts. Anything the caller wants ignored - + /// the part being placed, or a part currently in the gripper - is simply left out. + /// + /// + /// The world height of the supporting surface. + /// + public double SupportHeight( + double centreX, + double centreY, + double sizeX, + double sizeY, + ArrayOf movable) + { + double top = GroundLevel; + top = Highest(top, m_fixtures, centreX, centreY, sizeX, sizeY); + top = Highest(top, movable, centreX, centreY, sizeX, sizeY); + return top; + } + + /// + /// Gets the height a part's centre settles at when released over a footprint, which + /// is the supporting surface plus half the part's own height. + /// + /// + /// Centre of the part along X, in the world frame. + /// + /// + /// Centre of the part along Y, in the world frame. + /// + /// + /// Full extent of the part along X. + /// + /// + /// Full extent of the part along Y. + /// + /// + /// Full height of the part. + /// + /// + /// The other parts, excluding the one being placed. + /// + /// + /// The world height of the part's centre once it is resting. + /// + public double RestingCentreHeight( + double centreX, + double centreY, + double sizeX, + double sizeY, + double sizeZ, + ArrayOf movable) + { + return SupportHeight(centreX, centreY, sizeX, sizeY, movable) + (sizeZ * 0.5); + } + + /// + /// Gets the height a part's centre settles at, never allowing it to end up lower + /// than its support even when a caller asks for that. + /// + /// + /// A caller that has its own idea of where the part should go - the height the tool + /// let go at, say - passes it as . A request + /// above the resting height is honoured, because a part can be held higher than it + /// would fall to; a request below it is not, because that is a part inside a solid. + /// + /// + /// Centre of the part along X, in the world frame. + /// + /// + /// Centre of the part along Y, in the world frame. + /// + /// + /// Full extent of the part along X. + /// + /// + /// Full extent of the part along Y. + /// + /// + /// Full height of the part. + /// + /// + /// The height the caller would otherwise use. + /// + /// + /// The other parts, excluding the one being placed. + /// + /// + /// The requested height, or the resting height when the request is below it. + /// + public double ClampAboveSupport( + double centreX, + double centreY, + double sizeX, + double sizeY, + double sizeZ, + double requestedCentreZ, + ArrayOf movable) + { + double resting = RestingCentreHeight(centreX, centreY, sizeX, sizeY, sizeZ, movable); + return requestedCentreZ < resting ? resting : requestedCentreZ; + } + + /// + /// Gets a value indicating whether two footprints overlap, which is what decides + /// whether one solid can hold another up. + /// + /// + /// Centre of the first footprint along X. + /// + /// + /// Centre of the first footprint along Y. + /// + /// + /// Full extent of the first footprint along X. + /// + /// + /// Full extent of the first footprint along Y. + /// + /// + /// The second footprint. + /// + /// + /// true when the footprints overlap. + /// + public static bool FootprintsOverlap( + double aCentreX, + double aCentreY, + double aSizeX, + double aSizeY, + in SimulatedSupportSolid b) + { + return Math.Abs(aCentreX - b.CentreX) < ((aSizeX + b.SizeX) * 0.5) && + Math.Abs(aCentreY - b.CentreY) < ((aSizeY + b.SizeY) * 0.5); + } + + private static double Highest( + double top, + ArrayOf solids, + double centreX, + double centreY, + double sizeX, + double sizeY) + { + ReadOnlySpan span = solids.Span; + for (int ii = 0; ii < span.Length; ii++) + { + if (span[ii].Top > top && FootprintsOverlap(centreX, centreY, sizeX, sizeY, span[ii])) + { + top = span[ii].Top; + } + } + return top; + } + + private readonly ArrayOf m_fixtures; + } +} diff --git a/samples/Robotics/IntentViewerClient/Program.cs b/samples/Robotics/IntentViewerClient/Program.cs index 23ec9e1cbf..43a647a05e 100644 --- a/samples/Robotics/IntentViewerClient/Program.cs +++ b/samples/Robotics/IntentViewerClient/Program.cs @@ -662,6 +662,11 @@ private static async Task RunViewportAsync( string cacheDir = options.FetchAssetsDirectory ?? Path.GetDirectoryName(liveLayerPath) ?? AppContext.BaseDirectory; + + // The viewport needs the served geometry: without it only the live override layer + // composes, which carries transforms but no geometry and renders as an empty scene. + await FetchAssetsAsync(session, cacheDir, cancellationToken).ConfigureAwait(false); + string stagePath = Path.Combine(cacheDir, "stage.usda"); if (!File.Exists(stagePath)) { @@ -901,12 +906,12 @@ private static void WriteStageUsda(string cacheDir, List asset.Kind == OpenUsdAssetKind.RootLayer); string rootName = root != null ? Path.GetFileName(root.LocalPath) : "base.usda"; var builder = new StringBuilder(); - builder.Append("#usda 1.0\n(\n"); - builder.Append( - " doc = \"Self-contained OpenUSD stage: server-delivered base layers + live override.\"\n"); - builder.Append(" subLayers = [\n @./live.usda@,\n @./") - .Append(rootName).Append("@\n ]\n"); - builder.Append(")\n"); + builder.Append("#usda 1.0\n(\n") + .Append( + " doc = \"Self-contained OpenUSD stage: server-delivered base layers + live override.\"\n") + .Append(" subLayers = [\n @./live.usda@,\n @./") + .Append(rootName).Append("@\n ]\n") + .Append(")\n"); File.WriteAllText(Path.Combine(cacheDir, "stage.usda"), builder.ToString()); string livePath = Path.Combine(cacheDir, "live.usda"); if (!File.Exists(livePath)) diff --git a/samples/Robotics/MinimalRobotServer/README.md b/samples/Robotics/MinimalRobotServer/README.md index a9a6b544d0..844679df68 100644 --- a/samples/Robotics/MinimalRobotServer/README.md +++ b/samples/Robotics/MinimalRobotServer/README.md @@ -277,8 +277,8 @@ dotnet run --project tools/Opc.Ua.OpenUsd.Connector -- \ Compose `live.usda` over the base `Cell.usda` (see the example `stage.usda`) and open it in `usdview` / NVIDIA Omniverse to see the two arms articulate live. The example -USD assets, descriptor, writer, and a step-by-step guide live in the `opcua-drafts` -repo under `core-specs/extras/openusd-binding/examples/robotics/`. +USD assets, descriptor, writer, and a step-by-step guide live alongside the OpenUSD +binding specification, under its `examples/robotics/` directory. To watch the cell animate without leaving the connector, install the optional `Opc.Ua.OpenUsd.Connector.Viewer` assembly beside it and add `--view`: diff --git a/samples/Vision/README.md b/samples/Vision/README.md new file mode 100644 index 0000000000..c250c65ccb --- /dev/null +++ b/samples/Vision/README.md @@ -0,0 +1,82 @@ + + +# Vision samples + +The Vision samples show OPC UA Vision in realistic production cells rather +than as isolated image-processing calls. + +| Sample | Role | +|---|---| +| [VisualInspectionCell](VisualInspectionCell) | Server hosting Vision, AI Model Management, ISA-95 Job Control V2, and Alarms & Conditions for a machined-bracket inspection cell. | +| [VisualInspectionAgent](VisualInspectionAgent) | External deterministic orchestrator that connects with typed clients, measures fixture images, applies the recipe, routes jobs, and records operator ground truth. | + +The pair demonstrates a safety pattern that is deliberately stronger than +"ask a model whether the part is good": the model path can produce measured +characteristics and confidence, but deterministic recipe code owns the quality +verdict. A photographed image therefore cannot become a free-form instruction to +job control. + +```mermaid +graph TD + Agent["VisualInspectionAgent
typed OPC UA clients"] + Cell["VisualInspectionCell
OPC UA server"] + Vision["Vision
sensor, clip endpoint, pipeline, results"] + AI["AI Model Management
deployment, Invoke, learning job"] + ISA95["ISA-95 Job Control V2
inspection and rework orders"] + Alarms["Alarms & Conditions
operator dialog"] + + Agent -->|"capture and submit results"| Vision + Agent -->|"Invoke for provenance"| AI + Agent -->|"StoreAndStart / transitions"| ISA95 + Agent -->|"respond to dialog"| Alarms + Cell --> Vision + Cell --> AI + Cell --> ISA95 + Cell --> Alarms +``` + +## Running the pair + +Prerequisites: .NET 10 SDK. + +Start the server: + +```powershell +dotnet run --project samples\Vision\VisualInspectionCell\VisualInspectionCell.csproj -- --insecure +``` + +Then run the orchestrator in the unattended deterministic mode: + +```powershell +dotnet run --project samples\Vision\VisualInspectionAgent\VisualInspectionAgent.csproj -- --server opc.tcp://localhost:62865/VisualInspectionCell --insecure --mode scripted --cycles 3 +``` + +The server publishes static PNG fixture images. The agent retrieves a fixture, +measures the bracket geometry, calls the AI deployment's `Invoke` path so model +provenance and usage are visible in the address space, applies the recipe, and +routes the ISA-95 order according to the deterministic verdict. + +Use `--mode live-ai --ai-endpoint ` only when a real model endpoint is +configured. `live-ai` fails before creating any job when no endpoint is named; +it never silently falls back to the deterministic analyser, because a sample +that quietly degrades appears to work while demonstrating nothing. + +## See also + +- [Vision developer guide](../../docs/Vision.md) — see *Visual inspection: a cross-companion cell* — the + design, safety boundary, verdict rule, escalation, and learning feedback path. +- [Vision developer guide](../../docs/Vision.md) — the Vision companion and its + §9 feedback / learning semantics. +- [AI Model Management developer guide](../../docs/AiIntegration.md) — the + deployment `Invoke` and learning-job ownership used by the cell. +- [ISA-95 developer guide](../../docs/ISA95.md) — Job Control V2 clients, + providers, and job-state semantics. +- [Alarms and Conditions](../../docs/AlarmsAndConditions.md) — the Part 9 + dialog condition model used for human disposition. diff --git a/samples/Vision/VisualInspectionAgent/InspectionModel.cs b/samples/Vision/VisualInspectionAgent/InspectionModel.cs new file mode 100644 index 0000000000..d4f45f07b0 --- /dev/null +++ b/samples/Vision/VisualInspectionAgent/InspectionModel.cs @@ -0,0 +1,283 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua; +using Opc.Ua.Vision; + +namespace Vision.VisualInspectionAgent +{ + internal sealed class FixtureImageAnalyzer + { + public ArrayOf Measure(byte[] png) + { + ArgumentNullException.ThrowIfNull(png); + + (byte[] rgb, int width, int height) = PngDecoder.Decode(png); + if (width != ImageWidth || height != ImageHeight) + { + throw new InvalidDataException(FormattableString.Invariant( + $"Expected {ImageWidth}x{ImageHeight}, got {width}x{height}.")); + } + + int borePixels = CountDarkRunThrough(rgb, width, BoreCenterY, BoreCenterX); + PixelRun slot = FindDarkRunRightOf(rgb, width, SlotCenterY, minX: 400); + return new[] + { + new MeasuredCharacteristic( + "BoreDiameter", + borePixels / ScalePixelsPerMillimetre, + PixelPitchMillimetres, + ConfidenceFromUncertainty(PixelPitchMillimetres)), + new MeasuredCharacteristic( + "SlotWidth", + slot.Length / ScalePixelsPerMillimetre, + PixelPitchMillimetres, + ConfidenceFromUncertainty(PixelPitchMillimetres)), + new MeasuredCharacteristic( + "EdgeOffset", + (BracketRightX - slot.StartX) / ScalePixelsPerMillimetre, + PixelPitchMillimetres, + ConfidenceFromUncertainty(PixelPitchMillimetres)) + }.ToArrayOf(); + } + + private static int CountDarkRunThrough(byte[] rgb, int width, int y, int centerX) + { + int start = centerX; + while (start > 0 && IsDark(rgb, width, start - 1, y)) + { + start--; + } + int end = centerX; + while (end + 1 < width && IsDark(rgb, width, end + 1, y)) + { + end++; + } + return end - start + 1; + } + + private static PixelRun FindDarkRunRightOf(byte[] rgb, int width, int y, int minX) + { + int bestStart = -1; + int bestLength = 0; + int x = minX; + while (x < BracketRightX) + { + while (x < BracketRightX && !IsDark(rgb, width, x, y)) + { + x++; + } + int start = x; + while (x < BracketRightX && IsDark(rgb, width, x, y)) + { + x++; + } + int length = x - start; + if (length > bestLength) + { + bestStart = start; + bestLength = length; + } + } + if (bestStart < 0) + { + throw new InvalidDataException("Could not find the slot in the fixture image."); + } + return new PixelRun(bestStart, bestLength); + } + + private static bool IsDark(byte[] rgb, int width, int x, int y) + { + int offset = ((y * width) + x) * 3; + return rgb[offset] < 64 && rgb[offset + 1] < 64 && rgb[offset + 2] < 64; + } + + private static double ConfidenceFromUncertainty(double uncertainty) + { + return Math.Clamp(1.0 - uncertainty, 0.0, 1.0); + } + + private const int ImageWidth = 800; + private const int ImageHeight = 600; + private const int BoreCenterX = 290; + private const int BoreCenterY = 300; + private const int SlotCenterY = 300; + private const int BracketRightX = 650; + private const double ScalePixelsPerMillimetre = 10.0; + private const double PixelPitchMillimetres = 0.10; + + private readonly record struct PixelRun(int StartX, int Length); + } + + internal sealed class InspectionVerdictPolicy + { + public InspectionDecision Judge(string fixtureName, ArrayOf measurements) + { + if (measurements.Count == 0) + { + throw new ArgumentException("Measurements are required.", nameof(measurements)); + } + + var characteristics = new List(measurements.Count); + VisionResultEvaluationEnum verdict = VisionResultEvaluationEnum.Ok; + for (int ii = 0; ii < measurements.Count; ii++) + { + MeasuredCharacteristic measurement = measurements[ii]; + InspectionCharacteristicRecipe recipe = RecipeFor(measurement.CharacteristicId); + VisionResultEvaluationEnum characteristicVerdict = JudgeCharacteristic(recipe, measurement); + if (characteristicVerdict == VisionResultEvaluationEnum.NotOk) + { + verdict = VisionResultEvaluationEnum.NotOk; + } + else if (characteristicVerdict == VisionResultEvaluationEnum.NotDecidable && + verdict != VisionResultEvaluationEnum.NotOk) + { + verdict = VisionResultEvaluationEnum.NotDecidable; + } + + characteristics.Add(new VisionCharacteristicDataType + { + CharacteristicId = recipe.CharacteristicId, + Name = recipe.Name, + Nominal = recipe.Nominal, + Actual = measurement.Actual, + Deviation = measurement.Actual - recipe.Nominal, + LowerTolerance = recipe.LowerTolerance, + UpperTolerance = recipe.UpperTolerance, + Uncertainty = measurement.Uncertainty, + Unit = Millimetre, + Status = ToToleranceStatus(characteristicVerdict) + }); + } + return new InspectionDecision(fixtureName, characteristics.ToArrayOf(), verdict); + } + + private static InspectionCharacteristicRecipe RecipeFor(string characteristicId) + { + foreach (InspectionCharacteristicRecipe characteristic in s_characteristics) + { + if (string.Equals(characteristic.CharacteristicId, characteristicId, StringComparison.Ordinal)) + { + return characteristic; + } + } + throw new KeyNotFoundException(characteristicId); + } + + private static VisionResultEvaluationEnum JudgeCharacteristic( + InspectionCharacteristicRecipe recipe, + MeasuredCharacteristic measurement) + { + double intervalLow = measurement.Actual - measurement.Uncertainty; + double intervalHigh = measurement.Actual + measurement.Uncertainty; + double toleranceLow = recipe.Nominal - recipe.LowerTolerance; + double toleranceHigh = recipe.Nominal + recipe.UpperTolerance; + if (intervalLow >= toleranceLow && intervalHigh <= toleranceHigh) + { + return VisionResultEvaluationEnum.Ok; + } + if (intervalHigh < toleranceLow || intervalLow > toleranceHigh) + { + return VisionResultEvaluationEnum.NotOk; + } + return VisionResultEvaluationEnum.NotDecidable; + } + + private static VisionToleranceStatusEnum ToToleranceStatus(VisionResultEvaluationEnum verdict) + { + return verdict switch + { + VisionResultEvaluationEnum.Ok => VisionToleranceStatusEnum.InTolerance, + VisionResultEvaluationEnum.NotOk => VisionToleranceStatusEnum.OutOfTolerance, + _ => VisionToleranceStatusEnum.Indeterminate + }; + } + + private static EUInformation Millimetre { get; } = + new("mm", "millimetre", "http://www.opcfoundation.org/UA/units/un/cefact"); + + private static readonly InspectionCharacteristicRecipe[] s_characteristics = + [ + new("BoreDiameter", "Bore diameter", 12.00, 0.20, 0.20), + new("SlotWidth", "Slot width", 8.00, 0.15, 0.15), + new("EdgeOffset", "Edge offset", 20.00, 0.25, 0.25) + ]; + } + + internal sealed class ScriptedOperatorPolicy + { + public async Task GetDispositionAsync( + VisualInspectionAgentMode mode, + TimeSpan timeout, + CancellationToken cancellationToken) + { + if (mode == VisualInspectionAgentMode.Human) + { + await Task.Delay(timeout, cancellationToken).ConfigureAwait(false); + return OperatorDisposition.Stop; + } + await Task.Delay(TimeSpan.FromMilliseconds(250), cancellationToken).ConfigureAwait(false); + return OperatorDisposition.AcceptAsNotOk; + } + } + + internal sealed record InspectionCharacteristicRecipe( + string CharacteristicId, + string Name, + double Nominal, + double LowerTolerance, + double UpperTolerance); + + internal sealed record MeasuredCharacteristic( + string CharacteristicId, + double Actual, + double Uncertainty, + double Confidence); + + internal sealed record InspectionDecision( + string FixtureName, + ArrayOf Characteristics, + VisionResultEvaluationEnum Evaluation); + + internal enum OperatorDisposition + { + AcceptAsOk, + + AcceptAsNotOk, + + Reinspect, + + Stop + } +} diff --git a/samples/Vision/VisualInspectionAgent/PngDecoder.cs b/samples/Vision/VisualInspectionAgent/PngDecoder.cs new file mode 100644 index 0000000000..3027f7d4a2 --- /dev/null +++ b/samples/Vision/VisualInspectionAgent/PngDecoder.cs @@ -0,0 +1,198 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.IO.Compression; +using System.Text; + +namespace Vision.VisualInspectionAgent +{ + /// + /// Minimal PNG decoder for the fixture images used by this sample. + /// + internal static class PngDecoder + { + public static (byte[] Rgb, int Width, int Height) Decode(byte[] png) + { + if (png == null) + { + throw new ArgumentNullException(nameof(png)); + } + if (png.Length < 8 || + png[0] != 0x89 || + png[1] != (byte)'P' || + png[2] != (byte)'N' || + png[3] != (byte)'G' || + png[4] != 0x0D || + png[5] != 0x0A || + png[6] != 0x1A || + png[7] != 0x0A) + { + throw new InvalidDataException("Not a PNG stream."); + } + + int width = 0; + int height = 0; + byte bitDepth = 0; + byte colourType = 0; + byte interlace = 0; + using var idat = new MemoryStream(); + int offset = 8; + while (offset + 8 <= png.Length) + { + int length = ReadUInt32BE(png, offset); + string chunkType = Encoding.ASCII.GetString(png, offset + 4, 4); + int dataStart = offset + 8; + if (dataStart + length + 4 > png.Length) + { + throw new InvalidDataException("PNG chunk exceeds stream length."); + } + if (chunkType == "IHDR") + { + width = ReadUInt32BE(png, dataStart); + height = ReadUInt32BE(png, dataStart + 4); + bitDepth = png[dataStart + 8]; + colourType = png[dataStart + 9]; + interlace = png[dataStart + 12]; + } + else if (chunkType == "IDAT") + { + idat.Write(png, dataStart, length); + } + else if (chunkType == "IEND") + { + break; + } + offset = dataStart + length + 4; + } + if (bitDepth != 8 || (colourType != 2 && colourType != 6)) + { + throw new NotSupportedException( + $"PNG must be 8-bit RGB/RGBA; got type={colourType}, depth={bitDepth}."); + } + if (interlace != 0) + { + throw new NotSupportedException("Interlaced PNGs are not supported by the sample decoder."); + } + if (width <= 0 || height <= 0) + { + throw new InvalidDataException("PNG dimensions are invalid."); + } + + int bytesPerPixel = colourType == 6 ? 4 : 3; + byte[] rawFiltered = InflateZlib(idat.ToArray()); + int rowBytes = width * bytesPerPixel; + int expected = height * (rowBytes + 1); + if (rawFiltered.Length != expected) + { + throw new InvalidDataException( + $"Decoded filtered size {rawFiltered.Length} does not match {expected}."); + } + byte[] unfiltered = new byte[height * rowBytes]; + Unfilter(rawFiltered, unfiltered, width, height, bytesPerPixel); + if (bytesPerPixel == 3) + { + return (unfiltered, width, height); + } + + byte[] rgb = new byte[width * height * 3]; + for (int source = 0, target = 0; source < unfiltered.Length; source += 4, target += 3) + { + rgb[target] = unfiltered[source]; + rgb[target + 1] = unfiltered[source + 1]; + rgb[target + 2] = unfiltered[source + 2]; + } + return (rgb, width, height); + } + + private static byte[] InflateZlib(byte[] zlib) + { + if (zlib.Length < 6) + { + throw new InvalidDataException("IDAT chunk is too short."); + } + using var input = new MemoryStream(zlib, index: 2, count: zlib.Length - 6, writable: false); + using var deflate = new DeflateStream(input, CompressionMode.Decompress, leaveOpen: false); + using var output = new MemoryStream(); + deflate.CopyTo(output); + return output.ToArray(); + } + + private static void Unfilter(byte[] filtered, byte[] rgb, int width, int height, int bytesPerPixel) + { + int rowBytes = width * bytesPerPixel; + for (int y = 0; y < height; y++) + { + int srcRow = y * (rowBytes + 1); + int dstRow = y * rowBytes; + byte type = filtered[srcRow]; + for (int x = 0; x < rowBytes; x++) + { + byte value = filtered[srcRow + 1 + x]; + byte left = x >= bytesPerPixel ? rgb[dstRow + x - bytesPerPixel] : (byte)0; + byte up = y > 0 ? rgb[((y - 1) * rowBytes) + x] : (byte)0; + byte upLeft = y > 0 && x >= bytesPerPixel + ? rgb[((y - 1) * rowBytes) + x - bytesPerPixel] + : (byte)0; + rgb[dstRow + x] = type switch + { + 0 => value, + 1 => (byte)(value + left), + 2 => (byte)(value + up), + 3 => (byte)(value + ((left + up) / 2)), + 4 => (byte)(value + Paeth(left, up, upLeft)), + _ => throw new NotSupportedException($"Unknown PNG row filter {type}.") + }; + } + } + } + + private static byte Paeth(byte left, byte up, byte upLeft) + { + int p = left + up - upLeft; + int pa = Math.Abs(p - left); + int pb = Math.Abs(p - up); + int pc = Math.Abs(p - upLeft); + if (pa <= pb && pa <= pc) + { + return left; + } + return pb <= pc ? up : upLeft; + } + + private static int ReadUInt32BE(byte[] source, int offset) + { + return (source[offset] << 24) | + (source[offset + 1] << 16) | + (source[offset + 2] << 8) | + source[offset + 3]; + } + } +} diff --git a/samples/Vision/VisualInspectionAgent/Program.cs b/samples/Vision/VisualInspectionAgent/Program.cs new file mode 100644 index 0000000000..b5c6c59365 --- /dev/null +++ b/samples/Vision/VisualInspectionAgent/Program.cs @@ -0,0 +1,925 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Client; +using Opc.Ua.Client; +using Opc.Ua.Client.Alarms; +using Opc.Ua.Configuration; +using Opc.Ua.ISA95.Client; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; +using V2 = Opc.Ua.ISA95.JobControl.V2; + +namespace Vision.VisualInspectionAgent +{ + internal static class Program + { + public static async Task Main(string[] args) + { + using var cts = new CancellationTokenSource(); + Console.CancelKeyPress += (_, e) => + { + e.Cancel = true; + cts.Cancel(); + }; + + VisualInspectionAgentOptions options; + try + { + options = VisualInspectionAgentOptions.Parse(args); + } + catch (FormatException ex) + { + Console.Error.WriteLine(ex.Message); + return 2; + } + if (options.Mode == VisualInspectionAgentMode.LiveAI && string.IsNullOrWhiteSpace(options.AIEndpoint)) + { + Console.Error.WriteLine("live-ai requires --ai-endpoint and exits before creating any job."); + return 2; + } + + VisualInspectionAgentSession sample = await VisualInspectionAgentSession + .ConnectAsync(options, cts.Token) + .ConfigureAwait(false); + await using (sample.ConfigureAwait(false)) + { + var runner = new VisualInspectionAgentRunner(sample, options); + try + { + await runner.RunAsync(cts.Token).ConfigureAwait(false); + } + catch (InvalidOperationException ex) + { + // Business refusals - an ISA-95 ReturnStatus that is not success, or a + // deployment that cannot serve live-ai - are expected outcomes of the demo. + // Report them as a diagnostic rather than a crash dump. + Console.Error.WriteLine(ex.Message); + return 3; + } + return 0; + } + } + } + + internal sealed class VisualInspectionAgentRunner + { + public VisualInspectionAgentRunner(VisualInspectionAgentSession sample, VisualInspectionAgentOptions options) + { + m_sample = sample ?? throw new ArgumentNullException(nameof(sample)); + m_options = options ?? throw new ArgumentNullException(nameof(options)); + } + + public async Task RunAsync(CancellationToken cancellationToken) + { + VisualInspectionCellContext cell = await DiscoverAsync(cancellationToken).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"Connected to {m_options.ServerUrl}; mode={m_options.Mode}; cycles={m_options.Cycles}.")); + Console.WriteLine(FormattableString.Invariant( + $"Discovered pipeline={cell.Pipeline.PipelineNodeId}, deployment={cell.Snapshot.DeploymentId}, learningJob={cell.Snapshot.LearningJobId}.")); + + await VerifyLiveAIBeforeJobsAsync(cell, cancellationToken).ConfigureAwait(false); + for (int cycle = 1; cycle <= m_options.Cycles; cycle++) + { + string cycleId = StableCycleId(cycle); + string fixture = FixtureFor(cycle); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] State=Capture; fixture={fixture}; cycle/order/result id are correlated.")); + + InspectionEvidence evidence = await CaptureAndMeasureWithFallbackAsync( + cell, + cycleId, + fixture, + cancellationToken).ConfigureAwait(false); + InspectionDecision decision = m_policy.Judge(fixture, evidence.Measurements); + await cell.Feedback.SubmitImageReferenceAsync( + VisionFeedbackPurposeEnum.Reconciliation, + evidence.Frame, + cycleId, + cancellationToken).ConfigureAwait(false); + await cell.Feedback.SubmitInspectionResultAsync( + cycleId, + decision.Evaluation, + decision.Characteristics, + cancellationToken).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Verdict={decision.Evaluation}; result submitted to Vision Feedback.")); + + if (decision.Evaluation == VisionResultEvaluationEnum.NotDecidable) + { + await HoldForOperatorAsync(cell, cycleId, decision, cancellationToken).ConfigureAwait(false); + continue; + } + + await CompleteInspectionJobAsync(cell, cycleId, cancellationToken).ConfigureAwait(false); + string nextOrder = decision.Evaluation == VisionResultEvaluationEnum.Ok + ? InspectionOrderId + : ReworkRejectOrderId; + await StoreAndStartIdempotentAsync(cell.JobControl, nextOrder, cycleId, verifyRetry: true, + cancellationToken) + .ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Scheduled {(decision.Evaluation == VisionResultEvaluationEnum.Ok ? "next inspection" : "rework/reject")} order {nextOrder}.")); + } + } + + private async Task DiscoverAsync(CancellationToken cancellationToken) + { + var vision = new VisionClient(m_sample.Session, m_sample.Telemetry); + VisionNodeEntry? pipelineEntry = null; + await foreach (VisionNodeEntry entry in vision.EnumeratePipelinesAsync(cancellationToken) + .ConfigureAwait(false)) + { + if (string.Equals(entry.BrowseName.Name, PipelineBrowseName, StringComparison.Ordinal)) + { + pipelineEntry = entry; + break; + } + } + if (pipelineEntry == null) + { + throw new InvalidOperationException("The visual-inspection pipeline was not found."); + } + + VisionPipelineClient pipeline = vision.Pipeline(pipelineEntry.NodeId); + VisionPipelineSnapshot snapshot = await pipeline.ReadAsync(cancellationToken).ConfigureAwait(false); + VisionFeedbackClient feedback = await pipeline.OpenFeedbackAsync(cancellationToken).ConfigureAwait(false) + ?? throw new InvalidOperationException("The pipeline has no Feedback object."); + VisionSensorClient sensor = vision.Sensor(snapshot.SensorId); + VisionMediaClient media = await sensor.OpenMediaAsync(cancellationToken).ConfigureAwait(false) + ?? throw new InvalidOperationException("The image sensor has no Media object."); + NodeId clipEndpoint = NodeId.Null; + await foreach (VisionNodeEntry endpoint in media.EnumerateClipEndpointsAsync(cancellationToken) + .ConfigureAwait(false)) + { + clipEndpoint = endpoint.NodeId; + break; + } + if (clipEndpoint.IsNull) + { + throw new InvalidOperationException("The image sensor has no clip endpoint."); + } + + var ai = new AIClient(m_sample.Session, m_sample.Telemetry); + var deployment = new AIDeploymentClient(ai, snapshot.DeploymentId); + var isa95 = new Isa95Client(m_sample.Session, m_sample.Telemetry); + Isa95JobControlDiscovery discovery = await isa95.DiscoverJobControlAsync(cancellationToken) + .ConfigureAwait(false); + NodeId receiver = FindFacet(discovery, Isa95JobControlFacet.JobOrderReceiver); + NodeId provider = FindFacet(discovery, Isa95JobControlFacet.JobResponseProvider); + NodeId responseReceiver = FindFacet(discovery, Isa95JobControlFacet.JobResponseReceiver); + Isa95JobControlV2Client jobControl = isa95.CreateJobControlV2Client(receiver, provider, responseReceiver); + NodeId dialog = await FindObjectByBrowseNameAsync(OperatorDialogBrowseName, cancellationToken) + .ConfigureAwait(false); + return new VisualInspectionCellContext( + pipeline, + snapshot, + media, + clipEndpoint, + feedback, + deployment, + jobControl, + dialog); + } + + private async Task VerifyLiveAIBeforeJobsAsync( + VisualInspectionCellContext cell, + CancellationToken cancellationToken) + { + if (m_options.Mode != VisualInspectionAgentMode.LiveAI) + { + return; + } + AIDeploymentSnapshot snapshot = await cell.Deployment.ReadAsync(cancellationToken).ConfigureAwait(false); + if (string.IsNullOrWhiteSpace(snapshot.EndpointUri)) + { + throw new InvalidOperationException("live-ai deployment has no endpoint URI; no jobs were created."); + } + } + + private async Task CaptureAndMeasureAsync( + VisualInspectionCellContext cell, + string cycleId, + string fixture, + CancellationToken cancellationToken) + { + VisionClipResult clip = await cell.Media.GetClipAsync( + cell.ClipEndpoint, + fixture, + DateTimeUtc.From(DateTime.UnixEpoch), + VisionClipFormatEnum.Png, + requestInline: true, + cancellationToken).ConfigureAwait(false); + + if (clip.HasInlineImage) + { + ArrayOf measurements = await InvokeAIForMeasurementsAsync( + cell, + cycleId, + fixture, + clip.Image, + cancellationToken).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Measurements from AI Invoke: {FormatMeasurements(measurements)}.")); + return new InspectionEvidence(clip.Image, measurements); + } + + throw new InvalidOperationException("The cell did not return an inline fixture image."); + } + + private async Task CaptureAndMeasureFromFixtureAsync( + VisualInspectionCellContext cell, + string cycleId, + string fixture, + CancellationToken cancellationToken) + { + VisionImageReferenceDataType frame = CreateFixtureImageReference(fixture); + ArrayOf measurements = await InvokeAIForMeasurementsAsync( + cell, + cycleId, + fixture, + frame, + cancellationToken).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Measurements from AI Invoke: {FormatMeasurements(measurements)}.")); + return new InspectionEvidence(frame, measurements); + } + + private async Task CaptureAndMeasureWithFallbackAsync( + VisualInspectionCellContext cell, + string cycleId, + string fixture, + CancellationToken cancellationToken) + { + try + { + return await CaptureAndMeasureAsync(cell, cycleId, fixture, cancellationToken).ConfigureAwait(false); + } + catch (ServiceResultException ex) when (ex.StatusCode == StatusCodes.BadNodeIdUnknown) + { + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Cell clip endpoint rejected direct fixture capture ({ex.StatusCode}); using the packaged fixture image reference and still invoking the cell AI deployment.")); + return await CaptureAndMeasureFromFixtureAsync(cell, cycleId, fixture, cancellationToken) + .ConfigureAwait(false); + } + } + + private async Task> InvokeAIForMeasurementsAsync( + VisualInspectionCellContext cell, + string cycleId, + string fixture, + VisionImageReferenceDataType frame, + CancellationToken cancellationToken) + { + if (cell.Snapshot.DeploymentId.IsNull) + { + throw new InvalidOperationException("The pipeline does not name an AI deployment."); + } + string payload = BuildAIPayload(cycleId, fixture, frame); + for (int attempt = 1; attempt <= MaxOperationalAttempts; attempt++) + { + try + { + AIInvokeResult result = await cell.Deployment.InvokeAsync( + ByteString.From(Encoding.UTF8.GetBytes(payload)), + "application/vnd.opcfoundation.visual-inspection.measurements+json", + ArrayOf.Empty, + 5000, + cancellationToken: cancellationToken).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] AI deployment Invoke returned {result.ResponseContentType}; evidence only.")); + return ParseMeasurements(result.ResponsePayload); + } + catch (ServiceResultException ex) when (ex.StatusCode == StatusCodes.BadNotImplemented) + { + AIInvokeResult result = await InvokeAIMethodByInstanceNodeAsync( + cell.Snapshot.DeploymentId, + ByteString.From(Encoding.UTF8.GetBytes(payload)), + "application/vnd.opcfoundation.visual-inspection.measurements+json", + cancellationToken).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] AI deployment Invoke instance method returned {result.ResponseContentType}; evidence only.")); + return ParseMeasurements(result.ResponsePayload); + } + catch (Exception ex) when (attempt < MaxOperationalAttempts && IsOperationalFailure(ex)) + { + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Operational AI/camera error attempt {attempt}: {ex.Message}; retrying.")); + } + } + + throw new InvalidOperationException("AI Invoke failed after bounded retries; holding without a quality verdict."); + } + + private async Task InvokeAIMethodByInstanceNodeAsync( + NodeId deploymentId, + ByteString payload, + string contentType, + CancellationToken cancellationToken) + { + NodeId methodId = await FindChildByBrowseNameAsync( + deploymentId, + "Invoke", + NodeClass.Method, + cancellationToken).ConfigureAwait(false); + if (methodId.IsNull) + { + throw new ServiceResultException(StatusCodes.BadMethodInvalid); + } + var request = new CallMethodRequest + { + ObjectId = deploymentId, + MethodId = methodId, + InputArguments = + [ + Variant.From(payload), + Variant.From(string.Empty), + Variant.From(contentType), + Variant.FromStructure(ArrayOf.Empty), + Variant.From(5000.0) + ] + }; + CallResponse response = await m_sample.Session.CallAsync( + null, + [request], + cancellationToken).ConfigureAwait(false); + CallMethodResult result = response.Results[0]; + if (StatusCode.IsBad(result.StatusCode)) + { + throw new ServiceResultException(result.StatusCode); + } + ArrayOf output = result.OutputArguments; + return new AIInvokeResult + { + ResponsePayload = output.Count > 0 && output[0].TryGetValue(out ByteString responsePayload) + ? responsePayload + : ByteString.Empty, + ResponseContentType = output.Count > 1 && output[1].TryGetValue(out string responseContentType) + ? responseContentType + : string.Empty + }; + } + + private async Task CompleteInspectionJobAsync( + VisualInspectionCellContext cell, + string cycleId, + CancellationToken cancellationToken) + { + await EnsureInspectionOrderExecutingAsync(cell, cycleId, cancellationToken).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Completing ISA-95 inspection order {InspectionOrderId} from the execution state.")); + await EnsureJobMethodSucceededAsync( + cell.JobControl.StopAsync(InspectionOrderId, Comment(cycleId, "complete"), cancellationToken), + "Complete inspection", + cycleId).ConfigureAwait(false); + await EnsureJobMethodSucceededAsync( + cell.JobControl.ClearAsync(InspectionOrderId, Comment(cycleId, "close"), cancellationToken), + "Close inspection", + cycleId).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Inspection job completed and closed; no ReturnStatus=0x8.")); + } + + /// + /// Re-establishes the inspection order when an earlier run left none executing. + /// + /// + /// A run that ends on NotDecidable holds without scheduling the next inspection, + /// so the order the cell seeded at start-up is already closed by the time the next run + /// begins. Completing it then fails, which used to end the sample on an unhandled + /// exception rather than simply carrying on. + /// + private static async Task EnsureInspectionOrderExecutingAsync( + VisualInspectionCellContext cell, + string cycleId, + CancellationToken cancellationToken) + { + ulong returnStatus = await cell.JobControl.StoreAndStartAsync( + NewCatalogueOrder(InspectionOrderId, cycleId), + Comment(cycleId, "ensure-executing"), + cancellationToken).ConfigureAwait(false); + if ((returnStatus & Isa95ReturnStatusSuccess) != 0) + { + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] No inspection order was executing; re-established {InspectionOrderId}.")); + return; + } + if ((returnStatus & Isa95ReturnStatusUnableToAccept) == 0) + { + ThrowJobMethodFailure("Ensure inspection order", cycleId, returnStatus); + } + } + + private async Task StoreAndStartIdempotentAsync( + Isa95JobControlV2Client jobControl, + string orderId, + string cycleId, + bool verifyRetry, + CancellationToken cancellationToken) + { + if (await OrderExistsAsync(jobControl, orderId, cancellationToken).ConfigureAwait(false)) + { + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Order {orderId} already exists; retry suppressed to avoid a duplicate.")); + return; + } + ulong returnStatus = await jobControl.StoreAndStartAsync( + NewCatalogueOrder(orderId, cycleId), + Comment(cycleId, "store-start"), + cancellationToken).ConfigureAwait(false); + if ((returnStatus & Isa95ReturnStatusSuccess) == 0) + { + if ((returnStatus & Isa95ReturnStatusUnableToAccept) != 0) + { + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] StoreAndStart ReturnStatus=0x{returnStatus:X}; stable order {orderId} already exists, so no duplicate was created.")); + return; + } + ThrowJobMethodFailure("StoreAndStart", cycleId, returnStatus); + } + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] StoreAndStart ReturnStatus=0x{returnStatus:X}.")); + if (verifyRetry) + { + await StoreAndStartIdempotentAsync( + jobControl, + orderId, + FormattableString.Invariant($"{cycleId}-retry"), + verifyRetry: false, + cancellationToken).ConfigureAwait(false); + } + } + + private async Task OrderExistsAsync( + Isa95JobControlV2Client jobControl, + string orderId, + CancellationToken cancellationToken) + { + try + { + (V2.ISA95JobResponseDataType response, ulong returnStatus) = await jobControl + .RequestJobResponseByJobOrderIdAsync(orderId, cancellationToken) + .ConfigureAwait(false); + return (returnStatus & Isa95ReturnStatusSuccess) != 0 && response != null; + } + catch (ServiceResultException ex) when (StatusCode.IsUncertain(ex.StatusCode)) + { + return false; + } + } + + private async Task HoldForOperatorAsync( + VisualInspectionCellContext cell, + string cycleId, + InspectionDecision decision, + CancellationToken cancellationToken) + { + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] NotDecidable: holding; no job is scheduled until the dialog is answered.")); + ulong before = await ReadSamplesCollectedAsync(cell, cancellationToken).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] SamplesCollected before operator correction: {before}.")); + OperatorDisposition disposition = await m_operatorPolicy.GetDispositionAsync( + m_options.Mode, + m_options.OperatorTimeout, + cancellationToken).ConfigureAwait(false); + if (disposition == OperatorDisposition.Stop) + { + Console.WriteLine(FormattableString.Invariant($"[{cycleId}] Operator requested stop.")); + return; + } + if (!cell.OperatorDialogId.IsNull) + { + await new AlarmClient(m_sample.Session, m_sample.Telemetry) + .RespondAsync(cell.OperatorDialogId, ToDialogResponse(disposition), cancellationToken) + .ConfigureAwait(false); + } + if (disposition == OperatorDisposition.Reinspect) + { + await StoreAndStartIdempotentAsync( + cell.JobControl, + InspectionOrderId, + cycleId, + verifyRetry: false, + cancellationToken) + .ConfigureAwait(false); + return; + } + + bool retractAll = disposition == OperatorDisposition.AcceptAsNotOk; + await cell.Feedback.SubmitCorrectionAsync( + cycleId, + VisionFeedbackPurposeEnum.GroundTruthLabel, + ArrayOf.Empty, + retractAll ? ArrayOf.Empty : decision.Characteristics, + LocalizedText.From(disposition.ToString()), + ByteString.Empty, + retractAll, + cancellationToken).ConfigureAwait(false); + ulong afterFirst = await ReadSamplesCollectedAsync(cell, cancellationToken).ConfigureAwait(false); + await cell.Feedback.SubmitCorrectionAsync( + cycleId, + VisionFeedbackPurposeEnum.GroundTruthLabel, + ArrayOf.Empty, + retractAll ? ArrayOf.Empty : decision.Characteristics, + LocalizedText.From(disposition.ToString()), + ByteString.Empty, + retractAll, + cancellationToken).ConfigureAwait(false); + ulong afterDuplicate = await ReadSamplesCollectedAsync(cell, cancellationToken).ConfigureAwait(false); + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] Operator {disposition}; SamplesCollected {before}->{afterFirst}->{afterDuplicate} after duplicate correction.")); + } + + private async Task FindObjectByBrowseNameAsync( + string browseName, + CancellationToken cancellationToken) + { + var pending = new Queue(); + var visited = new HashSet(); + pending.Enqueue(global::Opc.Ua.ObjectIds.ObjectsFolder); + visited.Add(global::Opc.Ua.ObjectIds.ObjectsFolder); + while (pending.Count > 0) + { + NodeId current = pending.Dequeue(); + (ArrayOf> descriptions, ArrayOf errors) = + await m_sample.Session.ManagedBrowseAsync( + requestHeader: null, + view: null, + nodesToBrowse: [current], + maxResultsToReturn: 0, + browseDirection: BrowseDirection.Forward, + referenceTypeId: default, + includeSubtypes: true, + nodeClassMask: (uint)NodeClass.Object, + ct: cancellationToken).ConfigureAwait(false); + if (errors.Count > 0 && StatusCode.IsBad(errors[0].StatusCode)) + { + throw new ServiceResultException(errors[0]); + } + if (descriptions.Count == 0) + { + continue; + } + foreach (ReferenceDescription reference in descriptions[0]) + { + NodeId nodeId = ExpandedNodeId.ToNodeId(reference.NodeId, m_sample.Session.NamespaceUris); + if (nodeId.IsNull) + { + continue; + } + if (string.Equals(reference.BrowseName.Name, browseName, StringComparison.Ordinal)) + { + return nodeId; + } + if (visited.Add(nodeId)) + { + pending.Enqueue(nodeId); + } + } + } + return NodeId.Null; + } + + private async Task ReadSamplesCollectedAsync( + VisualInspectionCellContext cell, + CancellationToken cancellationToken) + { + if (cell.Snapshot.LearningJobId.IsNull) + { + return 0; + } + NodeId samples = await FindChildByBrowseNameAsync( + cell.Snapshot.LearningJobId, + "SamplesCollected", + NodeClass.Variable, + cancellationToken).ConfigureAwait(false); + if (samples.IsNull) + { + return 0; + } + DataValue value = await m_sample.Session.ReadValueAsync(samples, cancellationToken).ConfigureAwait(false); + return value.WrappedValue.TryGetValue(out ulong count) ? count : 0; + } + + private async Task FindChildByBrowseNameAsync( + NodeId parent, + string browseName, + NodeClass nodeClass, + CancellationToken cancellationToken) + { + (ArrayOf> descriptions, ArrayOf errors) = + await m_sample.Session.ManagedBrowseAsync( + requestHeader: null, + view: null, + nodesToBrowse: [parent], + maxResultsToReturn: 0, + browseDirection: BrowseDirection.Forward, + referenceTypeId: default, + includeSubtypes: true, + nodeClassMask: (uint)nodeClass, + ct: cancellationToken).ConfigureAwait(false); + if (errors.Count > 0 && StatusCode.IsBad(errors[0].StatusCode)) + { + throw new ServiceResultException(errors[0]); + } + if (descriptions.Count == 0) + { + return NodeId.Null; + } + foreach (ReferenceDescription reference in descriptions[0]) + { + if (string.Equals(reference.BrowseName.Name, browseName, StringComparison.Ordinal)) + { + return ExpandedNodeId.ToNodeId(reference.NodeId, m_sample.Session.NamespaceUris); + } + } + return NodeId.Null; + } + + private static NodeId FindFacet(Isa95JobControlDiscovery discovery, Isa95JobControlFacet facet) + { + foreach (Isa95JobControlEndpoint endpoint in discovery.V2Endpoints) + { + if (endpoint.Facet == facet) + { + return endpoint.NodeId; + } + } + throw new InvalidOperationException($"ISA-95 V2 facet {facet} was not found."); + } + + private static async Task EnsureJobMethodSucceededAsync( + ValueTask operation, + string operationName, + string cycleId) + { + ulong returnStatus = await operation.ConfigureAwait(false); + if ((returnStatus & Isa95ReturnStatusSuccess) == 0) + { + ThrowJobMethodFailure(operationName, cycleId, returnStatus); + } + Console.WriteLine(FormattableString.Invariant( + $"[{cycleId}] {operationName} ReturnStatus=0x{returnStatus:X}.")); + } + + private static void ThrowJobMethodFailure(string operationName, string cycleId, ulong returnStatus) + { + throw new InvalidOperationException(FormattableString.Invariant( + $"[{cycleId}] {operationName} returned ISA-95 ReturnStatus=0x{returnStatus:X}; business failures are not success even when the OPC UA service status is Uncertain.")); + } + + private static ArrayOf Comment(string cycleId, string action) + { + return new[] { LocalizedText.From(FormattableString.Invariant($"{cycleId}:{action}")) }.ToArrayOf(); + } + + private static V2.ISA95JobOrderDataType NewCatalogueOrder(string orderId, string cycleId) + { + return new V2.ISA95JobOrderDataType + { + JobOrderID = orderId, + Priority = string.Equals(orderId, InspectionOrderId, StringComparison.Ordinal) ? (short)10 : (short)20, + Description = new[] + { + LocalizedText.From(FormattableString.Invariant($"Allowlisted catalogue order for {cycleId}.")) + }.ToArrayOf() + }; + } + + private static string BuildAIPayload( + string cycleId, + string fixture, + VisionImageReferenceDataType frame) + { + var builder = new StringBuilder(); + builder.Append(CultureInfo.InvariantCulture, + $"{{\"cycleId\":\"{cycleId}\",\"fixture\":\"{fixture}\",\"image\":\"") + .Append(frame.Uri) + .Append("\"}"); + return builder.ToString(); + } + + private static ArrayOf ParseMeasurements(ByteString responsePayload) + { + if (responsePayload.Length == 0) + { + throw new InvalidDataException("AI Invoke returned an empty response."); + } + using JsonDocument document = JsonDocument.Parse(responsePayload.ToArray()); + JsonElement root = document.RootElement; + double confidence = ReadRequiredDouble(root, "confidence"); + if (!root.TryGetProperty("measurements", out JsonElement measurements) || + measurements.ValueKind != JsonValueKind.Array) + { + throw new InvalidDataException("AI Invoke response did not contain a measurements array."); + } + var parsed = new List(); + foreach (JsonElement measurement in measurements.EnumerateArray()) + { + string characteristicId = ReadRequiredString(measurement, "characteristicId"); + double actual = ReadRequiredDouble(measurement, "actual"); + double uncertainty = ReadRequiredDouble(measurement, "uncertainty"); + parsed.Add(new MeasuredCharacteristic(characteristicId, actual, uncertainty, confidence)); + } + if (parsed.Count == 0) + { + throw new InvalidDataException("AI Invoke response contained no measurements."); + } + return parsed.ToArrayOf(); + } + + private static VisionImageReferenceDataType CreateFixtureImageReference(string fixture) + { + string path = Path.Combine(FixtureDirectory, fixture); + ByteString png = ByteString.From(File.ReadAllBytes(path)); + return new VisionImageReferenceDataType + { + Uri = FormattableString.Invariant($"opcua-inline://visual-inspection-cell/fixtures/{fixture}"), + Digest = ByteString.From(SHA256.HashData(png.Span)), + DigestAlgorithm = "SHA-256", + Format = VisionClipFormatEnum.Png, + PixelFormat = "RGB8", + Width = 800, + Height = 600, + SizeBytes = (uint)png.Length, + Timestamp = DateTimeUtc.From(DateTime.UnixEpoch) + }; + } + + private static string FixtureDirectory + { + get + { + string output = Path.Combine(AppContext.BaseDirectory, "Fixtures"); + if (Directory.Exists(output)) + { + return output; + } + + string project = Path.GetFullPath(Path.Combine( + AppContext.BaseDirectory, + "..", + "..", + "..", + "..", + "..", + "VisualInspectionCell", + "Fixtures")); + if (Directory.Exists(project)) + { + return project; + } + + return Path.Combine( + Environment.CurrentDirectory, + "samples", + "Vision", + "VisualInspectionCell", + "Fixtures"); + } + } + + private static string ReadRequiredString(JsonElement element, string name) + { + if (element.TryGetProperty(name, out JsonElement property) && + property.ValueKind == JsonValueKind.String && + property.GetString() is { Length: > 0 } value) + { + return value; + } + throw new InvalidDataException($"AI Invoke response missing string property '{name}'."); + } + + private static double ReadRequiredDouble(JsonElement element, string name) + { + if (!element.TryGetProperty(name, out JsonElement property)) + { + throw new InvalidDataException($"AI Invoke response missing numeric property '{name}'."); + } + if (property.ValueKind == JsonValueKind.Number && property.TryGetDouble(out double numeric)) + { + return numeric; + } + if (property.ValueKind == JsonValueKind.String && + double.TryParse( + property.GetString(), + NumberStyles.Float, + CultureInfo.InvariantCulture, + out double text)) + { + return text; + } + throw new InvalidDataException($"AI Invoke response property '{name}' is not numeric."); + } + + private static bool IsOperationalFailure(Exception exception) + { + return exception is ServiceResultException or JsonException or InvalidDataException or IOException; + } + + private static string FormatMeasurements(ArrayOf measurements) + { + var formatted = new List(measurements.Count); + for (int ii = 0; ii < measurements.Count; ii++) + { + MeasuredCharacteristic measurement = measurements[ii]; + formatted.Add(FormattableString.Invariant( + $"{measurement.CharacteristicId}={measurement.Actual:0.00}mm confidence={measurement.Confidence:0.00}")); + } + return string.Join(", ", formatted); + } + + private static string StableCycleId(int cycle) + { + return FormattableString.Invariant($"vis-agent-cycle-{cycle:000}"); + } + + private static string FixtureFor(int cycle) + { + return ((cycle - 1) % 3) switch + { + 0 => "bracket-ok.png", + 1 => "bracket-not-ok.png", + _ => "bracket-ambiguous.png" + }; + } + + private static int ToDialogResponse(OperatorDisposition disposition) + { + return disposition switch + { + OperatorDisposition.AcceptAsOk => 0, + OperatorDisposition.AcceptAsNotOk => 1, + OperatorDisposition.Reinspect => 2, + _ => 3 + }; + } + + private const string PipelineBrowseName = "BracketInspectionPipeline"; + private const string OperatorDialogBrowseName = "OperatorDispositionDialog"; + private const string InspectionOrderId = "VIS-INSP-BRACKET-001"; + private const string ReworkRejectOrderId = "VIS-REWORK-REJECT-001"; + private const ulong Isa95ReturnStatusSuccess = 1UL; + private const ulong Isa95ReturnStatusUnableToAccept = 0x10UL; + private const int MaxOperationalAttempts = 2; + + private readonly InspectionVerdictPolicy m_policy = new(); + private readonly ScriptedOperatorPolicy m_operatorPolicy = new(); + private readonly VisualInspectionAgentOptions m_options; + private readonly VisualInspectionAgentSession m_sample; + } + + internal sealed record VisualInspectionCellContext( + VisionPipelineClient Pipeline, + VisionPipelineSnapshot Snapshot, + VisionMediaClient Media, + NodeId ClipEndpoint, + VisionFeedbackClient Feedback, + AIDeploymentClient Deployment, + Isa95JobControlV2Client JobControl, + NodeId OperatorDialogId); + + internal sealed record InspectionEvidence( + VisionImageReferenceDataType Frame, + ArrayOf Measurements); +} diff --git a/samples/Vision/VisualInspectionAgent/README.md b/samples/Vision/VisualInspectionAgent/README.md new file mode 100644 index 0000000000..31bcd17a28 --- /dev/null +++ b/samples/Vision/VisualInspectionAgent/README.md @@ -0,0 +1,117 @@ + + +# Visual Inspection Agent + +`VisualInspectionAgent` is the external deterministic orchestrator for the +[VisualInspectionCell](../VisualInspectionCell) server. It uses typed OPC UA +clients for Vision, AI Model Management, ISA-95 Job Control V2, and Alarms & +Conditions; it does not call server internals. + +The model path never owns the production verdict. The agent captures a fixture +image, measures characteristics, routes the measurements through the AI +companion deployment's `Invoke` method for provenance, and then applies the +recipe rule locally before touching job control. + +```mermaid +flowchart TD + Start["Start cycle"] --> Capture["Get fixture PNG from FixtureImages"] + Capture --> Measure["Measure BoreDiameter, SlotWidth, EdgeOffset"] + Measure --> Invoke["Call deployment Invoke for provenance"] + Invoke --> Judge["Apply deterministic recipe rule"] + Judge --> Decision{"Verdict"} + Decision -->|"Ok"| CompleteOk["Complete and close inspection job"] + CompleteOk --> NextInspection["StoreAndStart VIS-INSP-BRACKET-001"] + Decision -->|"NotOk"| CompleteBad["Complete and close inspection job"] + CompleteBad --> Rework["StoreAndStart VIS-REWORK-REJECT-001"] + Decision -->|"NotDecidable"| Hold["Hold for operator dialog"] + Hold --> Correction["Submit ground-truth correction"] +``` + +## Running + +Start the cell first: + +```powershell +dotnet run --project samples\Vision\VisualInspectionCell\VisualInspectionCell.csproj -- --insecure +``` + +Then run the agent: + +```powershell +dotnet run --project samples\Vision\VisualInspectionAgent\VisualInspectionAgent.csproj -- --server opc.tcp://localhost:62865/VisualInspectionCell --insecure --mode scripted --cycles 3 +``` + +## Options + +| Option | Meaning | +|---|---| +| `--server ` | Server endpoint. Default `opc.tcp://localhost:62865/VisualInspectionCell`. | +| `--insecure` | Demo-only certificate convenience. | +| `--mode scripted\|live-ai\|human` | Selects the operator/model mode. Default `scripted`. | +| `--cycles ` | Number of fixture cycles. Default `3`, minimum `1`. | +| `--operator-timeout ` | Bounded human wait. Default `10`, minimum `1`. | +| `--ai-endpoint ` | Required for `live-ai`. | + +## Exit codes + +| Code | Meaning | +|---|---| +| `0` | The requested cycles ran to completion. | +| `2` | The command line was rejected, for example an unknown `--mode` or `live-ai` without `--ai-endpoint`. No session is opened and no job is created. | +| `3` | The cell refused the run, for example an ISA-95 `ReturnStatus` that is not success, or a deployment that cannot serve `live-ai`. Reported as a single diagnostic line, not a stack trace. | + +Mode names are matched without regard to case, hyphens or underscores, so +`live-ai`, `live_ai` and `LiveAI` are the same mode. An unrecognised mode is +rejected rather than quietly treated as `scripted`, because a silent downgrade +would run the simulated analyser while the operator believed a real model was +deciding. + +## Modes + +- `scripted` — the unattended path. The analyser and operator disposition are + deterministic, and `--cycles N` makes the run finite. +- `live-ai` — requires `--ai-endpoint`. If no endpoint is configured, the agent + exits before creating any ISA-95 job. There is no silent fallback to the + simulated analyser, because a silently degraded sample appears to work while + demonstrating nothing. +- `human` — waits for a real dialog subscriber for a bounded time. A timeout + stops or holds; it never auto-approves and never blocks forever. + +## Job policy + +Quality and job execution state are separate facts. A defective part does not +mean the inspection job failed. + +| Verdict | Inspection job | Next job | +|---|---|---| +| `Ok` | complete, close | schedule next inspection | +| `NotOk` | complete, close | schedule rework/reject order | +| `NotDecidable` | hold | none until the operator answers | + +Scheduling chooses from the fixed catalogue exposed by the cell and calls ISA-95 +V2 `StoreAndStart`. The agent never invents a job-order payload outside that +catalogue. + +## Operator feedback and learning + +For `NotDecidable`, the agent responds to `OperatorDispositionDialog` and sends +Vision feedback with `VisionFeedbackPurposeEnum.GroundTruthLabel`. An accepted +positive correction carries the measured characteristics. An accepted negative +example uses the `RetractAll` path. The same correction is submitted twice in +the current orchestrator to demonstrate that the cell counts a learning sample +idempotently per stable sample id. + +## Related docs + +- [Vision developer guide](../../../docs/Vision.md) — see *Visual inspection: a cross-companion cell* +- [VisualInspectionCell](../VisualInspectionCell) +- [Vision developer guide](../../../docs/Vision.md) +- [AI Model Management developer guide](../../../docs/AiIntegration.md) +- [ISA-95 developer guide](../../../docs/ISA95.md) diff --git a/samples/Vision/VisualInspectionAgent/VisualInspectionAgent.csproj b/samples/Vision/VisualInspectionAgent/VisualInspectionAgent.csproj new file mode 100644 index 0000000000..60dfc596fc --- /dev/null +++ b/samples/Vision/VisualInspectionAgent/VisualInspectionAgent.csproj @@ -0,0 +1,30 @@ + + + net10.0 + Exe + false + VisualInspectionAgent + VisualInspectionAgent + Orchestrating client for the Vision visual-inspection sample cell. + Vision.VisualInspectionAgent + enable + $(NoWarn);CA1014;CA1812;CA1822;CA1852 + false + win-x64 + false + true + + + + + + + + + + + + + + + diff --git a/samples/Vision/VisualInspectionAgent/VisualInspectionAgentOptions.cs b/samples/Vision/VisualInspectionAgent/VisualInspectionAgentOptions.cs new file mode 100644 index 0000000000..e9f9939b71 --- /dev/null +++ b/samples/Vision/VisualInspectionAgent/VisualInspectionAgentOptions.cs @@ -0,0 +1,119 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Globalization; +using System.Linq; + +namespace Vision.VisualInspectionAgent +{ + internal sealed record VisualInspectionAgentOptions + { + public string ServerUrl { get; init; } = "opc.tcp://localhost:62865/VisualInspectionCell"; + + public bool Insecure { get; init; } + + public VisualInspectionAgentMode Mode { get; init; } = VisualInspectionAgentMode.Scripted; + + public int Cycles { get; init; } = 3; + + public TimeSpan OperatorTimeout { get; init; } = TimeSpan.FromSeconds(10); + + public string? AIEndpoint { get; init; } + + public static VisualInspectionAgentOptions Parse(string[] args) + { + return new VisualInspectionAgentOptions + { + ServerUrl = GetOption(args, "--server") ?? "opc.tcp://localhost:62865/VisualInspectionCell", + Insecure = HasFlag(args, "--insecure"), + Mode = ParseMode(GetOption(args, "--mode")), + Cycles = Math.Max(1, int.TryParse( + GetOption(args, "--cycles"), NumberStyles.Integer, CultureInfo.InvariantCulture, out int cycles) + ? cycles + : 3), + OperatorTimeout = TimeSpan.FromSeconds(Math.Max(1, int.TryParse( + GetOption(args, "--operator-timeout"), NumberStyles.Integer, CultureInfo.InvariantCulture, + out int timeout) + ? timeout + : 10)), + AIEndpoint = GetOption(args, "--ai-endpoint") + }; + } + + private static VisualInspectionAgentMode ParseMode(string? value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return VisualInspectionAgentMode.Scripted; + } + + // The documented spellings are hyphenated ("live-ai"), the enum members are not. + // Silently defaulting an unrecognised mode would drop live-ai into the simulated + // analyser, which is exactly the degraded run the mode exists to prevent. + string candidate = value.Replace("-", string.Empty, StringComparison.Ordinal) + .Replace("_", string.Empty, StringComparison.Ordinal); + if (Enum.TryParse(candidate, ignoreCase: true, out VisualInspectionAgentMode mode) && + Enum.IsDefined(mode)) + { + return mode; + } + + throw new FormatException( + FormattableString.Invariant( + $"Unknown --mode '{value}'. Expected one of: scripted, live-ai, human.")); + } + + private static string? GetOption(string[] args, string name) + { + for (int ii = 0; ii < args.Length - 1; ii++) + { + if (string.Equals(args[ii], name, StringComparison.OrdinalIgnoreCase)) + { + return args[ii + 1]; + } + } + return null; + } + + private static bool HasFlag(string[] args, string name) + { + return args.Any(a => string.Equals(a, name, StringComparison.OrdinalIgnoreCase)); + } + } + + internal enum VisualInspectionAgentMode + { + Scripted, + + LiveAI, + + Human + } +} diff --git a/samples/Vision/VisualInspectionAgent/VisualInspectionAgentSession.cs b/samples/Vision/VisualInspectionAgent/VisualInspectionAgentSession.cs new file mode 100644 index 0000000000..fd0d899945 --- /dev/null +++ b/samples/Vision/VisualInspectionAgent/VisualInspectionAgentSession.cs @@ -0,0 +1,169 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Client; +using Opc.Ua.Client.Subscriptions.Streaming; +using Opc.Ua.Configuration; + +namespace Vision.VisualInspectionAgent +{ + internal sealed class VisualInspectionAgentSession : IAsyncDisposable + { + private VisualInspectionAgentSession( + ISession session, + IStreamingSubscription streaming, + ApplicationConfiguration configuration, + ITelemetryContext telemetry) + { + Session = session; + Streaming = streaming; + m_configuration = configuration; + Telemetry = telemetry; + } + + public ISession Session { get; } + + public IStreamingSubscription Streaming { get; } + + public ITelemetryContext Telemetry { get; } + + public static async Task ConnectAsync( + VisualInspectionAgentOptions options, + CancellationToken cancellationToken) + { + if (options == null) + { + throw new ArgumentNullException(nameof(options)); + } + + ITelemetryContext telemetry = DefaultTelemetry.Create(builder => + builder.SetMinimumLevel(LogLevel.Warning)); + string pkiRoot = GetPrivateStateRoot(); + var configuration = new ApplicationConfiguration(telemetry) + { + ApplicationName = "VisualInspectionAgent", + ApplicationUri = "urn:localhost:OPCFoundation:VisualInspectionAgent", + ApplicationType = ApplicationType.Client, + SecurityConfiguration = new SecurityConfiguration + { + ApplicationCertificate = new CertificateIdentifier + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(pkiRoot, "own"), + SubjectName = "CN=VisualInspectionAgent, O=OPC Foundation" + }, + TrustedIssuerCertificates = new CertificateTrustList + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(pkiRoot, "issuer") + }, + TrustedPeerCertificates = new CertificateTrustList + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(pkiRoot, "trusted") + }, + RejectedCertificateStore = new CertificateTrustList + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(pkiRoot, "rejected") + }, + AutoAcceptUntrustedCertificates = options.Insecure + }, + TransportQuotas = new TransportQuotas { MaxMessageSize = 8 * 1024 * 1024 }, + ClientConfiguration = new ClientConfiguration(), + ServerConfiguration = new ServerConfiguration() + }; + await configuration.ValidateAsync(ApplicationType.Client, cancellationToken).ConfigureAwait(false); + + var appInstance = new ApplicationInstance(configuration, telemetry); + await appInstance.CheckApplicationInstanceCertificatesAsync(true, ct: cancellationToken) + .ConfigureAwait(false); + await appInstance.DisposeAsync().ConfigureAwait(false); + configuration.CertificateManager ??= CertificateManagerFactory.Create( + configuration.SecurityConfiguration, telemetry); + if (options.Insecure) + { + configuration.CertificateManager.AcceptError = static (_, _) => true; + Console.Error.WriteLine("WARNING: --insecure accepts any server certificate for the sample only."); + } + + EndpointDescription? endpointDescription = await CoreClientUtils.SelectEndpointAsync( + configuration, + options.ServerUrl, + useSecurity: true, + discoverTimeout: 15000, + telemetry, + cancellationToken).ConfigureAwait(false); + if (endpointDescription is null) + { + throw ServiceResultException.Create(StatusCodes.BadTimeout, "Could not reach {0}.", options.ServerUrl); + } + + var endpoint = new ConfiguredEndpoint( + null, + endpointDescription, + EndpointConfiguration.Create(configuration)); + ManagedSession session = await new ManagedSessionBuilder(configuration, telemetry) + .UseEndpoint(endpoint) + .WithSessionName("VisualInspectionAgent") + .WithSessionTimeout(TimeSpan.FromSeconds(60)) + .WithUserIdentity(new UserIdentity(new AnonymousIdentityToken())) + .ConnectAsync(cancellationToken).ConfigureAwait(false); + return new VisualInspectionAgentSession(session, session.DefaultStreaming, configuration, telemetry); + } + + public async ValueTask DisposeAsync() + { + await Streaming.DisposeAsync().ConfigureAwait(false); + await Session.CloseAsync(CancellationToken.None).ConfigureAwait(false); + await Session.DisposeAsync().ConfigureAwait(false); + (m_configuration.CertificateManager as IDisposable)?.Dispose(); + } + + private static string GetPrivateStateRoot() + { + string baseDirectory = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData); + if (string.IsNullOrEmpty(baseDirectory)) + { + baseDirectory = AppContext.BaseDirectory; + } + string root = Path.Combine(baseDirectory, "OPC Foundation", "VisualInspectionAgent", "pki"); + Directory.CreateDirectory(root); + return root; + } + + private readonly ApplicationConfiguration m_configuration; + } +} diff --git a/samples/Vision/VisualInspectionCell/AiNodeManagerRegistry.cs b/samples/Vision/VisualInspectionCell/AiNodeManagerRegistry.cs new file mode 100644 index 0000000000..3c082939ce --- /dev/null +++ b/samples/Vision/VisualInspectionCell/AiNodeManagerRegistry.cs @@ -0,0 +1,175 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.AI.Server; +using Opc.Ua.Server; +using Opc.Ua.Server.Hosting; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Vision.VisualInspectionCell +{ + internal sealed class AINodeManagerRegistry : IServerStartupTask + { + public AINodeManagerRegistry(ILogger logger) + { + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public AINodeManager? NodeManager => Volatile.Read(ref m_nodeManager); + + public ValueTask OnServerStartedAsync( + IServerContext server, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + if (server is IServerInternal internalServer) + { + AINodeManager? manager = internalServer.NodeManager.AsyncNodeManagers + .OfType() + .FirstOrDefault(); + Volatile.Write(ref m_nodeManager, manager); + if (manager != null) + { + FixDeploymentMethodDeclarations(manager); + BindVisionPipeline(internalServer, manager); + } + if (m_logger.IsEnabled(LogLevel.Information)) + { + m_logger.AIManagerCaptured(manager?.LearningJobId.ToString() ?? string.Empty); + } + } + return default; + } + + private void BindVisionPipeline(IServerInternal server, AINodeManager ai) + { + VisionNodeManager? vision = server.NodeManager.AsyncNodeManagers + .OfType() + .FirstOrDefault(); + InferencePipelineState? pipeline = FindPipeline(vision); + if (pipeline == null) + { + return; + } + if (pipeline.Deployment != null) + { + pipeline.Deployment.Value = ai.PrimaryDeploymentId; + } + if (pipeline.LearningJob != null) + { + pipeline.LearningJob.Value = ai.LearningJobId; + } + pipeline.ClearChangeMasks(vision!.SystemContext, true); + if (m_logger.IsEnabled(LogLevel.Information)) + { + m_logger.VisionPipelineBound(ai.PrimaryDeploymentId.ToString(), ai.LearningJobId.ToString()); + } + } + + private void FixDeploymentMethodDeclarations(AINodeManager ai) + { + FixMethodDeclaration(ai, ai.PrimaryDeploymentId, Opc.Ua.AI.BrowseNames.Invoke, 6136); + FixMethodDeclaration(ai, ai.PrimaryDeploymentId, Opc.Ua.AI.BrowseNames.InvokeAsync, 6139); + FixMethodDeclaration(ai, ai.PrimaryDeploymentId, Opc.Ua.AI.BrowseNames.GetCapabilities, 6056); + FixMethodDeclaration(ai, ai.PrimaryDeploymentId, Opc.Ua.AI.BrowseNames.BeginTransfer, 6157); + if (!ai.FallbackDeploymentId.IsNull) + { + FixMethodDeclaration(ai, ai.FallbackDeploymentId, Opc.Ua.AI.BrowseNames.Invoke, 6136); + FixMethodDeclaration(ai, ai.FallbackDeploymentId, Opc.Ua.AI.BrowseNames.InvokeAsync, 6139); + FixMethodDeclaration(ai, ai.FallbackDeploymentId, Opc.Ua.AI.BrowseNames.GetCapabilities, 6056); + FixMethodDeclaration(ai, ai.FallbackDeploymentId, Opc.Ua.AI.BrowseNames.BeginTransfer, 6157); + } + } + + private static void FixMethodDeclaration( + AINodeManager ai, + NodeId deploymentId, + string browseName, + uint methodDeclarationId) + { + if (deploymentId.IsNull) + { + return; + } + BaseInstanceState deployment = ai.FindPredefinedNode(deploymentId); + var children = new List(); + deployment.GetChildren(ai.SystemContext, children); + MethodState? method = children.OfType().FirstOrDefault( + child => string.Equals(child.BrowseName.Name, browseName, StringComparison.Ordinal)); + if (method != null) + { + method.MethodDeclarationId = new NodeId(methodDeclarationId, deploymentId.NamespaceIndex); + } + } + + private static InferencePipelineState? FindPipeline(VisionNodeManager? vision) + { + FolderState? pipelines = vision?.Root.Pipelines; + if (pipelines == null || vision == null) + { + return null; + } + var children = new List(); + pipelines.GetChildren(vision.SystemContext, children); + return children.OfType().FirstOrDefault( + child => string.Equals( + child.BrowseName.Name, + VisualInspectionCell.PipelineBrowseName, + StringComparison.Ordinal)); + } + + private readonly ILogger m_logger; + private AINodeManager? m_nodeManager; + } + + internal static partial class AINodeManagerRegistryLog + { + [LoggerMessage(EventId = VisualInspectionCellEventIds.AI + 1, + Level = LogLevel.Information, + Message = "Captured AI node manager; LearningJob={LearningJobNodeId}.")] + public static partial void AIManagerCaptured( + this ILogger logger, + string learningJobNodeId); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.AI + 2, + Level = LogLevel.Information, + Message = "Bound Vision pipeline to Deployment={DeploymentNodeId}, LearningJob={LearningJobNodeId}.")] + public static partial void VisionPipelineBound( + this ILogger logger, + string deploymentNodeId, + string learningJobNodeId); + } +} diff --git a/samples/Vision/VisualInspectionCell/EventIds.cs b/samples/Vision/VisualInspectionCell/EventIds.cs new file mode 100644 index 0000000000..09129b0f62 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/EventIds.cs @@ -0,0 +1,45 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Vision.VisualInspectionCell +{ + /// + /// Event ids for source-generated log messages in this sample. + /// + internal static class VisualInspectionCellEventIds + { + public const int Configurator = 0; + public const int Media = 20; + public const int Inference = 40; + public const int Feedback = 80; + public const int Isa95 = 120; + public const int AI = 160; + public const int Dialog = 200; + } +} diff --git a/samples/Vision/VisualInspectionCell/FixtureImageAnalyzer.cs b/samples/Vision/VisualInspectionCell/FixtureImageAnalyzer.cs new file mode 100644 index 0000000000..0d919a8a15 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/FixtureImageAnalyzer.cs @@ -0,0 +1,130 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.IO; + +namespace Vision.VisualInspectionCell +{ + /// + /// Measures the bracket geometry from pixels instead of from fixture names. + /// + internal sealed class FixtureImageAnalyzer + { + public IReadOnlyList Measure(string fixturePath) + { + if (string.IsNullOrWhiteSpace(fixturePath)) + { + throw new ArgumentException("A fixture path is required.", nameof(fixturePath)); + } + + byte[] png = File.ReadAllBytes(fixturePath); + (byte[] rgb, int width, int height) = PngDecoder.Decode(png); + if (width != ImageWidth || height != ImageHeight) + { + throw new InvalidDataException($"Expected {ImageWidth}x{ImageHeight}, got {width}x{height}."); + } + + int borePixels = CountDarkRunThrough(rgb, width, BoreCenterY, BoreCenterX); + PixelRun slot = FindDarkRunRightOf(rgb, width, SlotCenterY, minX: 400); + double bore = borePixels / ScalePixelsPerMillimetre; + double slotWidth = slot.Length / ScalePixelsPerMillimetre; + double edgeOffset = (BracketRightX - slot.StartX) / ScalePixelsPerMillimetre; + return + [ + new MeasuredCharacteristic("BoreDiameter", bore, PixelPitchMillimetres), + new MeasuredCharacteristic("SlotWidth", slotWidth, PixelPitchMillimetres), + new MeasuredCharacteristic("EdgeOffset", edgeOffset, PixelPitchMillimetres) + ]; + } + + private static int CountDarkRunThrough(byte[] rgb, int width, int y, int centerX) + { + int start = centerX; + while (start > 0 && IsDark(rgb, width, start - 1, y)) + { + start--; + } + int end = centerX; + while (end + 1 < width && IsDark(rgb, width, end + 1, y)) + { + end++; + } + return end - start + 1; + } + + private static PixelRun FindDarkRunRightOf(byte[] rgb, int width, int y, int minX) + { + int bestStart = -1; + int bestLength = 0; + int x = minX; + while (x < BracketRightX) + { + while (x < BracketRightX && !IsDark(rgb, width, x, y)) + { + x++; + } + int start = x; + while (x < BracketRightX && IsDark(rgb, width, x, y)) + { + x++; + } + int length = x - start; + if (length > bestLength) + { + bestStart = start; + bestLength = length; + } + } + if (bestStart < 0) + { + throw new InvalidDataException("Could not find the slot in the fixture image."); + } + return new PixelRun(bestStart, bestLength); + } + + private static bool IsDark(byte[] rgb, int width, int x, int y) + { + int offset = ((y * width) + x) * 3; + return rgb[offset] < 64 && rgb[offset + 1] < 64 && rgb[offset + 2] < 64; + } + + private const int ImageWidth = 800; + private const int ImageHeight = 600; + private const int BoreCenterX = 290; + private const int BoreCenterY = 300; + private const int SlotCenterY = 300; + private const int BracketRightX = 650; + private const double ScalePixelsPerMillimetre = 10.0; + private const double PixelPitchMillimetres = 0.10; + + private readonly record struct PixelRun(int StartX, int Length); + } +} diff --git a/samples/Vision/VisualInspectionCell/Fixtures/bracket-ambiguous.png b/samples/Vision/VisualInspectionCell/Fixtures/bracket-ambiguous.png new file mode 100644 index 0000000000..cab3238388 Binary files /dev/null and b/samples/Vision/VisualInspectionCell/Fixtures/bracket-ambiguous.png differ diff --git a/samples/Vision/VisualInspectionCell/Fixtures/bracket-not-ok.png b/samples/Vision/VisualInspectionCell/Fixtures/bracket-not-ok.png new file mode 100644 index 0000000000..c9093676bc Binary files /dev/null and b/samples/Vision/VisualInspectionCell/Fixtures/bracket-not-ok.png differ diff --git a/samples/Vision/VisualInspectionCell/Fixtures/bracket-ok.png b/samples/Vision/VisualInspectionCell/Fixtures/bracket-ok.png new file mode 100644 index 0000000000..77935f09f6 Binary files /dev/null and b/samples/Vision/VisualInspectionCell/Fixtures/bracket-ok.png differ diff --git a/samples/Vision/VisualInspectionCell/Fixtures/generate_fixtures.py b/samples/Vision/VisualInspectionCell/Fixtures/generate_fixtures.py new file mode 100644 index 0000000000..6d01a568db --- /dev/null +++ b/samples/Vision/VisualInspectionCell/Fixtures/generate_fixtures.py @@ -0,0 +1,125 @@ +"""Generates the three inspection fixtures for the VisualInspectionCell sample. + +The sample's deterministic analyser measures these images by counting pixels +and converting to millimetres through a known calibration scale, so the +geometry written here IS the ground truth the verdict is judged against. +Regenerate with: python tools/generate_fixtures.py + +At 10 px/mm a feature edge can only fall on a pixel boundary, so a measurement +carries a quantisation uncertainty of one pixel, 0.10 mm. That is not an +invented number to make the demonstration work - it is the camera's pixel +pitch, and it is what makes the Uncertainty field of +VisionCharacteristicDataType mean something physical. + +One recipe, three parts: + + bore diameter nominal 12.00 mm, tolerance +/- 0.20 + slot width nominal 8.00 mm, tolerance +/- 0.15 + edge offset nominal 20.00 mm, tolerance +/- 0.25 + + bracket-ok bore 12.00, slot 8.00. Every interval falls wholly + inside its tolerance band, so the verdict is Ok. + bracket-not-ok bore 12.60. The interval [12.50, 12.70] lies wholly + outside the [11.80, 12.20] band, so the verdict is NotOk. + bracket-ambiguous slot draws as 8.10 - 8.15 mm is 81.5 px and cannot be + drawn. Its interval [8.00, 8.20] straddles the 8.15 + tolerance limit, so no verdict is possible and the + characteristic is NotDecidable. This is precisely the + case the specification defines that value for, and the + case that escalates to a human. +""" + +import struct +import sys +import zlib +from pathlib import Path + +WIDTH = 800 +HEIGHT = 600 +SCALE_PX_PER_MM = 10.0 + +BACKGROUND = (28, 30, 34) +BRACKET = (176, 180, 188) +CUT = (18, 19, 22) + + +def new_canvas(): + return [[BACKGROUND for _ in range(WIDTH)] for _ in range(HEIGHT)] + + +def fill_rect(px, x0, y0, x1, y1, colour): + for y in range(max(0, int(y0)), min(HEIGHT, int(y1))): + for x in range(max(0, int(x0)), min(WIDTH, int(x1))): + px[y][x] = colour + + +def fill_circle(px, cx, cy, radius, colour): + r2 = radius * radius + for y in range(max(0, int(cy - radius) - 1), min(HEIGHT, int(cy + radius) + 2)): + for x in range(max(0, int(cx - radius) - 1), min(WIDTH, int(cx + radius) + 2)): + dx = x + 0.5 - cx + dy = y + 0.5 - cy + if dx * dx + dy * dy <= r2: + px[y][x] = colour + + +def render(bore_mm, slot_mm, edge_mm): + """Draws a bracket whose features measure exactly the given millimetres.""" + px = new_canvas() + + body_x0, body_y0 = 150, 110 + body_x1, body_y1 = 650, 490 + fill_rect(px, body_x0, body_y0, body_x1, body_y1, BRACKET) + + # The bore: a through hole, measured across its diameter. + bore_r = (bore_mm * SCALE_PX_PER_MM) / 2.0 + fill_circle(px, 290.0, 300.0, bore_r, CUT) + + # The slot: measured across its width. Its left edge sits edge_mm from the + # bracket's right-hand edge, which is the third characteristic. + slot_w = slot_mm * SCALE_PX_PER_MM + slot_left = body_x1 - (edge_mm * SCALE_PX_PER_MM) + fill_rect(px, slot_left, 200.0, slot_left + slot_w, 400.0, CUT) + + return px + + +def write_png(path, px): + raw = bytearray() + for row in px: + raw.append(0) + for r, g, b in row: + raw += bytes((r, g, b)) + + def chunk(tag, data): + out = struct.pack(">I", len(data)) + tag + data + return out + struct.pack(">I", zlib.crc32(tag + data) & 0xFFFFFFFF) + + header = struct.pack(">IIBBBBB", WIDTH, HEIGHT, 8, 2, 0, 0, 0) + blob = ( + b"\x89PNG\r\n\x1a\n" + + chunk(b"IHDR", header) + + chunk(b"IDAT", zlib.compress(bytes(raw), 9)) + + chunk(b"IEND", b"") + ) + Path(path).write_bytes(blob) + return len(blob) + + +def main(): + out = Path(sys.argv[1] if len(sys.argv) > 1 else ".") + out.mkdir(parents=True, exist_ok=True) + + parts = { + "bracket-ok.png": (12.00, 8.00, 20.00), + "bracket-not-ok.png": (12.60, 8.00, 20.00), + "bracket-ambiguous.png": (12.00, 8.15, 20.00), + } + + for name, (bore, slot, edge) in parts.items(): + size = write_png(out / name, render(bore, slot, edge)) + print(f"{name:26} bore={bore:5.2f} slot={slot:5.2f} edge={edge:5.2f} {size:6d} bytes") + + +if __name__ == "__main__": + main() diff --git a/samples/Vision/VisualInspectionCell/InspectionJobControlProvider.cs b/samples/Vision/VisualInspectionCell/InspectionJobControlProvider.cs new file mode 100644 index 0000000000..fef6fe9804 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/InspectionJobControlProvider.cs @@ -0,0 +1,214 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua; +using Opc.Ua.ISA95.Server.Providers; +using V2 = Opc.Ua.ISA95.JobControl.V2; + +namespace Vision.VisualInspectionCell +{ + internal sealed class InspectionJobControlProvider : + IIsa95JobOrderReceiverV2, + IIsa95JobResponseProviderV2, + IIsa95JobResponseReceiverV2, + IIsa95JobStatusSourceV2, + IIsa95JobExecutionController, + IIsa95JobOrderCatalog, + IIsa95JobOrderCatalogChangeSource, + IDisposable + { + public InspectionJobControlProvider(TimeProvider? timeProvider = null) + { + m_provider = new InMemoryIsa95JobControlProvider( + new Isa95JobControlProviderOptions + { + MaxJobOrders = AllowedOrders.Count, + MaxJobResponses = 16, + ResponseRetention = TimeSpan.Zero + }, + timeProvider ?? TimeProvider.System); + } + + public ushort MaxDownloadableJobOrders => m_provider.MaxDownloadableJobOrders; + + public ValueTask> GetJobOrdersV1Async( + CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + return new ValueTask>( + ArrayOf.Empty); + } + + public ValueTask> GetJobOrdersV2Async( + CancellationToken cancellationToken = default) + { + return m_provider.GetJobOrdersV2Async(cancellationToken); + } + + public async ValueTask ReceiveJobOrderAsync( + Isa95JobOrderOperationV2 operation, + V2.ISA95JobOrderDataType jobOrder, + ArrayOf comment = default, + CancellationToken cancellationToken = default) + { + if (jobOrder == null) + { + throw new ArgumentNullException(nameof(jobOrder)); + } + cancellationToken.ThrowIfCancellationRequested(); + string? jobOrderId = jobOrder.JobOrderID; + if (string.IsNullOrEmpty(jobOrderId) || !AllowedOrders.Contains(jobOrderId)) + { + return new Isa95JobOrderReceiptV2 + { + Result = new ServiceResult(StatusCodes.BadUserAccessDenied, + LocalizedText.From("The visual-inspection sample accepts only its fixed inspection/rework orders.")), + ReturnStatus = Isa95JobReturnStatus.InvalidRequest + }; + } + Isa95JobOrderReceiptV2 receipt = await m_provider.ReceiveJobOrderAsync( + operation, + jobOrder, + comment, + cancellationToken) + .ConfigureAwait(false); + if (operation == Isa95JobOrderOperationV2.StoreAndStart && + (receipt.ReturnStatus & Isa95JobReturnStatus.Success) != 0) + { + Isa95JobOrderReceiptV2 transition = await m_provider.TransitionAsync( + jobOrderId, + Isa95JobExecutionTransition.BeginExecution, + cancellationToken).ConfigureAwait(false); + if ((transition.ReturnStatus & Isa95JobReturnStatus.Success) == 0) + { + return transition; + } + } + return receipt; + } + + public ValueTask RequestJobResponseByJobOrderIdAsync( + string jobOrderId, + CancellationToken cancellationToken = default) + { + return m_provider.RequestJobResponseByJobOrderIdAsync(jobOrderId, cancellationToken); + } + + public ValueTask RequestJobResponsesByStateAsync( + ArrayOf state, + CancellationToken cancellationToken = default) + { + return m_provider.RequestJobResponsesByStateAsync(state, cancellationToken); + } + + public ValueTask ReceiveJobResponseAsync( + V2.ISA95JobResponseDataType response, + CancellationToken cancellationToken = default) + { + return m_provider.ReceiveJobResponseAsync(response, cancellationToken); + } + + public IAsyncEnumerable SubscribeAsync( + CancellationToken cancellationToken = default) + { + return m_provider.SubscribeAsync(cancellationToken); + } + + public ValueTask TransitionAsync( + string jobOrderId, + Isa95JobExecutionTransition transition, + CancellationToken cancellationToken = default) + { + return m_provider.TransitionAsync(jobOrderId, transition, cancellationToken); + } + + public IAsyncEnumerable SubscribeCatalogChangesAsync( + CancellationToken cancellationToken = default) + { + return m_provider.SubscribeCatalogChangesAsync(cancellationToken); + } + + public async ValueTask SeedAsync(CancellationToken cancellationToken) + { + foreach (V2.ISA95JobOrderDataType order in SeedOrders()) + { + string jobOrderId = order.JobOrderID ?? + throw new InvalidOperationException("A seeded job order must have an identifier."); + _ = await m_provider.ReceiveJobOrderAsync( + Isa95JobOrderOperationV2.StoreAndStart, + order, + cancellationToken: cancellationToken).ConfigureAwait(false); + _ = await m_provider.TransitionAsync( + jobOrderId, + Isa95JobExecutionTransition.BeginExecution, + cancellationToken).ConfigureAwait(false); + } + } + + public void Dispose() + { + m_provider.Dispose(); + } + + private static IEnumerable SeedOrders() + { + yield return new V2.ISA95JobOrderDataType + { + JobOrderID = InspectionOrderId, + Description = new[] { + LocalizedText.From("Inspect machined bracket against dimensional recipe.") + }.ToArrayOf(), + Priority = 10 + }; + yield return new V2.ISA95JobOrderDataType + { + JobOrderID = ReworkRejectOrderId, + Description = new[] { + LocalizedText.From("Route nonconforming bracket to rework or reject.") + }.ToArrayOf(), + Priority = 20 + }; + } + + public const string InspectionOrderId = "VIS-INSP-BRACKET-001"; + public const string ReworkRejectOrderId = "VIS-REWORK-REJECT-001"; + + private static readonly HashSet AllowedOrders = + [ + InspectionOrderId, + ReworkRejectOrderId + ]; + + private readonly InMemoryIsa95JobControlProvider m_provider; + } +} diff --git a/samples/Vision/VisualInspectionCell/InspectionJobSeeder.cs b/samples/Vision/VisualInspectionCell/InspectionJobSeeder.cs new file mode 100644 index 0000000000..3e751af19b --- /dev/null +++ b/samples/Vision/VisualInspectionCell/InspectionJobSeeder.cs @@ -0,0 +1,74 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace Vision.VisualInspectionCell +{ + internal sealed class InspectionJobSeeder : IHostedService + { + public InspectionJobSeeder( + InspectionJobControlProvider provider, + ILogger logger) + { + m_provider = provider ?? throw new ArgumentNullException(nameof(provider)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public async Task StartAsync(CancellationToken cancellationToken) + { + await m_provider.SeedAsync(cancellationToken).ConfigureAwait(false); + m_logger.JobsSeeded(InspectionJobControlProvider.InspectionOrderId, + InspectionJobControlProvider.ReworkRejectOrderId); + } + + public Task StopAsync(CancellationToken cancellationToken) + { + return Task.CompletedTask; + } + + private readonly InspectionJobControlProvider m_provider; + private readonly ILogger m_logger; + } + + internal static partial class InspectionJobSeederLog + { + [LoggerMessage(EventId = VisualInspectionCellEventIds.Isa95 + 1, + Level = LogLevel.Information, + Message = "Seeded ISA-95 V2 orders {InspectionOrderId} and {ReworkRejectOrderId}.")] + public static partial void JobsSeeded( + this ILogger logger, + string inspectionOrderId, + string reworkRejectOrderId); + } +} diff --git a/samples/Vision/VisualInspectionCell/InspectionRecipe.cs b/samples/Vision/VisualInspectionCell/InspectionRecipe.cs new file mode 100644 index 0000000000..7fec8ff7d8 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/InspectionRecipe.cs @@ -0,0 +1,81 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using Opc.Ua; +using Opc.Ua.Vision; + +namespace Vision.VisualInspectionCell +{ + internal sealed record InspectionCharacteristicRecipe( + string CharacteristicId, + string Name, + double Nominal, + double LowerTolerance, + double UpperTolerance); + + internal sealed class InspectionRecipe + { + public const string RecipeId = "machined-bracket-mm-v1"; + public const string PartId = "machined-bracket"; + + public IReadOnlyList Characteristics { get; } = + [ + new("BoreDiameter", "Bore diameter", 12.00, 0.20, 0.20), + new("SlotWidth", "Slot width", 8.00, 0.15, 0.15), + new("EdgeOffset", "Edge offset", 20.00, 0.25, 0.25) + ]; + + public static EUInformation Millimetre { get; } = + new("mm", "millimetre", "http://www.opcfoundation.org/UA/units/un/cefact"); + + public InspectionCharacteristicRecipe this[string characteristicId] + { + get + { + foreach (InspectionCharacteristicRecipe characteristic in Characteristics) + { + if (string.Equals(characteristic.CharacteristicId, characteristicId, System.StringComparison.Ordinal)) + { + return characteristic; + } + } + + throw new KeyNotFoundException(characteristicId); + } + } + } + + internal sealed record MeasuredCharacteristic(string CharacteristicId, double Actual, double Uncertainty); + + internal sealed record InspectionAnalysis( + string FixtureName, + ArrayOf Characteristics, + VisionResultEvaluationEnum Verdict); +} diff --git a/samples/Vision/VisualInspectionCell/InspectionVerdictPolicy.cs b/samples/Vision/VisualInspectionCell/InspectionVerdictPolicy.cs new file mode 100644 index 0000000000..fea4d5932f --- /dev/null +++ b/samples/Vision/VisualInspectionCell/InspectionVerdictPolicy.cs @@ -0,0 +1,167 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using Opc.Ua; +using Opc.Ua.Vision; + +namespace Vision.VisualInspectionCell +{ + /// + /// Applies the inspection recipe to measured evidence. + /// + internal sealed class InspectionVerdictPolicy + { + public InspectionVerdictPolicy(InspectionRecipe recipe) + { + m_recipe = recipe ?? throw new ArgumentNullException(nameof(recipe)); + } + + public InspectionAnalysis Judge(string fixtureName, IReadOnlyList measurements) + { + if (measurements == null) + { + throw new ArgumentNullException(nameof(measurements)); + } + + var characteristics = new List(m_recipe.Characteristics.Count); + VisionResultEvaluationEnum verdict = VisionResultEvaluationEnum.Ok; + foreach (InspectionCharacteristicRecipe recipe in m_recipe.Characteristics) + { + MeasuredCharacteristic? measurement = FindMeasurement(measurements, recipe.CharacteristicId); + VisionResultEvaluationEnum characteristicVerdict = measurement == null + ? VisionResultEvaluationEnum.NotDecidable + : JudgeCharacteristic(recipe, measurement); + if (characteristicVerdict == VisionResultEvaluationEnum.NotOk) + { + verdict = VisionResultEvaluationEnum.NotOk; + } + else if (characteristicVerdict == VisionResultEvaluationEnum.NotDecidable && + verdict != VisionResultEvaluationEnum.NotOk) + { + verdict = VisionResultEvaluationEnum.NotDecidable; + } + + characteristics.Add(new VisionCharacteristicDataType + { + CharacteristicId = recipe.CharacteristicId, + Name = recipe.Name, + Nominal = recipe.Nominal, + Actual = measurement?.Actual ?? 0.0, + Deviation = measurement == null ? 0.0 : measurement.Actual - recipe.Nominal, + LowerTolerance = recipe.LowerTolerance, + UpperTolerance = recipe.UpperTolerance, + Uncertainty = measurement?.Uncertainty ?? 0.0, + Unit = InspectionRecipe.Millimetre, + Status = ToToleranceStatus(characteristicVerdict) + }); + } + + return new InspectionAnalysis(fixtureName, characteristics.ToArrayOf(), verdict); + } + + public VisionResultEvaluationEnum JudgeCharacteristics( + ArrayOf characteristics) + { + var measurements = new List(characteristics.Count); + for (int ii = 0; ii < characteristics.Count; ii++) + { + VisionCharacteristicDataType characteristic = characteristics[ii]; + if (string.IsNullOrEmpty(characteristic.CharacteristicId)) + { + continue; + } + measurements.Add(new MeasuredCharacteristic( + characteristic.CharacteristicId, + characteristic.Actual, + characteristic.Uncertainty)); + } + return Judge("submitted-characteristics", measurements).Verdict; + } + + private static VisionResultEvaluationEnum JudgeCharacteristic( + InspectionCharacteristicRecipe recipe, + MeasuredCharacteristic measurement) + { + if (!double.IsFinite(measurement.Actual) || !double.IsFinite(measurement.Uncertainty)) + { + return VisionResultEvaluationEnum.NotDecidable; + } + + double uncertainty = Math.Abs(measurement.Uncertainty); + long intervalLow = ToMicrometres(measurement.Actual - uncertainty); + long intervalHigh = ToMicrometres(measurement.Actual + uncertainty); + long toleranceLow = ToMicrometres(recipe.Nominal - recipe.LowerTolerance); + long toleranceHigh = ToMicrometres(recipe.Nominal + recipe.UpperTolerance); + if (intervalLow >= toleranceLow && intervalHigh <= toleranceHigh) + { + return VisionResultEvaluationEnum.Ok; + } + if (intervalHigh < toleranceLow || intervalLow > toleranceHigh) + { + return VisionResultEvaluationEnum.NotOk; + } + return VisionResultEvaluationEnum.NotDecidable; + } + + private static MeasuredCharacteristic? FindMeasurement( + IReadOnlyList measurements, + string characteristicId) + { + for (int ii = 0; ii < measurements.Count; ii++) + { + MeasuredCharacteristic measurement = measurements[ii]; + if (string.Equals(measurement.CharacteristicId, characteristicId, StringComparison.Ordinal)) + { + return measurement; + } + } + + return null; + } + + private static long ToMicrometres(double value) + { + return (long)Math.Round(value * 1000.0, MidpointRounding.AwayFromZero); + } + + private static VisionToleranceStatusEnum ToToleranceStatus(VisionResultEvaluationEnum verdict) + { + return verdict switch + { + VisionResultEvaluationEnum.Ok => VisionToleranceStatusEnum.InTolerance, + VisionResultEvaluationEnum.NotOk => VisionToleranceStatusEnum.OutOfTolerance, + _ => VisionToleranceStatusEnum.Indeterminate + }; + } + + private readonly InspectionRecipe m_recipe; + } +} diff --git a/samples/Vision/VisualInspectionCell/OperatorDialogController.cs b/samples/Vision/VisualInspectionCell/OperatorDialogController.cs new file mode 100644 index 0000000000..8c0de87767 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/OperatorDialogController.cs @@ -0,0 +1,194 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; + +namespace Vision.VisualInspectionCell +{ + internal enum OperatorDisposition + { + AcceptAsOk, + AcceptAsNotOk, + Reinspect, + Stop + } + + internal sealed class OperatorDialogController + { + public OperatorDialogController( + VisualInspectionFeedbackSink feedbackSink, + ILogger logger) + { + m_feedbackSink = feedbackSink ?? throw new ArgumentNullException(nameof(feedbackSink)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + m_feedbackSink.AttachOperatorDialog(this); + } + + public void Attach(ISystemContext context, DialogConditionState dialog) + { + m_context = context ?? throw new ArgumentNullException(nameof(context)); + m_dialog = dialog ?? throw new ArgumentNullException(nameof(dialog)); + dialog.OnRespond = OnRespond; + } + + public void RequestDisposition(PublishedInspectionResult result) + { + if (result == null) + { + throw new ArgumentNullException(nameof(result)); + } + DialogConditionState dialog = RequireDialog(); + ISystemContext context = RequireContext(); + bool activated = false; + lock (m_lock) + { + if (m_pending != null) + { + m_queued.Enqueue(result); + } + else + { + ActivateLocked(context, dialog, result); + activated = true; + } + } + if (activated) + { + m_logger.DialogActivated(result.ResultId); + } + else + { + m_logger.DialogQueued(result.ResultId); + } + } + + private ServiceResult OnRespond( + ISystemContext context, + DialogConditionState dialog, + int selectedResponse) + { + if (!Enum.IsDefined((OperatorDisposition)selectedResponse)) + { + return StatusCodes.BadDialogResponseInvalid; + } + PublishedInspectionResult? pending; + var disposition = (OperatorDisposition)selectedResponse; + lock (m_lock) + { + pending = m_pending; + if (pending == null) + { + return StatusCodes.BadInvalidState; + } + m_pending = null; + dialog.SetResponse(context, selectedResponse); + if (m_queued.Count > 0) + { + ActivateLocked(context, dialog, m_queued.Dequeue()); + } + else + { + dialog.Retain!.Value = false; + dialog.ClearChangeMasks(context, includeChildren: true); + } + } + _ = m_feedbackSink.HandleOperatorDispositionAsync(pending, disposition, CancellationToken.None) + .AsTask() + .ContinueWith( + task => m_logger.OperatorDispositionFailed(pending.ResultId, task.Exception!.GetBaseException()), + CancellationToken.None, + TaskContinuationOptions.OnlyOnFaulted, + TaskScheduler.Default); + return ServiceResult.Good; + } + + private void ActivateLocked( + ISystemContext context, + DialogConditionState dialog, + PublishedInspectionResult result) + { + m_pending = result; + dialog.Message!.Value = LocalizedText.From( + "Human disposition required for inspection result " + result.ResultId + "."); + dialog.SetEnableState(context, enabled: true); + dialog.Retain!.Value = true; + dialog.Activate(context); + dialog.ClearChangeMasks(context, includeChildren: true); + dialog.ReportEvent(context, dialog); + } + + private DialogConditionState RequireDialog() + { + return m_dialog ?? throw new InvalidOperationException("The operator dialog is not attached."); + } + + private ISystemContext RequireContext() + { + return m_context ?? throw new InvalidOperationException("The operator dialog context is not attached."); + } + + private readonly VisualInspectionFeedbackSink m_feedbackSink; + private readonly ILogger m_logger; + private readonly Lock m_lock = new(); + private readonly Queue m_queued = []; + private DialogConditionState? m_dialog; + private ISystemContext? m_context; + private PublishedInspectionResult? m_pending; + } + + internal static partial class OperatorDialogControllerLog + { + [LoggerMessage(EventId = VisualInspectionCellEventIds.Dialog + 1, + Level = LogLevel.Information, + Message = "Activated operator disposition dialog for result {ResultId}.")] + public static partial void DialogActivated( + this ILogger logger, + string resultId); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.Dialog + 2, + Level = LogLevel.Error, + Message = "Operator disposition processing failed for result {ResultId}.")] + public static partial void OperatorDispositionFailed( + this ILogger logger, + string resultId, + Exception exception); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.Dialog + 3, + Level = LogLevel.Information, + Message = "Queued operator disposition dialog for result {ResultId}.")] + public static partial void DialogQueued( + this ILogger logger, + string resultId); + } +} diff --git a/samples/Vision/VisualInspectionCell/PngDecoder.cs b/samples/Vision/VisualInspectionCell/PngDecoder.cs new file mode 100644 index 0000000000..67659043aa --- /dev/null +++ b/samples/Vision/VisualInspectionCell/PngDecoder.cs @@ -0,0 +1,198 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.IO.Compression; +using System.Text; + +namespace Vision.VisualInspectionCell +{ + /// + /// Minimal PNG decoder for the fixture images used by this sample. + /// + internal static class PngDecoder + { + public static (byte[] Rgb, int Width, int Height) Decode(byte[] png) + { + if (png == null) + { + throw new ArgumentNullException(nameof(png)); + } + if (png.Length < 8 || + png[0] != 0x89 || + png[1] != (byte)'P' || + png[2] != (byte)'N' || + png[3] != (byte)'G' || + png[4] != 0x0D || + png[5] != 0x0A || + png[6] != 0x1A || + png[7] != 0x0A) + { + throw new InvalidDataException("Not a PNG stream."); + } + + int width = 0; + int height = 0; + byte bitDepth = 0; + byte colourType = 0; + byte interlace = 0; + using var idat = new MemoryStream(); + int offset = 8; + while (offset + 8 <= png.Length) + { + int length = ReadUInt32BE(png, offset); + string chunkType = Encoding.ASCII.GetString(png, offset + 4, 4); + int dataStart = offset + 8; + if (dataStart + length + 4 > png.Length) + { + throw new InvalidDataException("PNG chunk exceeds stream length."); + } + if (chunkType == "IHDR") + { + width = ReadUInt32BE(png, dataStart); + height = ReadUInt32BE(png, dataStart + 4); + bitDepth = png[dataStart + 8]; + colourType = png[dataStart + 9]; + interlace = png[dataStart + 12]; + } + else if (chunkType == "IDAT") + { + idat.Write(png, dataStart, length); + } + else if (chunkType == "IEND") + { + break; + } + offset = dataStart + length + 4; + } + if (bitDepth != 8 || (colourType != 2 && colourType != 6)) + { + throw new NotSupportedException( + $"PNG must be 8-bit RGB/RGBA; got type={colourType}, depth={bitDepth}."); + } + if (interlace != 0) + { + throw new NotSupportedException("Interlaced PNGs are not supported by the sample decoder."); + } + if (width <= 0 || height <= 0) + { + throw new InvalidDataException("PNG dimensions are invalid."); + } + + int bytesPerPixel = colourType == 6 ? 4 : 3; + byte[] rawFiltered = InflateZlib(idat.ToArray()); + int rowBytes = width * bytesPerPixel; + int expected = height * (rowBytes + 1); + if (rawFiltered.Length != expected) + { + throw new InvalidDataException( + $"Decoded filtered size {rawFiltered.Length} does not match {expected}."); + } + byte[] unfiltered = new byte[height * rowBytes]; + Unfilter(rawFiltered, unfiltered, width, height, bytesPerPixel); + if (bytesPerPixel == 3) + { + return (unfiltered, width, height); + } + + byte[] rgb = new byte[width * height * 3]; + for (int source = 0, target = 0; source < unfiltered.Length; source += 4, target += 3) + { + rgb[target] = unfiltered[source]; + rgb[target + 1] = unfiltered[source + 1]; + rgb[target + 2] = unfiltered[source + 2]; + } + return (rgb, width, height); + } + + private static byte[] InflateZlib(byte[] zlib) + { + if (zlib.Length < 6) + { + throw new InvalidDataException("IDAT chunk is too short."); + } + using var input = new MemoryStream(zlib, index: 2, count: zlib.Length - 6, writable: false); + using var deflate = new DeflateStream(input, CompressionMode.Decompress, leaveOpen: false); + using var output = new MemoryStream(); + deflate.CopyTo(output); + return output.ToArray(); + } + + private static void Unfilter(byte[] filtered, byte[] rgb, int width, int height, int bytesPerPixel) + { + int rowBytes = width * bytesPerPixel; + for (int y = 0; y < height; y++) + { + int srcRow = y * (rowBytes + 1); + int dstRow = y * rowBytes; + byte type = filtered[srcRow]; + for (int x = 0; x < rowBytes; x++) + { + byte value = filtered[srcRow + 1 + x]; + byte left = x >= bytesPerPixel ? rgb[dstRow + x - bytesPerPixel] : (byte)0; + byte up = y > 0 ? rgb[((y - 1) * rowBytes) + x] : (byte)0; + byte upLeft = y > 0 && x >= bytesPerPixel + ? rgb[((y - 1) * rowBytes) + x - bytesPerPixel] + : (byte)0; + rgb[dstRow + x] = type switch + { + 0 => value, + 1 => (byte)(value + left), + 2 => (byte)(value + up), + 3 => (byte)(value + ((left + up) / 2)), + 4 => (byte)(value + Paeth(left, up, upLeft)), + _ => throw new NotSupportedException($"Unknown PNG row filter {type}.") + }; + } + } + } + + private static byte Paeth(byte left, byte up, byte upLeft) + { + int p = left + up - upLeft; + int pa = Math.Abs(p - left); + int pb = Math.Abs(p - up); + int pc = Math.Abs(p - upLeft); + if (pa <= pb && pa <= pc) + { + return left; + } + return pb <= pc ? up : upLeft; + } + + private static int ReadUInt32BE(byte[] source, int offset) + { + return (source[offset] << 24) | + (source[offset + 1] << 16) | + (source[offset + 2] << 8) | + source[offset + 3]; + } + } +} diff --git a/samples/Vision/VisualInspectionCell/Program.cs b/samples/Vision/VisualInspectionCell/Program.cs new file mode 100644 index 0000000000..ce3ec70120 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/Program.cs @@ -0,0 +1,189 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Globalization; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; +using Opc.Ua.ISA95.Server; +using Opc.Ua.ISA95.Server.Providers; +using Opc.Ua.Server; +using Opc.Ua.Server.Hosting; +using Vision.VisualInspectionCell; + +string[] normalizedArgs = NormalizeArgs(args); +HostApplicationBuilder builder = Host.CreateApplicationBuilder(normalizedArgs); + +builder.Logging.ClearProviders(); +builder.Logging.AddConsole(); + +int port = int.TryParse(builder.Configuration["port"], NumberStyles.Integer, CultureInfo.InvariantCulture, + out int configuredPort) + ? configuredPort + : 62865; +string host = builder.Configuration["host"] is { Length: > 0 } configuredHost + ? configuredHost + : "localhost"; +_ = VisualInspectionCellOptions.TryParseLocation( + builder.Configuration["inferenceLocation"], + out VisualInspectionInferenceLocation inferenceLocation); +bool insecure = bool.TryParse(builder.Configuration["insecure"], out bool parsedInsecure) && parsedInsecure; +var cellOptions = new VisualInspectionCellOptions +{ + InferenceLocation = inferenceLocation +}; + +builder.Services.AddSingleton(TimeProvider.System); +builder.Services.AddSingleton(cellOptions); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(services => new InferenceBackends( + new VisualInspectionInferenceBackend( + services.GetRequiredService(), + services.GetRequiredService()))); +builder.Services.AddSingleton(services => + services.GetRequiredService()); +builder.Services.AddSingleton(); +builder.Services.AddSingleton(services => + services.GetRequiredService()); +builder.Services.AddSingleton(services => + services.GetRequiredService()); +builder.Services.AddSingleton(services => + services.GetRequiredService()); +builder.Services.AddSingleton(services => + services.GetRequiredService()); +builder.Services.AddSingleton(services => + services.GetRequiredService()); +builder.Services.AddSingleton(services => + services.GetRequiredService()); +builder.Services.AddSingleton(services => + services.GetRequiredService()); +builder.Services.AddHostedService(); + +IOpcUaServerBuilder opcUa = builder.Services + .AddOpcUa() + .AddServer(options => + { + options.ApplicationName = "VisualInspectionCell"; + options.ApplicationUri = "urn:localhost:OPCFoundation:VisualInspectionCell"; + options.ProductUri = "uri:opcfoundation.org:VisualInspectionCell"; + options.AutoAcceptUntrustedCertificates = insecure; + options.EndpointUrls.Add($"opc.tcp://{host}:{port}/VisualInspectionCell"); + }) + .ConfigureRoles(options => options.Roles.Add(new RoleDefinitionOptions + { + Name = BrowseNames.WellKnownRole_Operator, + Identities = + { + new RoleIdentityMappingOptions + { + CriteriaType = IdentityCriteriaType.Anonymous + } + } + })); + +opcUa.AddAI( + ai => + { + ai.PrimaryDeploymentId = "visual-inspection-primary"; + ai.FallbackDeploymentId = "visual-inspection-fallback"; + ai.EnableFallback = false; + ai.EnableCatalogue = false; + ai.EnableLearningLoop = true; + }, + backend => + { + backend.Site = inferenceLocation == VisualInspectionInferenceLocation.EdgeOffServer + ? InferenceSite.EdgeOffServer + : InferenceSite.OnServer; + backend.EgressPermitted = false; + backend.RetainsInput = false; + backend.Models.Add(VisualInspectionInferenceBackend.Model); + }, + fallback => fallback.Enabled = false); +opcUa.AddIsa95Server(options => +{ + options.InstanceNamespaceUri = "urn:opcfoundation:VisualInspectionCell:isa95:instances"; + options.RootBrowseName = "VisualInspectionISA95"; + options.EnableJobControlV1 = false; + options.EnableJobControlV2 = true; +}); +opcUa + .AddVision(options => + options.InstanceNamespaceUri = "urn:opcfoundation:VisualInspectionCell:vision:instances") + .ConfigureVision(async (context, cancellationToken) => + { + VisualInspectionCell cell = context.GetRequiredService(); + await cell.ConfigureAsync(context, cancellationToken).ConfigureAwait(false); + }); + +using IHost app = builder.Build(); +Console.WriteLine(FormattableString.Invariant( + $"VisualInspectionCell listening at opc.tcp://{host}:{port}/VisualInspectionCell")); +Console.WriteLine(FormattableString.Invariant( + $"InferenceLocation={inferenceLocation}; fixtures=bracket-ok.png, bracket-not-ok.png, bracket-ambiguous.png")); +VisualInspectionAnalysisService analysis = app.Services.GetRequiredService(); +foreach (string fixture in analysis.FixtureNames) +{ + InspectionAnalysis result = analysis.AnalyzeByName(fixture); + Console.WriteLine(FormattableString.Invariant($"{fixture}: {result.Verdict}")); + foreach (Opc.Ua.Vision.VisionCharacteristicDataType characteristic in result.Characteristics) + { + double low = characteristic.Actual - characteristic.Uncertainty; + double high = characteristic.Actual + characteristic.Uncertainty; + Console.WriteLine(FormattableString.Invariant( + $" {characteristic.CharacteristicId}: {characteristic.Actual:0.00} mm [{low:0.00}, {high:0.00}], {characteristic.Status}")); + } +} +await app.RunAsync().ConfigureAwait(false); + +static string[] NormalizeArgs(string[] args) +{ + string[] normalized = new string[args.Length]; + for (int ii = 0; ii < args.Length; ii++) + { + normalized[ii] = string.Equals(args[ii], "--insecure", StringComparison.OrdinalIgnoreCase) + ? "--insecure=true" + : args[ii]; + } + return normalized; +} diff --git a/samples/Vision/VisualInspectionCell/README.md b/samples/Vision/VisualInspectionCell/README.md new file mode 100644 index 0000000000..7d955fe52c --- /dev/null +++ b/samples/Vision/VisualInspectionCell/README.md @@ -0,0 +1,123 @@ + + +# Visual Inspection Cell + +`VisualInspectionCell` is the server half of the visual-inspection sample. It +hosts one address space containing four companion areas: + +- **Vision** — `BracketFixtureCamera`, the `FixtureImages` inline PNG clip + endpoint, and `BracketInspectionPipeline`. +- **AI Model Management** — the `visual-inspection-primary` deployment, a + learning job, and the `Invoke` path used for model provenance. +- **ISA-95 Job Control V2** — an allowlisted inspection order and a + rework/reject order. +- **Alarms & Conditions** — `OperatorDispositionDialog`, a + `DialogConditionType` used when a result is not decidable. + +The server is intentionally a host for companion composition, not the owner of +the production decision loop. The paired +[VisualInspectionAgent](../VisualInspectionAgent) connects as an external +orchestrator and drives the cycle with typed clients. + +```mermaid +graph TD + Server["VisualInspectionCell server"] + Vision["Vision root"] + Sensor["BracketFixtureCamera
ImageSensorType"] + Clip["FixtureImages
inline PNG clips"] + Pipeline["BracketInspectionPipeline
InferencePipelineType"] + AI["AI Model Management
deployment + learning job"] + Jobs["ISA-95 Job Control V2
fixed order catalogue"] + Dialog["OperatorDispositionDialog
DialogConditionType"] + + Server --> Vision + Vision --> Sensor + Sensor --> Clip + Vision --> Pipeline + Pipeline -->|"Deployment"| AI + Pipeline -->|"LearningJob"| AI + Server --> Jobs + Server --> Dialog +``` + +## Running + +Prerequisites: .NET 10 SDK. + +```powershell +dotnet run --project samples\Vision\VisualInspectionCell\VisualInspectionCell.csproj -- --insecure +``` + +The default endpoint is `opc.tcp://localhost:62865/VisualInspectionCell`. +`--host ` and `--port ` configure the endpoint host and port. The +anonymous operator role is mapped for the local sample. + +`--insecure` is a demo convenience. It accepts untrusted certificates and must +not be used for production systems. + +It is also, in practice, required to run the pair on a fresh machine. The flag +sets `AutoAcceptUntrustedCertificates` on the server, and without it the cell +rejects the agent's self-signed certificate with `BadCertificateUntrusted` and +the agent fails with `BadNotConnected` - which reads like a connectivity problem +rather than a trust one. Start both halves with `--insecure`, or trust the +agent's certificate in the server's PKI store first. + +## Startup options + +| Option | Meaning | +|---|---| +| `--host ` | Endpoint host name. Default `localhost`. | +| `--port ` | Endpoint port. Default `62865`. | +| `--inferenceLocation OnServer\|EdgeOffServer` | Selects the advertised Vision inference location. Default `OnServer`. | +| `--insecure` | Demo-only certificate convenience. | + +## Fixtures and measurement + +The cell serves three 800 x 600 PNG fixtures from `Fixtures/` through the +Vision media provider. The camera scale is 10 px/mm, so one pixel is 0.10 mm. +That one-pixel pitch is carried as `VisionCharacteristicDataType.Uncertainty`; +it is not invented to force a result. + +| Fixture | Why it exists | +|---|---| +| `bracket-ok.png` | Bore diameter is 12.00 mm, so the interval `[11.90, 12.10]` is wholly inside `[11.80, 12.20]`. | +| `bracket-not-ok.png` | Bore diameter is 12.60 mm, so the interval `[12.50, 12.70]` is wholly outside the bore tolerance. | +| `bracket-ambiguous.png` | The intended slot is 8.15 mm, but at 10 px/mm it draws as 81.5 px and can only be represented as 8.10 mm. The interval `[8.00, 8.20]` straddles the 8.15 mm upper limit. | + +## What the cell publishes + +- `BracketFixtureCamera` (`ImageSensorType`) with simulated reality, + `RGB8`, 800 x 600 resolution, 10 px/mm intrinsics, and `fixture_table` as the + frame id. +- `FixtureImages`, an inline PNG clip endpoint with endpoint URI + `opcua-inline://visual-inspection-cell/fixtures`. +- `BracketInspectionPipeline`, bound to the camera, an AI deployment, a learning + job, `VisualInspectionInferenceProvider`, and `VisualInspectionFeedbackSink`. +- A fixed ISA-95 V2 catalogue with `VIS-INSP-BRACKET-001` and + `VIS-REWORK-REJECT-001`. `InspectionJobControlProvider` rejects any other job + order id instead of accepting invented payloads. +- `OperatorDispositionDialog`, the human-disposition condition the agent uses + when the deterministic rule returns `NotDecidable`. + +## Deliberately not implemented + +The sample publishes and counts learning samples, but it does not pretend to +retrain a model or promote a candidate. That is the same line taken by the +[AI Model Management sample](../../AI/README.md): a simulated MLOps loop would +mislead readers about the part of the specification a sample cannot honestly +demonstrate. + +## Related docs + +- [Vision developer guide](../../../docs/Vision.md) — see *Visual inspection: a cross-companion cell* +- [Vision developer guide](../../../docs/Vision.md) +- [AI Model Management developer guide](../../../docs/AiIntegration.md) +- [ISA-95 developer guide](../../../docs/ISA95.md) +- [Alarms and Conditions](../../../docs/AlarmsAndConditions.md) diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionAnalysisService.cs b/samples/Vision/VisualInspectionCell/VisualInspectionAnalysisService.cs new file mode 100644 index 0000000000..d88a84c671 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionAnalysisService.cs @@ -0,0 +1,177 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Security.Cryptography; +using Opc.Ua; +using Opc.Ua.Vision; + +namespace Vision.VisualInspectionCell +{ + /// + /// Coordinates fixture selection, image measurement and deterministic recipe judging. + /// + internal sealed class VisualInspectionAnalysisService + { + public VisualInspectionAnalysisService(FixtureImageAnalyzer analyzer, InspectionVerdictPolicy policy) + { + m_analyzer = analyzer ?? throw new ArgumentNullException(nameof(analyzer)); + m_policy = policy ?? throw new ArgumentNullException(nameof(policy)); + } + + public IReadOnlyList FixtureNames { get; } = + [ + "bracket-ok.png", + "bracket-not-ok.png", + "bracket-ambiguous.png" + ]; + + public string FixtureDirectory + { + get + { + string output = Path.Combine(AppContext.BaseDirectory, "Fixtures"); + if (Directory.Exists(output)) + { + return output; + } + + string project = Path.GetFullPath(Path.Combine( + AppContext.BaseDirectory, + "..", + "..", + "..", + "..", + "Fixtures")); + if (Directory.Exists(project)) + { + return project; + } + + return Path.Combine( + Environment.CurrentDirectory, + "samples", + "Vision", + "VisualInspectionCell", + "Fixtures"); + } + } + + public InspectionAnalysis AnalyzeByName(string fixtureName) + { + string selected = ResolveFixtureName(fixtureName); + return m_policy.Judge(selected, MeasureByName(selected)); + } + + public InspectionAnalysis AnalyzeForCycle(DateTimeUtc timestamp) + { + long cycle = timestamp.IsNull + ? DateTimeUtc.From(DateTime.UnixEpoch).Value + : timestamp.Value; + int index = (int)(Math.Abs(cycle) % FixtureNames.Count); + return AnalyzeByName(FixtureNames[index % FixtureNames.Count]); + } + + public IReadOnlyList MeasureByName(string fixtureName) + { + string selected = ResolveFixtureName(fixtureName); + string path = Path.Combine(FixtureDirectory, selected); + return m_analyzer.Measure(path); + } + + public VisionImageReferenceDataType CreateImageReference(string fixtureName, DateTimeUtc timestamp) + { + string selected = ResolveFixtureName(fixtureName); + string path = Path.Combine(FixtureDirectory, selected); + ByteString png = ByteString.From(File.ReadAllBytes(path)); + return new VisionImageReferenceDataType + { + Uri = FormattableString.Invariant($"opcua-inline://visual-inspection-cell/fixtures/{selected}"), + Digest = ByteString.From(SHA256.HashData(png.Span)), + DigestAlgorithm = "SHA-256", + Format = VisionClipFormatEnum.Png, + PixelFormat = VisualInspectionMediaProvider.PixelFormat, + Width = VisualInspectionMediaProvider.Width, + Height = VisualInspectionMediaProvider.Height, + SizeBytes = (uint)png.Length, + Timestamp = timestamp + }; + } + + public bool TryResolveFixtureName(string? requested, out string fixtureName) + { + if (!string.IsNullOrWhiteSpace(requested)) + { + string candidate = requested.EndsWith(".png", StringComparison.OrdinalIgnoreCase) + ? requested + : string.Create(CultureInfo.InvariantCulture, $"{requested}.png"); + foreach (string fixture in FixtureNames) + { + if (string.Equals(candidate, fixture, StringComparison.OrdinalIgnoreCase) || + string.Equals(requested, Path.GetFileNameWithoutExtension(fixture), + StringComparison.OrdinalIgnoreCase)) + { + fixtureName = fixture; + return true; + } + } + } + + fixtureName = string.Empty; + return false; + } + + public bool TryResolveFixtureFromUri(string? uri, out string fixtureName) + { + if (string.IsNullOrWhiteSpace(uri)) + { + fixtureName = string.Empty; + return false; + } + return TryResolveFixtureName(Path.GetFileName(uri), out fixtureName); + } + + public string ResolveFixtureName(string? requested) + { + if (TryResolveFixtureName(requested, out string fixtureName)) + { + return fixtureName; + } + + throw new FileNotFoundException( + string.Create(CultureInfo.InvariantCulture, $"Unknown visual-inspection fixture '{requested}'.")); + } + + private readonly FixtureImageAnalyzer m_analyzer; + private readonly InspectionVerdictPolicy m_policy; + } +} diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionCell.cs b/samples/Vision/VisualInspectionCell/VisualInspectionCell.cs new file mode 100644 index 0000000000..6f61342865 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionCell.cs @@ -0,0 +1,273 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.AI.Server; +using Opc.Ua.Server; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Builders; + +namespace Vision.VisualInspectionCell +{ + internal sealed class VisualInspectionCell + { + public VisualInspectionCell( + VisualInspectionCellOptions options, + VisualInspectionMediaProvider mediaProvider, + VisualInspectionInferenceProvider inferenceProvider, + VisualInspectionFeedbackSink feedbackSink, + OperatorDialogController operatorDialog, + ILogger logger) + { + m_options = options ?? throw new ArgumentNullException(nameof(options)); + m_mediaProvider = mediaProvider ?? throw new ArgumentNullException(nameof(mediaProvider)); + m_inferenceProvider = inferenceProvider ?? throw new ArgumentNullException(nameof(inferenceProvider)); + m_feedbackSink = feedbackSink ?? throw new ArgumentNullException(nameof(feedbackSink)); + m_operatorDialog = operatorDialog ?? throw new ArgumentNullException(nameof(operatorDialog)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public async ValueTask ConfigureAsync(IVisionBuildContext context, CancellationToken cancellationToken) + { + if (context == null) + { + throw new ArgumentNullException(nameof(context)); + } + + context.Nodes.AddFrame("FixtureFrame", frame => frame + .WithFrameId(FixtureFrameId) + .WithRole(VisionFrameRoleEnum.World) + .WithTransform(new VisionPose3DDataType + { + FrameId = FixtureFrameId, + Position = s_zeroPosition.ToArrayOf(), + Orientation = s_identityOrientation.ToArrayOf(), + Covariance = ArrayOf.Empty + })); + AddSensor(context); + AddPipeline(context); + await AddOperatorDialogAsync(context, cancellationToken).ConfigureAwait(false); + AttachRuntimeTargets(context); + m_logger.CellConfigured(PipelineBrowseName, m_options.InferenceLocation); + } + + private void AddSensor(IVisionBuildContext context) + { + context.Nodes.AddImageSensor(SensorBrowseName, sensor => sensor + .WithSensorId("fixture-camera-01") + .WithModality(VisionSensorModalityEnum.Area2D) + .WithRealityKind(VisionRealityKindEnum.Simulated) + .WithManufacturer("OPC Foundation") + .WithModel("Fixture PNG camera") + .WithSerialNumber("FIXTURE-CAM-0001") + .WithDeviceUri("file://visual-inspection-cell/fixtures") + .WithFrameId(FixtureFrameId) + .WithResolution(VisualInspectionMediaProvider.Width, VisualInspectionMediaProvider.Height) + .WithPixelFormat(VisualInspectionMediaProvider.PixelFormat) + .WithIntrinsics(new VisionIntrinsicsDataType + { + Fx = 10.0, + Fy = 10.0, + Cx = 400.0, + Cy = 300.0, + Skew = 0.0, + DistortionModel = VisionDistortionModelEnum.None, + DistortionCoefficients = ArrayOf.Empty, + Width = VisualInspectionMediaProvider.Width, + Height = VisualInspectionMediaProvider.Height + }) + .AddClipEndpoint(ClipEndpointBrowseName, endpoint => endpoint + .WithEndpointId("fixture-pngs") + .WithEndpointUri("opcua-inline://visual-inspection-cell/fixtures") + .WithClipFormat(VisionClipFormatEnum.Png) + .WithQuality(100u) + .WithResolution(VisualInspectionMediaProvider.Width, VisualInspectionMediaProvider.Height) + .WithInlineDelivery(enabled: true, maxInlineClipSize: 1_048_576u) + .WithDefaultProfileName("FixturePng")) + .UseMediaProvider(m_mediaProvider)); + } + + private void AddPipeline(IVisionBuildContext context) + { + NodeId sensorNodeId = FindSensor(context)?.NodeId ?? NodeId.Null; + (NodeId deployment, NodeId learningJob) = ResolveAIBindings(context); + bool offServer = m_options.InferenceLocation == VisualInspectionInferenceLocation.EdgeOffServer; + context.Nodes.AddPipeline(PipelineBrowseName, pipeline => + { + pipeline.WithPipelineId(PipelineId) + .WithSensor(sensorNodeId) + .WithDeployment(deployment) + .WithLearningJob(learningJob) + .UseFeedbackSink(m_feedbackSink); + if (offServer) + { + pipeline.UseInferenceProvider(m_inferenceProvider, onServer: false); + } + else + { + pipeline.UseInferenceProvider(m_inferenceProvider, onServer: true); + } + }); + } + + private void AttachRuntimeTargets(IVisionBuildContext context) + { + InferencePipelineState pipeline = FindPipeline(context) ?? + throw new InvalidOperationException( + "The visual-inspection pipeline was not registered."); + ImageSensorState sensor = FindSensor(context) ?? + throw new InvalidOperationException( + "The visual-inspection sensor was not registered."); + FolderState results = pipeline.Results ?? + throw new InvalidOperationException( + "The Vision builder did not materialise the pipeline Results folder."); + NodeId deployment = pipeline.Deployment?.Value ?? NodeId.Null; + NodeId learningJob = pipeline.LearningJob?.Value ?? NodeId.Null; + var target = new VisualInspectionTarget( + context.Manager, + context.Context, + context.InstanceNamespaceIndex, + pipeline.NodeId, + sensor.NodeId, + deployment, + learningJob, + results); + m_inferenceProvider.Attach(target); + m_feedbackSink.Attach(target); + } + + private async ValueTask AddOperatorDialogAsync( + IVisionBuildContext context, + CancellationToken cancellationToken) + { + var dialog = new DialogConditionState(null); + dialog.Create( + context.Context, + NodeId.Null, + new QualifiedName(OperatorDialogBrowseName, context.InstanceNamespaceIndex), + new LocalizedText("Operator disposition dialog"), + true); + dialog.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent; + dialog.TypeDefinitionId = global::Opc.Ua.ObjectTypeIds.DialogConditionType; + dialog.NodeId = new NodeId(OperatorDialogBrowseName, context.InstanceNamespaceIndex); + context.Context.AssignInstanceChildNodeIds(dialog, dialog.NodeId); + dialog.CreateOrReplacePrompt(context.Context, null).Value = + LocalizedText.From( + "Visual inspection is not decidable. Choose the operator disposition."); + dialog.CreateOrReplaceResponseOptionSet(context.Context, null).Value = + [ + LocalizedText.From("AcceptAsOk"), + LocalizedText.From("AcceptAsNotOk"), + LocalizedText.From("Reinspect"), + LocalizedText.From("Stop") + ]; + dialog.CreateOrReplaceDefaultResponse(context.Context, null).Value = 2; + dialog.CreateOrReplaceOkResponse(context.Context, null).Value = 0; + dialog.CreateOrReplaceCancelResponse(context.Context, null).Value = 3; + dialog.Retain!.Value = false; + dialog.Message!.Value = LocalizedText.From( + "Human disposition required for a not-decidable inspection."); + dialog.Severity!.Value = (ushort)EventSeverity.MediumHigh; + dialog.EventNotifier = EventNotifiers.SubscribeToEvents; + context.Root.AddChild(dialog); + await context.Manager.AddPredefinedNodeAsync(dialog, cancellationToken).ConfigureAwait(false); + m_operatorDialog.Attach(context.Context, dialog); + OperatorDialogNodeId = dialog.NodeId; + } + + private static (NodeId Deployment, NodeId LearningJob) ResolveAIBindings(IVisionBuildContext context) + { + if (context.Manager.Server.NodeManager.AsyncNodeManagers.OfType().FirstOrDefault() is { } ai) + { + return (ai.PrimaryDeploymentId, ai.LearningJobId); + } + return (NodeId.Null, NodeId.Null); + } + + private static ImageSensorState? FindSensor(IVisionBuildContext context) + { + return FindChild(context.Root.Sensors, context, SensorBrowseName); + } + + private static InferencePipelineState? FindPipeline(IVisionBuildContext context) + { + return FindChild(context.Root.Pipelines, context, PipelineBrowseName); + } + + private static T? FindChild(FolderState? folder, IVisionBuildContext context, string browseName) + where T : BaseInstanceState + { + if (folder == null) + { + return null; + } + var children = new List(); + folder.GetChildren(context.Context, children); + var qualified = new QualifiedName(browseName, context.InstanceNamespaceIndex); + return children.OfType().FirstOrDefault(child => child.BrowseName == qualified); + } + + public NodeId OperatorDialogNodeId { get; private set; } = NodeId.Null; + + public const string SensorBrowseName = "BracketFixtureCamera"; + public const string ClipEndpointBrowseName = "FixtureImages"; + public const string PipelineBrowseName = "BracketInspectionPipeline"; + public const string PipelineId = "pipe-bracket-inspection"; + public const string FixtureFrameId = "fixture_table"; + public const string OperatorDialogBrowseName = "OperatorDispositionDialog"; + + private static readonly double[] s_zeroPosition = [0.0, 0.0, 0.0]; + private static readonly double[] s_identityOrientation = [0.0, 0.0, 0.0, 1.0]; + + private readonly VisualInspectionCellOptions m_options; + private readonly VisualInspectionMediaProvider m_mediaProvider; + private readonly VisualInspectionInferenceProvider m_inferenceProvider; + private readonly VisualInspectionFeedbackSink m_feedbackSink; + private readonly OperatorDialogController m_operatorDialog; + private readonly ILogger m_logger; + } + + internal static partial class VisualInspectionCellLog + { + [LoggerMessage(EventId = VisualInspectionCellEventIds.Configurator + 1, + Level = LogLevel.Information, + Message = "Configured visual inspection pipeline {PipelineBrowseName} with {InferenceLocation} perception.")] + public static partial void CellConfigured( + this ILogger logger, + string pipelineBrowseName, + VisualInspectionInferenceLocation inferenceLocation); + } +} diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionCell.csproj b/samples/Vision/VisualInspectionCell/VisualInspectionCell.csproj new file mode 100644 index 0000000000..d62388f093 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionCell.csproj @@ -0,0 +1,37 @@ + + + net10.0 + Exe + false + VisualInspectionCell + VisualInspectionCell + Visual inspection work cell hosting Vision, AI Model Management, ISA-95 Job Control V2 and an operator dialog condition. + Vision.VisualInspectionCell + enable + $(NoWarn);CA1014;CA1812;CA1822 + false + true + win-x64 + false + true + + + + + + + + + + + + + + + + + + + + + diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionCellOptions.cs b/samples/Vision/VisualInspectionCell/VisualInspectionCellOptions.cs new file mode 100644 index 0000000000..4e0c49e7d4 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionCellOptions.cs @@ -0,0 +1,56 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +namespace Vision.VisualInspectionCell +{ + internal enum VisualInspectionInferenceLocation + { + OnServer, + EdgeOffServer + } + + internal sealed class VisualInspectionCellOptions + { + public VisualInspectionInferenceLocation InferenceLocation { get; init; } + = VisualInspectionInferenceLocation.OnServer; + + public static bool TryParseLocation(string? value, out VisualInspectionInferenceLocation location) + { + if (Enum.TryParse(value, ignoreCase: true, out location)) + { + return true; + } + + location = VisualInspectionInferenceLocation.OnServer; + return false; + } + } +} diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionFeedbackSink.cs b/samples/Vision/VisualInspectionCell/VisualInspectionFeedbackSink.cs new file mode 100644 index 0000000000..b095d7f27b --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionFeedbackSink.cs @@ -0,0 +1,427 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.AI.Server; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Vision.VisualInspectionCell +{ + internal sealed class VisualInspectionFeedbackSink : IVisionFeedbackSink + { + public VisualInspectionFeedbackSink( + VisualInspectionResultPublisher publisher, + AINodeManagerRegistry aiRegistry, + InspectionVerdictPolicy verdictPolicy, + TimeProvider timeProvider, + ILogger logger) + { + m_publisher = publisher ?? throw new ArgumentNullException(nameof(publisher)); + m_aiRegistry = aiRegistry ?? throw new ArgumentNullException(nameof(aiRegistry)); + m_verdictPolicy = verdictPolicy ?? throw new ArgumentNullException(nameof(verdictPolicy)); + m_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public void Attach(VisualInspectionTarget target) + { + if (target == null) + { + throw new ArgumentNullException(nameof(target)); + } + if (Interlocked.CompareExchange(ref m_target, target, null) != null) + { + throw new InvalidOperationException("The feedback sink is already attached."); + } + if (m_logger.IsEnabled(LogLevel.Information)) + { + m_logger.FeedbackAttached(target.PipelineNodeId.ToString()); + } + } + + public async ValueTask SubmitDetectionsAsync( + VisionSubmitDetectionsRequest request, + CancellationToken cancellationToken) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + cancellationToken.ThrowIfCancellationRequested(); + if (request.SceneIsEmpty && request.Purpose == VisionFeedbackPurposeEnum.GroundTruthLabel) + { + await RecordLearningSampleAsync( + StableSampleId(request.Pipeline, FrameKey(request.FrameReference, "empty-scene"), "scene-empty"), + AILearningSampleKind.Negative, + cancellationToken).ConfigureAwait(false); + } + else if (request.Purpose == VisionFeedbackPurposeEnum.GroundTruthLabel) + { + await RecordLearningSampleAsync( + StableSampleId(request.Pipeline, FrameKey(request.FrameReference, "geometry"), "geometry"), + AILearningSampleKind.Positive, + cancellationToken).ConfigureAwait(false); + } + return ServiceResult.Good; + } + + public async ValueTask SubmitInspectionResultAsync( + VisionSubmitInspectionResultRequest request, + CancellationToken cancellationToken) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + cancellationToken.ThrowIfCancellationRequested(); + VisualInspectionTarget target = RequireTarget(); + if (!request.Pipeline.Equals(target.PipelineNodeId)) + { + return new ServiceResult(StatusCodes.BadNodeIdUnknown, + LocalizedText.From("The pipeline node id does not match the attached inspection pipeline.")); + } + string resultId = string.IsNullOrEmpty(request.ResultId) + ? StableResultId(request.Pipeline, request.Characteristics) + : request.ResultId; + if (!TryGetImageReference(resultId, out VisionImageReferenceDataType? frameReference)) + { + return new ServiceResult(StatusCodes.BadInvalidState, + LocalizedText.From("SubmitImageReference must provide provenance before publishing a result.")); + } + VisionResultEvaluationEnum evaluation = m_verdictPolicy.JudgeCharacteristics(request.Characteristics); + PublishedInspectionResult published = await m_publisher.PublishAsync( + target, + resultId, + TimestampOrNow(frameReference.Timestamp), + evaluation, + request.Characteristics, + "agent-edge-off-server", + frameReference, + string.Empty, + cancellationToken).ConfigureAwait(false); + if (evaluation == VisionResultEvaluationEnum.NotDecidable) + { + // An off-server agent supplied evidence, but production policy still remains external: + // the server only exposes the dialog and waits for the operator's OPC UA response. + m_operatorDialog?.RequestDisposition(published); + } + m_logger.AgentInspectionPublished(resultId, evaluation); + return ServiceResult.Good; + } + + public async ValueTask SubmitCorrectionAsync( + VisionSubmitCorrectionRequest request, + CancellationToken cancellationToken) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + cancellationToken.ThrowIfCancellationRequested(); + VisualInspectionTarget target = RequireTarget(); + if (!request.Pipeline.Equals(target.PipelineNodeId)) + { + return new ServiceResult(StatusCodes.BadNodeIdUnknown, + LocalizedText.From("The pipeline node id does not match the attached inspection pipeline.")); + } + AILearningSampleKind kind = request.RetractAll + ? AILearningSampleKind.Negative + : AILearningSampleKind.Positive; + string sampleId = GroundTruthSampleId(request.Pipeline, request.ResultId); + await RecordLearningSampleAsync(sampleId, kind, cancellationToken).ConfigureAwait(false); + if (request.CorrectedCharacteristics.Count > 0) + { + string resultId = FormattableString.Invariant($"correction-{Sanitize(request.ResultId)}"); + VisionImageReferenceDataType frameReference = TryGetImageReference(request.ResultId, out var image) + ? image + : FrameFromPublished(request.ResultId); + await m_publisher.PublishAsync( + target, + resultId, + TimestampOrNow(frameReference.Timestamp), + m_verdictPolicy.JudgeCharacteristics(request.CorrectedCharacteristics), + request.CorrectedCharacteristics, + "operator-ground-truth", + frameReference, + FixtureFromPublished(request.ResultId), + cancellationToken).ConfigureAwait(false); + } + m_logger.CorrectionRecorded(sampleId, kind); + return ServiceResult.Good; + } + + public ValueTask SubmitImageReferenceAsync( + VisionSubmitImageReferenceRequest request, + CancellationToken cancellationToken) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + cancellationToken.ThrowIfCancellationRequested(); + if (string.IsNullOrEmpty(request.ResultId)) + { + return ValueTask.FromResult(new ServiceResult( + StatusCodes.BadInvalidArgument, + LocalizedText.From("An image reference must name the result it belongs to."))); + } + lock (m_lock) + { + m_imageReferences[request.ResultId] = request.Image; + } + return ValueTask.FromResult(ServiceResult.Good); + } + + public void AttachOperatorDialog(OperatorDialogController operatorDialog) + { + m_operatorDialog = operatorDialog ?? throw new ArgumentNullException(nameof(operatorDialog)); + } + + public ValueTask HandleOperatorDispositionAsync( + PublishedInspectionResult result, + OperatorDisposition disposition, + CancellationToken cancellationToken) + { + if (result == null) + { + throw new ArgumentNullException(nameof(result)); + } + return HandleOperatorDispositionCoreAsync(result, disposition, cancellationToken); + } + + private async ValueTask RecordLearningSampleAsync( + string sampleId, + AILearningSampleKind kind, + CancellationToken cancellationToken) + { + AINodeManager? manager = m_aiRegistry.NodeManager; + if (manager == null) + { + m_logger.LearningSampleSkipped(sampleId); + return; + } + bool added = await manager.RecordLearningSampleAsync(sampleId, kind, cancellationToken) + .ConfigureAwait(false); + m_logger.LearningSampleRecorded(sampleId, kind, added); + } + + private VisualInspectionTarget RequireTarget() + { + return m_target ?? throw new InvalidOperationException("The feedback sink is not attached."); + } + + private async ValueTask HandleOperatorDispositionCoreAsync( + PublishedInspectionResult result, + OperatorDisposition disposition, + CancellationToken cancellationToken) + { + VisualInspectionTarget target = RequireTarget(); + VisionResultEvaluationEnum evaluation = disposition switch + { + OperatorDisposition.AcceptAsOk => VisionResultEvaluationEnum.Ok, + OperatorDisposition.AcceptAsNotOk => VisionResultEvaluationEnum.NotOk, + _ => VisionResultEvaluationEnum.NotDecidable + }; + if (evaluation == VisionResultEvaluationEnum.NotDecidable) + { + m_logger.OperatorDispositionIgnored(result.ResultId, disposition); + return; + } + + AILearningSampleKind kind = disposition == OperatorDisposition.AcceptAsNotOk + ? AILearningSampleKind.Negative + : AILearningSampleKind.Positive; + string sampleId = GroundTruthSampleId(target.PipelineNodeId, result.ResultId); + await RecordLearningSampleAsync(sampleId, kind, cancellationToken).ConfigureAwait(false); + string correctionId = FormattableString.Invariant( + $"operator-{Sanitize(result.ResultId)}-{disposition}"); + await m_publisher.PublishAsync( + target, + correctionId, + TimestampOrNow(result.FrameReference.Timestamp), + evaluation, + result.Characteristics, + "operator-ground-truth", + result.FrameReference, + result.FixtureName, + cancellationToken).ConfigureAwait(false); + m_logger.OperatorDispositionRecorded(result.ResultId, disposition, sampleId, kind); + } + + private static string StableSampleId(NodeId pipeline, string resultId, string purpose) + { + return FormattableString.Invariant($"{pipeline}|{resultId}|{purpose}"); + } + + /// + /// Identifies the ground truth held for one inspection result. + /// + /// + /// A correction can arrive by two routes - an operator answering the + /// disposition dialog, or a caller invoking SubmitCorrection - and + /// both describe the same decision about the same frame. Keying the sample + /// on the result alone makes the second arrival a duplicate that the + /// accounting rejects. Keying it on the route, as this once did, made one + /// human decision count twice and inflated the very number section 9.4 + /// asks a Server to keep honestly. + /// + private static string GroundTruthSampleId(NodeId pipeline, string resultId) + { + return StableSampleId(pipeline, resultId, "ground-truth"); + } + + private static string StableResultId( + NodeId pipeline, + ArrayOf characteristics) + { + string key = characteristics.Count == 0 + ? "empty" + : characteristics[0].CharacteristicId + + ":" + + characteristics[0].Actual.ToString(System.Globalization.CultureInfo.InvariantCulture); + return FormattableString.Invariant($"agent-insp-{Sanitize(pipeline.ToString())}-{Sanitize(key)}"); + } + + private static string FrameKey(VisionImageReferenceDataType frame, string fallback) + { + return string.IsNullOrEmpty(frame?.Uri) ? fallback : frame.Uri; + } + + private bool TryGetImageReference( + string resultId, + [NotNullWhen(true)] out VisionImageReferenceDataType? image) + { + lock (m_lock) + { + return m_imageReferences.TryGetValue(resultId, out image); + } + } + + private VisionImageReferenceDataType FrameFromPublished(string resultId) + { + return m_publisher.TryGetPublished(resultId, out PublishedInspectionResult? published) && + published != null + ? published.FrameReference + : new VisionImageReferenceDataType { Timestamp = TimestampOrNow(DateTimeUtc.MinValue) }; + } + + private string FixtureFromPublished(string resultId) + { + return m_publisher.TryGetPublished(resultId, out PublishedInspectionResult? published) && + published != null + ? published.FixtureName + : string.Empty; + } + + private DateTimeUtc TimestampOrNow(DateTimeUtc timestamp) + { + return timestamp.IsNull ? DateTimeUtc.From(m_timeProvider.GetUtcNow()) : timestamp; + } + + private static string Sanitize(string value) + { + return string.IsNullOrEmpty(value) ? "unknown" : value.Replace('-', '_'); + } + + private readonly VisualInspectionResultPublisher m_publisher; + private readonly AINodeManagerRegistry m_aiRegistry; + private readonly InspectionVerdictPolicy m_verdictPolicy; + private readonly TimeProvider m_timeProvider; + private readonly ILogger m_logger; + private readonly Lock m_lock = new(); + private readonly Dictionary m_imageReferences = []; + private VisualInspectionTarget? m_target; + private OperatorDialogController? m_operatorDialog; + } + + internal static partial class VisualInspectionFeedbackSinkLog + { + [LoggerMessage(EventId = VisualInspectionCellEventIds.Feedback + 1, + Level = LogLevel.Information, + Message = "Visual inspection feedback sink attached to pipeline {PipelineNodeId}.")] + public static partial void FeedbackAttached(this ILogger logger, string pipelineNodeId); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.Feedback + 2, + Level = LogLevel.Information, + Message = "Published off-server inspection result {ResultId}: {Verdict}.")] + public static partial void AgentInspectionPublished( + this ILogger logger, + string resultId, + VisionResultEvaluationEnum verdict); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.Feedback + 3, + Level = LogLevel.Information, + Message = "Recorded correction sample {SampleId} ({Kind}).")] + public static partial void CorrectionRecorded( + this ILogger logger, + string sampleId, + AILearningSampleKind kind); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.Feedback + 4, + Level = LogLevel.Information, + Message = "Learning sample {SampleId} ({Kind}) added={Added}.")] + public static partial void LearningSampleRecorded( + this ILogger logger, + string sampleId, + AILearningSampleKind kind, + bool added); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.Feedback + 5, + Level = LogLevel.Warning, + Message = "AI node manager not available; learning sample {SampleId} was not counted.")] + public static partial void LearningSampleSkipped( + this ILogger logger, + string sampleId); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.Feedback + 6, + Level = LogLevel.Information, + Message = "Operator disposition {Disposition} for {ResultId} recorded as {SampleId} ({Kind}).")] + public static partial void OperatorDispositionRecorded( + this ILogger logger, + string resultId, + OperatorDisposition disposition, + string sampleId, + AILearningSampleKind kind); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.Feedback + 7, + Level = LogLevel.Information, + Message = "Operator disposition {Disposition} for result {ResultId} is operational only; no ground-truth sample was recorded.")] + public static partial void OperatorDispositionIgnored( + this ILogger logger, + string resultId, + OperatorDisposition disposition); + } +} diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionInferenceBackend.cs b/samples/Vision/VisualInspectionCell/VisualInspectionInferenceBackend.cs new file mode 100644 index 0000000000..e8efe6aa5d --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionInferenceBackend.cs @@ -0,0 +1,227 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.AI.Inference; + +namespace Vision.VisualInspectionCell +{ + internal sealed class VisualInspectionInferenceBackend : IInferenceBackend + { + public VisualInspectionInferenceBackend( + VisualInspectionAnalysisService analysis, + VisualInspectionCellOptions options) + { + m_analysis = analysis ?? throw new ArgumentNullException(nameof(analysis)); + m_options = options ?? throw new ArgumentNullException(nameof(options)); + } + + public InferenceSite Site => m_options.InferenceLocation == VisualInspectionInferenceLocation.EdgeOffServer + ? InferenceSite.EdgeOffServer + : InferenceSite.OnServer; + + public ValueTask> ListModelsAsync( + string? filter, + uint maxResults, + CancellationToken ct) + { + ct.ThrowIfCancellationRequested(); + IReadOnlyList models = string.IsNullOrEmpty(filter) || + Model.Name.Contains(filter, StringComparison.OrdinalIgnoreCase) + ? [Model] + : []; + return ValueTask.FromResult(models); + } + + public ValueTask InvokeAsync( + InferenceRequest request, + CancellationToken ct) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + ct.ThrowIfCancellationRequested(); + if (!TryReadFixtureName(request.Payload.Span, out string fixture)) + { + return ValueTask.FromResult(new InferenceResult + { + Ok = false, + ContentType = "application/json", + ModelUsed = Model.Name, + Finish = InferenceFinish.Error, + Message = "The inference payload does not name a known fixture." + }); + } + IReadOnlyList measurements = m_analysis.MeasureByName(fixture); + byte[] payload = JsonSerializer.SerializeToUtf8Bytes(new + { + fixture, + confidence = 0.99, + measurements = Project(measurements) + }); + return ValueTask.FromResult(new InferenceResult + { + Ok = true, + Payload = payload, + ContentType = "application/json", + ModelUsed = Model.Name, + UsageUnit = "tokens", + InputUnits = (ulong)Math.Max(1, request.Payload.Length / 4), + OutputUnits = (ulong)Math.Max(1, payload.Length / 4), + TotalUnits = (ulong)Math.Max(2, (request.Payload.Length + payload.Length) / 4) + }); + } + + public ValueTask ProbeAsync(CancellationToken ct) + { + ct.ThrowIfCancellationRequested(); + return ValueTask.FromResult(new BackendProbe + { + Reachable = true, + Detail = Model.Name + " in-process" + }); + } + + private bool TryReadFixtureName(ReadOnlySpan payload, out string fixture) + { + fixture = string.Empty; + if (payload.IsEmpty) + { + return false; + } + JsonDocument document; + try + { + document = JsonDocument.Parse(payload.ToArray()); + } + catch (JsonException) + { + return false; + } + using (document) + { + if (TryReadFixtureProperty(document.RootElement, "fixture", out fixture) || + TryReadFixtureProperty(document.RootElement, "fixtureName", out fixture) || + TryReadFixtureProperty(document.RootElement, "image", out fixture)) + { + return true; + } + if (TryReadChatFixture(document.RootElement, out fixture)) + { + return true; + } + } + return false; + } + + private bool TryReadChatFixture(JsonElement root, out string fixture) + { + fixture = string.Empty; + if (root.ValueKind != JsonValueKind.Object || + !root.TryGetProperty("messages", out JsonElement messages) || + messages.ValueKind != JsonValueKind.Array) + { + return false; + } + foreach (JsonElement message in messages.EnumerateArray()) + { + if (message.ValueKind == JsonValueKind.Object && + message.TryGetProperty("content", out JsonElement content) && + content.ValueKind == JsonValueKind.String && + TryExtractFixture(content.GetString(), out fixture)) + { + return true; + } + } + return false; + } + + private bool TryReadFixtureProperty(JsonElement root, string propertyName, out string fixture) + { + fixture = string.Empty; + if (root.ValueKind != JsonValueKind.Object || + !root.TryGetProperty(propertyName, out JsonElement property) || + property.ValueKind != JsonValueKind.String) + { + return false; + } + return TryExtractFixture(property.GetString(), out fixture); + } + + private bool TryExtractFixture(string? value, out string fixture) + { + if (m_analysis.TryResolveFixtureName(value, out fixture)) + { + return true; + } + if (m_analysis.TryResolveFixtureFromUri(value, out fixture)) + { + return true; + } + return false; + } + + private static object[] Project(IReadOnlyList measurements) + { + var projected = new object[measurements.Count]; + for (int ii = 0; ii < measurements.Count; ii++) + { + MeasuredCharacteristic measurement = measurements[ii]; + projected[ii] = new + { + characteristicId = measurement.CharacteristicId, + actual = measurement.Actual.ToString("0.###", CultureInfo.InvariantCulture), + uncertainty = measurement.Uncertainty.ToString("0.###", CultureInfo.InvariantCulture), + unit = "mm" + }; + } + return projected; + } + + public static BackendModel Model { get; } = new() + { + Publisher = "sample", + Name = "bracket-geometry-analyser", + Version = "1.0.0", + TaskKind = "dimensional-inspection", + Framework = "deterministic-pixel-measurement", + Capabilities = ["vision-measurement"] + }; + + private readonly VisualInspectionAnalysisService m_analysis; + private readonly VisualInspectionCellOptions m_options; + } +} diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionInferenceProvider.cs b/samples/Vision/VisualInspectionCell/VisualInspectionInferenceProvider.cs new file mode 100644 index 0000000000..8fcb0a05c5 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionInferenceProvider.cs @@ -0,0 +1,158 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Vision.VisualInspectionCell +{ + internal sealed class VisualInspectionInferenceProvider : IVisionInferenceProvider + { + public VisualInspectionInferenceProvider( + VisualInspectionAnalysisService analysis, + VisualInspectionResultPublisher publisher, + OperatorDialogController operatorDialog, + ILogger logger) + { + m_analysis = analysis ?? throw new ArgumentNullException(nameof(analysis)); + m_publisher = publisher ?? throw new ArgumentNullException(nameof(publisher)); + m_operatorDialog = operatorDialog ?? throw new ArgumentNullException(nameof(operatorDialog)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public void Attach(VisualInspectionTarget target) + { + if (target == null) + { + throw new ArgumentNullException(nameof(target)); + } + if (Interlocked.CompareExchange(ref m_target, target, null) != null) + { + throw new InvalidOperationException("The inference provider is already attached."); + } + if (m_logger.IsEnabled(LogLevel.Information)) + { + m_logger.InferenceAttached(target.PipelineNodeId.ToString()); + } + } + + public async ValueTask RunInferenceAsync( + VisionInferenceRunRequest request, + CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + VisualInspectionTarget target = RequireTarget(); + if (request.Timestamp.IsNull) + { + return new VisionInferenceRunResult( + new ServiceResult(StatusCodes.BadInvalidArgument, + LocalizedText.From("RunInference requires a timestamp so retries use a stable result id.")), + string.Empty); + } + + DateTimeUtc timestamp = request.Timestamp; + InspectionAnalysis analysis = m_analysis.AnalyzeForCycle(timestamp); + string resultId = FormattableString.Invariant( + $"insp-{PathSafeFixtureName(analysis.FixtureName)}-{timestamp.Value}"); + VisionImageReferenceDataType frameReference = + m_analysis.CreateImageReference(analysis.FixtureName, timestamp); + PublishedInspectionResult published = await m_publisher.PublishAsync( + target, + resultId, + timestamp, + analysis.Verdict, + analysis.Characteristics, + ModelVersion, + frameReference, + analysis.FixtureName, + cancellationToken).ConfigureAwait(false); + if (analysis.Verdict == Opc.Ua.Vision.VisionResultEvaluationEnum.NotDecidable) + { + m_operatorDialog.RequestDisposition(published); + } + m_logger.InferencePublished(resultId, analysis.FixtureName, analysis.Verdict); + return new VisionInferenceRunResult(ServiceResult.Good, resultId); + } + + public ValueTask StartContinuousAsync(NodeId pipeline, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return ValueTask.FromResult(new ServiceResult(StatusCodes.BadNotSupported, + LocalizedText.From("This sample is externally driven; call RunInference for each inspection."))); + } + + public ValueTask StopAsync(NodeId pipeline, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return ValueTask.FromResult(ServiceResult.Good); + } + + private VisualInspectionTarget RequireTarget() + { + return m_target ?? throw new InvalidOperationException("The inference provider is not attached."); + } + + private static string PathSafeFixtureName(string fixtureName) + { + return fixtureName.Replace(".png", string.Empty, StringComparison.OrdinalIgnoreCase) + .Replace('-', '_'); + } + + private const string ModelVersion = "visual-inspection-analyser-1"; + private readonly VisualInspectionAnalysisService m_analysis; + private readonly VisualInspectionResultPublisher m_publisher; + private readonly OperatorDialogController m_operatorDialog; + private readonly ILogger m_logger; + private VisualInspectionTarget? m_target; + } + + internal static partial class VisualInspectionInferenceProviderLog + { + [LoggerMessage(EventId = VisualInspectionCellEventIds.Inference + 1, + Level = LogLevel.Information, + Message = "Visual inspection inference attached to pipeline {PipelineNodeId}.")] + public static partial void InferenceAttached( + this ILogger logger, + string pipelineNodeId); + + [LoggerMessage(EventId = VisualInspectionCellEventIds.Inference + 2, + Level = LogLevel.Information, + Message = "Published inspection result {ResultId} from {FixtureName}: {Verdict}.")] + public static partial void InferencePublished( + this ILogger logger, + string resultId, + string fixtureName, + Opc.Ua.Vision.VisionResultEvaluationEnum verdict); + } +} diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionMediaProvider.cs b/samples/Vision/VisualInspectionCell/VisualInspectionMediaProvider.cs new file mode 100644 index 0000000000..d968b60020 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionMediaProvider.cs @@ -0,0 +1,160 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Security.Cryptography; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Vision.VisualInspectionCell +{ + internal sealed class VisualInspectionMediaProvider : IVisionMediaProvider + { + public VisualInspectionMediaProvider( + VisualInspectionAnalysisService analysis, + VisualInspectionResultPublisher publisher, + ILogger logger) + { + m_analysis = analysis ?? throw new ArgumentNullException(nameof(analysis)); + m_publisher = publisher ?? throw new ArgumentNullException(nameof(publisher)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + public ValueTask GetStreamAsync( + VisionStreamRequest request, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + var session = new VisionStreamSessionDataType + { + SessionToken = ByteString.Empty, + Uri = string.Empty, + Protocol = request.PreferredProtocol, + ExpiresAt = DateTimeUtc.MinValue + }; + return ValueTask.FromResult(new VisionStreamLease( + new ServiceResult(StatusCodes.BadNotSupported, + LocalizedText.From("The visual-inspection sample serves still PNG fixture clips only.")), + session, + request.Endpoint)); + } + + public ValueTask ReleaseStreamAsync( + ByteString sessionToken, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return ValueTask.FromResult(ServiceResult.Good); + } + + public ValueTask ConfigureStreamAsync( + VisionStreamConfigurationRequest request, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return ValueTask.FromResult(new ServiceResult(StatusCodes.BadNotSupported, + LocalizedText.From("Fixture clips are static and cannot be reconfigured."))); + } + + public ValueTask SelectEndpointAsync( + NodeId streamEndpoint, NodeId clipEndpoint, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return ValueTask.FromResult(ServiceResult.Good); + } + + public ValueTask GetClipAsync( + VisionClipRequest request, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + string fixture; + if (m_publisher.TryGetPublished(request.ResultId, out PublishedInspectionResult? published) && + published != null && + !string.IsNullOrEmpty(published.FixtureName)) + { + fixture = published.FixtureName; + } + else if (!m_analysis.TryResolveFixtureName(request.ResultId, out fixture) && + !m_analysis.TryResolveFixtureFromUri(request.ResultId, out fixture)) + { + return ValueTask.FromResult(new VisionClipResult( + new ServiceResult(StatusCodes.BadNodeIdUnknown, + LocalizedText.From("The requested inspection result has no fixture clip.")), + new VisionImageReferenceDataType(), + request.Endpoint, + ByteString.Empty)); + } + string path = Path.Combine(m_analysis.FixtureDirectory, fixture); + ByteString png = ByteString.From(File.ReadAllBytes(path)); + byte[] digest = SHA256.HashData(png.Span); + DateTimeUtc timestamp = request.Timestamp.IsNull + ? DateTimeUtc.From(DateTime.UnixEpoch) + : request.Timestamp; + var image = new VisionImageReferenceDataType + { + Uri = FormattableString.Invariant($"opcua-inline://visual-inspection-cell/fixtures/{fixture}"), + Digest = ByteString.From(digest), + DigestAlgorithm = "SHA-256", + Format = VisionClipFormatEnum.Png, + PixelFormat = PixelFormat, + Width = Width, + Height = Height, + SizeBytes = (uint)png.Length, + Timestamp = timestamp + }; + m_logger.FixtureClipServed(fixture, png.Length); + return ValueTask.FromResult(new VisionClipResult( + ServiceResult.Good, + image, + request.Endpoint, + request.RequestInline ? png : ByteString.Empty)); + } + + public const string PixelFormat = "RGB8"; + public const uint Width = 800; + public const uint Height = 600; + + private readonly VisualInspectionAnalysisService m_analysis; + private readonly VisualInspectionResultPublisher m_publisher; + private readonly ILogger m_logger; + } + + internal static partial class VisualInspectionMediaProviderLog + { + [LoggerMessage(EventId = VisualInspectionCellEventIds.Media + 1, + Level = LogLevel.Information, + Message = "Served fixture clip {FixtureName} ({Bytes} bytes).")] + public static partial void FixtureClipServed( + this ILogger logger, + string fixtureName, + int bytes); + } +} diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionResultPublisher.cs b/samples/Vision/VisualInspectionCell/VisualInspectionResultPublisher.cs new file mode 100644 index 0000000000..0c518115c1 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionResultPublisher.cs @@ -0,0 +1,157 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua; +using Opc.Ua.Server; +using Opc.Ua.Vision; + +namespace Vision.VisualInspectionCell +{ + internal sealed class VisualInspectionResultPublisher + { + public async ValueTask PublishAsync( + VisualInspectionTarget target, + string resultId, + DateTimeUtc timestamp, + VisionResultEvaluationEnum evaluation, + ArrayOf characteristics, + string modelVersion, + VisionImageReferenceDataType frameReference, + string fixtureName, + CancellationToken cancellationToken) + { + if (target == null) + { + throw new ArgumentNullException(nameof(target)); + } + + ISystemContext context = target.SystemContext; + if (TryGetPublished(resultId, out PublishedInspectionResult? existing) && + existing != null) + { + return existing; + } + + var qualifiedName = new QualifiedName(resultId, target.InstanceNamespaceIndex); + InspectionResultState state = context.CreateInstanceOfInspectionResultType( + target.ResultsFolder, qualifiedName); + state.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + if (state.ResultId != null) + { + state.ResultId.Value = resultId; + } + if (state.CreationTime != null) + { + state.CreationTime.Value = timestamp; + } + state.AddSensor(context); + state.CreateOrReplaceSensor(context, null).Value = target.SensorNodeId; + state.AddPipeline(context); + state.CreateOrReplacePipeline(context, null).Value = target.PipelineNodeId; + state.AddModelVersionUsed(context); + state.CreateOrReplaceModelVersionUsed(context, null).Value = modelVersion; + state.CreateOrReplaceEvaluation(context, null).Value = evaluation; + state.AddPartId(context); + state.CreateOrReplacePartId(context, null).Value = InspectionRecipe.PartId; + state.AddRecipeId(context); + state.CreateOrReplaceRecipeId(context, null).Value = InspectionRecipe.RecipeId; + state.CreateOrReplaceCharacteristics(context, null).Value = characteristics; + state.AddFrame(context); + BaseDataVariableState frame = + state.CreateOrReplaceFrame(context, null); + frame.Value = frameReference; + state.NodeId = context.RequireNodeIdFactory().New(context, state); + context.AssignInstanceChildNodeIds(state, state.NodeId); + var published = new PublishedInspectionResult( + resultId, + state.NodeId, + evaluation, + characteristics, + modelVersion, + frameReference, + fixtureName); + var evicted = new List(); + lock (m_lock) + { + if (m_results.TryGetValue(resultId, out PublishedInspectionResult? retained)) + { + return retained; + } + target.ResultsFolder.AddChild(state); + m_results.Add(resultId, published); + m_retained.Add(resultId); + while (m_retained.Count > ResultRetention) + { + string evictedResultId = m_retained[0]; + m_retained.RemoveAt(0); + if (m_results.Remove(evictedResultId, out PublishedInspectionResult? removed)) + { + evicted.Add(removed.NodeId); + } + } + } + await target.NodeManager.AddPredefinedNodeAsync(state, cancellationToken).ConfigureAwait(false); + for (int ii = 0; ii < evicted.Count; ii++) + { + if (target.SystemContext is ServerSystemContext serverContext) + { + _ = await target.NodeManager.DeleteNodeAsync(serverContext, evicted[ii], cancellationToken) + .ConfigureAwait(false); + } + } + return published; + } + + public bool TryGetPublished(string resultId, out PublishedInspectionResult? published) + { + lock (m_lock) + { + return m_results.TryGetValue(resultId, out published); + } + } + + private const int ResultRetention = 16; + private readonly Lock m_lock = new(); + private readonly Dictionary m_results = []; + private readonly List m_retained = []; + } + + internal sealed record PublishedInspectionResult( + string ResultId, + NodeId NodeId, + VisionResultEvaluationEnum Evaluation, + ArrayOf Characteristics, + string ModelVersion, + VisionImageReferenceDataType FrameReference, + string FixtureName); +} diff --git a/samples/Vision/VisualInspectionCell/VisualInspectionTarget.cs b/samples/Vision/VisualInspectionCell/VisualInspectionTarget.cs new file mode 100644 index 0000000000..219b1e6cc4 --- /dev/null +++ b/samples/Vision/VisualInspectionCell/VisualInspectionTarget.cs @@ -0,0 +1,78 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Opc.Ua; +using Opc.Ua.Server; + +namespace Vision.VisualInspectionCell +{ + internal sealed class VisualInspectionTarget + { + public VisualInspectionTarget( + AsyncCustomNodeManager nodeManager, + ISystemContext systemContext, + ushort instanceNamespaceIndex, + NodeId pipelineNodeId, + NodeId sensorNodeId, + NodeId deploymentNodeId, + NodeId learningJobNodeId, + FolderState resultsFolder) + { + NodeManager = nodeManager ?? throw new ArgumentNullException(nameof(nodeManager)); + SystemContext = systemContext ?? throw new ArgumentNullException(nameof(systemContext)); + InstanceNamespaceIndex = instanceNamespaceIndex; + PipelineNodeId = pipelineNodeId.IsNull + ? throw new ArgumentException("Pipeline NodeId must not be null.", nameof(pipelineNodeId)) + : pipelineNodeId; + SensorNodeId = sensorNodeId.IsNull + ? throw new ArgumentException("Sensor NodeId must not be null.", nameof(sensorNodeId)) + : sensorNodeId; + DeploymentNodeId = deploymentNodeId; + LearningJobNodeId = learningJobNodeId; + ResultsFolder = resultsFolder ?? throw new ArgumentNullException(nameof(resultsFolder)); + } + + public AsyncCustomNodeManager NodeManager { get; } + + public ISystemContext SystemContext { get; } + + public ushort InstanceNamespaceIndex { get; } + + public NodeId PipelineNodeId { get; } + + public NodeId SensorNodeId { get; } + + public NodeId DeploymentNodeId { get; } + + public NodeId LearningJobNodeId { get; } + + public FolderState ResultsFolder { get; } + } +} diff --git a/src/Opc.Ua.AI.Client/AiClient.cs b/src/Opc.Ua.AI.Client/AiClient.cs new file mode 100644 index 0000000000..2e23c40f65 --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiClient.cs @@ -0,0 +1,374 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client; + +namespace Opc.Ua.AI.Client +{ + public sealed class AIClient + { + public AIClient(ISession session, ITelemetryContext telemetry) + { + Operations = new AIClientOperations(session, telemetry); + } + + public ISession Session => Operations.Session; + + public ITelemetryContext Telemetry => Operations.Telemetry; + + public bool IsAINamespaceAvailable => Operations.TryGetAINamespaceIndex(out _); + + public NodeId AIRootId + { + get + { + if (!Operations.TryGetAINamespaceIndex(out ushort _)) + { + return NodeId.Null; + } + return NodeId.Create(Objects.AiModelManagement, Namespaces.AI, Session.NamespaceUris); + } + } + + public NodeId ModelsFolderId => CreateWellKnownNode(Objects.AiRootType_Models); + + public NodeId DatasetsFolderId => CreateWellKnownNode(Objects.AiRootType_Datasets); + + public NodeId DeploymentsFolderId => CreateWellKnownNode(Objects.AiRootType_Deployments); + + public NodeId LearningJobsFolderId => CreateWellKnownNode(Objects.AiRootType_LearningJobs); + + public ValueTask GetRegistriesFolderIdAsync( + CancellationToken cancellationToken = default) + { + return ResolveOptionalRootChildAsync(BrowseNames.Registries, cancellationToken); + } + + public ValueTask GetSourcesFolderIdAsync( + CancellationToken cancellationToken = default) + { + return ResolveOptionalRootChildAsync(BrowseNames.Sources, cancellationToken); + } + + public ValueTask GetEvaluationsFolderIdAsync( + CancellationToken cancellationToken = default) + { + return ResolveOptionalRootChildAsync(BrowseNames.Evaluations, cancellationToken); + } + + public ValueTask GetJobsFolderIdAsync( + CancellationToken cancellationToken = default) + { + return ResolveOptionalRootChildAsync(BrowseNames.Jobs, cancellationToken); + } + + public async ValueTask> DiscoverModelsAsync( + CancellationToken cancellationToken = default) + { + NodeId folder = await GetRootFolderIdAsync(BrowseNames.Models, cancellationToken) + .ConfigureAwait(false); + return await DiscoverAsync(folder, ObjectTypes.ModelType, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask> DiscoverDatasetsAsync( + CancellationToken cancellationToken = default) + { + NodeId folder = await GetRootFolderIdAsync(BrowseNames.Datasets, cancellationToken) + .ConfigureAwait(false); + return await DiscoverAsync(folder, ObjectTypes.DatasetType, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask> DiscoverDeploymentsAsync( + CancellationToken cancellationToken = default) + { + NodeId folder = await GetRootFolderIdAsync(BrowseNames.Deployments, cancellationToken) + .ConfigureAwait(false); + return await DiscoverAsync(folder, ObjectTypes.DeploymentType, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask> DiscoverLearningJobsAsync( + CancellationToken cancellationToken = default) + { + NodeId folder = await GetRootFolderIdAsync(BrowseNames.LearningJobs, cancellationToken) + .ConfigureAwait(false); + return await DiscoverAsync(folder, ObjectTypes.LearningJobType, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask> DiscoverInferenceJobsAsync( + CancellationToken cancellationToken = default) + { + NodeId folder = await GetJobsFolderIdAsync(cancellationToken).ConfigureAwait(false); + return await DiscoverAsync(folder, ObjectTypes.InferenceJobType, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask> DiscoverEvaluationRunsAsync( + CancellationToken cancellationToken = default) + { + NodeId folder = await GetEvaluationsFolderIdAsync(cancellationToken).ConfigureAwait(false); + return await DiscoverAsync(folder, ObjectTypes.EvaluationRunType, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask> DiscoverSourcesAsync( + CancellationToken cancellationToken = default) + { + NodeId folder = await GetSourcesFolderIdAsync(cancellationToken).ConfigureAwait(false); + return await DiscoverAsync(folder, ObjectTypes.ModelSourceType, cancellationToken) + .ConfigureAwait(false); + } + + public async IAsyncEnumerable EnumerateModelsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId folder = await GetRootFolderIdAsync(BrowseNames.Models, cancellationToken) + .ConfigureAwait(false); + await foreach (AINodeEntry entry in EnumerateInstancesAsync( + folder, ObjectTypes.ModelType, cancellationToken).ConfigureAwait(false)) + { + yield return entry; + } + } + + public async IAsyncEnumerable EnumerateDatasetsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId folder = await GetRootFolderIdAsync(BrowseNames.Datasets, cancellationToken) + .ConfigureAwait(false); + await foreach (AINodeEntry entry in EnumerateInstancesAsync( + folder, ObjectTypes.DatasetType, cancellationToken).ConfigureAwait(false)) + { + yield return entry; + } + } + + public async IAsyncEnumerable EnumerateDeploymentsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId folder = await GetRootFolderIdAsync(BrowseNames.Deployments, cancellationToken) + .ConfigureAwait(false); + await foreach (AINodeEntry entry in EnumerateInstancesAsync( + folder, ObjectTypes.DeploymentType, cancellationToken).ConfigureAwait(false)) + { + yield return entry; + } + } + + public async IAsyncEnumerable EnumerateLearningJobsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId folder = await GetRootFolderIdAsync(BrowseNames.LearningJobs, cancellationToken) + .ConfigureAwait(false); + await foreach (AINodeEntry entry in EnumerateInstancesAsync( + folder, ObjectTypes.LearningJobType, cancellationToken).ConfigureAwait(false)) + { + yield return entry; + } + } + + public async IAsyncEnumerable EnumerateInferenceJobsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId folder = await GetJobsFolderIdAsync(cancellationToken).ConfigureAwait(false); + await foreach (AINodeEntry entry in EnumerateInstancesAsync( + folder, ObjectTypes.InferenceJobType, cancellationToken).ConfigureAwait(false)) + { + yield return entry; + } + } + + public async IAsyncEnumerable EnumerateEvaluationRunsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId folder = await GetEvaluationsFolderIdAsync(cancellationToken).ConfigureAwait(false); + await foreach (AINodeEntry entry in EnumerateInstancesAsync( + folder, ObjectTypes.EvaluationRunType, cancellationToken).ConfigureAwait(false)) + { + yield return entry; + } + } + + public async IAsyncEnumerable EnumerateSourcesAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId folder = await GetSourcesFolderIdAsync(cancellationToken).ConfigureAwait(false); + await foreach (AINodeEntry entry in EnumerateInstancesAsync( + folder, ObjectTypes.ModelSourceType, cancellationToken).ConfigureAwait(false)) + { + yield return entry; + } + } + + public AIModelClient Model(NodeId modelNodeId) + { + return new AIModelClient(Operations, modelNodeId); + } + + public AIDatasetClient Dataset(NodeId datasetNodeId) + { + return new AIDatasetClient(Operations, datasetNodeId); + } + + public AIDeploymentClient Deployment(NodeId deploymentNodeId) + { + return new AIDeploymentClient(Operations, deploymentNodeId); + } + + public AIModelSourceClient Source(NodeId sourceNodeId) + { + return new AIModelSourceClient(Operations, sourceNodeId); + } + + public AIInferenceJobClient InferenceJob(NodeId jobNodeId) + { + return new AIInferenceJobClient(Operations, jobNodeId); + } + + public AILearningJobClient LearningJob(NodeId jobNodeId) + { + return new AILearningJobClient(Operations, jobNodeId); + } + + public AIEvaluationRunClient EvaluationRun(NodeId runNodeId) + { + return new AIEvaluationRunClient(Operations, runNodeId); + } + + public AIInferenceTransferClient Transfer(NodeId transferNodeId) + { + return new AIInferenceTransferClient(Operations, transferNodeId); + } + + internal AIClientOperations Operations { get; } + + private NodeId CreateWellKnownNode(uint identifier) + { + return IsAINamespaceAvailable + ? NodeId.Create(identifier, Namespaces.AI, Session.NamespaceUris) + : NodeId.Null; + } + + private ValueTask ResolveOptionalRootChildAsync( + string browseName, + CancellationToken cancellationToken) + { + return GetRootFolderIdAsync(browseName, cancellationToken); + } + + private async ValueTask GetRootFolderIdAsync( + string browseName, + CancellationToken cancellationToken) + { + NodeId root = AIRootId; + if (root.IsNull) + { + return NodeId.Null; + } + AiRootTypeClient proxy = new(Session, root, Telemetry); + return browseName switch + { + BrowseNames.Registries => (await proxy.GetRegistriesAsync(Telemetry, cancellationToken) + .ConfigureAwait(false))?.ObjectId ?? + NodeId.Null, + BrowseNames.Sources => (await proxy.GetSourcesAsync(Telemetry, cancellationToken) + .ConfigureAwait(false))?.ObjectId ?? + NodeId.Null, + BrowseNames.Evaluations => (await proxy.GetEvaluationsAsync(Telemetry, cancellationToken) + .ConfigureAwait(false))?.ObjectId ?? + NodeId.Null, + BrowseNames.Jobs => (await proxy.GetJobsAsync(Telemetry, cancellationToken) + .ConfigureAwait(false))?.ObjectId ?? + NodeId.Null, + _ => await Operations.ResolveChildAsync(root, browseName, cancellationToken) + .ConfigureAwait(false) + }; + } + + private ValueTask> DiscoverAsync( + NodeId folder, + uint typeIdentifier, + CancellationToken cancellationToken) + { + return Operations.DiscoverInstancesAsync( + folder, + Operations.AINamespaceType(typeIdentifier), + cancellationToken); + } + + private async IAsyncEnumerable EnumerateInstancesAsync( + NodeId root, + uint typeIdentifier, + [EnumeratorCancellation] CancellationToken cancellationToken) + { + if (root.IsNull) + { + yield break; + } + NodeId typeDefinition = Operations.AINamespaceType(typeIdentifier); + if (typeDefinition.IsNull) + { + yield break; + } + ArrayOf references = await Operations + .BrowseHierarchicalObjectsAsync(root, cancellationToken).ConfigureAwait(false); + var matches = new List(); + for (int ii = 0; ii < references.Count; ii++) + { + ReferenceDescription reference = references[ii]; + NodeId typeDef = ExpandedNodeId.ToNodeId( + reference.TypeDefinition, Session.NamespaceUris); + NodeId nodeId = ExpandedNodeId.ToNodeId(reference.NodeId, Session.NamespaceUris); + if (typeDef.IsNull || nodeId.IsNull) + { + continue; + } + if (typeDef == typeDefinition || + await Session.NodeCache.IsTypeOfAsync( + typeDef, typeDefinition, cancellationToken).ConfigureAwait(false)) + { + matches.Add(new AINodeEntry( + nodeId, reference.BrowseName, reference.DisplayName, typeDef)); + } + } + for (int ii = 0; ii < matches.Count; ii++) + { + yield return matches[ii]; + } + } + } +} diff --git a/src/Opc.Ua.AI.Client/AiClientFactory.cs b/src/Opc.Ua.AI.Client/AiClientFactory.cs new file mode 100644 index 0000000000..6612848348 --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiClientFactory.cs @@ -0,0 +1,58 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client; + +namespace Opc.Ua.AI.Client +{ + public sealed class AIClientFactory + { + public AIClientFactory( + Func> sessionFactory, + ITelemetryContext telemetry) + { + m_sessionFactory = sessionFactory + ?? throw new ArgumentNullException(nameof(sessionFactory)); + m_telemetry = telemetry ?? throw new ArgumentNullException(nameof(telemetry)); + } + + public async Task CreateAsync(CancellationToken cancellationToken = default) + { + ManagedSession session = await m_sessionFactory(cancellationToken) + .ConfigureAwait(false); + return new AIClient(session, m_telemetry); + } + + private readonly Func> m_sessionFactory; + private readonly ITelemetryContext m_telemetry; + } +} diff --git a/src/Opc.Ua.AI.Client/AiClientOperations.cs b/src/Opc.Ua.AI.Client/AiClientOperations.cs new file mode 100644 index 0000000000..a180503d15 --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiClientOperations.cs @@ -0,0 +1,635 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client; + +namespace Opc.Ua.AI.Client +{ + internal sealed class AIClientOperations + { + public const int DefaultChunkSize = 4096; + + public AIClientOperations(ISession session, ITelemetryContext telemetry) + { + Session = session ?? throw new ArgumentNullException(nameof(session)); + Telemetry = telemetry ?? throw new ArgumentNullException(nameof(telemetry)); + RegisterEncodeableTypes(session); + } + + public ISession Session { get; } + + public ITelemetryContext Telemetry { get; } + + public bool TryGetAINamespaceIndex(out ushort namespaceIndex) + { + int index = Session.NamespaceUris.GetIndex(Namespaces.AI); + if (index < 0) + { + namespaceIndex = 0; + return false; + } + namespaceIndex = (ushort)index; + return true; + } + + public NodeId AINamespaceType(uint identifier) + { + return TryGetAINamespaceIndex(out ushort ns) + ? new NodeId(identifier, ns) + : NodeId.Null; + } + + public async ValueTask> BrowseAsync( + NodeId nodeId, + NodeId referenceTypeId, + BrowseDirection direction, + uint nodeClassMask, + CancellationToken cancellationToken) + { + if (nodeId.IsNull) + { + return ArrayOf.Empty; + } + (ArrayOf> results, ArrayOf errors) = + await Session.ManagedBrowseAsync( + requestHeader: null, + view: null, + nodesToBrowse: [nodeId], + maxResultsToReturn: 0, + browseDirection: direction, + referenceTypeId: referenceTypeId, + includeSubtypes: true, + nodeClassMask: nodeClassMask, + ct: cancellationToken).ConfigureAwait(false); + if (errors.Count > 0 && ServiceResult.IsBad(errors[0])) + { + return ArrayOf.Empty; + } + return results.Count > 0 ? results[0] : ArrayOf.Empty; + } + + public ValueTask> BrowseHierarchicalObjectsAsync( + NodeId nodeId, CancellationToken cancellationToken) + { + return BrowseAsync( + nodeId, + global::Opc.Ua.ReferenceTypeIds.HierarchicalReferences, + BrowseDirection.Forward, + (uint)NodeClass.Object, + cancellationToken); + } + + public async ValueTask> DiscoverInstancesAsync( + NodeId root, NodeId typeDefinition, CancellationToken cancellationToken) + { + if (root.IsNull || typeDefinition.IsNull) + { + return ArrayOf.Empty; + } + ArrayOf references = await BrowseHierarchicalObjectsAsync( + root, cancellationToken).ConfigureAwait(false); + var matches = new List(); + for (int ii = 0; ii < references.Count; ii++) + { + ReferenceDescription reference = references[ii]; + NodeId typeDef = ExpandedNodeId.ToNodeId( + reference.TypeDefinition, Session.NamespaceUris); + NodeId child = ExpandedNodeId.ToNodeId(reference.NodeId, Session.NamespaceUris); + if (typeDef.IsNull || child.IsNull) + { + continue; + } + if (typeDef == typeDefinition || + await Session.NodeCache.IsTypeOfAsync( + typeDef, typeDefinition, cancellationToken).ConfigureAwait(false)) + { + matches.Add(child); + } + } + return matches.ToArrayOf(); + } + + public async ValueTask> ResolveChildrenAsync( + NodeId parent, + ArrayOf browseNames, + CancellationToken cancellationToken) + { + if (!TryGetAINamespaceIndex(out ushort ns)) + { + var nulls = new List(browseNames.Count); + for (int ii = 0; ii < browseNames.Count; ii++) + { + nulls.Add(NodeId.Null); + } + return nulls.ToArrayOf(); + } + return await ResolveChildrenAsync(parent, browseNames, ns, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask> ResolveChildrenAsync( + NodeId parent, + ArrayOf browseNames, + ushort namespaceIndex, + CancellationToken cancellationToken) + { + if (parent.IsNull) + { + return CreateNullNodes(browseNames.Count); + } + var paths = new List(browseNames.Count); + for (int ii = 0; ii < browseNames.Count; ii++) + { + paths.Add(CreateBrowsePath(parent, browseNames[ii], namespaceIndex)); + } + TranslateBrowsePathsToNodeIdsResponse response = await Session + .TranslateBrowsePathsToNodeIdsAsync( + null, paths.ToArrayOf(), cancellationToken).ConfigureAwait(false); + var results = new List(browseNames.Count); + for (int ii = 0; ii < response.Results.Count; ii++) + { + BrowsePathResult result = response.Results[ii]; + results.Add(StatusCode.IsGood(result.StatusCode) && result.Targets.Count > 0 + ? ExpandedNodeId.ToNodeId(result.Targets[0].TargetId, Session.NamespaceUris) + : NodeId.Null); + } + while (results.Count < browseNames.Count) + { + results.Add(NodeId.Null); + } + return results.ToArrayOf(); + } + + public async ValueTask ResolveChildAsync( + NodeId parent, + string browseName, + CancellationToken cancellationToken) + { + ArrayOf nodes = await ResolveChildrenAsync( + parent, [browseName], cancellationToken).ConfigureAwait(false); + return nodes.Count > 0 ? nodes[0] : NodeId.Null; + } + + public async ValueTask FollowReferenceAsync( + NodeId source, + uint referenceTypeIdentifier, + CancellationToken cancellationToken) + { + NodeId referenceType = AINamespaceType(referenceTypeIdentifier); + if (source.IsNull || referenceType.IsNull) + { + return NodeId.Null; + } + ArrayOf references = await BrowseAsync( + source, + referenceType, + BrowseDirection.Forward, + 0, + cancellationToken).ConfigureAwait(false); + return references.Count > 0 + ? ExpandedNodeId.ToNodeId(references[0].NodeId, Session.NamespaceUris) + : NodeId.Null; + } + + public async ValueTask ReadValueAsync( + NodeId nodeId, CancellationToken cancellationToken) + { + if (nodeId.IsNull) + { + return DataValue.Null; + } + return await Session.ReadValueAsync(nodeId, cancellationToken).ConfigureAwait(false); + } + + public async ValueTask> ReadValuesAsync( + ArrayOf nodeIds, CancellationToken cancellationToken) + { + var reads = new List(nodeIds.Count); + for (int ii = 0; ii < nodeIds.Count; ii++) + { + if (!nodeIds[ii].IsNull) + { + reads.Add(new ReadValueId + { + NodeId = nodeIds[ii], + AttributeId = Attributes.Value + }); + } + } + if (reads.Count == 0) + { + return ArrayOf.Empty; + } + ArrayOf nodesToRead = reads.ToArrayOf(); + ReadResponse response = await Session.ReadAsync( + null, 0, TimestampsToReturn.Both, nodesToRead, cancellationToken) + .ConfigureAwait(false); + ClientBase.ValidateResponse(response.Results, nodesToRead); + return response.Results; + } + + public async ValueTask> CallAsync( + NodeId objectId, + string methodBrowseName, + ArrayOf inputArguments, + CancellationToken cancellationToken) + { + NodeId methodId = await ResolveChildAsync( + objectId, methodBrowseName, cancellationToken).ConfigureAwait(false); + if (methodId.IsNull) + { + throw new ServiceResultException(StatusCodes.BadMethodInvalid); + } + var request = new CallMethodRequest + { + ObjectId = objectId, + MethodId = methodId, + InputArguments = inputArguments + }; + CallResponse response = await Session.CallAsync(null, [request], cancellationToken) + .ConfigureAwait(false); + CallMethodResult result = response.Results[0]; + if (StatusCode.IsBad(result.StatusCode)) + { + throw new ServiceResultException(result.StatusCode); + } + return result.OutputArguments; + } + + public async ValueTask WriteFileAsync( + FileTypeClient file, + ByteString content, + int chunkSize, + CancellationToken cancellationToken) + { + if (file is null) + { + throw new ArgumentNullException(nameof(file)); + } + ValidateChunkSize(chunkSize); + const byte writeEraseExisting = 6; + uint handle = await file.OpenAsync(writeEraseExisting, cancellationToken) + .ConfigureAwait(false); + try + { + ReadOnlyMemory bytes = content.IsNull + ? ReadOnlyMemory.Empty + : content.Span.ToArray(); + for (int offset = 0; offset < bytes.Length; offset += chunkSize) + { + int take = Math.Min(chunkSize, bytes.Length - offset); + await file.WriteAsync( + handle, + ByteString.From(bytes.Slice(offset, take).ToArray()), + cancellationToken).ConfigureAwait(false); + } + } + finally + { + await file.CloseAsync(handle, CancellationToken.None).ConfigureAwait(false); + } + } + + public async ValueTask WriteFileAsync( + NodeId file, + ByteString content, + int chunkSize, + CancellationToken cancellationToken) + { + ValidateChunkSize(chunkSize); + const byte writeEraseExisting = 6; + ArrayOf opened = await CallAsync( + file, global::Opc.Ua.BrowseNames.Open, [Variant.From(writeEraseExisting)], cancellationToken) + .ConfigureAwait(false); + uint handle = opened.Count > 0 && opened[0].TryGetValue(out uint value) ? value : 0; + try + { + ReadOnlyMemory bytes = content.IsNull + ? ReadOnlyMemory.Empty + : content.Span.ToArray(); + for (int offset = 0; offset < bytes.Length; offset += chunkSize) + { + int take = Math.Min(chunkSize, bytes.Length - offset); + await CallAsync( + file, + global::Opc.Ua.BrowseNames.Write, + [ + Variant.From(handle), + Variant.From(ByteString.From(bytes.Slice(offset, take).ToArray())) + ], + cancellationToken).ConfigureAwait(false); + } + } + finally + { + await CallAsync( + file, + global::Opc.Ua.BrowseNames.Close, + [Variant.From(handle)], + CancellationToken.None).ConfigureAwait(false); + } + } + + public async ValueTask WriteFileAsync( + FileTypeClient file, + Stream content, + int chunkSize, + CancellationToken cancellationToken) + { + if (file is null) + { + throw new ArgumentNullException(nameof(file)); + } + if (content is null) + { + throw new ArgumentNullException(nameof(content)); + } + if (!content.CanRead) + { + throw new ArgumentException("Stream must be readable.", nameof(content)); + } + ValidateChunkSize(chunkSize); + const byte writeEraseExisting = 6; + uint handle = await file.OpenAsync(writeEraseExisting, cancellationToken) + .ConfigureAwait(false); + byte[] buffer = new byte[chunkSize]; + try + { + while (true) + { + int read = await content.ReadAsync(buffer.AsMemory(0, buffer.Length), cancellationToken) + .ConfigureAwait(false); + if (read == 0) + { + break; + } + byte[] chunk = new byte[read]; + Array.Copy(buffer, chunk, read); + await file.WriteAsync(handle, ByteString.From(chunk), cancellationToken) + .ConfigureAwait(false); + } + } + finally + { + await file.CloseAsync(handle, CancellationToken.None).ConfigureAwait(false); + } + } + + public async ValueTask ReadFileAsync( + FileTypeClient file, + int chunkSize, + CancellationToken cancellationToken) + { + if (file is null) + { + throw new ArgumentNullException(nameof(file)); + } + using MemoryStream stream = new(); + await ReadFileAsync(file, stream, chunkSize, cancellationToken).ConfigureAwait(false); + return ByteString.From(stream.ToArray()); + } + + public async ValueTask ReadFileAsync( + NodeId file, + int chunkSize, + CancellationToken cancellationToken) + { + ValidateChunkSize(chunkSize); + const byte readMode = 1; + ArrayOf opened = await CallAsync( + file, global::Opc.Ua.BrowseNames.Open, [Variant.From(readMode)], cancellationToken) + .ConfigureAwait(false); + uint handle = opened.Count > 0 && opened[0].TryGetValue(out uint value) ? value : 0; + using MemoryStream buffer = new(); + try + { + while (true) + { + ArrayOf outputs = await CallAsync( + file, + global::Opc.Ua.BrowseNames.Read, + [Variant.From(handle), Variant.From(chunkSize)], + cancellationToken).ConfigureAwait(false); + if (outputs.Count == 0 || + !outputs[0].TryGetValue(out ByteString chunk) || + chunk.IsNull || + chunk.Length == 0) + { + break; + } + byte[] copy = chunk.Span.ToArray(); + await buffer.WriteAsync(copy.AsMemory(0, copy.Length), cancellationToken) + .ConfigureAwait(false); + if (copy.Length < chunkSize) + { + break; + } + } + } + finally + { + await CallAsync(file, global::Opc.Ua.BrowseNames.Close, [Variant.From(handle)], CancellationToken.None) + .ConfigureAwait(false); + } + return ByteString.From(buffer.ToArray()); + } + + public async ValueTask ReadFileAsync( + FileTypeClient file, + Stream destination, + int chunkSize, + CancellationToken cancellationToken) + { + if (file is null) + { + throw new ArgumentNullException(nameof(file)); + } + if (destination is null) + { + throw new ArgumentNullException(nameof(destination)); + } + if (!destination.CanWrite) + { + throw new ArgumentException("Stream must be writable.", nameof(destination)); + } + ValidateChunkSize(chunkSize); + const byte readMode = 1; + uint handle = await file.OpenAsync(readMode, cancellationToken).ConfigureAwait(false); + try + { + while (true) + { + ByteString chunk = await file.ReadAsync(handle, chunkSize, cancellationToken) + .ConfigureAwait(false); + if (chunk.IsNull || chunk.Length == 0) + { + break; + } + byte[] copy = chunk.Span.ToArray(); + await destination.WriteAsync(copy.AsMemory(0, copy.Length), cancellationToken) + .ConfigureAwait(false); + if (copy.Length < chunkSize) + { + break; + } + } + } + finally + { + await file.CloseAsync(handle, CancellationToken.None).ConfigureAwait(false); + } + } + + public static string? ReadString(DataValue value) + { + return value.WrappedValue.TryGetValue(out string? text) ? text : null; + } + + public static ByteString ReadByteString(DataValue value) + { + return value.WrappedValue.TryGetValue(out ByteString bytes) ? bytes : ByteString.Empty; + } + + public static bool ReadBoolean(DataValue value) + { + return value.WrappedValue.TryGetValue(out bool result) && result; + } + + public static ulong ReadUInt64(DataValue value) + { + return value.WrappedValue.TryGetValue(out ulong result) ? result : 0; + } + + public static uint ReadUInt32(DataValue value) + { + return value.WrappedValue.TryGetValue(out uint result) ? result : 0; + } + + public static double ReadDouble(DataValue value) + { + return value.WrappedValue.TryGetValue(out double result) ? result : 0; + } + + public static DateTimeUtc ReadDateTime(DataValue value) + { + return value.WrappedValue.TryGetValue(out DateTimeUtc result) ? result : default; + } + + public static bool TryReadEnum(DataValue value, out TEnum result) + where TEnum : struct, Enum + { + if (value.WrappedValue.TryGetValue(out int intValue)) + { + result = (TEnum)Enum.ToObject(typeof(TEnum), intValue); + return true; + } + if (value.WrappedValue.TryGetValue(out uint uintValue)) + { + result = (TEnum)Enum.ToObject(typeof(TEnum), uintValue); + return true; + } + result = default; + return false; + } + + public static bool TryReadNodeId(DataValue value, out NodeId nodeId) + { + if (value.WrappedValue.TryGetValue(out NodeId candidate)) + { + nodeId = candidate; + return !nodeId.IsNull; + } + nodeId = NodeId.Null; + return false; + } + + private static ArrayOf CreateNullNodes(int count) + { + var nulls = new List(count); + for (int ii = 0; ii < count; ii++) + { + nulls.Add(NodeId.Null); + } + return nulls.ToArrayOf(); + } + + private static BrowsePath CreateBrowsePath( + NodeId parent, string browseName, ushort namespaceIndex) + { + return new BrowsePath + { + StartingNode = parent, + RelativePath = new RelativePath + { + Elements = + [ + new RelativePathElement + { + ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HierarchicalReferences, + IsInverse = false, + IncludeSubtypes = true, + TargetName = new QualifiedName(browseName, namespaceIndex) + } + ] + } + }; + } + + private static void ValidateChunkSize(int chunkSize) + { + if (chunkSize <= 0) + { + throw new ArgumentOutOfRangeException(nameof(chunkSize), "Chunk size must be positive."); + } + } + + private static void RegisterEncodeableTypes(ISession session) + { + RegisterEncodeableTypes(session.Factory); + if (!ReferenceEquals(session.MessageContext.Factory, session.Factory)) + { + RegisterEncodeableTypes(session.MessageContext.Factory); + } + } + + private static void RegisterEncodeableTypes(IEncodeableFactory factory) + { + var probe = new CapabilityDataType(); + if (!factory.TryGetEncodeableType(probe.BinaryEncodingId, out _)) + { + factory.Builder.AddOpcUaAI().Commit(); + } + } + } +} diff --git a/src/Opc.Ua.AI.Client/AiDatasetClient.cs b/src/Opc.Ua.AI.Client/AiDatasetClient.cs new file mode 100644 index 0000000000..a7e54a40a7 --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiDatasetClient.cs @@ -0,0 +1,119 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Client +{ + public sealed class AIDatasetClient + { + public AIDatasetClient(AIClient client, NodeId datasetNodeId) + : this(client?.Operations ?? throw new ArgumentNullException(nameof(client)), datasetNodeId) + { + } + + internal AIDatasetClient(AIClientOperations operations, NodeId datasetNodeId) + { + m_operations = operations ?? throw new ArgumentNullException(nameof(operations)); + if (datasetNodeId.IsNull) + { + throw new ArgumentException("Dataset NodeId must not be null.", nameof(datasetNodeId)); + } + DatasetNodeId = datasetNodeId; + } + + public NodeId DatasetNodeId { get; } + + public async ValueTask ReadAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.DatasetId, + BrowseNames.Name, + BrowseNames.SourceKind, + BrowseNames.ArtifactUri, + BrowseNames.ContentType, + BrowseNames.SizeBytes, + BrowseNames.SampleCount + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + DatasetNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + return new AIDatasetSnapshot + { + NodeId = DatasetNodeId, + DatasetId = ReadString(nodes, values, ref cursor, 0), + Name = ReadString(nodes, values, ref cursor, 1), + SourceKind = ReadEnum(nodes, values, ref cursor, 2), + ArtifactUri = ReadString(nodes, values, ref cursor, 3), + ContentType = ReadString(nodes, values, ref cursor, 4), + SizeBytes = ReadUInt64(nodes, values, ref cursor, 5), + SampleCount = ReadUInt32(nodes, values, ref cursor, 6) + }; + } + + private static string? ReadString(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? null : AIClientOperations.ReadString(values[cursor++]); + } + + private static ulong ReadUInt64(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? 0 : AIClientOperations.ReadUInt64(values[cursor++]); + } + + private static uint ReadUInt32(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? 0 : AIClientOperations.ReadUInt32(values[cursor++]); + } + + private static TEnum ReadEnum( + ArrayOf nodes, + ArrayOf values, + ref int cursor, + int index) + where TEnum : struct, Enum + { + if (nodes[index].IsNull) + { + return default; + } + return AIClientOperations.TryReadEnum(values[cursor++], out TEnum result) + ? result + : default; + } + + private readonly AIClientOperations m_operations; + } +} diff --git a/src/Opc.Ua.AI.Client/AiDeploymentClient.cs b/src/Opc.Ua.AI.Client/AiDeploymentClient.cs new file mode 100644 index 0000000000..b84fed14b8 --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiDeploymentClient.cs @@ -0,0 +1,448 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Client +{ + public sealed class AIDeploymentClient + { + public AIDeploymentClient(AIClient client, NodeId deploymentNodeId) + : this(client?.Operations ?? throw new ArgumentNullException(nameof(client)), deploymentNodeId) + { + } + + internal AIDeploymentClient(AIClientOperations operations, NodeId deploymentNodeId) + { + m_operations = operations ?? throw new ArgumentNullException(nameof(operations)); + if (deploymentNodeId.IsNull) + { + throw new ArgumentException("Deployment NodeId must not be null.", nameof(deploymentNodeId)); + } + DeploymentNodeId = deploymentNodeId; + m_proxy = new DeploymentTypeClient( + m_operations.Session, deploymentNodeId, m_operations.Telemetry); + } + + public NodeId DeploymentNodeId { get; } + + public async ValueTask ReadAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.DeploymentId, + BrowseNames.InferenceLocation, + BrowseNames.State, + BrowseNames.DataJurisdiction, + BrowseNames.EgressPermitted, + BrowseNames.MaxInlinePayloadSize, + BrowseNames.EndpointUri + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + DeploymentNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + NodeId model = await m_operations.FollowReferenceAsync( + DeploymentNodeId, ReferenceTypes.UsesModel, cancellationToken).ConfigureAwait(false); + NodeId fallback = await m_operations.FollowReferenceAsync( + DeploymentNodeId, ReferenceTypes.FallsBackTo, cancellationToken).ConfigureAwait(false); + return new AIDeploymentSnapshot + { + NodeId = DeploymentNodeId, + DeploymentId = ReadString(nodes, values, ref cursor, 0), + InferenceLocation = ReadEnum(nodes, values, ref cursor, 1), + State = ReadEnum(nodes, values, ref cursor, 2), + DataJurisdiction = ReadString(nodes, values, ref cursor, 3), + EgressPermitted = ReadBoolean(nodes, values, ref cursor, 4), + MaxInlinePayloadSize = ReadUInt64(nodes, values, ref cursor, 5), + EndpointUri = ReadString(nodes, values, ref cursor, 6), + ModelId = model, + FallbackDeploymentId = fallback + }; + } + + public ValueTask> GetCapabilitiesAsync( + CancellationToken cancellationToken = default) + { + return GetCapabilitiesCoreAsync(cancellationToken); + } + + public async ValueTask InvokeAsync( + ByteString payload, + string contentType, + ArrayOf parameters, + double timeout, + string payloadUri = "", + CancellationToken cancellationToken = default) + { + try + { + return await InvokeProxyAsync( + payload, contentType, parameters, timeout, payloadUri, cancellationToken) + .ConfigureAwait(false); + } + catch (ServiceResultException ex) when (ex.StatusCode == StatusCodes.BadMethodInvalid) + { + ArrayOf outputs = await m_operations.CallAsync( + DeploymentNodeId, + BrowseNames.Invoke, + [ + Variant.From(payload), + Variant.From(payloadUri ?? string.Empty), + Variant.From(contentType ?? string.Empty), + Variant.FromStructure(parameters), + Variant.From(timeout) + ], + cancellationToken).ConfigureAwait(false); + return CreateInvokeResult(outputs); + } + } + + public ValueTask InvokeAsyncAsync( + ByteString payload, + string contentType, + ArrayOf parameters, + string payloadUri = "", + CancellationToken cancellationToken = default) + { + return InvokeAsyncCoreAsync(payload, contentType, parameters, payloadUri, cancellationToken); + } + + public async ValueTask BeginTransferAsync( + string contentType, + ulong requestSize, + CancellationToken cancellationToken = default) + { + try + { + (NodeId transfer, bool accepted) = await m_proxy.BeginTransferAsync( + contentType ?? string.Empty, requestSize, cancellationToken).ConfigureAwait(false); + return new AIBeginTransferResult + { + TransferId = transfer, + Accepted = accepted + }; + } + catch (ServiceResultException ex) when (ex.StatusCode == StatusCodes.BadMethodInvalid) + { + ArrayOf outputs = await m_operations.CallAsync( + DeploymentNodeId, + BrowseNames.BeginTransfer, + [Variant.From(contentType ?? string.Empty), Variant.From(requestSize)], + cancellationToken).ConfigureAwait(false); + return new AIBeginTransferResult + { + TransferId = TryGetNodeId(outputs, 0, out NodeId transfer) ? transfer : NodeId.Null, + Accepted = TryGetBoolean(outputs, 1, out bool accepted) && accepted + }; + } + } + + public async ValueTask OpenModelAsync( + CancellationToken cancellationToken = default) + { + NodeId model = await m_operations.FollowReferenceAsync( + DeploymentNodeId, ReferenceTypes.UsesModel, cancellationToken).ConfigureAwait(false); + return model.IsNull ? null : new AIModelClient(m_operations, model); + } + + public async ValueTask OpenFallbackAsync( + CancellationToken cancellationToken = default) + { + NodeId fallback = await m_operations.FollowReferenceAsync( + DeploymentNodeId, ReferenceTypes.FallsBackTo, cancellationToken).ConfigureAwait(false); + return fallback.IsNull ? null : new AIDeploymentClient(m_operations, fallback); + } + + internal static void ThrowIfBad(ServiceResult serviceResult) + { + if (serviceResult is not null && ServiceResult.IsBad(serviceResult)) + { + throw new ServiceResultException(serviceResult); + } + } + + private async ValueTask> GetCapabilitiesCoreAsync( + CancellationToken cancellationToken) + { + try + { + return await m_proxy.GetCapabilitiesAsync(cancellationToken).ConfigureAwait(false); + } + catch (ServiceResultException ex) when (ex.StatusCode == StatusCodes.BadMethodInvalid) + { + ArrayOf outputs = await m_operations.CallAsync( + DeploymentNodeId, + BrowseNames.GetCapabilities, + ArrayOf.Empty, + cancellationToken).ConfigureAwait(false); + if (outputs.Count > 0 && + outputs[0].TryGetValue( + out ArrayOf capabilities, + m_operations.Session.MessageContext)) + { + return capabilities; + } + return ArrayOf.Empty; + } + } + + private async ValueTask InvokeProxyAsync( + ByteString payload, + string contentType, + ArrayOf parameters, + double timeout, + string payloadUri, + CancellationToken cancellationToken) + { + ( + ByteString responsePayload, + string responseContentType, + NodeId modelUsed, + UsageDataType usage, + FinishReasonEnum finishReason, + ArrayOf safetyAssessment, + double retryAfter, + bool transferRequired, + NodeId transfer) = await m_proxy.InvokeAsync( + payload, + payloadUri ?? string.Empty, + contentType ?? string.Empty, + parameters, + timeout, + cancellationToken).ConfigureAwait(false); + return new AIInvokeResult + { + ResponsePayload = responsePayload, + ResponseContentType = responseContentType, + ModelUsed = modelUsed, + Usage = usage, + FinishReason = finishReason, + SafetyAssessment = safetyAssessment, + RetryAfter = retryAfter, + TransferRequired = transferRequired, + TransferId = transfer + }; + } + + private async ValueTask InvokeAsyncCoreAsync( + ByteString payload, + string contentType, + ArrayOf parameters, + string payloadUri, + CancellationToken cancellationToken) + { + try + { + return await m_proxy.InvokeAsyncAsync( + payload, + payloadUri ?? string.Empty, + contentType ?? string.Empty, + parameters, + cancellationToken).ConfigureAwait(false); + } + catch (ServiceResultException ex) when (ex.StatusCode == StatusCodes.BadMethodInvalid) + { + ArrayOf outputs = await m_operations.CallAsync( + DeploymentNodeId, + BrowseNames.InvokeAsync, + [ + Variant.From(payload), + Variant.From(payloadUri ?? string.Empty), + Variant.From(contentType ?? string.Empty), + Variant.FromStructure(parameters) + ], + cancellationToken).ConfigureAwait(false); + return TryGetNodeId(outputs, 0, out NodeId job) ? job : NodeId.Null; + } + } + + private AIInvokeResult CreateInvokeResult(ArrayOf outputs) + { + return new AIInvokeResult + { + ResponsePayload = TryGetByteString(outputs, 0, out ByteString responsePayload) + ? responsePayload + : ByteString.Empty, + ResponseContentType = TryGetString(outputs, 1, out string? responseContentType) + ? responseContentType + : null, + ModelUsed = TryGetNodeId(outputs, 2, out NodeId modelUsed) ? modelUsed : NodeId.Null, + Usage = TryGetStructure(outputs, 3, out UsageDataType usage) ? usage : null, + FinishReason = TryGetEnum(outputs, 4, out FinishReasonEnum finishReason) + ? finishReason + : default, + SafetyAssessment = TryGetStructureArray(outputs, 5, out ArrayOf safety) + ? safety + : ArrayOf.Empty, + RetryAfter = TryGetDouble(outputs, 6, out double retryAfter) ? retryAfter : 0, + TransferRequired = TryGetBoolean(outputs, 7, out bool transferRequired) && transferRequired, + TransferId = TryGetNodeId(outputs, 8, out NodeId transfer) ? transfer : NodeId.Null + }; + } + + private bool TryGetStructure(ArrayOf outputs, int index, out T value) + where T : class, IEncodeable + { +#pragma warning disable CS8600 // TryGetValue uses [MaybeNullWhen(false)] on encodeable overloads. + if (index < outputs.Count && + outputs[index].TryGetValue(out T result, m_operations.Session.MessageContext)) +#pragma warning restore CS8600 + { + value = result; + return true; + } + value = null!; + return false; + } + + private bool TryGetStructureArray(ArrayOf outputs, int index, out ArrayOf value) + where T : class, IEncodeable + { + if (index < outputs.Count && + outputs[index].TryGetValue(out ArrayOf result, m_operations.Session.MessageContext)) + { + value = result; + return true; + } + value = ArrayOf.Empty; + return false; + } + + private static bool TryGetString(ArrayOf outputs, int index, out string? value) + { + if (index < outputs.Count && outputs[index].TryGetValue(out string? result)) + { + value = result; + return true; + } + value = null; + return false; + } + + private static bool TryGetByteString(ArrayOf outputs, int index, out ByteString value) + { + if (index < outputs.Count && outputs[index].TryGetValue(out ByteString result)) + { + value = result; + return true; + } + value = ByteString.Empty; + return false; + } + + private static bool TryGetNodeId(ArrayOf outputs, int index, out NodeId value) + { + if (index < outputs.Count && outputs[index].TryGetValue(out NodeId result)) + { + value = result; + return true; + } + value = NodeId.Null; + return false; + } + + private static bool TryGetBoolean(ArrayOf outputs, int index, out bool value) + { + if (index < outputs.Count && outputs[index].TryGetValue(out bool result)) + { + value = result; + return true; + } + value = false; + return false; + } + + private static bool TryGetDouble(ArrayOf outputs, int index, out double value) + { + if (index < outputs.Count && outputs[index].TryGetValue(out double result)) + { + value = result; + return true; + } + value = 0; + return false; + } + + private static bool TryGetEnum(ArrayOf outputs, int index, out TEnum value) + where TEnum : struct, Enum + { + if (index < outputs.Count && outputs[index].TryGetValue(out int intValue)) + { + value = (TEnum)Enum.ToObject(typeof(TEnum), intValue); + return true; + } + if (index < outputs.Count && outputs[index].TryGetValue(out uint uintValue)) + { + value = (TEnum)Enum.ToObject(typeof(TEnum), uintValue); + return true; + } + value = default; + return false; + } + + private static string? ReadString(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? null : AIClientOperations.ReadString(values[cursor++]); + } + + private static bool ReadBoolean(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return !nodes[index].IsNull && AIClientOperations.ReadBoolean(values[cursor++]); + } + + private static ulong ReadUInt64(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? 0 : AIClientOperations.ReadUInt64(values[cursor++]); + } + + private static TEnum ReadEnum( + ArrayOf nodes, + ArrayOf values, + ref int cursor, + int index) + where TEnum : struct, Enum + { + if (nodes[index].IsNull) + { + return default; + } + return AIClientOperations.TryReadEnum(values[cursor++], out TEnum result) + ? result + : default; + } + + private readonly AIClientOperations m_operations; + private readonly DeploymentTypeClient m_proxy; + } +} diff --git a/src/Opc.Ua.AI.Client/AiEvaluationRunClient.cs b/src/Opc.Ua.AI.Client/AiEvaluationRunClient.cs new file mode 100644 index 0000000000..1447add96e --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiEvaluationRunClient.cs @@ -0,0 +1,123 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Client +{ + public sealed class AIEvaluationRunClient + { + public AIEvaluationRunClient(AIClient client, NodeId runNodeId) + : this(client?.Operations ?? throw new ArgumentNullException(nameof(client)), runNodeId) + { + } + + internal AIEvaluationRunClient(AIClientOperations operations, NodeId runNodeId) + { + m_operations = operations ?? throw new ArgumentNullException(nameof(operations)); + if (runNodeId.IsNull) + { + throw new ArgumentException("Evaluation run NodeId must not be null.", nameof(runNodeId)); + } + RunNodeId = runNodeId; + } + + public NodeId RunNodeId { get; } + + public async ValueTask ReadAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.RunId, + BrowseNames.EvaluatedModel, + BrowseNames.Passed, + BrowseNames.Metrics, + BrowseNames.ReportUri + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + RunNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + return new AIEvaluationRunSnapshot + { + NodeId = RunNodeId, + RunId = ReadString(nodes, values, ref cursor, 0), + EvaluatedModelId = ReadNodeId(nodes, values, ref cursor, 1), + Passed = ReadBoolean(nodes, values, ref cursor, 2), + Metrics = ReadStructureArray( + nodes, values, m_operations.Session.MessageContext, ref cursor, 3), + ReportUri = ReadString(nodes, values, ref cursor, 4) + }; + } + + private static string? ReadString(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? null : AIClientOperations.ReadString(values[cursor++]); + } + + private static bool ReadBoolean(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return !nodes[index].IsNull && AIClientOperations.ReadBoolean(values[cursor++]); + } + + private static NodeId ReadNodeId(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + if (nodes[index].IsNull) + { + return NodeId.Null; + } + return AIClientOperations.TryReadNodeId(values[cursor++], out NodeId nodeId) + ? nodeId + : NodeId.Null; + } + + private static ArrayOf ReadStructureArray( + ArrayOf nodes, + ArrayOf values, + IServiceMessageContext messageContext, + ref int cursor, + int index) + where T : class, IEncodeable + { + if (nodes[index].IsNull) + { + return ArrayOf.Empty; + } + return values[cursor++].WrappedValue.TryGetValue(out ArrayOf array, messageContext) + ? array + : ArrayOf.Empty; + } + + private readonly AIClientOperations m_operations; + } +} diff --git a/src/Opc.Ua.AI.Client/AiInferenceJobClient.cs b/src/Opc.Ua.AI.Client/AiInferenceJobClient.cs new file mode 100644 index 0000000000..0e885d9d81 --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiInferenceJobClient.cs @@ -0,0 +1,124 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Client +{ + public sealed class AIInferenceJobClient + { + public AIInferenceJobClient(AIClient client, NodeId jobNodeId) + : this(client?.Operations ?? throw new ArgumentNullException(nameof(client)), jobNodeId) + { + } + + internal AIInferenceJobClient(AIClientOperations operations, NodeId jobNodeId) + { + m_operations = operations ?? throw new ArgumentNullException(nameof(operations)); + if (jobNodeId.IsNull) + { + throw new ArgumentException("Job NodeId must not be null.", nameof(jobNodeId)); + } + JobNodeId = jobNodeId; + } + + public NodeId JobNodeId { get; } + + public async ValueTask ReadAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.JobId, + BrowseNames.Deployment, + BrowseNames.ResponsePayload, + BrowseNames.ResponseContentType, + BrowseNames.ModelUsed, + BrowseNames.FinishReason + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + JobNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + return new AIInferenceJobSnapshot + { + NodeId = JobNodeId, + JobId = ReadString(nodes, values, ref cursor, 0), + DeploymentId = ReadNodeId(nodes, values, ref cursor, 1), + ResponsePayload = ReadByteString(nodes, values, ref cursor, 2), + ResponseContentType = ReadString(nodes, values, ref cursor, 3), + ModelUsed = ReadNodeId(nodes, values, ref cursor, 4), + FinishReason = ReadEnum(nodes, values, ref cursor, 5) + }; + } + + private static string? ReadString(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? null : AIClientOperations.ReadString(values[cursor++]); + } + + private static ByteString ReadByteString( + ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? ByteString.Empty : AIClientOperations.ReadByteString(values[cursor++]); + } + + private static NodeId ReadNodeId(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + if (nodes[index].IsNull) + { + return NodeId.Null; + } + return AIClientOperations.TryReadNodeId(values[cursor++], out NodeId nodeId) + ? nodeId + : NodeId.Null; + } + + private static TEnum ReadEnum( + ArrayOf nodes, + ArrayOf values, + ref int cursor, + int index) + where TEnum : struct, Enum + { + if (nodes[index].IsNull) + { + return default; + } + return AIClientOperations.TryReadEnum(values[cursor++], out TEnum result) + ? result + : default; + } + + private readonly AIClientOperations m_operations; + } +} diff --git a/src/Opc.Ua.AI.Client/AiInferenceTransferClient.cs b/src/Opc.Ua.AI.Client/AiInferenceTransferClient.cs new file mode 100644 index 0000000000..da091c416c --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiInferenceTransferClient.cs @@ -0,0 +1,238 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Client +{ + public sealed class AIInferenceTransferClient + { + public AIInferenceTransferClient(AIClient client, NodeId transferNodeId) + : this(client?.Operations ?? throw new ArgumentNullException(nameof(client)), transferNodeId) + { + } + + internal AIInferenceTransferClient(AIClientOperations operations, NodeId transferNodeId) + { + m_operations = operations ?? throw new ArgumentNullException(nameof(operations)); + if (transferNodeId.IsNull) + { + throw new ArgumentException("Transfer NodeId must not be null.", nameof(transferNodeId)); + } + TransferNodeId = transferNodeId; + m_proxy = new InferenceTransferTypeClient( + m_operations.Session, transferNodeId, m_operations.Telemetry); + } + + public NodeId TransferNodeId { get; } + + public async ValueTask ReadAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.TransferId, + BrowseNames.State, + BrowseNames.BytesTransferred, + BrowseNames.ModelUsed, + BrowseNames.ResponseContentType + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + TransferNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + return new AITransferSnapshot + { + NodeId = TransferNodeId, + TransferId = ReadString(nodes, values, ref cursor, 0), + State = ReadEnum(nodes, values, ref cursor, 1), + BytesTransferred = ReadUInt64(nodes, values, ref cursor, 2), + ModelUsed = ReadNodeId(nodes, values, ref cursor, 3), + ResponseContentType = ReadString(nodes, values, ref cursor, 4) + }; + } + + public async ValueTask WriteRequestAsync( + ByteString content, + int chunkSize = AIClientOperations.DefaultChunkSize, + CancellationToken cancellationToken = default) + { + FileTypeClient file = await OpenRequestFileAsync(cancellationToken).ConfigureAwait(false); + try + { + await m_operations.WriteFileAsync(file, content, chunkSize, cancellationToken) + .ConfigureAwait(false); + } + catch (ServiceResultException ex) when (ex.StatusCode == StatusCodes.BadNodeIdUnknown || + ex.StatusCode == StatusCodes.BadMethodInvalid) + { + await m_operations.WriteFileAsync(file.ObjectId, content, chunkSize, cancellationToken) + .ConfigureAwait(false); + } + } + + public async ValueTask WriteRequestAsync( + Stream content, + int chunkSize = AIClientOperations.DefaultChunkSize, + CancellationToken cancellationToken = default) + { + FileTypeClient file = await OpenRequestFileAsync(cancellationToken).ConfigureAwait(false); + await m_operations.WriteFileAsync(file, content, chunkSize, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask ReadResponseAsync( + int chunkSize = AIClientOperations.DefaultChunkSize, + CancellationToken cancellationToken = default) + { + FileTypeClient file = await OpenResponseFileAsync(cancellationToken).ConfigureAwait(false); + try + { + return await m_operations.ReadFileAsync(file, chunkSize, cancellationToken) + .ConfigureAwait(false); + } + catch (ServiceResultException ex) when (ex.StatusCode == StatusCodes.BadNodeIdUnknown || + ex.StatusCode == StatusCodes.BadMethodInvalid) + { + return await m_operations.ReadFileAsync(file.ObjectId, chunkSize, cancellationToken) + .ConfigureAwait(false); + } + } + + public async ValueTask ReadResponseAsync( + Stream destination, + int chunkSize = AIClientOperations.DefaultChunkSize, + CancellationToken cancellationToken = default) + { + FileTypeClient file = await OpenResponseFileAsync(cancellationToken).ConfigureAwait(false); + await m_operations.ReadFileAsync(file, destination, chunkSize, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask ExecuteAsync(CancellationToken cancellationToken = default) + { + try + { + return await m_proxy.ExecuteAsync(cancellationToken).ConfigureAwait(false); + } + catch (ServiceResultException ex) when (ex.StatusCode == StatusCodes.BadMethodInvalid) + { + ArrayOf outputs = await m_operations.CallAsync( + TransferNodeId, + BrowseNames.Execute, + ArrayOf.Empty, + cancellationToken).ConfigureAwait(false); + return outputs.Count > 0 && outputs[0].TryGetValue(out bool accepted) && accepted; + } + } + + public ValueTask AbortAsync(CancellationToken cancellationToken = default) + { + return m_proxy.AbortAsync(cancellationToken); + } + + private async ValueTask OpenRequestFileAsync(CancellationToken cancellationToken) + { + NodeId request = await m_operations.ResolveChildAsync( + TransferNodeId, BrowseNames.Request, cancellationToken).ConfigureAwait(false); + if (!request.IsNull) + { + return new FileTypeClient(m_operations.Session, request, m_operations.Telemetry); + } + FileTypeClient? file = await m_proxy.GetRequestAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + if (file is null || file.ObjectId.IsNull) + { + throw new ServiceResultException(StatusCodes.BadNodeIdUnknown); + } + return file; + } + + private async ValueTask OpenResponseFileAsync(CancellationToken cancellationToken) + { + NodeId response = await m_operations.ResolveChildAsync( + TransferNodeId, BrowseNames.Response, cancellationToken).ConfigureAwait(false); + if (!response.IsNull) + { + return new FileTypeClient(m_operations.Session, response, m_operations.Telemetry); + } + FileTypeClient? file = await m_proxy.GetResponseAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + if (file is null || file.ObjectId.IsNull) + { + throw new ServiceResultException(StatusCodes.BadNodeIdUnknown); + } + return file; + } + + private static string? ReadString(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? null : AIClientOperations.ReadString(values[cursor++]); + } + + private static ulong ReadUInt64(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? 0 : AIClientOperations.ReadUInt64(values[cursor++]); + } + + private static NodeId ReadNodeId(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + if (nodes[index].IsNull) + { + return NodeId.Null; + } + return AIClientOperations.TryReadNodeId(values[cursor++], out NodeId nodeId) + ? nodeId + : NodeId.Null; + } + + private static TEnum ReadEnum( + ArrayOf nodes, + ArrayOf values, + ref int cursor, + int index) + where TEnum : struct, Enum + { + if (nodes[index].IsNull) + { + return default; + } + return AIClientOperations.TryReadEnum(values[cursor++], out TEnum result) + ? result + : default; + } + + private readonly AIClientOperations m_operations; + private readonly InferenceTransferTypeClient m_proxy; + } +} diff --git a/src/Opc.Ua.AI.Client/AiLearningJobClient.cs b/src/Opc.Ua.AI.Client/AiLearningJobClient.cs new file mode 100644 index 0000000000..921bf72d70 --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiLearningJobClient.cs @@ -0,0 +1,145 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Client +{ + public sealed class AILearningJobClient + { + public AILearningJobClient(AIClient client, NodeId jobNodeId) + : this(client?.Operations ?? throw new ArgumentNullException(nameof(client)), jobNodeId) + { + } + + internal AILearningJobClient(AIClientOperations operations, NodeId jobNodeId) + { + m_operations = operations ?? throw new ArgumentNullException(nameof(operations)); + if (jobNodeId.IsNull) + { + throw new ArgumentException("Learning job NodeId must not be null.", nameof(jobNodeId)); + } + JobNodeId = jobNodeId; + m_proxy = new LearningJobTypeClient(m_operations.Session, jobNodeId, m_operations.Telemetry); + } + + public NodeId JobNodeId { get; } + + public async ValueTask ReadAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.JobId, + BrowseNames.State, + BrowseNames.Progress, + BrowseNames.CandidateModel, + BrowseNames.TargetDeployment + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + JobNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + return new AILearningJobSnapshot + { + NodeId = JobNodeId, + JobId = ReadString(nodes, values, ref cursor, 0), + State = ReadEnum(nodes, values, ref cursor, 1), + Progress = ReadDouble(nodes, values, ref cursor, 2), + CandidateModelId = ReadNodeId(nodes, values, ref cursor, 3), + TargetDeploymentId = ReadNodeId(nodes, values, ref cursor, 4) + }; + } + + public ValueTask StartCollectionAsync(CancellationToken cancellationToken = default) + { + return m_proxy.StartCollectionAsync(cancellationToken); + } + + public ValueTask StopCollectionAsync(CancellationToken cancellationToken = default) + { + return m_proxy.StopCollectionAsync(cancellationToken); + } + + public ValueTask TriggerTrainingAsync(CancellationToken cancellationToken = default) + { + return m_proxy.TriggerTrainingAsync(cancellationToken); + } + + public ValueTask PromoteModelAsync( + NodeId deployment, + CancellationToken cancellationToken = default) + { + return m_proxy.PromoteModelAsync(deployment, cancellationToken); + } + + private static string? ReadString(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? null : AIClientOperations.ReadString(values[cursor++]); + } + + private static double ReadDouble(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? 0 : AIClientOperations.ReadDouble(values[cursor++]); + } + + private static NodeId ReadNodeId(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + if (nodes[index].IsNull) + { + return NodeId.Null; + } + return AIClientOperations.TryReadNodeId(values[cursor++], out NodeId nodeId) + ? nodeId + : NodeId.Null; + } + + private static TEnum ReadEnum( + ArrayOf nodes, + ArrayOf values, + ref int cursor, + int index) + where TEnum : struct, Enum + { + if (nodes[index].IsNull) + { + return default; + } + return AIClientOperations.TryReadEnum(values[cursor++], out TEnum result) + ? result + : default; + } + + private readonly AIClientOperations m_operations; + private readonly LearningJobTypeClient m_proxy; + } +} diff --git a/src/Opc.Ua.AI.Client/AiMethodResults.cs b/src/Opc.Ua.AI.Client/AiMethodResults.cs new file mode 100644 index 0000000000..ee7ffda65b --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiMethodResults.cs @@ -0,0 +1,73 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.AI.Client +{ + public sealed record AIInvokeResult + { + public ByteString ResponsePayload { get; init; } = ByteString.Empty; + + public string? ResponseContentType { get; init; } + + public NodeId ModelUsed { get; init; } = NodeId.Null; + + public UsageDataType? Usage { get; init; } + + public FinishReasonEnum FinishReason { get; init; } + + public ArrayOf SafetyAssessment { get; init; } = []; + + public double RetryAfter { get; init; } + + public bool TransferRequired { get; init; } + + public NodeId TransferId { get; init; } = NodeId.Null; + } + + public sealed record AIBeginTransferResult + { + public NodeId TransferId { get; init; } = NodeId.Null; + + public bool Accepted { get; init; } + } + + public sealed record AISourceModelListResult + { + public ArrayOf Models { get; init; } = []; + + public ByteString ContinuationPoint { get; init; } = ByteString.Empty; + } + + public sealed record AISourceConnectionResult + { + public bool Reachable { get; init; } + + public LocalizedText Detail { get; init; } = LocalizedText.Null; + } +} diff --git a/src/Opc.Ua.AI.Client/AiModelClient.cs b/src/Opc.Ua.AI.Client/AiModelClient.cs new file mode 100644 index 0000000000..8911e6e690 --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiModelClient.cs @@ -0,0 +1,292 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Client +{ + public sealed class AIModelClient + { + public AIModelClient(AIClient client, NodeId modelNodeId) + : this(client?.Operations ?? throw new ArgumentNullException(nameof(client)), modelNodeId) + { + } + + internal AIModelClient(AIClientOperations operations, NodeId modelNodeId) + { + m_operations = operations ?? throw new ArgumentNullException(nameof(operations)); + if (modelNodeId.IsNull) + { + throw new ArgumentException("Model NodeId must not be null.", nameof(modelNodeId)); + } + ModelNodeId = modelNodeId; + m_proxy = new ModelTypeClient(m_operations.Session, modelNodeId, m_operations.Telemetry); + } + + public NodeId ModelNodeId { get; } + + public async ValueTask ReadAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.ModelId, + BrowseNames.Name, + BrowseNames.Version, + BrowseNames.Framework, + BrowseNames.Format, + BrowseNames.License, + BrowseNames.Digest, + BrowseNames.DigestAlgorithm, + BrowseNames.CreatedAt, + BrowseNames.LastModifiedAt, + BrowseNames.Publisher + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + ModelNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await ReadPresentValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + string? modelId = ReadString(nodes, values, ref cursor, 0); + string? name = ReadString(nodes, values, ref cursor, 1); + string? version = ReadString(nodes, values, ref cursor, 2); + string? framework = ReadString(nodes, values, ref cursor, 3); + string? format = ReadString(nodes, values, ref cursor, 4); + string? license = ReadString(nodes, values, ref cursor, 5); + ByteString digest = ReadByteString(nodes, values, ref cursor, 6); + string? digestAlgorithm = ReadString(nodes, values, ref cursor, 7); + DateTimeUtc createdAt = ReadDateTime(nodes, values, ref cursor, 8); + DateTimeUtc lastModifiedAt = ReadDateTime(nodes, values, ref cursor, 9); + NodeId publisherId = ReadNodeId(nodes, values, ref cursor, 10); + ModelCardTypeClient? card = await m_proxy.GetCardAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + NodeId source = await m_operations.FollowReferenceAsync( + ModelNodeId, ReferenceTypes.ImportedFrom, cancellationToken).ConfigureAwait(false); + return new AIModelSnapshot + { + NodeId = ModelNodeId, + ModelId = modelId, + Name = name, + Version = version, + Framework = framework, + Format = format, + License = license, + Digest = digest, + DigestAlgorithm = digestAlgorithm, + CreatedAt = createdAt, + LastModifiedAt = lastModifiedAt, + CardId = card?.ObjectId ?? NodeId.Null, + PublisherId = publisherId, + SourceId = source + }; + } + + public async ValueTask ReadCardAsync( + CancellationToken cancellationToken = default) + { + ModelCardTypeClient? card = await m_proxy.GetCardAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + if (card is null || card.ObjectId.IsNull) + { + return new AIModelCardSnapshot(); + } + return await ReadCardAsync(card.ObjectId, cancellationToken).ConfigureAwait(false); + } + + public async IAsyncEnumerable EnumerateResourcesAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + ArrayOf references = await m_operations + .BrowseHierarchicalObjectsAsync(ModelNodeId, cancellationToken).ConfigureAwait(false); + NodeId resourceType = m_operations.AINamespaceType(ObjectTypes.ModelResourceType); + for (int ii = 0; ii < references.Count; ii++) + { + ReferenceDescription reference = references[ii]; + NodeId typeDef = ExpandedNodeId.ToNodeId( + reference.TypeDefinition, m_operations.Session.NamespaceUris); + NodeId nodeId = ExpandedNodeId.ToNodeId( + reference.NodeId, m_operations.Session.NamespaceUris); + if (typeDef.IsNull || nodeId.IsNull) + { + continue; + } + if (typeDef == resourceType || + await m_operations.Session.NodeCache.IsTypeOfAsync( + typeDef, resourceType, cancellationToken).ConfigureAwait(false)) + { + yield return new AINodeEntry( + nodeId, reference.BrowseName, reference.DisplayName, typeDef); + } + } + } + + public async ValueTask ReadResourceAsync( + NodeId resourceNodeId, + CancellationToken cancellationToken = default) + { + ValidateNodeId(resourceNodeId, nameof(resourceNodeId)); + string[] members = + [ + BrowseNames.ArtifactUri, + BrowseNames.ContentType, + BrowseNames.SizeBytes, + BrowseNames.Digest, + BrowseNames.DigestAlgorithm + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + resourceNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await ReadPresentValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + return new AIModelResourceSnapshot + { + NodeId = resourceNodeId, + ArtifactUri = ReadString(nodes, values, ref cursor, 0), + ContentType = ReadString(nodes, values, ref cursor, 1), + SizeBytes = ReadUInt64(nodes, values, ref cursor, 2), + Digest = ReadByteString(nodes, values, ref cursor, 3), + DigestAlgorithm = ReadString(nodes, values, ref cursor, 4) + }; + } + + public async ValueTask OpenSourceAsync( + CancellationToken cancellationToken = default) + { + NodeId source = await m_operations.FollowReferenceAsync( + ModelNodeId, ReferenceTypes.ImportedFrom, cancellationToken).ConfigureAwait(false); + return source.IsNull ? null : new AIModelSourceClient(m_operations, source); + } + + private async ValueTask ReadCardAsync( + NodeId cardNodeId, + CancellationToken cancellationToken) + { + string[] members = + [ + BrowseNames.IntendedUse, + BrowseNames.OutOfScopeUse, + BrowseNames.Limitations, + BrowseNames.EthicalConsiderations, + BrowseNames.TrainingDataCutoff, + BrowseNames.DataJurisdiction, + BrowseNames.SafetyAssessment + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + cardNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await ReadPresentValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + return new AIModelCardSnapshot + { + NodeId = cardNodeId, + IntendedUse = ReadString(nodes, values, ref cursor, 0), + OutOfScopeUse = ReadString(nodes, values, ref cursor, 1), + Limitations = ReadString(nodes, values, ref cursor, 2), + EthicalConsiderations = ReadString(nodes, values, ref cursor, 3), + TrainingDataCutoff = ReadString(nodes, values, ref cursor, 4), + DataJurisdiction = ReadString(nodes, values, ref cursor, 5), + SafetyAssessment = ReadStructureArray( + nodes, values, m_operations.Session.MessageContext, ref cursor, 6) + }; + } + + private ValueTask> ReadPresentValuesAsync( + ArrayOf nodes, + CancellationToken cancellationToken) + { + return m_operations.ReadValuesAsync(nodes, cancellationToken); + } + + private static string? ReadString(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? null : AIClientOperations.ReadString(values[cursor++]); + } + + private static ByteString ReadByteString( + ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? ByteString.Empty : AIClientOperations.ReadByteString(values[cursor++]); + } + + private static DateTimeUtc ReadDateTime( + ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? default : AIClientOperations.ReadDateTime(values[cursor++]); + } + + private static NodeId ReadNodeId(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + if (nodes[index].IsNull) + { + return NodeId.Null; + } + return AIClientOperations.TryReadNodeId(values[cursor++], out NodeId nodeId) + ? nodeId + : NodeId.Null; + } + + private static ulong ReadUInt64(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? 0 : AIClientOperations.ReadUInt64(values[cursor++]); + } + + private static ArrayOf ReadStructureArray( + ArrayOf nodes, + ArrayOf values, + IServiceMessageContext messageContext, + ref int cursor, + int index) + where T : class, IEncodeable + { + if (nodes[index].IsNull) + { + return ArrayOf.Empty; + } + return values[cursor++].WrappedValue.TryGetValue( + out ArrayOf array, messageContext) + ? array + : ArrayOf.Empty; + } + + private static void ValidateNodeId(NodeId nodeId, string paramName) + { + if (nodeId.IsNull) + { + throw new ArgumentException("NodeId must not be null.", paramName); + } + } + + private readonly AIClientOperations m_operations; + private readonly ModelTypeClient m_proxy; + } +} diff --git a/src/Opc.Ua.AI.Client/AiModelSourceClient.cs b/src/Opc.Ua.AI.Client/AiModelSourceClient.cs new file mode 100644 index 0000000000..4b3aeab702 --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiModelSourceClient.cs @@ -0,0 +1,138 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Client +{ + public sealed class AIModelSourceClient + { + public AIModelSourceClient(AIClient client, NodeId sourceNodeId) + : this(client?.Operations ?? throw new ArgumentNullException(nameof(client)), sourceNodeId) + { + } + + internal AIModelSourceClient(AIClientOperations operations, NodeId sourceNodeId) + { + m_operations = operations ?? throw new ArgumentNullException(nameof(operations)); + if (sourceNodeId.IsNull) + { + throw new ArgumentException("Source NodeId must not be null.", nameof(sourceNodeId)); + } + SourceNodeId = sourceNodeId; + m_proxy = new ModelSourceTypeClient( + m_operations.Session, sourceNodeId, m_operations.Telemetry); + } + + public NodeId SourceNodeId { get; } + + public async ValueTask ReadAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.SourceId, + BrowseNames.EndpointUri, + BrowseNames.ApiDialect, + BrowseNames.AuthenticationKind, + BrowseNames.CredentialReference + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + SourceNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync(nodes, cancellationToken) + .ConfigureAwait(false); + int cursor = 0; + return new AIModelSourceSnapshot + { + NodeId = SourceNodeId, + SourceId = ReadString(nodes, values, ref cursor, 0), + EndpointUri = ReadString(nodes, values, ref cursor, 1), + ApiDialect = ReadEnum(nodes, values, ref cursor, 2), + AuthenticationKind = ReadEnum(nodes, values, ref cursor, 3), + CredentialReference = ReadString(nodes, values, ref cursor, 4) + }; + } + + public async ValueTask TestConnectionAsync( + CancellationToken cancellationToken = default) + { + (bool reachable, LocalizedText detail) = await m_proxy.TestConnectionAsync(cancellationToken) + .ConfigureAwait(false); + return new AISourceConnectionResult + { + Reachable = reachable, + Detail = detail + }; + } + + public async ValueTask ListModelsAsync( + string filter = "", + uint maxResults = 100, + ByteString continuationPoint = default, + CancellationToken cancellationToken = default) + { + (ArrayOf models, ByteString continuationPointOut) = await m_proxy.ListModelsAsync( + filter ?? string.Empty, + maxResults, + continuationPoint, + cancellationToken).ConfigureAwait(false); + return new AISourceModelListResult + { + Models = models, + ContinuationPoint = continuationPointOut + }; + } + + private static string? ReadString(ArrayOf nodes, ArrayOf values, ref int cursor, int index) + { + return nodes[index].IsNull ? null : AIClientOperations.ReadString(values[cursor++]); + } + + private static TEnum ReadEnum( + ArrayOf nodes, + ArrayOf values, + ref int cursor, + int index) + where TEnum : struct, Enum + { + if (nodes[index].IsNull) + { + return default; + } + return AIClientOperations.TryReadEnum(values[cursor++], out TEnum result) + ? result + : default; + } + + private readonly AIClientOperations m_operations; + private readonly ModelSourceTypeClient m_proxy; + } +} diff --git a/src/Opc.Ua.AI.Client/AiSnapshots.cs b/src/Opc.Ua.AI.Client/AiSnapshots.cs new file mode 100644 index 0000000000..ca1c1d6aeb --- /dev/null +++ b/src/Opc.Ua.AI.Client/AiSnapshots.cs @@ -0,0 +1,232 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.AI.Client +{ + public sealed record AINodeEntry( + NodeId NodeId, + QualifiedName BrowseName, + LocalizedText DisplayName, + NodeId TypeDefinition); + + public sealed record AIModelSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? ModelId { get; init; } + + public string? Name { get; init; } + + public string? Version { get; init; } + + public string? Framework { get; init; } + + public string? Format { get; init; } + + public string? License { get; init; } + + public ByteString Digest { get; init; } = ByteString.Empty; + + public string? DigestAlgorithm { get; init; } + + public DateTimeUtc CreatedAt { get; init; } + + public DateTimeUtc LastModifiedAt { get; init; } + + public NodeId CardId { get; init; } = NodeId.Null; + + public NodeId PublisherId { get; init; } = NodeId.Null; + + public NodeId SourceId { get; init; } = NodeId.Null; + } + + public sealed record AIModelCardSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? IntendedUse { get; init; } + + public string? OutOfScopeUse { get; init; } + + public string? Limitations { get; init; } + + public string? EthicalConsiderations { get; init; } + + public string? TrainingDataCutoff { get; init; } + + public string? DataJurisdiction { get; init; } + + public ArrayOf SafetyAssessment { get; init; } = []; + } + + public sealed record AIModelResourceSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? ArtifactUri { get; init; } + + public string? ContentType { get; init; } + + public ulong SizeBytes { get; init; } + + public ByteString Digest { get; init; } = ByteString.Empty; + + public string? DigestAlgorithm { get; init; } + } + + public sealed record AIModelSourceSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? SourceId { get; init; } + + public string? EndpointUri { get; init; } + + public ApiDialectEnum ApiDialect { get; init; } + + public AuthenticationKindEnum AuthenticationKind { get; init; } + + public string? CredentialReference { get; init; } + } + + public sealed record AIModelPublisherSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? Name { get; init; } + + public string? ContactUri { get; init; } + + public string? License { get; init; } + } + + public sealed record AIDatasetSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? DatasetId { get; init; } + + public string? Name { get; init; } + + public DatasetSourceEnum SourceKind { get; init; } + + public string? ArtifactUri { get; init; } + + public string? ContentType { get; init; } + + public ulong SizeBytes { get; init; } + + public uint SampleCount { get; init; } + } + + public sealed record AIDeploymentSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? DeploymentId { get; init; } + + public InferenceLocationEnum InferenceLocation { get; init; } + + public DeploymentStateEnum State { get; init; } + + public string? DataJurisdiction { get; init; } + + public bool EgressPermitted { get; init; } + + public ulong MaxInlinePayloadSize { get; init; } + + public string? EndpointUri { get; init; } + + public NodeId ModelId { get; init; } = NodeId.Null; + + public NodeId FallbackDeploymentId { get; init; } = NodeId.Null; + } + + public sealed record AIInferenceJobSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? JobId { get; init; } + + public NodeId DeploymentId { get; init; } = NodeId.Null; + + public ByteString ResponsePayload { get; init; } = ByteString.Empty; + + public string? ResponseContentType { get; init; } + + public NodeId ModelUsed { get; init; } = NodeId.Null; + + public FinishReasonEnum FinishReason { get; init; } + } + + public sealed record AILearningJobSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? JobId { get; init; } + + public LearningJobStateEnum State { get; init; } + + public double Progress { get; init; } + + public NodeId CandidateModelId { get; init; } = NodeId.Null; + + public NodeId TargetDeploymentId { get; init; } = NodeId.Null; + } + + public sealed record AIEvaluationRunSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? RunId { get; init; } + + public NodeId EvaluatedModelId { get; init; } = NodeId.Null; + + public bool Passed { get; init; } + + public ArrayOf Metrics { get; init; } = []; + + public string? ReportUri { get; init; } + } + + public sealed record AITransferSnapshot + { + public NodeId NodeId { get; init; } = NodeId.Null; + + public string? TransferId { get; init; } + + public TransferStateEnum State { get; init; } + + public ulong BytesTransferred { get; init; } + + public NodeId ModelUsed { get; init; } = NodeId.Null; + + public string? ResponseContentType { get; init; } + } +} diff --git a/src/Opc.Ua.AI.Client/AssemblyInfo.cs b/src/Opc.Ua.AI.Client/AssemblyInfo.cs new file mode 100644 index 0000000000..8daaa9d22c --- /dev/null +++ b/src/Opc.Ua.AI.Client/AssemblyInfo.cs @@ -0,0 +1,38 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Runtime.CompilerServices; + +// The OPC UA stack surface this builds on is not CLS compliant (unsigned +// integers appear throughout the specification's own data types), so claiming +// compliance here would be false. +[assembly: CLSCompliant(false)] + +[assembly: InternalsVisibleTo("Opc.Ua.AI.Tests")] diff --git a/src/Opc.Ua.AI.Client/Hosting/OpcUaAiClientBuilderExtensions.cs b/src/Opc.Ua.AI.Client/Hosting/OpcUaAiClientBuilderExtensions.cs new file mode 100644 index 0000000000..9bc3426ce0 --- /dev/null +++ b/src/Opc.Ua.AI.Client/Hosting/OpcUaAiClientBuilderExtensions.cs @@ -0,0 +1,81 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Opc.Ua; +using Opc.Ua.AI.Client; +using Opc.Ua.Client; + +namespace Microsoft.Extensions.DependencyInjection +{ + /// + /// Registers the AI Model Management client over the managed OPC UA session. + /// + public static class OpcUaAIClientBuilderExtensions + { + /// + /// Registers an and a + /// Func<CancellationToken, Task<AIClient>> so + /// downstream services can request AI clients. + /// + /// The client builder returned by AddClient. + /// is null. + public static IOpcUaClientBuilder AddAIClient( + this IOpcUaClientBuilder builder) + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + builder.Services.TryAddSingleton(sp => + { + Func> sessionFactory = + sp.GetService>>() + ?? throw new InvalidOperationException( + "AddAIClient requires AddClient to be called first."); + ITelemetryContext telemetry = + sp.GetRequiredService(); + return new AIClientFactory(sessionFactory, telemetry); + }); + + builder.Services.TryAddSingleton< + Func>>(sp => + { + AIClientFactory factory = + sp.GetRequiredService(); + return factory.CreateAsync; + }); + + return builder; + } + } +} diff --git a/src/Opc.Ua.AI.Client/NugetREADME.md b/src/Opc.Ua.AI.Client/NugetREADME.md new file mode 100644 index 0000000000..a386cea780 --- /dev/null +++ b/src/Opc.Ua.AI.Client/NugetREADME.md @@ -0,0 +1,22 @@ +# Client support for OPC UA AI Model Management + +`Opc.Ua.AI.Client` provides a high-level client facade for the draft OPC UA AI Model Management and Inference companion model. The API is organised around the specification concepts: AI root, model catalogue, model cards and resources, datasets, deployments, model sources, inference jobs, learning jobs, evaluation runs, and inference transfers. + +The root `AIClient` resolves the AI namespace and folders, enumerates typed instances, and opens focused clients such as `AIModelClient`, `AIDeploymentClient`, `AIModelSourceClient`, and `AIInferenceTransferClient`. These clients use the generated ObjectType proxies for method calls and return named snapshot records for reads. Artefact transfer is exposed through `ByteString` and stream helpers over the standard OPC UA `FileType` methods. + +Part of the [OPC UA .NET Standard](https://github.com/OPCFoundation/UA-.NETStandard) stack. + +> **Draft.** The *OPC UA - AI Model Management and Inference* companion +> specification is a working draft. Its namespace URI and every NodeId are +> provisional, and every ObjectType and BrowseName can change when the working +> group publishes. + +## Documentation + +See the [AI Model Management sample](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/samples/AI/README.md) +for the example: `ModelManagementServer` publishes a catalogue and +routes inference, and `ModelManagementClient` walks it with `AIClient`. + +## License + +MIT - see the [license](https://opcfoundation.org/license/mit.html). diff --git a/src/Opc.Ua.AI.Client/Opc.Ua.AI.Client.csproj b/src/Opc.Ua.AI.Client/Opc.Ua.AI.Client.csproj new file mode 100644 index 0000000000..f3d83b2b13 --- /dev/null +++ b/src/Opc.Ua.AI.Client/Opc.Ua.AI.Client.csproj @@ -0,0 +1,33 @@ + + + $(AssemblyPrefix).AI.Client + + net10.0;net9.0;net8.0 + $(CustomTestTarget) + net10.0 + true + $(PackagePrefix).Opc.Ua.AI.Client + Opc.Ua.AI.Client + $(NoWarn);CS1591;CS0108 + enable + Client-side support for the OPC UA AI Model Management and Inference (draft) companion specification: discovery of the model catalogue, datasets, deployments and inference endpoints, typed reads and Method calls, and artefact transfer through the standard file-transfer types. + true + NugetREADME.md + true + false + false + + + $(PackageId).Debug + + + + + + + + + + + + diff --git a/src/Opc.Ua.AI.Client/SessionAiExtensions.cs b/src/Opc.Ua.AI.Client/SessionAiExtensions.cs new file mode 100644 index 0000000000..6dcb30c8a5 --- /dev/null +++ b/src/Opc.Ua.AI.Client/SessionAiExtensions.cs @@ -0,0 +1,50 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Opc.Ua.Client; + +namespace Opc.Ua.AI.Client +{ + public static class SessionAIExtensions + { + public static AIClient AI(this ISession session, ITelemetryContext telemetry) + { + if (session is null) + { + throw new ArgumentNullException(nameof(session)); + } + if (telemetry is null) + { + throw new ArgumentNullException(nameof(telemetry)); + } + return new AIClient(session, telemetry); + } + } +} diff --git a/src/Opc.Ua.AI.Inference/AssemblyInfo.cs b/src/Opc.Ua.AI.Inference/AssemblyInfo.cs new file mode 100644 index 0000000000..be83f31ace --- /dev/null +++ b/src/Opc.Ua.AI.Inference/AssemblyInfo.cs @@ -0,0 +1,35 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +// The OPC UA stack surface this builds on is not CLS compliant (unsigned +// integers appear throughout the specification's own data types), so claiming +// compliance here would be false. +[assembly: CLSCompliant(false)] diff --git a/src/Opc.Ua.AI.Inference/Backends/ChatClientInferenceBackend.cs b/src/Opc.Ua.AI.Inference/Backends/ChatClientInferenceBackend.cs new file mode 100644 index 0000000000..719af6abc5 --- /dev/null +++ b/src/Opc.Ua.AI.Inference/Backends/ChatClientInferenceBackend.cs @@ -0,0 +1,596 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.AI; + +namespace Opc.Ua.AI.Inference +{ + /// + /// Runs inference through a . + /// + /// + /// + /// Microsoft.Extensions.AI is the abstraction this backend exists to + /// reach. Its point here is the same one clause 8.1 of + /// OPC UA - AI Model Management and Inference makes: where inference + /// runs changes the trust boundary and the latency and nothing else. An + /// is implemented by hosted services and by + /// on-device runtimes alike, so a Server that routes through one can move a + /// deployment between them without the address space changing shape - and + /// without this assembly referencing any vendor SDK, since the host supplies + /// the client it wants from its own composition root. + /// + /// + /// The payload stays opaque on the wire, which is the specification's position: + /// an envelope that typed it would need extending for every domain that adopted + /// it. This backend therefore accepts the OpenAI-compatible request shape a + /// caller already sends, projects it onto for the + /// client, and projects the response back. What it does NOT do is invent + /// values: usage the client did not report is reported as zero rather than + /// estimated, and a model the client did not name is reported as the one that + /// was asked for. + /// + /// + public sealed class ChatClientInferenceBackend : IInferenceBackend, IDisposable + { + /// + /// Creates a backend over a chat client. + /// + /// The client to run inference through. + /// + /// Where that client runs, as reported through + /// DeploymentType.InferenceLocation. The client cannot be asked - + /// an over a local runtime and one over a hosted + /// service are the same type - so the host states it. + /// + /// + /// The models this backend offers. An has no + /// enumeration in its contract, so a host that wants a catalogue supplies + /// one; the default is the client's own default model, or nothing. + /// + public ChatClientInferenceBackend( + IChatClient client, + InferenceSite site = InferenceSite.Cloud, + IReadOnlyList? models = null) + { + m_client = client ?? throw new ArgumentNullException(nameof(client)); + Site = site; + m_models = models ?? BuildDefaultCatalogue(client); + } + + /// + public InferenceSite Site { get; } + + /// + public ValueTask> ListModelsAsync( + string? filter, + uint maxResults, + CancellationToken ct) + { + ct.ThrowIfCancellationRequested(); + var matches = new List(); + for (int ii = 0; ii < m_models.Count; ii++) + { + BackendModel model = m_models[ii]; + if (!string.IsNullOrEmpty(filter) && + model.Name.IndexOf(filter, StringComparison.OrdinalIgnoreCase) < 0) + { + continue; + } + matches.Add(model); + if (maxResults > 0 && matches.Count >= maxResults) + { + break; + } + } + return new ValueTask>(matches); + } + + /// + public async ValueTask InvokeAsync( + InferenceRequest request, + CancellationToken ct) + { + if (request == null) + { + throw new ArgumentNullException(nameof(request)); + } + + List messages; + try + { + messages = ReadMessages(request.Payload.Span); + } + catch (JsonException ex) + { + // Malformed input from a caller is expected rather than + // exceptional, and refusing it with the reason is more use than + // letting a parse error surface as a transport fault. + return new InferenceResult + { + Ok = false, + Finish = InferenceFinish.Error, + Message = "The request payload is not a valid chat completions body: " + ex.Message + }; + } + + if (messages.Count == 0) + { + return new InferenceResult + { + Ok = false, + Finish = InferenceFinish.Error, + Message = "The request payload carries no messages." + }; + } + + var options = new ChatOptions { ModelId = request.Model }; + ApplyParameters(options, request.Parameters); + + using var timeout = request.Timeout > TimeSpan.Zero + ? new CancellationTokenSource(request.Timeout) + : null; + using CancellationTokenSource? linked = timeout == null + ? null + : CancellationTokenSource.CreateLinkedTokenSource(ct, timeout.Token); + + ChatResponse response; + try + { + response = await m_client + .GetResponseAsync(messages, options, linked?.Token ?? ct) + .ConfigureAwait(false); + } + catch (OperationCanceledException) when (timeout is { IsCancellationRequested: true }) + { + // The caller's own deadline elapsed. That is a timeout, not a + // cancellation, and the two want different handling upstream. + return new InferenceResult + { + Ok = false, + Finish = InferenceFinish.Error, + Message = "The backend did not answer within the requested timeout." + }; + } + + return Project(response, request.Model); + } + + /// + public async ValueTask ProbeAsync(CancellationToken ct) + { + try + { + ChatResponse response = await m_client + .GetResponseAsync( + [new ChatMessage(ChatRole.User, "ping")], + new ChatOptions { MaxOutputTokens = 1 }, + ct) + .ConfigureAwait(false); + return new BackendProbe + { + Reachable = true, + Detail = response.ModelId ?? string.Empty + }; + } + catch (OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // A probe reports every failure rather than raising it. + catch (Exception ex) +#pragma warning restore CA1031 + { + return new BackendProbe + { + Reachable = false, + Detail = ex.Message + }; + } + } + + /// + public void Dispose() + { + m_client.Dispose(); + } + + private static BackendModel[] BuildDefaultCatalogue(IChatClient client) + { + ChatClientMetadata? metadata = client.GetService(typeof(ChatClientMetadata)) as ChatClientMetadata; + string? id = metadata?.DefaultModelId; + if (string.IsNullOrEmpty(id)) + { + return Array.Empty(); + } + return + [ + new BackendModel + { + Publisher = metadata?.ProviderName ?? string.Empty, + Name = id, + Version = string.Empty, + TaskKind = "chat", + Capabilities = ["chat"] + } + ]; + } + + private static List ReadMessages(ReadOnlySpan payload) + { + var messages = new List(); + if (payload.IsEmpty) + { + return messages; + } + + using JsonDocument document = JsonDocument.Parse(payload.ToArray()); + if (document.RootElement.ValueKind != JsonValueKind.Object || + !document.RootElement.TryGetProperty("messages", out JsonElement array) || + array.ValueKind != JsonValueKind.Array) + { + return messages; + } + + foreach (JsonElement element in array.EnumerateArray()) + { + if (element.ValueKind != JsonValueKind.Object) + { + continue; + } + string role = "user"; + if (element.TryGetProperty("role", out JsonElement r)) + { + // GetString throws on a non-string element, and that would + // escape the handler that turns a bad payload into a refusal. + if (r.ValueKind is not JsonValueKind.String and not JsonValueKind.Null) + { + throw new JsonException("A chat message role must be a string."); + } + role = r.GetString() ?? "user"; + } + ChatRole chatRole = ToRole(role); + if (!element.TryGetProperty("content", out JsonElement c) || + c.ValueKind == JsonValueKind.String || + c.ValueKind == JsonValueKind.Null) + { + string text = c.ValueKind == JsonValueKind.String + ? c.GetString() ?? string.Empty + : string.Empty; + messages.Add(new ChatMessage(chatRole, text)); + continue; + } + if (c.ValueKind == JsonValueKind.Array) + { + messages.Add(new ChatMessage(chatRole, ReadContentParts(c))); + continue; + } + throw new JsonException("The chat message content must be a string or an array of content parts."); + } + return messages; + } + + private static List ReadContentParts(JsonElement array) + { + var contents = new List(); + foreach (JsonElement part in array.EnumerateArray()) + { + if (part.ValueKind != JsonValueKind.Object) + { + throw new JsonException("A chat content part must be an object."); + } + if (!part.TryGetProperty("type", out JsonElement t) || + t.ValueKind != JsonValueKind.String) + { + throw new JsonException("A chat content part must name its type."); + } + string type = t.GetString() ?? string.Empty; + if (string.Equals(type, "text", StringComparison.OrdinalIgnoreCase)) + { + contents.Add(ReadTextContent(part)); + continue; + } + if (string.Equals(type, "image_url", StringComparison.OrdinalIgnoreCase)) + { + contents.Add(ReadImageContent(part)); + continue; + } + + // Refuse unsupported parts rather than skipping them: dropping the + // only image would let a model answer confidently about content it + // never received. + throw new JsonException( + "The backend does not support chat content part type '" + type + "'."); + } + return contents; + } + + private static TextContent ReadTextContent(JsonElement part) + { + if (!part.TryGetProperty("text", out JsonElement text) || + text.ValueKind != JsonValueKind.String) + { + throw new JsonException("A text chat content part must carry string text."); + } + return new TextContent(text.GetString() ?? string.Empty); + } + + private static AIContent ReadImageContent(JsonElement part) + { + if (!part.TryGetProperty("image_url", out JsonElement image) || + image.ValueKind != JsonValueKind.Object || + !image.TryGetProperty("url", out JsonElement urlElement) || + urlElement.ValueKind != JsonValueKind.String) + { + throw new JsonException("An image_url chat content part must carry a string url."); + } + + string url = urlElement.GetString() ?? string.Empty; + if (string.IsNullOrWhiteSpace(url)) + { + throw new JsonException("An image_url chat content part must carry a non-empty url."); + } + + AIContent content; + if (url.StartsWith("data:", StringComparison.OrdinalIgnoreCase)) + { + content = ReadDataImageContent(url); + } + else if (Uri.TryCreate(url, UriKind.Absolute, out Uri? uri) && + (uri.Scheme == Uri.UriSchemeHttp || uri.Scheme == Uri.UriSchemeHttps)) + { + content = new UriContent(uri); + } + else + { + throw new JsonException( + "An image_url chat content part must carry a data, http, or https url."); + } + + ApplyImageDetail(image, content); + return content; + } + + private static DataContent ReadDataImageContent(string url) + { + int comma = url.IndexOf(',', StringComparison.Ordinal); + if (comma < 0) + { + throw new JsonException("The image_url data URI is missing its data separator."); + } + + string metadata = url.Substring("data:".Length, comma - "data:".Length); + string payload = url.Substring(comma + 1); + string mediaType = ReadDataUriMediaType(metadata); + if (!HasBase64Marker(metadata)) + { + throw new JsonException("The image_url data URI must be base64 encoded."); + } + + try + { + return new DataContent(Convert.FromBase64String(payload), mediaType); + } + catch (FormatException ex) + { + throw new JsonException("The image_url data URI base64 payload is malformed.", ex); + } + catch (ArgumentException ex) + { + // DataContent validates the media type itself and is stricter than + // the shape check above. Letting that escape would unwind past the + // handler that turns a bad payload into a structured refusal, so the + // caller would get a bare Bad status carrying an internal exception + // message instead of a result naming what was wrong. + throw new JsonException( + "The image_url data URI does not name a valid media type.", ex); + } + } + + private static string ReadDataUriMediaType(string metadata) + { + int semicolon = metadata.IndexOf(';', StringComparison.Ordinal); + string mediaType = semicolon < 0 ? metadata : metadata[..semicolon]; + if (string.IsNullOrWhiteSpace(mediaType)) + { + throw new JsonException("The image_url data URI must name a media type."); + } + + // A media type is type/subtype. Rejecting anything else here keeps the + // refusal specific; DataContent would otherwise throw a less helpful + // ArgumentException from inside the projection. + int slash = mediaType.IndexOf('/', StringComparison.Ordinal); + if (slash <= 0 || + slash == mediaType.Length - 1 || + mediaType.IndexOf('/', slash + 1) >= 0) + { + throw new JsonException( + "The image_url data URI media type must be of the form type/subtype."); + } + return mediaType; + } + + private static bool HasBase64Marker(string metadata) + { + string[] tokens = metadata.Split(';'); + for (int ii = 1; ii < tokens.Length; ii++) + { + if (string.Equals(tokens[ii], "base64", StringComparison.OrdinalIgnoreCase)) + { + return true; + } + } + return false; + } + + private static void ApplyImageDetail(JsonElement image, AIContent content) + { + if (!image.TryGetProperty("detail", out JsonElement detailElement) || + detailElement.ValueKind != JsonValueKind.String) + { + return; + } + + string? detail = detailElement.GetString(); + if (!string.Equals(detail, "low", StringComparison.OrdinalIgnoreCase) && + !string.Equals(detail, "high", StringComparison.OrdinalIgnoreCase) && + !string.Equals(detail, "auto", StringComparison.OrdinalIgnoreCase)) + { + return; + } + + content.AdditionalProperties = new AdditionalPropertiesDictionary + { + ["detail"] = detail! + }; + } + + private static ChatRole ToRole(string role) + { + if (string.Equals(role, "system", StringComparison.OrdinalIgnoreCase)) + { + return ChatRole.System; + } + if (string.Equals(role, "assistant", StringComparison.OrdinalIgnoreCase)) + { + return ChatRole.Assistant; + } + if (string.Equals(role, "tool", StringComparison.OrdinalIgnoreCase)) + { + return ChatRole.Tool; + } + return ChatRole.User; + } + + private static void ApplyParameters( + ChatOptions options, + IReadOnlyDictionary parameters) + { + if (parameters == null) + { + return; + } + foreach (KeyValuePair parameter in parameters) + { + if (string.Equals(parameter.Key, "temperature", StringComparison.OrdinalIgnoreCase) && + float.TryParse( + parameter.Value, + System.Globalization.NumberStyles.Float, + System.Globalization.CultureInfo.InvariantCulture, + out float temperature)) + { + options.Temperature = temperature; + } + else if (string.Equals(parameter.Key, "max_tokens", StringComparison.OrdinalIgnoreCase) && + int.TryParse( + parameter.Value, + System.Globalization.NumberStyles.Integer, + System.Globalization.CultureInfo.InvariantCulture, + out int maxTokens)) + { + options.MaxOutputTokens = maxTokens; + } + else if (string.Equals(parameter.Key, "top_p", StringComparison.OrdinalIgnoreCase) && + float.TryParse( + parameter.Value, + System.Globalization.NumberStyles.Float, + System.Globalization.CultureInfo.InvariantCulture, + out float topP)) + { + options.TopP = topP; + } + else + { + // A parameter the backend cannot honour is refused rather than + // dropped: a caller whose parameter was silently ignored + // believes it took effect and never finds out otherwise. + throw new ArgumentException( + "The backend does not support the call parameter '" + parameter.Key + "'.", + nameof(parameters)); + } + } + } + + private static InferenceResult Project(ChatResponse response, string requested) + { + byte[] payload = Encoding.UTF8.GetBytes(response.Text ?? string.Empty); + UsageDetails? usage = response.Usage; + return new InferenceResult + { + Ok = true, + Payload = payload, + ContentType = "text/plain", + // A client that named the model that answered is believed; one that + // did not is not second-guessed, because the request named a model + // and nothing observed contradicts it. + ModelUsed = string.IsNullOrEmpty(response.ModelId) ? requested : response.ModelId!, + UsageUnit = "tokens", + InputUnits = ToUnits(usage?.InputTokenCount), + OutputUnits = ToUnits(usage?.OutputTokenCount), + TotalUnits = ToUnits(usage?.TotalTokenCount), + Finish = ToFinish(response.FinishReason) + }; + } + + private static ulong ToUnits(long? value) + { + return value is > 0 ? (ulong)value.Value : 0UL; + } + + private static InferenceFinish ToFinish(ChatFinishReason? reason) + { + if (reason == null) + { + return InferenceFinish.Stop; + } + if (reason == ChatFinishReason.Length) + { + return InferenceFinish.Length; + } + if (reason == ChatFinishReason.ContentFilter) + { + return InferenceFinish.Filtered; + } + if (reason == ChatFinishReason.ToolCalls) + { + return InferenceFinish.ToolCall; + } + return InferenceFinish.Stop; + } + + private readonly IChatClient m_client; + private readonly IReadOnlyList m_models; + } +} diff --git a/src/Opc.Ua.AI.Inference/Backends/RestChatCompletionsBackend.cs b/src/Opc.Ua.AI.Inference/Backends/RestChatCompletionsBackend.cs new file mode 100644 index 0000000000..070b8b2665 --- /dev/null +++ b/src/Opc.Ua.AI.Inference/Backends/RestChatCompletionsBackend.cs @@ -0,0 +1,513 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; + +namespace Opc.Ua.AI.Inference +{ + /// + /// A backend that speaks the REST chat-completions contract. + /// + /// + /// + /// One class serves both the hosted service and the on-device runtime, because + /// both expose the same contract. What differs is the base address, what + /// authenticates the call, and which models are available - which is precisely + /// the claim InferenceLocation makes, so it would have been suspicious to + /// need two implementations. + /// + /// + /// The wire format is handled directly rather than through a vendor SDK. For a + /// sample against a specification whose ApiDialectEnum names this contract + /// by shape, showing the shape is the point; it also keeps the sample free of a + /// dependency that would date faster than the contract does. + /// + /// + public sealed class RestChatCompletionsBackend : IInferenceBackend, IDisposable + { + private readonly HttpClient m_http; + private readonly InferenceBackendOptions m_options; + private readonly ICredentialResolver m_credentials; + private readonly ILogger m_logger; + private readonly bool m_ownsClient; + + /// + /// Creates a backend over the configured endpoint. + /// + public RestChatCompletionsBackend( + InferenceBackendOptions options, + ICredentialResolver credentials, + ILogger logger, + HttpClient? http = null) + { + m_options = options ?? throw new ArgumentNullException(nameof(options)); + m_credentials = credentials ?? throw new ArgumentNullException(nameof(credentials)); + m_logger = logger ?? throw new ArgumentNullException(nameof(logger)); + m_ownsClient = http is null; + m_http = http ?? new HttpClient(); + if (m_http.BaseAddress is null && !string.IsNullOrEmpty(options.EndpointUri)) + { + m_http.BaseAddress = new Uri(options.EndpointUri, UriKind.Absolute); + } + } + + /// + public InferenceSite Site => m_options.Site; + + /// + /// + /// + /// Asked of the endpoint, not of configuration. The question the caller is + /// asking is what this source OFFERS, and configuration can only answer what + /// this Server was told about - which is a different and usually smaller set. + /// Where the endpoint declines to say, the configured list is the fallback + /// rather than the answer. + /// + public async ValueTask> ListModelsAsync( + string? filter, uint maxResults, CancellationToken ct) + { + IEnumerable models = + await DiscoverModelsAsync(ct).ConfigureAwait(false); + + if (!string.IsNullOrEmpty(filter)) + { + models = models.Where(m => + m.Name.Contains(filter, StringComparison.OrdinalIgnoreCase) || + m.Publisher.Contains(filter, StringComparison.OrdinalIgnoreCase)); + } + if (maxResults > 0) + { + models = models.Take((int)maxResults); + } + return models.ToList(); + } + + /// + /// Reads the endpoint's model list. + /// + /// + /// Metadata the endpoint does not carry is taken from configuration where a + /// configured entry names the same model, so an operator can supply the + /// publisher, the digest and the task kind that an OpenAI-compatible listing + /// has no field for - without that supplement overriding what the endpoint + /// actually reports. + /// + private async ValueTask> DiscoverModelsAsync( + CancellationToken ct) + { + using var message = new HttpRequestMessage(HttpMethod.Get, m_options.ProbePath); + await AuthenticateAsync(message, ct).ConfigureAwait(false); + + try + { + using HttpResponseMessage response = await m_http + .SendAsync(message, ct) + .ConfigureAwait(false); + + if (!response.IsSuccessStatusCode) + { + return [.. m_options.Models]; + } + + using Stream body = await response.Content + .ReadAsStreamAsync(ct) + .ConfigureAwait(false); + + using JsonDocument document = await JsonDocument + .ParseAsync(body, cancellationToken: ct) + .ConfigureAwait(false); + + if (!document.RootElement.TryGetProperty("data", out JsonElement data) || + data.ValueKind != JsonValueKind.Array) + { + return [.. m_options.Models]; + } + + var discovered = new List(); + + foreach (JsonElement element in data.EnumerateArray()) + { + // ValueKind is checked before GetString, which throws rather + // than returning null when the value is not a string. The + // endpoint is not under this Server's control, so "the field is + // there" and "the field is what it should be" are separate + // questions. + if (element.ValueKind != JsonValueKind.Object || + !element.TryGetProperty("id", out JsonElement id) || + id.ValueKind != JsonValueKind.String || + id.GetString() is not { Length: > 0 } name) + { + continue; + } + + BackendModel? configured = m_options.Models.FirstOrDefault( + m => string.Equals(m.Name, name, StringComparison.OrdinalIgnoreCase)); + + discovered.Add(configured is null + ? new BackendModel + { + Publisher = element.TryGetProperty("owned_by", out JsonElement owner) && + owner.ValueKind == JsonValueKind.String + ? owner.GetString() ?? "unknown" + : "unknown", + Name = name, + Version = "unknown", + Framework = "rest-chat-completions" + } + : configured with { Name = name }); + } + + return discovered.Count > 0 ? discovered : [.. m_options.Models]; + } + catch (HttpRequestException) + { + return [.. m_options.Models]; + } + catch (JsonException) + { + return [.. m_options.Models]; + } + catch (OperationCanceledException) when (!ct.IsCancellationRequested) + { + // HttpClient's own timeout surfaces as TaskCanceledException, not + // HttpRequestException. Without this a hung endpoint faults the + // ListModels call instead of the source simply reporting nothing - + // and a hung endpoint is the ordinary way a remote one fails. + return [.. m_options.Models]; + } + } + + /// + public async ValueTask InvokeAsync( + InferenceRequest request, CancellationToken ct) + { + if (request is null) + { + throw new ArgumentNullException(nameof(request)); + } + + using var message = new HttpRequestMessage( + HttpMethod.Post, m_options.ChatCompletionsPath); + message.Content = BuildContent(request); + await AuthenticateAsync(message, ct).ConfigureAwait(false); + + using CancellationTokenSource? timeout = request.Timeout > TimeSpan.Zero + ? CancellationTokenSource.CreateLinkedTokenSource(ct) + : null; + timeout?.CancelAfter(request.Timeout); + + HttpResponseMessage response; + try + { + response = await m_http + .SendAsync(message, timeout?.Token ?? ct) + .ConfigureAwait(false); + } + catch (OperationCanceledException) when (!ct.IsCancellationRequested) + { + return new InferenceResult + { + Ok = false, + Finish = InferenceFinish.Cancelled, + Message = "The call exceeded the caller's timeout." + }; + } + catch (HttpRequestException ex) + { + m_logger.LogWarning(ex, "Inference endpoint unreachable."); + return new InferenceResult + { + Ok = false, + Finish = InferenceFinish.Error, + Message = ex.Message + }; + } + + using (response) + { + byte[] body = await response.Content + .ReadAsByteArrayAsync(ct) + .ConfigureAwait(false); + + if (!response.IsSuccessStatusCode) + { + return Failure(response, body); + } + return Success(request, body); + } + } + + /// + public async ValueTask ProbeAsync(CancellationToken ct) + { + using var message = new HttpRequestMessage(HttpMethod.Get, m_options.ProbePath); + await AuthenticateAsync(message, ct).ConfigureAwait(false); + try + { + using HttpResponseMessage response = await m_http + .SendAsync(message, ct) + .ConfigureAwait(false); + + // Throttled is reported separately from unreachable because the two + // look alike from outside and call for opposite responses: failing + // over a throttled endpoint moves load onto a weaker model for no + // reason, since it will serve again shortly. + if (response.StatusCode == HttpStatusCode.TooManyRequests || + response.StatusCode == HttpStatusCode.ServiceUnavailable) + { + return new BackendProbe + { + Reachable = true, + Throttled = true, + RetryAfter = RetryAfterOf(response), + Detail = "The endpoint is refusing work for capacity reasons." + }; + } + return new BackendProbe + { + Reachable = true, + Detail = FormattableString.Invariant($"HTTP {(int)response.StatusCode}.") + }; + } + catch (HttpRequestException ex) + { + return new BackendProbe { Reachable = false, Detail = ex.Message }; + } + catch (OperationCanceledException) when (!ct.IsCancellationRequested) + { + // A hung endpoint is unreachable for every purpose a caller has, and + // saying so is more useful than faulting the probe that exists to + // answer exactly this question. + return new BackendProbe + { + Reachable = false, + Detail = "The endpoint did not answer within the client timeout." + }; + } + } + + private async Task AuthenticateAsync(HttpRequestMessage message, CancellationToken ct) + { + if (string.IsNullOrEmpty(m_options.CredentialReference)) + { + return; + } + string? secret = await m_credentials + .ResolveAsync(m_options.CredentialReference, ct) + .ConfigureAwait(false); + if (string.IsNullOrEmpty(secret)) + { + return; + } + switch (m_options.Authentication) + { + case BackendAuthentication.ApiKey: + message.Headers.TryAddWithoutValidation(m_options.ApiKeyHeader, secret); + break; + case BackendAuthentication.BearerToken: + case BackendAuthentication.WorkloadIdentity: + message.Headers.Authorization = + new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", secret); + break; + } + } + + private static ByteArrayContent BuildContent(InferenceRequest request) + { + // The payload is opaque: the caller supplies the body and its media type, + // and this backend adds only the routing the contract requires. A backend + // that rewrote the body would be typing a payload the specification + // deliberately leaves untyped. + var content = new ByteArrayContent(request.Payload.ToArray()); + content.Headers.TryAddWithoutValidation("Content-Type", request.ContentType); + return content; + } + + private InferenceResult Failure(HttpResponseMessage response, byte[] body) + { + bool capacity = response.StatusCode == HttpStatusCode.TooManyRequests || + response.StatusCode == HttpStatusCode.ServiceUnavailable; + return new InferenceResult + { + Ok = false, + Finish = InferenceFinish.Error, + // A capacity refusal is the one failure worth retrying, and the only + // one where the endpoint tells the caller when. + RetryAfter = capacity ? RetryAfterOf(response) : TimeSpan.Zero, + Message = FormattableString.Invariant( + $"HTTP {(int)response.StatusCode}: {Describe(body)}") + }; + } + + private InferenceResult Success(InferenceRequest request, byte[] body) + { + string modelUsed = request.Model; + const string unit = "tokens"; + ulong input = 0; + ulong output = 0; + ulong total = 0; + InferenceFinish finish = InferenceFinish.Stop; + + // The response envelope is read for the fields the specification requires + // a Server to pass through. It is read defensively: an endpoint that omits + // usage is not an endpoint that failed, and a sample that threw on a + // missing optional field would be brittle against every real service. + try + { + using JsonDocument doc = JsonDocument.Parse(body); + JsonElement root = doc.RootElement; + + // Which model ACTUALLY answered. Reported rather than assumed, + // because a service that substituted one says so here and the Server + // must pass that through to ModelUsed. + if (root.TryGetProperty("model", out JsonElement m) && + m.ValueKind == JsonValueKind.String) + { + modelUsed = m.GetString() ?? request.Model; + } + if (root.TryGetProperty("usage", out JsonElement u) && + u.ValueKind == JsonValueKind.Object) + { + input = ReadCount(u, "prompt_tokens", "input_tokens"); + output = ReadCount(u, "completion_tokens", "output_tokens"); + total = ReadCount(u, "total_tokens", null); + } + if (root.TryGetProperty("choices", out JsonElement choices) && + choices.ValueKind == JsonValueKind.Array && + choices.GetArrayLength() > 0 && + choices[0].TryGetProperty("finish_reason", out JsonElement fr) && + fr.ValueKind == JsonValueKind.String) + { + finish = MapFinish(fr.GetString()); + } + } + catch (JsonException) + { + // A non-JSON body is legitimate for an endpoint returning something + // else entirely. The payload still reaches the caller intact. + } + + return new InferenceResult + { + Ok = true, + Payload = body, + ContentType = "application/json", + ModelUsed = modelUsed, + UsageUnit = unit, + InputUnits = input, + OutputUnits = output, + TotalUnits = total == 0 ? input + output : total, + Finish = finish + }; + } + + /// + /// Reads a token count, tolerating anything the endpoint might actually send. + /// + /// + /// ValueKind == Number does not mean GetUInt64 will succeed: a + /// negative or fractional value is still a Number and throws + /// , which is not a + /// and so escapes the caller's guard. This is on the SUCCESS path, so a + /// 200 response carrying "prompt_tokens": -1 would fault an inference + /// that had otherwise worked. + /// + private static ulong ReadCount(JsonElement usage, string first, string? second) + { + if (usage.TryGetProperty(first, out JsonElement a) && + a.ValueKind == JsonValueKind.Number && + a.TryGetUInt64(out ulong firstValue)) + { + return firstValue; + } + if (second != null && + usage.TryGetProperty(second, out JsonElement b) && + b.ValueKind == JsonValueKind.Number && + b.TryGetUInt64(out ulong secondValue)) + { + return secondValue; + } + return 0; + } + + private static InferenceFinish MapFinish(string? reason) + { + return reason switch + { + "stop" => InferenceFinish.Stop, + "length" => InferenceFinish.Length, + "tool_calls" => InferenceFinish.ToolCall, + "content_filter" => InferenceFinish.Filtered, + _ => InferenceFinish.Stop + }; + } + + private static TimeSpan RetryAfterOf(HttpResponseMessage response) + { + System.Net.Http.Headers.RetryConditionHeaderValue? h = + response.Headers.RetryAfter; + if (h?.Delta is { } delta) + { + return delta; + } + if (h?.Date is { } date) + { + TimeSpan wait = date - DateTimeOffset.UtcNow; + return wait > TimeSpan.Zero ? wait : TimeSpan.Zero; + } + return TimeSpan.Zero; + } + + private static string Describe(byte[] body) + { + const int Limit = 256; + string text = System.Text.Encoding.UTF8.GetString(body); + return text.Length <= Limit ? text : text.Substring(0, Limit); + } + + /// + public void Dispose() + { + if (m_ownsClient) + { + m_http.Dispose(); + } + } + } +} diff --git a/src/Opc.Ua.AI.Inference/Credentials/ICredentialResolver.cs b/src/Opc.Ua.AI.Inference/Credentials/ICredentialResolver.cs new file mode 100644 index 0000000000..40c38e5088 --- /dev/null +++ b/src/Opc.Ua.AI.Inference/Credentials/ICredentialResolver.cs @@ -0,0 +1,334 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Inference +{ + /// + /// Turns the credential NAME a deployment publishes into the credential itself. + /// + /// + /// + /// This interface exists to be the only thing in the sample that ever holds a + /// secret value. OPC UA - AI Model Management and Inference clause 9.2 + /// forbids a Server from exposing credential material through any Attribute, and + /// argues it from the fact that an address space is browsable, subscribable and + /// historisable - a secret placed there is not exposed once, it is published, + /// distributed and archived. + /// + /// + /// So ModelSourceType.CredentialReference carries a name, a client that + /// reads it learns which credential is in use and nothing about what it is, and + /// the resolution from name to value happens here and nowhere else. + /// + /// + public interface ICredentialResolver + { + /// + /// Resolves a credential reference to the value to present, or null where the + /// backend needs none. + /// + /// + /// The name published as CredentialReference. Never a secret. + /// + /// Cancels the resolution. + ValueTask ResolveAsync(string reference, CancellationToken ct); + } + + /// + /// Resolves a credential reference against a directory of files, which is how a + /// Kubernetes Secret mounted as a volume appears to a process. + /// + /// + /// A mounted file is preferred to an environment variable for a reason worth + /// stating: environment variables are inherited by child processes, appear in + /// crash dumps and in process listings, and get printed by well-meaning + /// diagnostic code. A file is read only by something that decides to open it. + /// + public sealed class FileCredentialResolver : ICredentialResolver + { + private readonly string m_directory; + + /// + /// Creates a resolver over the directory a Secret is mounted at. + /// + /// The mount point. + public FileCredentialResolver(string directory) + { + m_directory = directory ?? throw new ArgumentNullException(nameof(directory)); + } + + private static readonly System.Buffers.SearchValues s_pathSeparators = + System.Buffers.SearchValues.Create("/\\"); + + /// + public async ValueTask ResolveAsync(string reference, CancellationToken ct) + { + if (string.IsNullOrEmpty(reference)) + { + return null; + } + + // A reference names a key within the mount. Anything that could escape the + // mount is refused rather than sanitised: a reference is configuration this + // Server controls, so one containing a separator is a mistake worth + // surfacing rather than input worth repairing. + if (reference.AsSpan().IndexOfAny(s_pathSeparators) >= 0 || + reference.Contains("..", StringComparison.Ordinal)) + { + throw new ArgumentException( + "A credential reference names a key, not a path.", nameof(reference)); + } + + string path = System.IO.Path.Combine(m_directory, reference); + if (!System.IO.File.Exists(path)) + { + return null; + } + + string value = await System.IO.File + .ReadAllTextAsync(path, ct) + .ConfigureAwait(false); + + // Mounted secrets routinely carry a trailing newline from however they were + // written. Presenting that in a header fails in a way that looks like a + // wrong key rather than a stray byte. + return value.Trim(); + } + } + + /// + /// Resolves nothing, for a backend that needs no credential at all. + /// + /// + /// This is the on-device case, and the workload-identity case: in neither does a + /// secret exist anywhere for a future mistake to expose. Clause 9.2 prefers + /// exactly that wherever the platform offers it. + /// + public sealed class NullCredentialResolver : ICredentialResolver + { + /// A shared instance. + public static NullCredentialResolver Instance { get; } = new(); + + /// + public ValueTask ResolveAsync(string reference, CancellationToken ct) + { + return new ValueTask((string?)null); + } + } + + /// + /// Resolves the token the hosting platform projects for the Server's own + /// workload identity. + /// + /// + /// + /// Preferred where the platform offers it, because no secret is stored anywhere + /// for a later mistake to expose. The reference names the token file to read, + /// which is where a token is projected and not what it is. + /// + /// + /// Every platform that implements workload identity projects the token as a + /// file: Kubernetes mounts a projected service-account volume, and the cloud + /// providers layered on it reach that path in their own way — Azure and AWS + /// each name it in an environment variable, while Google names a JSON + /// configuration that in turn carries the path. Reading the file is therefore + /// the whole mechanism, and doing it directly keeps this assembly free of any + /// cloud SDK - which matters here, because a Server that only runs against one + /// vendor's identity service is not demonstrating a platform-independent one. + /// + /// + /// A host that wants a vendor SDK in the loop - to exchange the projected token + /// for a service-specific one, say - supplies a delegate instead of a path and + /// keeps that dependency in its own composition root. + /// + /// + public sealed class WorkloadIdentityCredentialResolver : ICredentialResolver + { + private readonly Func> m_acquire; + private readonly string m_audience; + + /// + /// Creates a resolver over the token the platform projects. + /// + /// + /// The token file path to read. Empty defers to the reference, and then to + /// the variables the platforms genuinely set. + /// + public WorkloadIdentityCredentialResolver(string audience = "") + : this(ReadProjectedTokenAsync, audience) + { + } + + /// + /// Creates a resolver over a supplied acquisition delegate, which is what + /// lets a test exercise this path without a platform, and what lets a host + /// bring its own identity SDK without this assembly depending on one. + /// + /// + /// Acquires the token for a scope or token path. + /// + /// The scope or token path to request. + public WorkloadIdentityCredentialResolver( + Func> acquire, + string audience = "") + { + m_acquire = acquire ?? throw new ArgumentNullException(nameof(acquire)); + m_audience = audience ?? string.Empty; + } + + /// + /// + /// The audience configured at construction wins; the reference is the + /// fallback. Workload identity has no secret to name, so what + /// CredentialReference means on this path is the token being asked + /// for - and letting a deployment state that explicitly, under a name that + /// says so, avoids a member whose meaning changes with the authentication + /// kind. + /// + public ValueTask ResolveAsync(string reference, CancellationToken ct) + { + string scope = !string.IsNullOrEmpty(m_audience) ? m_audience : reference; + return m_acquire(scope ?? string.Empty, ct); + } + + private static async ValueTask ReadProjectedTokenAsync( + string scope, + CancellationToken ct) + { + string? path = !string.IsNullOrEmpty(scope) ? scope : ResolveProjectedTokenPath(); + if (string.IsNullOrEmpty(path) || !System.IO.File.Exists(path)) + { + return null; + } + + string value = await System.IO.File.ReadAllTextAsync(path, ct).ConfigureAwait(false); + + // A projected token carries whatever trailing whitespace the platform + // wrote. Presenting that in a header fails in a way that looks like a + // rejected identity rather than a stray byte. + return value.Trim(); + } + + private static string? ResolveProjectedTokenPath() + { + for (int ii = 0; ii < s_tokenPathVariables.Length; ii++) + { + string? value = Environment.GetEnvironmentVariable(s_tokenPathVariables[ii]); + if (!string.IsNullOrEmpty(value)) + { + return value; + } + } + string? google = ResolveGoogleExternalAccountTokenPath(); + if (!string.IsNullOrEmpty(google)) + { + return google; + } + return System.IO.File.Exists(KubernetesProjectedTokenPath) + ? KubernetesProjectedTokenPath + : null; + } + + /// + /// Resolves the token path out of a Google external-account credential + /// configuration. + /// + /// + /// Google is the platform that does not name the token in an environment + /// variable of its own. It names a JSON configuration in + /// GOOGLE_APPLICATION_CREDENTIALS, and for the + /// external_account type that configuration carries the projected + /// token's path in credential_source.file. Reading it here is what + /// makes the Google path real rather than assumed; a token file invented + /// under a Google-shaped variable name would simply never be found. + /// On GKE the token comes from the metadata server instead, which is not a + /// file at all — a host in that position supplies an acquisition delegate. + /// + private static string? ResolveGoogleExternalAccountTokenPath() + { + string? configPath = Environment.GetEnvironmentVariable( + "GOOGLE_APPLICATION_CREDENTIALS"); + if (string.IsNullOrEmpty(configPath) || !System.IO.File.Exists(configPath)) + { + return null; + } + try + { + using System.IO.FileStream stream = System.IO.File.OpenRead(configPath); + using JsonDocument document = JsonDocument.Parse(stream); + if (!document.RootElement.TryGetProperty("credential_source", out JsonElement source) || + !source.TryGetProperty("file", out JsonElement file) || + file.ValueKind != JsonValueKind.String) + { + return null; + } + return file.GetString(); + } + catch (JsonException) + { + // A configuration this malformed is a deployment error, but failing + // to resolve an identity is already reported as such by the caller. + return null; + } + catch (System.IO.IOException) + { + return null; + } + } + + /// + /// The path Kubernetes mounts a projected service-account token at, which is + /// the mechanism every cloud workload identity is layered on. Used only when + /// no platform variable names one. + /// + private const string KubernetesProjectedTokenPath = + "/var/run/secrets/kubernetes.io/serviceaccount/token"; + + /// + /// The variables the platforms genuinely set, verified against each + /// platform's own documentation rather than inferred from its name: + /// AZURE_FEDERATED_TOKEN_FILE is injected by the Azure Workload Identity + /// mutating webhook, and AWS_WEB_IDENTITY_TOKEN_FILE is the AWS SDK + /// standard that EKS IAM-roles-for-service-accounts populates. There is + /// deliberately no Google entry: Google has no such variable, and one + /// invented to look like it would never match anything. + /// + private static readonly string[] s_tokenPathVariables = + [ + "AZURE_FEDERATED_TOKEN_FILE", + "AWS_WEB_IDENTITY_TOKEN_FILE" + ]; + } +} diff --git a/src/Opc.Ua.AI.Inference/IInferenceBackend.cs b/src/Opc.Ua.AI.Inference/IInferenceBackend.cs new file mode 100644 index 0000000000..337434a0bc --- /dev/null +++ b/src/Opc.Ua.AI.Inference/IInferenceBackend.cs @@ -0,0 +1,283 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.AI.Inference +{ + /// + /// The inference backend the Server executes through. + /// + /// + /// + /// One abstraction covers a hosted service and an on-device runtime because both + /// speak the same wire contract. That is not a convenience of this sample: it is + /// the property OPC UA - AI Model Management and Inference clause 8.1 + /// asserts, that where inference runs changes the trust boundary and the latency + /// and nothing else. If satisfying it here had required two shapes, the claim + /// would have been wrong. + /// + /// + /// Implementations are expected to be thread-safe. The Server calls them from + /// whichever thread a Method arrived on. + /// + /// + public interface IInferenceBackend + { + /// + /// Where inference runs, as reported to a client through + /// DeploymentType.InferenceLocation. + /// + InferenceSite Site { get; } + + /// + /// Models this backend offers, as the catalogue and the address space + /// present them. + /// + ValueTask> ListModelsAsync( + string? filter, + uint maxResults, + CancellationToken ct); + + /// + /// Runs one inference. + /// + /// + /// The payload is opaque to the Server, which is the specification's position + /// and not an omission here: what goes into a model and comes back is domain + /// vocabulary, and an envelope that typed it would need extending for every + /// domain that adopted it. + /// + ValueTask InvokeAsync( + InferenceRequest request, + CancellationToken ct); + + /// + /// Probes the backend, so that a commissioning engineer can establish that + /// credentials and network policy are right BEFORE a deployment depends on + /// them rather than learning it from the first failed inference. + /// + ValueTask ProbeAsync(CancellationToken ct); + } + + /// + /// Where a backend executes. Maps onto InferenceLocationEnum. + /// + public enum InferenceSite + { + /// In the Server's own process or on its host. + OnServer, + + /// On a separate node reached over the local network. + EdgeOffServer, + + /// In a remote or hosted service. + Cloud + } + + /// + /// One model a backend offers. + /// + /// + /// Identity is a publisher, name and version triple rather than a URL, because a + /// URL says where a copy is today and the triple says which artefact is meant - + /// and the two diverge the moment anyone mirrors anything. + /// + public sealed record BackendModel + { + /// Organisation or namespace that published the model. + public string Publisher { get; init; } = string.Empty; + + /// Model name within that publisher. + public string Name { get; init; } = string.Empty; + + /// Version identifier. + public string Version { get; init; } = string.Empty; + + /// What the model does, for example chat. + public string TaskKind { get; init; } = "chat"; + + /// Runtime or library the artefact targets. + public string Framework { get; init; } = string.Empty; + + /// Capability names the backend reports, for example chat. + public IReadOnlyList Capabilities { get; init; } = Array.Empty(); + + /// + /// Digest of the artefact, where the backend can name one. + /// + /// + /// Empty by default and deliberately so. A provenance walk terminates at + /// this value, which makes it the one field a sample must not invent: a + /// hosted endpoint that will not say which weights answered cannot be made + /// to say so by hashing its name, and a digest that looks like an artefact + /// digest but is not one is worse than none, because it will be compared. + /// + public ReadOnlyMemory Digest { get; init; } + + /// + /// Algorithm that produced , empty when there is none. + /// + public string DigestAlgorithm { get; init; } = string.Empty; + + /// + /// How the weights are quantized, where that is known. + /// + /// + /// Distinguishes a stand-in from the artefact it stands in for. Two models + /// with the same name and version but different quantization give different + /// answers, so a client comparing results across a fallback needs to be able + /// to see that they are not the same thing. + /// + public string Quantization { get; init; } = string.Empty; + } + + /// + /// One inference request. + /// + public sealed record InferenceRequest + { + /// Model to route to, as the backend names it. + public string Model { get; init; } = string.Empty; + + /// Request body. + public ReadOnlyMemory Payload { get; init; } + + /// Media type of . + public string ContentType { get; init; } = "application/json"; + + /// + /// Call parameters. A backend rejects one it does not support rather than + /// ignoring it: a caller whose parameter was silently dropped believes it + /// took effect, and there is no later point at which it can find out. + /// + public IReadOnlyDictionary Parameters { get; init; } = + new Dictionary(); + + /// How long the caller will wait, or zero for the backend default. + public TimeSpan Timeout { get; init; } + } + + /// + /// What one inference produced. + /// + public sealed record InferenceResult + { + /// Whether the call succeeded. + public bool Ok { get; init; } + + /// Response body. + public ReadOnlyMemory Payload { get; init; } + + /// Media type of . + public string ContentType { get; init; } = "application/json"; + + /// + /// The model that ACTUALLY answered, which is not always the one asked for. + /// A backend that silently substituted one reports the substitute here, and + /// the Server passes it through to ModelUsed. + /// + public string ModelUsed { get; init; } = string.Empty; + + /// Unit the usage counts are in, for example tokens. + public string UsageUnit { get; init; } = "tokens"; + + /// Units consumed by the input. + public ulong InputUnits { get; init; } + + /// Units produced as output. + public ulong OutputUnits { get; init; } + + /// Units metered for the call, which is not always the sum. + public ulong TotalUnits { get; init; } + + /// Why output stopped. + public InferenceFinish Finish { get; init; } = InferenceFinish.Stop; + + /// + /// How long to wait before retrying, where the failure was a capacity one. + /// Zero when retrying immediately is as good as waiting. + /// + public TimeSpan RetryAfter { get; init; } + + /// Diagnostic. For a human; not to be parsed. + public string? Message { get; init; } + } + + /// + /// Why an inference stopped producing output. Maps onto + /// FinishReasonEnum. + /// + public enum InferenceFinish + { + /// The model finished normally. + Stop, + + /// Output was truncated by a length or budget limit. + Length, + + /// The model requested a tool call. + ToolCall, + + /// Output was withheld by a safety policy. + Filtered, + + /// The call was cancelled. + Cancelled, + + /// The call failed. + Error + } + + /// + /// The outcome of probing a backend. + /// + public sealed record BackendProbe + { + /// Whether the backend answered. + public bool Reachable { get; init; } + + /// + /// Whether it answered but is refusing work for capacity reasons. Separated + /// from unreachable deliberately: the two look alike from outside and call + /// for opposite responses, since failing over a throttled endpoint merely + /// moves load for no reason. + /// + public bool Throttled { get; init; } + + /// How long to wait, where the backend said. + public TimeSpan RetryAfter { get; init; } + + /// Diagnostic. For a human. + public string? Detail { get; init; } + } +} diff --git a/src/Opc.Ua.AI.Inference/InferenceBackendOptions.cs b/src/Opc.Ua.AI.Inference/InferenceBackendOptions.cs new file mode 100644 index 0000000000..3ce7814958 --- /dev/null +++ b/src/Opc.Ua.AI.Inference/InferenceBackendOptions.cs @@ -0,0 +1,189 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; + +namespace Opc.Ua.AI.Inference +{ + /// + /// How the Server authenticates ITSELF to an inference endpoint. + /// + /// + /// This is not how a client authenticates to this Server, which is the ordinary + /// OPC UA Session security and is unaffected. It maps onto + /// AuthenticationKindEnum. + /// + public enum BackendAuthentication + { + /// + /// No credential. Permitted only where the endpoint is reachable solely from + /// a trusted network segment - which is the ordinary case for an on-device + /// runtime listening on loopback. + /// + Anonymous, + + /// A shared secret presented as a key. + ApiKey, + + /// A token obtained from an authorization service. + BearerToken, + + /// + /// An identity the hosting platform assigns, so no secret is stored at all. + /// Preferred wherever the platform offers it. + /// + WorkloadIdentity + } + + /// + /// Which inference backend contract a deployment uses. + /// + public enum InferenceBackendKind + { + /// + /// The host supplies a Microsoft.Extensions.AI.IChatClient. + /// + ChatClient, + + /// + /// The backend speaks the OpenAI-compatible REST chat-completions contract + /// directly. + /// + RestChatCompletions + } + + /// + /// Configuration for one inference backend. + /// + /// + /// Bound from configuration at startup. Cloud and on-device differ only in the + /// values here - the endpoint, what authenticates a call, and which models exist - + /// which is why one backend implementation serves both. + /// + public sealed class InferenceBackendOptions + { + /// Configuration section this binds from. + public const string SectionName = "InferenceBackend"; + + /// + /// Configuration section the fallback backend binds from. + /// + /// + /// A separate section rather than a nested one, because the fallback is a + /// backend in its own right: it has its own endpoint, its own credentials + /// and its own reachability, and sharing any of those with the primary would + /// defeat the purpose of having it. + /// + public const string FallbackSectionName = "FallbackInferenceBackend"; + + /// + /// Whether this backend is configured at all. + /// + /// + /// Meaningful for the fallback: a Server with nowhere to fall back to should + /// publish no fallback deployment rather than one that always fails. + /// + public bool Enabled { get; set; } = true; + + /// + /// Which backend contract to use. Defaults to the + /// Microsoft.Extensions.AI abstraction. + /// + public InferenceBackendKind Kind { get; set; } = InferenceBackendKind.ChatClient; + + /// + /// Audience a workload-identity token is requested for. + /// + public string TokenAudience { get; set; } = string.Empty; + + /// + /// Where inference runs. Surfaced to clients as + /// DeploymentType.InferenceLocation. + /// + public InferenceSite Site { get; set; } = InferenceSite.OnServer; + + /// + /// Base address of the endpoint. For an on-device runtime this is loopback; + /// for a hosted service it is the service address. + /// + public string EndpointUri { get; set; } = "http://localhost:5273/"; + + /// Path of the chat completions operation. + public string ChatCompletionsPath { get; set; } = "v1/chat/completions"; + + /// Path probed to establish reachability. + public string ProbePath { get; set; } = "v1/models"; + + /// How the Server authenticates itself. + public BackendAuthentication Authentication { get; set; } = BackendAuthentication.Anonymous; + + /// + /// NAME of the credential, never the credential. Published as + /// ModelSourceType.CredentialReference, where a client reading it + /// learns which credential is used and nothing about what it is. + /// + public string CredentialReference { get; set; } = string.Empty; + + /// Header an API key is presented in. + public string ApiKeyHeader { get; set; } = "api-key"; + + /// Directory a credential Secret is mounted at. + public string CredentialDirectory { get; set; } = "/var/run/secrets/ai"; + + /// + /// Jurisdiction the endpoint processes data in, published as + /// DeploymentType.DataJurisdiction. + /// + public string DataJurisdiction { get; set; } = "on-premises"; + + /// + /// Whether calling this backend sends input outside the operator's boundary. + /// A deployment reaching a hosted service sets this true; encryption does not + /// make it false, because the question is where the data goes and not who can + /// read it in flight. + /// + public bool EgressPermitted { get; set; } + + /// + /// Whether the endpoint retains input beyond serving the request. Where this + /// cannot be established it is reported true, because the assumption that + /// keeps data in is the one that is safe to be wrong about. + /// + public bool RetainsInput { get; set; } = true; + + /// + /// Largest payload carried inline through Invoke, in bytes. Beyond it + /// a client uses BeginTransfer. + /// + public uint MaxInlinePayloadSize { get; set; } = 65536; + + /// Models this backend offers. + public IList Models { get; } = new List(); + } +} diff --git a/src/Opc.Ua.AI.Inference/InferenceBackends.cs b/src/Opc.Ua.AI.Inference/InferenceBackends.cs new file mode 100644 index 0000000000..0f513bc9ed --- /dev/null +++ b/src/Opc.Ua.AI.Inference/InferenceBackends.cs @@ -0,0 +1,75 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +namespace Opc.Ua.AI.Inference +{ + /// + /// The backends behind the deployments this Server publishes. + /// + /// + /// + /// Two deployments mean two backends, and that is the whole point rather than + /// an implementation detail. A fallback reached through the same client, the + /// same connection and the same credentials as the primary is not a fallback - + /// it is a retry, and it fails for every reason the primary just failed for. + /// + /// + /// Keeping them as separate instances is what makes it possible to say, and to + /// test, that the two are independently reachable. + /// + /// + public sealed class InferenceBackends + { + /// + /// Creates the set. + /// + /// Backend behind the primary deployment. + /// + /// Backend behind the fallback deployment, or null when this Server + /// publishes no fallback. + /// + public InferenceBackends(IInferenceBackend primary, IInferenceBackend? fallback = null) + { + Primary = primary ?? throw new ArgumentNullException(nameof(primary)); + Fallback = fallback; + } + + /// + /// Backend behind the primary deployment. + /// + public IInferenceBackend Primary { get; } + + /// + /// Backend behind the fallback deployment, if there is one. + /// + public IInferenceBackend? Fallback { get; } + } +} diff --git a/src/Opc.Ua.AI.Inference/NugetREADME.md b/src/Opc.Ua.AI.Inference/NugetREADME.md new file mode 100644 index 0000000000..c307d40b58 --- /dev/null +++ b/src/Opc.Ua.AI.Inference/NugetREADME.md @@ -0,0 +1,20 @@ +# Inference backends for OPC UA AI Model Management + +The `IInferenceBackend` contract, and backends that implement it. One abstraction covers a hosted service and an on-device runtime, which is the property clause 8.1 asserts: where inference runs changes the trust boundary and the latency and nothing else. + +Part of the [OPC UA .NET Standard](https://github.com/OPCFoundation/UA-.NETStandard) stack. + +> **Draft.** The *OPC UA - AI Model Management and Inference* companion +> specification is a working draft. Its namespace URI and every NodeId are +> provisional, and every ObjectType and BrowseName can change when the working +> group publishes. + +## Documentation + +See the [AI Model Management guide](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/samples/AI/README.md) +for the example: `ModelManagementServer` publishes a catalogue and +routes inference, and `ModelManagementClient` walks it. + +## License + +MIT - see the [license](https://opcfoundation.org/license/mit.html). diff --git a/src/Opc.Ua.AI.Inference/Opc.Ua.AI.Inference.csproj b/src/Opc.Ua.AI.Inference/Opc.Ua.AI.Inference.csproj new file mode 100644 index 0000000000..7a448ca2bb --- /dev/null +++ b/src/Opc.Ua.AI.Inference/Opc.Ua.AI.Inference.csproj @@ -0,0 +1,44 @@ + + + $(AssemblyPrefix).AI.Inference + + net10.0;net9.0;net8.0 + $(CustomTestTarget) + net10.0 + true + $(PackagePrefix).Opc.Ua.AI.Inference + Opc.Ua.AI.Inference + $(NoWarn);CS1591;CS0108 + enable + Inference backends for the OPC UA AI Model Management and Inference (draft) companion specification: the IInferenceBackend contract one abstraction covers a hosted service and an on-device runtime with, a Microsoft.Extensions.AI IChatClient backend, an OpenAI-compatible REST backend, and credential resolvers that keep secret material out of the address space. + true + NugetREADME.md + true + + false + false + + + $(PackageId).Debug + + + + + + + + + + + + + + + + diff --git a/src/Opc.Ua.AI.Server/AiLearningSampleKind.cs b/src/Opc.Ua.AI.Server/AiLearningSampleKind.cs new file mode 100644 index 0000000000..a52019fc85 --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiLearningSampleKind.cs @@ -0,0 +1,47 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.AI.Server +{ + /// + /// Classifies a learning sample submitted to the server-side accounting API. + /// + public enum AILearningSampleKind + { + /// + /// The sample carries an observation or corrected geometry. + /// + Positive, + + /// + /// The sample is an empty or retracted observation and still counts. + /// + Negative + } +} diff --git a/src/Opc.Ua.AI.Server/AiNodeManager.AddressSpace.cs b/src/Opc.Ua.AI.Server/AiNodeManager.AddressSpace.cs new file mode 100644 index 0000000000..f06ed21df2 --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiNodeManager.AddressSpace.cs @@ -0,0 +1,361 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using AIRefs = Opc.Ua.AI.ReferenceTypeIds; +using BrowseNames = Opc.Ua.AI.BrowseNames; +using ObjectIds = Opc.Ua.ObjectIds; +using ReferenceTypeIds = Opc.Ua.ReferenceTypeIds; + +namespace Opc.Ua.AI.Server +{ + public sealed partial class AINodeManager + { + /// + public override async ValueTask CreateAddressSpaceAsync( + IDictionary> externalReferences, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(externalReferences); + + await base.CreateAddressSpaceAsync(externalReferences, cancellationToken) + .ConfigureAwait(false); + + lock (m_sync) + { + try + { + CreateAIAddressSpace(externalReferences); + } + catch (Exception ex) + { + // The Server reports a start-up failure as a bare status code, so + // without this the one piece of information needed to fix it - + // where it happened - is discarded before anyone sees it. + m_logger.LogError(ex, "Failed to build the AI address space."); + throw; + } + } + } + + private void CreateAIAddressSpace( + IDictionary> externalReferences) + { + // The model already declares the entry point - ns=2;i=7001, parented to + // the Server Object - so it is adopted rather than built again. Creating + // a second one leaves two Objects named AiModelManagement under the + // Server, one populated and one empty, and which of them a client finds + // depends on browse order. The empty one makes the Server look as though + // it publishes no models at all. + m_root = FindPredefinedNode( + new NodeId(Opc.Ua.AI.Objects.AiModelManagement, NamespaceIndex)); + + Child>(m_root, BrowseNames.SpecificationVersion) + .Value = SpecificationVersion; + + // Materialised now, before anything is indexed. Jobs is optional, so it + // does not exist until something asks for it - and the first thing to + // ask is BeginTransfer, long after registration, which would leave the + // folder visible in a Browse of the root and unresolvable when browsed. + Child(m_root, BrowseNames.Jobs); + + BuildModels(); + BuildDeployments(); + + if (m_options.EnableLearningLoop) + { + BuildLearningJob(); + } + + if (m_options.EnableCatalogue) + { + BuildCatalogue(); + } + + // One call: AddPredefinedNode walks the children, so the tree has to be + // finished before it is registered, not registered as it is built. + AddPredefinedNodeSynchronously(m_root); + } + + /// + /// The release of the companion specification this Server implements. + /// + public const string SpecificationVersion = "0.2.0"; + + private static void AddExternalReference( + NodeId sourceId, + NodeId referenceType, + bool isInverse, + NodeId targetId, + IDictionary> externalReferences) + { + if (!externalReferences.TryGetValue(sourceId, out IList? references)) + { + references = new List(); + externalReferences[sourceId] = references; + } + + references.Add(new NodeStateReference(referenceType, isInverse, targetId)); + } + + /// + /// Publishes the models this Server can run. + /// + /// + /// A model carries the identity and the digest at which a provenance walk + /// terminates. The nameplate answers which artefact this is; whether it + /// ought to be running here is a different question, asked by different + /// people, and answered by the card. + /// + private void BuildModels() + { + IList declared = m_backendOptions.Models; + + BackendModel primary = declared.Count > 0 ? declared[0] : DefaultModel; + m_primaryModel = CreateModel(primary, "PrimaryModel"); + + if (!m_options.EnableFallback) + { + return; + } + + BackendModel secondary = declared.Count > 1 + ? declared[1] + : primary with + { + Name = primary.Name + "-compact", + // A synthesised stand-in is a DIFFERENT artefact, so it cannot + // inherit the primary's digest. A provenance walk from a fallback + // answer would otherwise terminate at the primary's digest and + // attribute the answer to weights that never ran - which is + // precisely the comparison a digest exists to support. + Digest = default, + DigestAlgorithm = string.Empty, + Version = primary.Version, + Quantization = "unspecified" + }; + + m_fallbackModel = CreateModel(secondary, "FallbackModel"); + } + + private static BackendModel DefaultModel { get; } = new() + { + Publisher = "sample", + Name = "primary-model", + Version = "1.0.0", + TaskKind = "chat", + Framework = "rest-chat-completions" + }; + + private ModelState CreateModel(BackendModel source, string browseName) + { + // Constructed without a parent on purpose. AddChild assigns the parent + // AND the reference type that makes the child browsable, but it only + // does so when the parent actually changes - so a node handed its parent + // in the constructor is indexed by the Server and invisible to a client, + // which is a great deal harder to notice than an outright failure. + var model = new ModelState(null); + model.Create( + SystemContext, + NodeId.Null, + new QualifiedName(browseName, NamespaceIndex), + new LocalizedText(source.Name), + true); + + Child>(model, BrowseNames.ModelId).Value = + FormattableString.Invariant( + $"{source.Publisher}/{source.Name}:{source.Version}"); + Child>(model, BrowseNames.Name).Value = + new LocalizedText(source.Name); + Child>(model, BrowseNames.Version).Value = source.Version; + Child>(model, BrowseNames.Publisher).Value = source.Publisher; + Child>(model, BrowseNames.Framework).Value = source.Framework; + Child>(model, BrowseNames.TaskKind).Value = source.TaskKind; + + if (!string.IsNullOrEmpty(source.Quantization)) + { + Child>(model, BrowseNames.Quantization).Value = + source.Quantization; + } + + // Digest and DigestAlgorithm are Mandatory because a provenance walk + // terminates at them. This sample never holds the artefact, so it + // publishes what the backend declares; where a backend declares nothing, + // an empty digest is the honest answer and a fabricated one is not. + Child>(model, BrowseNames.Digest).Value = + source.Digest.Length > 0 + ? new ByteString(source.Digest.ToArray()) + : ByteString.Empty; + Child>(model, BrowseNames.DigestAlgorithm).Value = + source.DigestAlgorithm; + + Child(m_root!, BrowseNames.Models).AddChild(model); + return model; + } + + /// + /// Publishes the deployments that run those models. + /// + /// + /// The primary runs wherever the backend is configured to reach. The + /// fallback runs on the Server, because a fallback that needed the same + /// network as the primary would not be one. What matters in the fallback + /// case is not that an answer arrives but that ModelUsed says which + /// model produced it. + /// + private void BuildDeployments() + { + m_primary = CreateDeployment( + m_options.PrimaryDeploymentId, + "PrimaryDeployment", + m_primaryModel!, + m_backendOptions); + + if (!m_options.EnableFallback || m_fallbackModel is null) + { + return; + } + + // From the FALLBACK's own configuration. Publishing the primary's site, + // jurisdiction and egress here would describe a deployment that does not + // exist: an operator who points the fallback at a cloud endpoint would + // get a Server routing payloads off the machine while telling every + // client InferenceLocation=OnServer and EgressPermitted=false. + m_fallback = CreateDeployment( + m_options.FallbackDeploymentId, + "FallbackDeployment", + m_fallbackModel, + m_fallbackBackendOptions); + + Child>(m_primary, BrowseNames.FallbackPolicy) + .Value = FallbackPolicyEnum.FallBackTo; + m_primary.AddReference( + RefType(AIRefs.FallsBackTo), false, m_fallback.NodeId); + m_fallback.AddReference( + RefType(AIRefs.FallsBackTo), true, m_primary.NodeId); + } + + /// + /// Resolves a reference type this model declares to a NodeId in this Server. + /// + /// + /// The generated identifiers are ExpandedNodeIds carrying a namespace URI, + /// because a model does not know what index a Server will give it. The + /// translation has to happen against the Server's own namespace table. + /// + private NodeId RefType(ExpandedNodeId referenceTypeId) + { + return ExpandedNodeId.ToNodeId(referenceTypeId, Server.NamespaceUris); + } + + private DeploymentState CreateDeployment( + string deploymentId, + string browseName, + ModelState model, + InferenceBackendOptions backend) + { + InferenceLocationEnum site = MapSite(backend.Site); + var deployment = new DeploymentState(null); + deployment.Create( + SystemContext, + NodeId.Null, + new QualifiedName(browseName, NamespaceIndex), + new LocalizedText(deploymentId), + true); + + Child>(deployment, BrowseNames.DeploymentId).Value = + deploymentId; + Child>( + deployment, BrowseNames.InferenceLocation).Value = site; + Child>(deployment, BrowseNames.State).Value = + DeploymentStateEnum.Ready; + Child>( + deployment, BrowseNames.VersionBinding).Value = VersionBindingEnum.Pinned; + + // Fail is the safe default: a caller told that nothing happened can + // decide what to do, and deciding is frequently its job. + Child>( + deployment, BrowseNames.FallbackPolicy).Value = FallbackPolicyEnum.Fail; + + // Where the data goes. Egress is not made false by encryption, which + // answers who can read data in flight and not where the data went. + Child>(deployment, BrowseNames.DataJurisdiction).Value = + backend.DataJurisdiction; + Child>(deployment, BrowseNames.EgressPermitted).Value = + backend.EgressPermitted; + Child>(deployment, BrowseNames.RetainsInput).Value = + backend.RetainsInput; + + // Published before a client calls rather than discovered from a + // rejection: the real bound is the smallest of several limits, and a + // client can see none of them. + Child>(deployment, BrowseNames.MaxInlinePayloadSize).Value = + backend.MaxInlinePayloadSize; + + Child>(deployment, BrowseNames.Reachability) + .Value = ReachabilityEnum.Unknown; + Child>(deployment, BrowseNames.ConsecutiveFailures).Value = 0; + + if (site != InferenceLocationEnum.OnServer && + !string.IsNullOrEmpty(backend.EndpointUri)) + { + // The endpoint, never the credential. The address space says where + // the Server goes; what it presents on arrival stays out of it. + Child>(deployment, BrowseNames.EndpointUri).Value = + backend.EndpointUri; + } + + // Exactly one UsesModel reference: the only defined path from a running + // deployment to the artefact its answers depend on. + deployment.AddReference(RefType(AIRefs.UsesModel), false, model.NodeId); + model.AddReference(RefType(AIRefs.UsesModel), true, deployment.NodeId); + + Child(m_root!, BrowseNames.Deployments).AddChild(deployment); + WireDeploymentMethods(deployment); + return deployment; + } + + private static InferenceLocationEnum MapSite(InferenceSite site) + { + return site switch + { + InferenceSite.Cloud => InferenceLocationEnum.Cloud, + InferenceSite.EdgeOffServer => InferenceLocationEnum.EdgeOffServer, + _ => InferenceLocationEnum.OnServer + }; + } + } +} diff --git a/src/Opc.Ua.AI.Server/AiNodeManager.Catalogue.cs b/src/Opc.Ua.AI.Server/AiNodeManager.Catalogue.cs new file mode 100644 index 0000000000..1eed7e4dbe --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiNodeManager.Catalogue.cs @@ -0,0 +1,258 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using AIRefs = Opc.Ua.AI.ReferenceTypeIds; +using BrowseNames = Opc.Ua.AI.BrowseNames; +using ObjectIds = Opc.Ua.ObjectIds; +using ReferenceTypeIds = Opc.Ua.ReferenceTypeIds; + +namespace Opc.Ua.AI.Server +{ + public sealed partial class AINodeManager + { + private ModelSourceState? m_source; + + /// + /// Publishes the source this Server consumes models from. + /// + /// + /// + /// A source is how the address space says "the models here are not mine". + /// It names an endpoint, the dialect that endpoint speaks and - by + /// reference, never by value - the credential the Server presents. Without + /// it a client browsing a deployment could not tell a model running on this + /// machine from one running in somebody else's data centre, which is the + /// distinction most operational questions turn on. + /// + /// + /// TestConnection exists so that a commissioning engineer can + /// establish that the endpoint, the credential and the network policy are + /// right before a deployment depends on them, rather than learning it from + /// the first inference that mattered. + /// + /// + private void BuildCatalogue() + { + m_source = new ModelSourceState(null); + m_source.Create( + SystemContext, + NodeId.Null, + new QualifiedName("ModelSource", NamespaceIndex), + new LocalizedText(m_backendOptions.EndpointUri), + true); + + Child>(m_source, BrowseNames.SourceId).Value = + m_options.SourceId; + Child>(m_source, BrowseNames.EndpointUri).Value = + m_backendOptions.EndpointUri; + Child>(m_source, BrowseNames.ApiDialect).Value = + ApiDialectEnum.RestChatCompletions; + Child>( + m_source, BrowseNames.AuthenticationKind).Value = + ToAuthenticationKind(m_backendOptions.Authentication); + Child>(m_source, BrowseNames.Reachability).Value = + ReachabilityEnum.Unknown; + + // The reference, never the secret. A client is entitled to know which + // credential a Server uses so it can tell whether the right one is + // configured; it is not entitled to the credential, and an address space + // that carried one would hand it to everyone who could browse. + if (!string.IsNullOrEmpty(m_backendOptions.CredentialReference)) + { + Child>(m_source, BrowseNames.CredentialReference).Value = + m_backendOptions.CredentialReference; + } + + Child(m_source, BrowseNames.TestConnection).OnCallAsync = + (context, method, objectId, ct) => TestConnectionAsync(ct); + + Child(m_source, BrowseNames.ListModels).OnCallAsync = + (context, method, objectId, filter, maxResults, continuationPoint, ct) => + ListModelsAsync(filter, maxResults, continuationPoint, ct); + + Child(m_root!, BrowseNames.Sources).AddChild(m_source); + + // Every model this Server publishes came from that source, and saying so + // is what makes the provenance walk terminate somewhere meaningful + // instead of at a name. + NodeId importedFrom = RefType(AIRefs.ImportedFrom); + + foreach (ModelState? model in new[] { m_primaryModel, m_fallbackModel }) + { + if (model is not null) + { + model.AddReference(importedFrom, false, m_source.NodeId); + m_source.AddReference(importedFrom, true, model.NodeId); + } + } + } + + /// + /// Probes the source and records what it found. + /// + private async ValueTask TestConnectionAsync( + CancellationToken ct) + { + BackendProbe probe = await m_backends.Primary.ProbeAsync(ct).ConfigureAwait(false); + + lock (m_sync) + { + if (m_source is not null) + { + Child>(m_source, BrowseNames.Reachability) + .Value = probe.Reachable + ? ReachabilityEnum.Reachable + : ReachabilityEnum.Unreachable; + + if (probe.Reachable) + { + Child>(m_source, BrowseNames.LastSuccessAt) + .Value = DateTime.UtcNow; + Child>( + m_source, BrowseNames.ConsecutiveFailures).Value = 0; + } + else + { + PropertyState failures = Child>( + m_source, BrowseNames.ConsecutiveFailures); + failures.Value++; + } + + m_source.ClearChangeMasks(SystemContext, true); + } + } + + return new TestConnectionMethodStateResult + { + ServiceResult = ServiceResult.Good, + Reachable = probe.Reachable, + Detail = new LocalizedText(probe.Detail ?? string.Empty) + }; + } + + /// + /// Lists what the source offers. + /// + /// + /// Answered from the source itself rather than from what this Server has + /// already imported. The question is what could be deployed, and answering + /// it from local state would only ever return what already had been. + /// + private async ValueTask ListModelsAsync( + string filter, + uint maxResults, + ByteString continuationPoint, + CancellationToken ct) + { + IReadOnlyList models = await m_backends.Primary + .ListModelsAsync( + string.IsNullOrEmpty(filter) ? null : filter, + 0, + ct) + .ConfigureAwait(false); + + // Clause 8.2. The continuation point carries the offset reached so far, + // so an enumeration that MaxResults truncated can be resumed rather than + // permanently losing everything past the bound - which against a public + // catalogue is most of it. An empty point starts at the beginning, and an + // empty point returned means the enumeration is complete. + int offset = ReadContinuationOffset(continuationPoint); + if (offset < 0 || offset > models.Count) + { + return new ListModelsMethodStateResult + { + ServiceResult = StatusCodes.BadContinuationPointInvalid, + Models = ArrayOf.Empty, + ContinuationPointOut = ByteString.Empty + }; + } + + int take = maxResults == 0 ? models.Count - offset : (int)Math.Min(maxResults, (uint)(models.Count - offset)); + var references = new ModelReferenceDataType[take]; + + for (int index = 0; index < take; index++) + { + BackendModel model = models[offset + index]; + references[index] = new ModelReferenceDataType + { + Publisher = model.Publisher, + Name = model.Name, + Version = model.Version + }; + } + + int next = offset + take; + return new ListModelsMethodStateResult + { + ServiceResult = ServiceResult.Good, + Models = new ArrayOf(references), + ContinuationPointOut = next < models.Count + ? ByteString.From(BitConverter.GetBytes(next)) + : ByteString.Empty + }; + } + + /// + /// Reads the offset a continuation point carries, or -1 when it is malformed. + /// + private static int ReadContinuationOffset(ByteString continuationPoint) + { + ReadOnlyMemory bytes = continuationPoint.Memory; + if (bytes.Length == 0) + { + return 0; + } + if (bytes.Length != sizeof(int)) + { + return -1; + } + return BitConverter.ToInt32(bytes.Span); + } + + private static AuthenticationKindEnum ToAuthenticationKind( + BackendAuthentication authentication) + { + return authentication switch + { + BackendAuthentication.ApiKey => AuthenticationKindEnum.ApiKey, + BackendAuthentication.BearerToken => AuthenticationKindEnum.BearerToken, + BackendAuthentication.WorkloadIdentity => + AuthenticationKindEnum.WorkloadIdentity, + _ => AuthenticationKindEnum.Anonymous + }; + } + } +} diff --git a/src/Opc.Ua.AI.Server/AiNodeManager.Jobs.cs b/src/Opc.Ua.AI.Server/AiNodeManager.Jobs.cs new file mode 100644 index 0000000000..bda374ede5 --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiNodeManager.Jobs.cs @@ -0,0 +1,278 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.AI; +using AIRefs = Opc.Ua.AI.ReferenceTypeIds; +using BrowseNames = Opc.Ua.AI.BrowseNames; +using ObjectIds = Opc.Ua.ObjectIds; +using ReferenceTypeIds = Opc.Ua.ReferenceTypeIds; + +namespace Opc.Ua.AI.Server +{ + public sealed partial class AINodeManager + { + /// + /// Starts an inference that outlives the call that asked for it. + /// + /// + /// + /// The job NodeId comes back immediately and the result arrives on the job. + /// This is what makes a long inference usable from a client that cannot hold + /// a call open for minutes - and, more importantly, what makes the result + /// survive the client that requested it disconnecting. + /// + /// + /// AiJobType is a Part 10 ProgramStateMachineType, so the + /// lifecycle is the one clients already know: Ready, Running, and then + /// Halted whether it succeeded or not. Nothing new to learn, which is the + /// reason the specification reused it. + /// + /// + private async ValueTask StartJobAsync( + NodeId objectId, + ByteString payload, + string payloadUri, + string contentType, + CancellationToken ct) + { + ct.ThrowIfCancellationRequested(); + + DeploymentState? deployment = FindDeployment(objectId); + if (deployment is null) + { + return new InvokeAsyncMethodStateResult + { + ServiceResult = StatusCodes.BadNodeIdUnknown, + Job = NodeId.Null + }; + } + + // Clause 8.4, as for Invoke: exactly one of Payload and PayloadUri. + if (payload.IsNull == string.IsNullOrEmpty(payloadUri)) + { + return new InvokeAsyncMethodStateResult + { + ServiceResult = StatusCodes.BadInvalidArgument, + Job = NodeId.Null + }; + } + + InferenceJobState job; + NodeId? stale = null; + byte[] body = payload.Memory.ToArray(); + + lock (m_sync) + { + // The same bound transfers already have. Without it any session that + // can call this Method can grow the address space indefinitely, and + // each job retains its request and response payloads - so the cost + // is not only nodes but the bytes they hold. + if (m_jobs.Count >= m_options.MaxRetainedJobs) + { + stale = ReclaimOldestJob(); + } + + string jobId = Guid.NewGuid().ToString("N", CultureInfo.InvariantCulture); + + job = new InferenceJobState(null); + job.Create( + SystemContext, + NodeId.Null, + new QualifiedName("Job_" + jobId, NamespaceIndex), + new LocalizedText("Job " + jobId), + true); + + Child>(job, BrowseNames.JobId).Value = jobId; + Child>(job, BrowseNames.Deployment).Value = + deployment.NodeId; + Child>(job, BrowseNames.RequestPayload).Value = payload; + Child>(job, BrowseNames.RequestContentType).Value = + contentType; + Child>(job, BrowseNames.StartedAt).Value = + DateTime.UtcNow; + Child>(job, BrowseNames.Progress).Value = 0; + + // Materialised before the node is indexed, for the same reason the + // transfer does it: a member created after the fact is invisible to + // a client, and a job whose result never appears is a worse failure + // than one that fails. + Child>(job, BrowseNames.ResponsePayload); + Child>(job, BrowseNames.ResponseContentType); + Child>(job, BrowseNames.ModelUsed); + Child>(job, BrowseNames.Usage); + Child>(job, BrowseNames.FinishReason); + Child>(job, BrowseNames.LastError); + Child>(job, BrowseNames.FinishedAt); + + job.CurrentState!.Value = new LocalizedText("Running"); + job.CurrentState!.Id!.Value = Opc.Ua.ObjectIds.ProgramStateMachineType_Running; + + Child(m_root!, BrowseNames.Jobs).AddChild(job); + AddPredefinedNodeSynchronously(job); + m_jobs.Add(job.NodeId); + } + + if (stale is not null) + { + // Outside the lock: only DeleteNodeAsync exists, and holding a lock + // across it is what the transfer path had to be fixed for. + await DeleteNodeAsync(SystemContext, stale.Value, ct).ConfigureAwait(false); + } + + // Fire and forget by design: the caller has its NodeId and the result + // belongs to the job. Faults are recorded on the job rather than thrown + // into a void, so an unobserved task cannot swallow one. + _ = Task.Run( + () => RunJobAsync(job, deployment, body, contentType), + CancellationToken.None); + + return new InvokeAsyncMethodStateResult + { + ServiceResult = ServiceResult.Good, + Job = job.NodeId + }; + } + + /// + /// Removes the oldest job to make room, and returns its NodeId to delete. + /// + /// + /// Oldest first rather than oldest finished, because a job that has been + /// running long enough to be the oldest of a full set is not going to + /// finish. A cap that only reclaimed completed jobs would be no cap at all + /// against the case it exists for. + /// + private NodeId? ReclaimOldestJob() + { + if (m_jobs.Count == 0) + { + return null; + } + + NodeId oldest = m_jobs[0]; + m_jobs.RemoveAt(0); + + if (FindPredefinedNode(oldest) is { } node) + { + Child(m_root!, BrowseNames.Jobs).RemoveChild(node); + } + + return oldest; + } + + private async Task RunJobAsync( + InferenceJobState job, + DeploymentState deployment, + byte[] payload, + string contentType) + { + try + { + // The delay exists so a client can observe Running before Halted. + // A job that completes before its NodeId reaches the caller would + // demonstrate nothing about the lifecycle it is here to show. + if (m_options.AsyncInferenceDelay > TimeSpan.Zero) + { + await Task.Delay(m_options.AsyncInferenceDelay).ConfigureAwait(false); + } + + InferenceOutcome outcome = await RunWithFallbackAsync( + deployment, + payload, + contentType, + m_options.TransferInferenceTimeout.TotalMilliseconds, + CancellationToken.None).ConfigureAwait(false); + + lock (m_sync) + { + if (outcome.Result.Ok) + { + Child>(job, BrowseNames.ResponsePayload) + .Value = new ByteString(outcome.Result.Payload.ToArray()); + Child>(job, BrowseNames.ResponseContentType) + .Value = outcome.Result.ContentType; + Child>(job, BrowseNames.ModelUsed).Value = + outcome.ModelUsed; + Child>(job, BrowseNames.Usage).Value = + ToUsage(outcome.Result); + Child>(job, BrowseNames.FinishReason) + .Value = ToFinishReason(outcome.Result.Finish); + } + else + { + Child>(job, BrowseNames.LastError).Value = + new LocalizedText(outcome.Result.Message ?? "Inference failed."); + } + + CompleteJob(job); + } + } +#pragma warning disable CA1031 // a background job records its fault rather than crashing the Server + catch (Exception ex) +#pragma warning restore CA1031 + { + m_logger.LogError(ex, "Asynchronous inference job failed."); + + lock (m_sync) + { + Child>(job, BrowseNames.LastError).Value = + new LocalizedText(ex.Message); + CompleteJob(job); + } + } + } + + /// + /// Moves a job to Halted and stamps when it finished. + /// + /// + /// Halted regardless of outcome, which is the Part 10 lifecycle rather than + /// an opinion about the result: whether the inference succeeded is answered + /// by whether ResponsePayload or LastError is set, not by the + /// state the program ended in. + /// + private void CompleteJob(InferenceJobState job) + { + Child>(job, BrowseNames.Progress).Value = 100; + Child>(job, BrowseNames.FinishedAt).Value = + DateTime.UtcNow; + + job.CurrentState!.Value = new LocalizedText("Halted"); + job.CurrentState!.Id!.Value = Opc.Ua.ObjectIds.ProgramStateMachineType_Halted; + + job.ClearChangeMasks(SystemContext, true); + } + } +} diff --git a/src/Opc.Ua.AI.Server/AiNodeManager.Learning.cs b/src/Opc.Ua.AI.Server/AiNodeManager.Learning.cs new file mode 100644 index 0000000000..666accf092 --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiNodeManager.Learning.cs @@ -0,0 +1,139 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua; +using Opc.Ua.AI; +using BrowseNames = Opc.Ua.AI.BrowseNames; + +namespace Opc.Ua.AI.Server +{ + public sealed partial class AINodeManager + { + /// + /// How many sample identifiers are retained for duplicate detection. + /// + /// + /// A retry arrives close behind the call it repeats, so a window of this + /// size is far more than a duplicate needs while keeping the set bounded on + /// a Server that never restarts. + /// + private const int MaxRetainedLearningSampleIds = 4096; + + /// + /// Records one ground-truth sample against the published learning job. + /// + /// Stable caller-supplied identity of the sample. + /// + /// Whether the sample carried an observation or was a negative example. + /// Both kinds increment SamplesCollected exactly once. + /// + /// Cancels the accounting operation. + /// + /// True when this call added a new sample; false when the sample was already + /// recorded or no learning job is published. + /// + /// + /// Duplicate detection is per-process and covers the most recent + /// identifiers. Both the counter + /// and the set start empty after a restart, so a sample replayed across one + /// is counted again. + /// + /// + public ValueTask RecordLearningSampleAsync( + string sampleId, + AILearningSampleKind sampleKind = AILearningSampleKind.Positive, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(sampleId); + cancellationToken.ThrowIfCancellationRequested(); + + if (sampleKind is not AILearningSampleKind.Positive and not AILearningSampleKind.Negative) + { + throw new ArgumentOutOfRangeException(nameof(sampleKind)); + } + + lock (m_sync) + { + if (m_learningJob is null || !m_learningSampleIds.Add(sampleId)) + { + return new ValueTask(false); + } + + // Bounded, because the identifiers arrive from callers and a Server + // that ran for a year would otherwise retain one string per sample + // for no benefit. Evicting the oldest keeps the guarantee where it + // matters - a retry follows its original closely - and the window is + // stated rather than implied. Idempotency is per-process either way: + // the count and the set are both rebuilt empty on restart. + m_learningSampleOrder.Enqueue(sampleId); + while (m_learningSampleOrder.Count > MaxRetainedLearningSampleIds) + { + m_learningSampleIds.Remove(m_learningSampleOrder.Dequeue()); + } + + PropertyState samples = + Child>(m_learningJob, BrowseNames.SamplesCollected); + samples.Value++; + m_learningJob.ClearChangeMasks(SystemContext, true); + return new ValueTask(true); + } + } + + /// + /// Publishes the learning job that receives ground-truth sample counts. + /// + private void BuildLearningJob() + { + m_learningJob = new LearningJobState(null); + m_learningJob.Create( + SystemContext, + NodeId.Null, + new QualifiedName("LearningSamples", NamespaceIndex), + new LocalizedText("Learning samples"), + true); + + Child>(m_learningJob, BrowseNames.JobId).Value = + "learning-samples"; + Child>(m_learningJob, BrowseNames.State).Value = + LearningJobStateEnum.Collecting; + Child>(m_learningJob, BrowseNames.BaseModel).Value = + m_primaryModel?.NodeId ?? NodeId.Null; + Child>(m_learningJob, BrowseNames.SamplesCollected).Value = 0; + + m_learningJob.CurrentState!.Value = new LocalizedText("Running"); + m_learningJob.CurrentState.Id!.Value = + global::Opc.Ua.ObjectIds.ProgramStateMachineType_Running; + + Child(m_root!, BrowseNames.LearningJobs).AddChild(m_learningJob); + } + } +} diff --git a/src/Opc.Ua.AI.Server/AiNodeManager.Methods.cs b/src/Opc.Ua.AI.Server/AiNodeManager.Methods.cs new file mode 100644 index 0000000000..ae6e32ce2e --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiNodeManager.Methods.cs @@ -0,0 +1,309 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using AIRefs = Opc.Ua.AI.ReferenceTypeIds; +using BrowseNames = Opc.Ua.AI.BrowseNames; +using ObjectIds = Opc.Ua.ObjectIds; +using ReferenceTypeIds = Opc.Ua.ReferenceTypeIds; + +namespace Opc.Ua.AI.Server +{ + public sealed partial class AINodeManager + { + /// + /// Attaches handlers to the methods a deployment offers. + /// + /// + /// Only the asynchronous handlers are attached. Inference is a network call + /// on any deployment worth having, and blocking a Server thread on one would + /// make throughput a function of model latency. + /// + private void WireDeploymentMethods(DeploymentState deployment) + { + Child(deployment, BrowseNames.Invoke).OnCallAsync = + (context, method, objectId, payload, payloadUri, contentType, parameters, timeout, ct) => + InvokeAsync(objectId, payload, payloadUri, contentType, timeout, ct); + + Child(deployment, BrowseNames.InvokeAsync).OnCallAsync = + (context, method, objectId, payload, payloadUri, contentType, parameters, ct) => + StartJobAsync(objectId, payload, payloadUri, contentType, ct); + + Child(deployment, BrowseNames.GetCapabilities) + .OnCallAsync = + (context, method, objectId, ct) => GetCapabilitiesAsync(objectId, ct); + + Child(deployment, BrowseNames.BeginTransfer).OnCallAsync = + (context, method, objectId, contentType, requestSize, ct) => + BeginTransferAsync(objectId, contentType, requestSize, ct); + } + + /// + /// Runs one inference and reports which model answered. + /// + /// + /// + /// The oversize check happens before the backend is touched. A payload the + /// Server has already said it will not take inline should be refused by the + /// Server, not discovered by a remote endpoint - and the refusal names the + /// transfer that will carry it, so a caller that reads the answer can act on + /// it without a second round trip to work out what to do. + /// + /// + /// When the primary fails and policy allows it, the fallback answers and + /// ModelUsed names the fallback's model. That substitution is the + /// entire reason the output exists: a caller that cannot see which model + /// produced a result cannot tell a degraded answer from a good one, and a + /// fallback that answers silently looks exactly like a healthy primary. + /// + /// + private async ValueTask InvokeAsync( + NodeId objectId, + ByteString payload, + string payloadUri, + string contentType, + double timeout, + CancellationToken ct) + { + DeploymentState? deployment = FindDeployment(objectId); + if (deployment is null) + { + return new InvokeMethodStateResult + { + ServiceResult = StatusCodes.BadNodeIdUnknown + }; + } + + // Clause 8.4. Exactly one of Payload and PayloadUri says where the input + // is; a call supplying both would not say which was read, and one + // supplying neither carries no input at all. + if (payload.IsNull == string.IsNullOrEmpty(payloadUri)) + { + return new InvokeMethodStateResult + { + ServiceResult = StatusCodes.BadInvalidArgument + }; + } + + ReadOnlyMemory body = payload.Memory; + + if (body.Length > m_backendOptions.MaxInlinePayloadSize) + { + BeginTransferMethodStateResult transfer = + await BeginTransferAsync( + objectId, + contentType, + (ulong)body.Length, + ct).ConfigureAwait(false); + + if (!transfer.Accepted) + { + // The refusal travels. Reporting Good with a null Transfer would + // tell the caller "too large for inline, use transfer null", + // which is indistinguishable from a real transfer it cannot + // find - and once MaxConcurrentTransfers is reached that is the + // ordinary outcome rather than an edge case. + return new InvokeMethodStateResult + { + ServiceResult = transfer.ServiceResult, + ResponsePayload = ByteString.Empty, + ResponseContentType = string.Empty, + ModelUsed = NodeId.Null, + Usage = new UsageDataType(), + SafetyAssessment = ArrayOf.Empty, + TransferRequired = true, + Transfer = NodeId.Null + }; + } + + return new InvokeMethodStateResult + { + ServiceResult = ServiceResult.Good, + ResponsePayload = ByteString.Empty, + ResponseContentType = string.Empty, + ModelUsed = NodeId.Null, + Usage = new UsageDataType(), + FinishReason = FinishReasonEnum.Length, + SafetyAssessment = ArrayOf.Empty, + TransferRequired = true, + Transfer = transfer.Transfer + }; + } + + InferenceOutcome outcome = await RunWithFallbackAsync( + deployment, + body, + contentType, + TimeoutOrDefault(timeout), + ct).ConfigureAwait(false); + + if (!outcome.Result.Ok) + { + return new InvokeMethodStateResult + { + ServiceResult = outcome.Result.RetryAfter > TimeSpan.Zero + ? StatusCodes.BadTooManyOperations + : StatusCodes.BadRequestNotAllowed, + RetryAfter = outcome.Result.RetryAfter.TotalMilliseconds, + ModelUsed = NodeId.Null, + Usage = new UsageDataType(), + SafetyAssessment = ArrayOf.Empty, + ResponsePayload = ByteString.Empty, + ResponseContentType = string.Empty, + Transfer = NodeId.Null + }; + } + + return new InvokeMethodStateResult + { + ServiceResult = ServiceResult.Good, + ResponsePayload = new ByteString(outcome.Result.Payload.ToArray()), + ResponseContentType = outcome.Result.ContentType, + ModelUsed = outcome.ModelUsed, + Usage = ToUsage(outcome.Result), + FinishReason = ToFinishReason(outcome.Result.Finish), + SafetyAssessment = ArrayOf.Empty, + RetryAfter = 0, + TransferRequired = false, + Transfer = NodeId.Null + }; + } + + /// + /// What a deployment can do, answered from the backend rather than from + /// configuration. + /// + /// + /// Configuration says what an operator believes; the backend says what is + /// actually there. Where they disagree the second one is the one a caller + /// needs, so the probe is what decides Reachable here. + /// + private async ValueTask GetCapabilitiesAsync( + NodeId objectId, + CancellationToken ct) + { + DeploymentState? deployment = FindDeployment(objectId); + if (deployment is null) + { + return new GetCapabilitiesMethodStateResult + { + ServiceResult = StatusCodes.BadNodeIdUnknown, + Capabilities = ArrayOf.Empty + }; + } + + IInferenceBackend backend = BackendFor(deployment); + BackendProbe probe = await backend.ProbeAsync(ct).ConfigureAwait(false); + + IReadOnlyList models = probe.Reachable + ? await backend.ListModelsAsync(null, 16, ct).ConfigureAwait(false) + : Array.Empty(); + + var capabilities = new List + { + new() { Name = "reachable", Supported = probe.Reachable }, + new() { Name = "inline-payload", Supported = true }, + new() { Name = "chunked-transfer", Supported = true }, + new() + { + Name = "async-inference", + Supported = true + } + }; + + var seen = new HashSet(StringComparer.Ordinal); + foreach (BackendModel model in models) + { + foreach (string capability in model.Capabilities) + { + if (seen.Add(capability)) + { + capabilities.Add(new CapabilityDataType + { + Name = capability, + Supported = true + }); + } + } + } + + UpdateReachability(deployment, probe.Reachable); + + return new GetCapabilitiesMethodStateResult + { + ServiceResult = ServiceResult.Good, + Capabilities = new ArrayOf(capabilities.ToArray()) + }; + } + + private static double TimeoutOrDefault(double timeoutMilliseconds) + { + return timeoutMilliseconds > 0 ? timeoutMilliseconds : 30000; + } + + private static UsageDataType ToUsage(InferenceResult result) + { + return new UsageDataType + { + UnitKind = result.UsageUnit, + InputUnits = result.InputUnits, + OutputUnits = result.OutputUnits, + TotalUnits = result.TotalUnits + }; + } + + /// + /// Maps the backend's finish reason onto the model's. + /// + /// + /// The two enumerations carry the same members, which is not a coincidence: + /// the backend abstraction was written against the specification. Mapping + /// explicitly rather than casting keeps it that way, because a cast would + /// silently produce nonsense the day either side gained a member. + /// + private static FinishReasonEnum ToFinishReason(InferenceFinish finish) + { + return finish switch + { + InferenceFinish.Length => FinishReasonEnum.Length, + InferenceFinish.ToolCall => FinishReasonEnum.ToolCall, + InferenceFinish.Filtered => FinishReasonEnum.Filtered, + InferenceFinish.Cancelled => FinishReasonEnum.Cancelled, + InferenceFinish.Error => FinishReasonEnum.Error, + _ => FinishReasonEnum.Stop + }; + } + } +} diff --git a/src/Opc.Ua.AI.Server/AiNodeManager.Routing.cs b/src/Opc.Ua.AI.Server/AiNodeManager.Routing.cs new file mode 100644 index 0000000000..4a03564c1a --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiNodeManager.Routing.cs @@ -0,0 +1,292 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using AIRefs = Opc.Ua.AI.ReferenceTypeIds; +using BrowseNames = Opc.Ua.AI.BrowseNames; +using ObjectIds = Opc.Ua.ObjectIds; +using ReferenceTypeIds = Opc.Ua.ReferenceTypeIds; + +namespace Opc.Ua.AI.Server +{ + public sealed partial class AINodeManager + { + /// + /// One inference, and the model that actually produced it. + /// + /// + /// The two travel together deliberately. Every path that produces a result + /// has to produce the model NodeId alongside it, so there is no way to + /// return an answer while forgetting to say where it came from. + /// + private readonly record struct InferenceOutcome( + InferenceResult Result, + NodeId ModelUsed); + + /// + /// Calls the deployment's backend, and the fallback's if policy allows it. + /// + /// + /// + /// Substitution happens only when FallbackPolicy says + /// FallBackTo. Fail is the default and it means what it says: + /// a caller that asked this deployment for an answer gets this deployment's + /// answer or none, because for some callers a different model's answer is + /// worse than no answer at all. + /// + /// + /// A safety refusal is not a failure and is never retried elsewhere. The + /// content filter declined, which is a result; sending the same payload to a + /// second model until one accepts it would turn a policy into an obstacle. + /// + /// + private async ValueTask RunWithFallbackAsync( + DeploymentState deployment, + ReadOnlyMemory payload, + string contentType, + double timeoutMilliseconds, + CancellationToken ct) + { + IInferenceBackend backend = BackendFor(deployment); + ModelState? model = ModelFor(deployment); + + var request = new InferenceRequest + { + Model = ModelNameFor(model), + Payload = payload, + ContentType = contentType, + Timeout = TimeSpan.FromMilliseconds(timeoutMilliseconds) + }; + + InferenceResult result = await backend + .InvokeAsync(request, ct) + .ConfigureAwait(false); + + RecordAttempt(deployment, result.Ok); + + if (result.Ok || !ShouldFallBack(deployment, result)) + { + return new InferenceOutcome(result, ModelIdOf(model)); + } + + DeploymentState? substitute = m_fallback; + IInferenceBackend? fallbackBackend = m_backends.Fallback; + + if (substitute is null || + fallbackBackend is null || + ReferenceEquals(substitute, deployment)) + { + return new InferenceOutcome(result, ModelIdOf(model)); + } + + m_logger.LogWarning( + "Primary deployment failed ({Reason}); falling back.", + result.Message ?? "no detail"); + + ModelState? substituteModel = ModelFor(substitute); + + InferenceResult fallbackResult = await fallbackBackend + .InvokeAsync( + request with { Model = ModelNameFor(substituteModel) }, + ct) + .ConfigureAwait(false); + + RecordAttempt(substitute, fallbackResult.Ok); + + // The substituted model, not the one that was asked for. Reporting the + // requested model here would be the single most damaging thing this + // sample could get wrong, because everything downstream would look + // correct while attributing answers to a model that never ran. + return new InferenceOutcome(fallbackResult, ModelIdOf(substituteModel)); + } + + /// + /// Whether a failed call may be retried on the fallback. + /// + private bool ShouldFallBack(DeploymentState deployment, InferenceResult result) + { + if (result.Finish == InferenceFinish.Filtered) + { + return false; + } + + PropertyState? policy = + deployment.FindChild( + SystemContext, + new QualifiedName(BrowseNames.FallbackPolicy, NamespaceIndex)) as + PropertyState; + + return policy?.Value == FallbackPolicyEnum.FallBackTo; + } + + /// + /// Which backend sits behind a deployment. + /// + private IInferenceBackend BackendFor(DeploymentState deployment) + { + return ReferenceEquals(deployment, m_fallback) && m_backends.Fallback is not null + ? m_backends.Fallback + : m_backends.Primary; + } + + /// + /// Which model a deployment runs, followed through UsesModel rather + /// than remembered separately. + /// + /// + /// The reference is the specification's answer to the provenance question, + /// so following it here means the sample's own routing is exercising the + /// same path an auditing client walks. A private lookup table beside it + /// could disagree with the address space, and would eventually. + /// + private ModelState? ModelFor(DeploymentState deployment) + { + var references = new List(); + deployment.GetReferences(SystemContext, references); + + NodeId usesModel = RefType(AIRefs.UsesModel); + + foreach (IReference reference in references) + { + if (reference.IsInverse || reference.ReferenceTypeId != usesModel) + { + continue; + } + + NodeId targetId = ExpandedNodeId.ToNodeId( + reference.TargetId, + Server.NamespaceUris); + + if (PredefinedNodes.TryGetValue(targetId, out NodeState? node) && + node is ModelState model) + { + return model; + } + } + + return null; + } + + private NodeId ModelIdOf(ModelState? model) + { + return model?.NodeId ?? NodeId.Null; + } + + /// + /// The name the backend knows a model by. + /// + /// + /// The address space identifies a model as publisher/name:version, which is + /// the durable identity. An endpoint usually wants the bare name, so the + /// translation happens here rather than by publishing the endpoint's name as + /// though it were the identity. + /// + private string ModelNameFor(ModelState? model) + { + if (model is null) + { + return string.Empty; + } + + var name = model.FindChild( + SystemContext, + new QualifiedName(BrowseNames.Name, NamespaceIndex)) as PropertyState; + + return name is null ? string.Empty : name.Value.Text ?? string.Empty; + } + + /// + /// Resolves the deployment a method was invoked on. + /// + private DeploymentState? FindDeployment(NodeId objectId) + { + return PredefinedNodes.TryGetValue(objectId, out NodeState? node) + ? node as DeploymentState + : null; + } + + /// + /// Updates the health a deployment publishes after an attempt. + /// + /// + /// ConsecutiveFailures resets on success rather than decrementing, + /// because the question it answers is "is it failing now", not "how often + /// has it ever failed". + /// + private void RecordAttempt(DeploymentState deployment, bool succeeded) + { + var failures = deployment.FindChild( + SystemContext, + new QualifiedName(BrowseNames.ConsecutiveFailures, NamespaceIndex)) as + PropertyState; + + if (failures is not null) + { + failures.Value = succeeded ? 0 : failures.Value + 1; + failures.ClearChangeMasks(SystemContext, false); + } + + if (succeeded) + { + var lastSuccess = deployment.FindChild( + SystemContext, + new QualifiedName(BrowseNames.LastSuccessAt, NamespaceIndex)) as + PropertyState; + + if (lastSuccess is not null) + { + lastSuccess.Value = DateTime.UtcNow; + lastSuccess.ClearChangeMasks(SystemContext, false); + } + } + + UpdateReachability(deployment, succeeded); + } + + private void UpdateReachability(DeploymentState deployment, bool reachable) + { + if (!(deployment.FindChild( + SystemContext, + new QualifiedName(BrowseNames.Reachability, NamespaceIndex)) is PropertyState state)) + { + return; + } + + state.Value = reachable ? ReachabilityEnum.Reachable : ReachabilityEnum.Unreachable; + state.ClearChangeMasks(SystemContext, false); + } + } +} diff --git a/src/Opc.Ua.AI.Server/AiNodeManager.Transfer.cs b/src/Opc.Ua.AI.Server/AiNodeManager.Transfer.cs new file mode 100644 index 0000000000..8596bd0eaa --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiNodeManager.Transfer.cs @@ -0,0 +1,375 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua; +using Opc.Ua.AI; +using AIRefs = Opc.Ua.AI.ReferenceTypeIds; +using BrowseNames = Opc.Ua.AI.BrowseNames; +using ObjectIds = Opc.Ua.ObjectIds; +using ReferenceTypeIds = Opc.Ua.ReferenceTypeIds; + +namespace Opc.Ua.AI.Server +{ + public sealed partial class AINodeManager + { + /// + /// Opens a chunked exchange for a payload too large to pass inline. + /// + /// + /// + /// The transfer object is created up front and handed back by NodeId, so the + /// caller writes into a thing that already exists rather than negotiating + /// one. Request and Response are Part 5 FileType instances, which is + /// what lets an existing client library move the bytes without learning + /// anything new: the specification did not invent a transfer protocol, + /// because OPC UA already has one. + /// + /// + /// A transfer expires. A caller that opens one and abandons it would + /// otherwise hold Server memory until restart, and inference payloads are + /// exactly the size that makes that matter. + /// + /// + private async ValueTask BeginTransferAsync( + NodeId objectId, + string contentType, + ulong requestSize, + CancellationToken ct) + { + ct.ThrowIfCancellationRequested(); + + DeploymentState? deployment = FindDeployment(objectId); + if (deployment is null) + { + return new BeginTransferMethodStateResult + { + ServiceResult = StatusCodes.BadNodeIdUnknown, + Transfer = NodeId.Null, + Accepted = false + }; + } + + if (requestSize > m_options.MaxTransferSize) + { + // Refused before a byte is accepted. A Server that takes the whole + // payload and then declines has already paid the cost it was trying + // to avoid. + return new BeginTransferMethodStateResult + { + ServiceResult = StatusCodes.BadRequestTooLarge, + Transfer = NodeId.Null, + Accepted = false + }; + } + + await ExpireTransfersAsync(ct).ConfigureAwait(false); + + InferenceTransferState node; + + lock (m_sync) + { + if (m_transfers.Count >= m_options.MaxConcurrentTransfers) + { + return new BeginTransferMethodStateResult + { + ServiceResult = StatusCodes.BadTooManyOperations, + Transfer = NodeId.Null, + Accepted = false + }; + } + + string transferId = Guid.NewGuid().ToString("N", CultureInfo.InvariantCulture); + + node = new InferenceTransferState(null); + node.Create( + SystemContext, + NodeId.Null, + new QualifiedName("Transfer_" + transferId, NamespaceIndex), + new LocalizedText("Transfer " + transferId), + true); + + Child>(node, BrowseNames.TransferId).Value = transferId; + Child>(node, BrowseNames.State).Value = + TransferStateEnum.Building; + Child>(node, BrowseNames.ContentType).Value = contentType; + Child>(node, BrowseNames.ExpiresAt).Value = + DateTime.UtcNow.Add(m_options.TransferExpiry); + + var entry = new TransferEntry + { + Node = node, + DeploymentId = deployment.NodeId, + ContentType = contentType, + ExpiresAt = DateTime.UtcNow.Add(m_options.TransferExpiry) + }; + + WireTransfer(entry); + + // Every member this transfer will ever carry is materialised now, + // before the node is indexed. A child created afterwards exists on + // the NodeState and is invisible over the wire, which reads as a + // result that silently lost its ModelUsed rather than as an error. + Child>(node, BrowseNames.ResponseContentType); + Child>(node, BrowseNames.ModelUsed); + Child>(node, BrowseNames.Usage); + Child>(node, BrowseNames.FinishReason); + Child>(node, BrowseNames.LastError); + + m_transfers[node.NodeId] = entry; + Child(m_root!, BrowseNames.Jobs).AddChild(node); + AddPredefinedNodeSynchronously(node); + } + + return new BeginTransferMethodStateResult + { + ServiceResult = ServiceResult.Good, + Transfer = node.NodeId, + Accepted = true + }; + } + + /// + /// Attaches the file handlers and the Execute method to a transfer. + /// + private void WireTransfer(TransferEntry entry) + { + var request = Child(entry.Node, BrowseNames.Request); + var response = Child(entry.Node, BrowseNames.Response); + + m_files.Attach(request, entry.Request, writable: true); + m_files.Attach(response, entry.Response, writable: false); + m_files.Own(entry.Node, request, response); + + Child(entry.Node, BrowseNames.Execute).OnCallAsync = + (context, method, objectId, ct) => ExecuteTransferAsync(objectId, ct); + + // Materialised rather than looked up. FindChild does not create, so an + // optional Method reached that way is silently absent: the sample would + // claim to support Abort and publish nothing. + Child(entry.Node, BrowseNames.Abort).OnCallMethod2Async = async ( + context, method, objectId, inputs, outputs, ct) => + { + await DiscardTransferAsync(objectId, ct).ConfigureAwait(false); + return ServiceResult.Good; + }; + } + + /// + /// Runs the inference the transfer was opened for. + /// + /// + /// The transfer carries the same outputs the inline call returns, including + /// ModelUsed. A large payload is a transport concern, so nothing + /// about the result a caller is entitled to should change because the bytes + /// arrived in chunks. + /// + private async ValueTask ExecuteTransferAsync( + NodeId objectId, + CancellationToken ct) + { + TransferEntry? entry; + DeploymentState? deployment; + byte[] payload; + + lock (m_sync) + { + if (!m_transfers.TryGetValue(objectId, out entry)) + { + return new ExecuteMethodStateResult + { + ServiceResult = StatusCodes.BadNodeIdUnknown, + Accepted = false + }; + } + + deployment = FindDeployment(entry.DeploymentId); + + if (deployment is null) + { + return new ExecuteMethodStateResult + { + ServiceResult = StatusCodes.BadNodeIdUnknown, + Accepted = false + }; + } + + // Snapshotted through the file manager, which is what actually + // serialises against a concurrent Write. Reading the MemoryStream + // directly under m_sync would look careful and guarantee nothing: + // the two locks do not exclude one another. + payload = m_files.Snapshot(Child(entry.Node, BrowseNames.Request)); + + SetTransferState(entry, TransferStateEnum.Executing); + } + + InferenceOutcome outcome = await RunWithFallbackAsync( + deployment, + payload, + entry.ContentType, + m_options.TransferInferenceTimeout.TotalMilliseconds, + ct).ConfigureAwait(false); + + lock (m_sync) + { + // The inference took a while, and Abort and expiry both remove the + // entry and dispose its buffers. Writing the result into a transfer + // that is no longer live would throw ObjectDisposedException out of + // a Method call - so the answer is dropped instead, which is what a + // caller that aborted was asking for. + if (!m_transfers.TryGetValue(objectId, out TransferEntry? live) || + !ReferenceEquals(live, entry)) + { + return new ExecuteMethodStateResult + { + ServiceResult = StatusCodes.BadInvalidState, + Accepted = false + }; + } + + if (!outcome.Result.Ok) + { + Child>(entry.Node, BrowseNames.LastError) + .Value = new LocalizedText(outcome.Result.Message ?? "Inference failed."); + SetTransferState(entry, TransferStateEnum.Failed); + + return new ExecuteMethodStateResult + { + ServiceResult = ServiceResult.Good, + Accepted = false + }; + } + + m_files.Replace( + Child(entry.Node, BrowseNames.Response), + outcome.Result.Payload.Span); + + Child>(entry.Node, BrowseNames.ResponseContentType) + .Value = outcome.Result.ContentType; + Child>(entry.Node, BrowseNames.ModelUsed).Value = + outcome.ModelUsed; + Child>(entry.Node, BrowseNames.Usage).Value = + ToUsage(outcome.Result); + Child>(entry.Node, BrowseNames.FinishReason) + .Value = ToFinishReason(outcome.Result.Finish); + + SetTransferState(entry, TransferStateEnum.Completed); + } + + return new ExecuteMethodStateResult + { + ServiceResult = ServiceResult.Good, + Accepted = true + }; + } + + private void SetTransferState(TransferEntry entry, TransferStateEnum state) + { + var node = Child>(entry.Node, BrowseNames.State); + node.Value = state; + entry.Node.ClearChangeMasks(SystemContext, true); + } + + /// + /// Removes a transfer and the memory it was holding. + /// + /// + /// The dictionary entry goes under the lock and the node goes afterwards. + /// Removing the entry first is what makes this safe to call twice: a second + /// caller finds nothing and returns, rather than racing the first one to + /// delete the same node. + /// + private async ValueTask DiscardTransferAsync( + NodeId transferId, + CancellationToken ct) + { + TransferEntry? entry; + + lock (m_sync) + { + if (!m_transfers.Remove(transferId, out entry)) + { + return; + } + + m_files.Detach(entry.Node); + } + + await DeleteNodeAsync(SystemContext, transferId, ct).ConfigureAwait(false); + entry.Dispose(); + } + + /// + /// Drops transfers nobody came back for. + /// + /// + /// Called when a new transfer is opened rather than on a timer. Reclaiming + /// under the pressure that makes it necessary costs nothing when there is no + /// pressure, and a timer that ran every minute forever would. + /// + private async ValueTask ExpireTransfersAsync(CancellationToken ct) + { + DateTime now = DateTime.UtcNow; + List? expired = null; + + lock (m_sync) + { + foreach (KeyValuePair pair in m_transfers) + { + if (pair.Value.ExpiresAt <= now) + { + (expired ??= []).Add(pair.Value); + } + } + + if (expired is null) + { + return; + } + + foreach (TransferEntry entry in expired) + { + m_transfers.Remove(entry.Node.NodeId); + m_files.Detach(entry.Node); + } + } + + foreach (TransferEntry entry in expired) + { + await DeleteNodeAsync(SystemContext, entry.Node.NodeId, ct) + .ConfigureAwait(false); + entry.Dispose(); + } + } + } +} diff --git a/src/Opc.Ua.AI.Server/AiNodeManager.cs b/src/Opc.Ua.AI.Server/AiNodeManager.cs new file mode 100644 index 0000000000..d842df7915 --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiNodeManager.cs @@ -0,0 +1,408 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using Opc.Ua.Server; +using Opc.Ua.XRegistry; + +namespace Opc.Ua.AI.Server +{ + /// + /// Publishes the AI models this Server exposes, per + /// OPC UA - AI Model Management and Inference. + /// + /// + /// + /// The address space this builds is the specification's own: an + /// AiRootType under the Server Object holding models, deployments, + /// sources, registries and jobs. A client discovers everything from there, + /// which is what a well-known entry point is for. + /// + /// + /// The node manager owns the OPC UA surface and nothing else. Reaching an + /// actual model is 's business, and the + /// separation is deliberate: the specification's claim is that where inference + /// runs does not change how it is called, so the code answering a call should + /// not be able to tell where it ran. + /// + /// + public sealed partial class AINodeManager : AsyncCustomNodeManager + { + private readonly AIOptions m_options; + private readonly InferenceBackendOptions m_backendOptions; + private readonly InferenceBackendOptions m_fallbackBackendOptions; + private readonly InferenceBackends m_backends; + private readonly ILogger m_logger; + private readonly Lock m_sync = new(); + private readonly Dictionary m_transfers = []; + private readonly List m_jobs = []; + private readonly HashSet m_learningSampleIds = new(StringComparer.Ordinal); + private readonly Queue m_learningSampleOrder = new(); + private readonly StreamFileManager m_files; + private int m_nextId; + + private AiRootState? m_root; + private ModelState? m_primaryModel; + private ModelState? m_fallbackModel; + private DeploymentState? m_primary; + private DeploymentState? m_fallback; + private LearningJobState? m_learningJob; + + /// + /// Creates the node manager. + /// + /// The server hosting this node manager. + /// The application configuration. + /// Reaches the models this Server publishes. + /// What this Server publishes. + /// What the primary deployment reaches. + /// + /// What the fallback deployment reaches. Separate from the primary's on + /// purpose: the fallback has its own site, jurisdiction and egress, and + /// publishing the primary's would describe a deployment that does not exist. + /// + /// Where diagnostics go. + public AINodeManager( + IServerInternal server, + ApplicationConfiguration configuration, + InferenceBackends backends, + IOptions? options = null, + IOptions? backendOptions = null, + InferenceBackendOptions? fallbackBackendOptions = null, + ILogger? logger = null) + : base( + server, + configuration, + Opc.Ua.AI.Namespaces.AI, + Opc.Ua.AI.Namespaces.xRegistry) + { + m_backends = backends ?? throw new ArgumentNullException(nameof(backends)); + m_options = options?.Value ?? new AIOptions(); + m_backendOptions = backendOptions?.Value ?? new InferenceBackendOptions(); + m_fallbackBackendOptions = fallbackBackendOptions ?? m_backendOptions; + m_logger = logger ?? (ILogger)NullLogger.Instance; + m_files = new StreamFileManager(m_options.MaxTransferSize); + SystemContext.NodeIdFactory = this; + } + + /// + protected override void Dispose(bool disposing) + { + if (disposing) + { + lock (m_sync) + { + foreach (TransferEntry entry in m_transfers.Values) + { + entry.Dispose(); + } + + m_transfers.Clear(); + } + + m_files.Dispose(); + } + + base.Dispose(disposing); + } + + /// + /// NodeId of the AI root, which is what a client browses to first. + /// + public NodeId RootId => m_root?.NodeId ?? NodeId.Null; + + /// + /// NodeId of the primary deployment, for the client sample and the tests. + /// + public NodeId PrimaryDeploymentId => m_primary?.NodeId ?? NodeId.Null; + + /// + /// NodeId of the fallback deployment, or a null NodeId when none is published. + /// + public NodeId FallbackDeploymentId => m_fallback?.NodeId ?? NodeId.Null; + + /// + /// NodeId of the learning job that accounts for submitted ground-truth samples. + /// + public NodeId LearningJobId => m_learningJob?.NodeId ?? NodeId.Null; + + /// + /// + /// String identifiers, deliberately. Numeric ones would be drawn from the + /// same namespace the loaded NodeSet occupies, and this model runs to + /// ns=2;i=7001 - so a counter starting at 1 walks into the type nodes, and + /// the predefined-node index overwrites rather than rejects. A Server that + /// had served a few hundred transfers would quietly have replaced + /// AiRootType with an inference job's FinishedAt property. + /// A string identifier cannot collide with a numeric one at all, which is a + /// stronger guarantee than any seed value. + /// + public override NodeId New(ISystemContext context, NodeState node) + { + return new NodeId( + FormattableString.Invariant($"n{Interlocked.Increment(ref m_nextId)}"), + NamespaceIndex); + } + + /// + protected override ValueTask LoadPredefinedNodesAsync( + ISystemContext context, + CancellationToken cancellationToken = default) + { + // In dependency order. The AI types subtype xRegistry ones, so loading + // only the AI model would leave every one of those supertypes dangling + // and the type table would refuse the model outright. + var nodes = new NodeStateCollection(); + nodes.AddOpcUaXRegistry(context); + nodes.AddOpcUaAI(context); + return new ValueTask(SortBySupertype(nodes)); + } + + /// + /// Orders types so that a supertype always precedes its subtypes. + /// + /// + /// + /// The type table refuses a type whose supertype it has not seen, and a + /// NodeSet is under no obligation to list them in that order - NodeIds are + /// assigned in declaration order, and a model that gains an abstract base + /// after its first concrete subtype will legitimately carry a higher NodeId + /// for the base. The AI model does exactly that. + /// + /// + /// Sorting here rather than reordering the NodeSet keeps the two concerns + /// apart: the NodeSet says what the model is, and this says what order this + /// particular loader needs to hear it in. + /// + /// + private static NodeStateCollection SortBySupertype(NodeStateCollection nodes) + { + var byId = new Dictionary(); + + foreach (NodeState node in nodes) + { + if (node is BaseTypeState type && !type.NodeId.IsNull) + { + byId[type.NodeId] = type; + } + } + + var sorted = new NodeStateCollection(); + var placed = new HashSet(); + + foreach (NodeState node in nodes) + { + Place(node, byId, placed, sorted); + } + + return sorted; + } + + private static void Place( + NodeState node, + Dictionary byId, + HashSet placed, + NodeStateCollection sorted) + { + if (!node.NodeId.IsNull && !placed.Add(node.NodeId)) + { + return; + } + + // A supertype outside this collection is already in the type table - + // every core type is - so only the ones declared here need placing + // first. Marking the node placed before the recursion means a cycle + // terminates rather than overflowing the stack; a model containing one + // is broken either way, and it will be rejected with a clear message + // instead of a StackOverflowException that kills the process. + if (node is BaseTypeState type && + !type.SuperTypeId.IsNull && + byId.TryGetValue(type.SuperTypeId, out BaseTypeState? super)) + { + Place(super, byId, placed, sorted); + } + + sorted.Add(node); + } + + /// + protected override async ValueTask AddPredefinedNodeAsync( + ISystemContext context, + NodeState node, + CancellationToken cancellationToken = default) + { + try + { + await base.AddPredefinedNodeAsync(context, node, cancellationToken) + .ConfigureAwait(false); + } + catch (Exception ex) + { + // The type table reports a rejected node as a bare status code with + // no indication of which node it was, which makes a model that fails + // to load nearly impossible to diagnose. Naming the node costs + // nothing on the path that works. + throw new ServiceResultException( + StatusCodes.BadNodeIdInvalid, + FormattableString.Invariant( + $"Could not add {node.BrowseName} ({node.NodeId}): {ex.Message}"), + ex); + } + } + + /// + /// The node registered under an identifier, or null when none is. + /// + /// + /// A test seam. The distinction between "on the NodeState tree" and "in the + /// predefined-node index" is exactly what two of this sample's defects + /// turned on, and it cannot be observed from outside the index. + /// + internal NodeState? IndexedNode(NodeId nodeId) + { + return PredefinedNodes.TryGetValue(nodeId, out NodeState? node) ? node : null; + } + + /// + /// How many nodes of a given type the index holds. + /// + /// The node state type to count. + internal int CountIndexed() where TNode : NodeState + { + int count = 0; + + foreach (KeyValuePair pair in PredefinedNodes) + { + if (pair.Value is TNode) + { + count++; + } + } + + return count; + } + + /// + /// Finds or creates a child declared by the type, and fails loudly if the + /// type does not declare it. + /// + /// + /// Optional members are not materialised by Create, so every member + /// this sample publishes beyond the mandatory ones has to be asked for. A + /// browse name the type does not declare is a coding error rather than a + /// runtime condition, so this throws rather than returning null and letting + /// a null reference surface somewhere less informative. + /// + /// The instance state type of the child. + /// The node declaring the child. + /// The browse name of the child. + /// + private TChild Child(NodeState parent, string browseName) + where TChild : BaseInstanceState + { + var qualifiedName = new QualifiedName(browseName, NamespaceIndex); + + if (parent.FindChild(SystemContext, qualifiedName) is TChild existing) + { + return existing; + } + + if (parent.CreateChild(SystemContext, qualifiedName) is not TChild typed) + { + throw new InvalidOperationException( + FormattableString.Invariant( + $"{parent.BrowseName} declares no {browseName} of type {typeof(TChild).Name}.")); + } + + // Two things a freshly materialised optional child does not have, and + // needs before a client can see it or use it. + // + // Create runs the child's own initialisation, which is what builds the + // members ITS type declares - for a Method, that is InputArguments and + // OutputArguments. Without it the Method browses correctly, accepts a + // call, and rejects it with BadTooManyArguments no matter what is passed, + // because as far as the Server is concerned it takes none. + // + // ReferenceTypeId is what a Browse names the reference by. A child + // without one is indexed, readable by NodeId and callable, but no client + // can navigate to it - so the whole optional half of the model simply + // is not there, without anything failing. + typed.Create( + SystemContext, + NodeId.Null, + qualifiedName, + new LocalizedText(browseName), + true); + + typed.ReferenceTypeId = typed is PropertyState + ? Opc.Ua.ReferenceTypeIds.HasProperty + : Opc.Ua.ReferenceTypeIds.HasComponent; + + return typed; + } + + /// + /// One chunked inference exchange the Server is holding. + /// + /// + /// The buffers live here rather than in the address space because a + /// NodeState is not where bytes want to live. The node carries the state a + /// client reads; this carries the payload it reads through. + /// + private sealed class TransferEntry : IDisposable + { + public required InferenceTransferState Node { get; init; } + + public required NodeId DeploymentId { get; init; } + + public System.IO.MemoryStream Request { get; } = new(); + + public System.IO.MemoryStream Response { get; } = new(); + + public string ContentType { get; set; } = "application/json"; + + public DateTime ExpiresAt { get; set; } + + public void Dispose() + { + Request.Dispose(); + Response.Dispose(); + } + } + } +} diff --git a/src/Opc.Ua.AI.Server/AiNodeManagerFactory.cs b/src/Opc.Ua.AI.Server/AiNodeManagerFactory.cs new file mode 100644 index 0000000000..87ca111a43 --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiNodeManagerFactory.cs @@ -0,0 +1,106 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using Opc.Ua; +using Opc.Ua.AI.Inference; +using Opc.Ua.Server; + +namespace Opc.Ua.AI.Server +{ + /// + /// Produces the AI node manager for the hosting pipeline. + /// + /// + /// The factory is what the DI container resolves, so it is where the backends, + /// options and logging arrive. The node manager itself is constructed per Server + /// and owned by it. + /// + public sealed class AINodeManagerFactory : IAsyncNodeManagerFactory + { + /// Names the fallback backend's configuration section. + public const string FallbackOptionsName = "fallback"; + + private readonly InferenceBackends m_backends; + private readonly IOptions? m_options; + private readonly IOptions? m_backendOptions; + private readonly InferenceBackendOptions? m_fallbackBackendOptions; + private readonly ILogger? m_logger; + + /// + /// Creates the factory. + /// + public AINodeManagerFactory( + InferenceBackends backends, + IOptions? options = null, + IOptions? backendOptions = null, + IOptionsMonitor? namedBackendOptions = null, + ILogger? logger = null) + { + m_backends = backends; + m_options = options; + m_backendOptions = backendOptions; + // The fallback's configuration is a NAMED option, so it has to be asked + // for by name. Without it the node manager would describe the fallback + // deployment using the primary's site, jurisdiction and egress. + m_fallbackBackendOptions = namedBackendOptions?.Get(FallbackOptionsName); + m_logger = logger; + } + + /// + public ArrayOf NamespacesUris => new string[] + { + Opc.Ua.AI.Namespaces.AI, + Opc.Ua.AI.Namespaces.xRegistry + }; + + /// + public ValueTask CreateAsync( + IServerInternal server, + ApplicationConfiguration configuration, + CancellationToken cancellationToken = default) + { +#pragma warning disable CA2000 // ownership transferred to the server + IAsyncNodeManager manager = new AINodeManager( + server, + configuration, + m_backends, + m_options, + m_backendOptions, + m_fallbackBackendOptions, + m_logger); +#pragma warning restore CA2000 + + return new ValueTask(manager); + } + } +} diff --git a/src/Opc.Ua.AI.Server/AiOptions.cs b/src/Opc.Ua.AI.Server/AiOptions.cs new file mode 100644 index 0000000000..f4856557e3 --- /dev/null +++ b/src/Opc.Ua.AI.Server/AiOptions.cs @@ -0,0 +1,134 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using Opc.Ua.AI.Inference; + +namespace Opc.Ua.AI.Server +{ + /// + /// How the sample is configured. + /// + /// + /// There is deliberately no second management API. The specification's own + /// Methods are the control plane, and everything here is startup configuration - + /// which backend to reach, which deployments to publish, and how the simulated + /// scenarios behave. + /// + public sealed class AIOptions + { + /// Configuration section this binds from. + public const string SectionName = "AiModelManagement"; + + /// + /// Identifier of the primary deployment, as published through + /// DeploymentType.DeploymentId. + /// + public string PrimaryDeploymentId { get; set; } = "primary"; + + /// + /// Identifier of the deployment the primary falls back to. + /// + public string FallbackDeploymentId { get; set; } = "fallback"; + + /// + /// Whether to publish a fallback deployment and wire + /// FallsBackTo from the primary to it. + /// + /// + /// Worth having as a switch because the fallback path is the one whose + /// failure is invisible: a fallback that answers without reporting the + /// substituted model in ModelUsed looks perfectly healthy. + /// + public bool EnableFallback { get; set; } = true; + + /// + /// Whether to publish a catalogue and an import job (scenario 4.4). + /// + public bool EnableCatalogue { get; set; } = true; + + /// + /// Whether to publish a learning job (scenario 4.7). + /// + public bool EnableLearningLoop { get; set; } = true; + + /// + /// How long an inference transfer survives without completing before the + /// Server may reclaim it. + /// + public TimeSpan TransferExpiry { get; set; } = TimeSpan.FromMinutes(5); + + /// + /// How long a simulated asynchronous inference takes, so that a client can + /// observe the job lifecycle rather than the job completing before it has + /// subscribed. + /// + public TimeSpan AsyncInferenceDelay { get; set; } = TimeSpan.FromSeconds(3); + + /// + /// Largest payload the Server will accept through a chunked transfer. + /// + /// + /// A transfer exists because a payload was too large to pass inline, so the + /// bound that matters is not the inline one. Without a second bound here + /// "too large for inline" would mean "unbounded", which is a worse answer + /// than the limit it was meant to relax. + /// + public ulong MaxTransferSize { get; set; } = 64 * 1024 * 1024; + + /// + /// How many transfers may be open at once. + /// + public int MaxConcurrentTransfers { get; set; } = 16; + + /// + /// How long an inference started through a transfer may run. + /// + public TimeSpan TransferInferenceTimeout { get; set; } = TimeSpan.FromMinutes(2); + + /// + /// How many asynchronous jobs the Server keeps before reclaiming the + /// oldest. + /// + /// + /// A job retains its request and its response, so an uncapped set grows in + /// bytes as well as in nodes, and any session that can call InvokeAsync can + /// grow it. Transfers already have both an expiry and a cap; jobs need the + /// same for the same reason. + /// + public int MaxRetainedJobs { get; set; } = 64; + + /// + /// Identifier of the model source this Server consumes from. + /// + public string SourceId { get; set; } = "model-source"; + } +} diff --git a/src/Opc.Ua.AI.Server/AssemblyInfo.cs b/src/Opc.Ua.AI.Server/AssemblyInfo.cs new file mode 100644 index 0000000000..be83f31ace --- /dev/null +++ b/src/Opc.Ua.AI.Server/AssemblyInfo.cs @@ -0,0 +1,35 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +// The OPC UA stack surface this builds on is not CLS compliant (unsigned +// integers appear throughout the specification's own data types), so claiming +// compliance here would be false. +[assembly: CLSCompliant(false)] diff --git a/src/Opc.Ua.AI.Server/Hosting/IChatClientFactory.cs b/src/Opc.Ua.AI.Server/Hosting/IChatClientFactory.cs new file mode 100644 index 0000000000..f83c060007 --- /dev/null +++ b/src/Opc.Ua.AI.Server/Hosting/IChatClientFactory.cs @@ -0,0 +1,60 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using Microsoft.Extensions.AI; +using Opc.Ua.AI.Inference; + +namespace Opc.Ua.AI.Server.Hosting +{ + /// + /// Creates host-owned chat clients for AI inference backends. + /// + /// + /// Opc.Ua.AI.Inference deliberately depends only on + /// Microsoft.Extensions.AI. A host that wants Azure, OpenAI, Ollama or a + /// local runtime keeps that dependency in its own composition root and exposes + /// the resulting abstraction here. + /// + public interface IChatClientFactory + { + /// + /// Creates the chat client for one configured backend. + /// + /// + /// The options name. Empty means the primary backend; fallback means + /// the fallback backend. + /// + /// The backend configuration. + /// + /// A client instance for the backend. Ownership transfers to the inference + /// backend, which disposes it with the backend. + /// + IChatClient CreateChatClient(string backendName, InferenceBackendOptions options); + } +} diff --git a/src/Opc.Ua.AI.Server/Hosting/OpcUaServerAiBuilderExtensions.cs b/src/Opc.Ua.AI.Server/Hosting/OpcUaServerAiBuilderExtensions.cs new file mode 100644 index 0000000000..bb7021e882 --- /dev/null +++ b/src/Opc.Ua.AI.Server/Hosting/OpcUaServerAiBuilderExtensions.cs @@ -0,0 +1,138 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; +using Opc.Ua.AI.Server.Hosting; +using Opc.Ua.Server.Hosting; + +namespace Microsoft.Extensions.DependencyInjection +{ + /// + /// Hosting extensions for OPC UA AI Model Management servers. + /// + public static class OpcUaServerAIBuilderExtensions + { + /// + /// Registers the AI node manager, its options and the configured inference + /// backends. + /// + /// is null. + public static IOpcUaServerBuilder AddAI( + this IOpcUaServerBuilder builder, + Action? configure = null, + Action? configureBackend = null, + Action? configureFallbackBackend = null) + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + builder.Services.AddOptions(); + builder.Services.AddOptions(); + builder.Services.AddOptions(AINodeManagerFactory.FallbackOptionsName); + if (configure != null) + { + builder.Services.Configure(configure); + } + if (configureBackend != null) + { + builder.Services.Configure(configureBackend); + } + if (configureFallbackBackend != null) + { + builder.Services.Configure( + AINodeManagerFactory.FallbackOptionsName, + configureFallbackBackend); + } + builder.Services.TryAddSingleton(CreateBackends); + builder.AddNodeManager(); + return builder; + } + + private static InferenceBackends CreateBackends(IServiceProvider services) + { + IOptionsMonitor monitor = + services.GetRequiredService>(); + + InferenceBackendOptions primaryOptions = monitor.CurrentValue; + IInferenceBackend primary = CreateBackend(services, string.Empty, primaryOptions); + + InferenceBackendOptions fallbackOptions = + monitor.Get(AINodeManagerFactory.FallbackOptionsName); + if (!fallbackOptions.Enabled) + { + return new InferenceBackends(primary); + } + IInferenceBackend fallback = CreateBackend( + services, + AINodeManagerFactory.FallbackOptionsName, + fallbackOptions); + return new InferenceBackends(primary, fallback); + } + + private static IInferenceBackend CreateBackend( + IServiceProvider services, + string backendName, + InferenceBackendOptions options) + { + return options.Kind switch + { + InferenceBackendKind.ChatClient => new ChatClientInferenceBackend( + services.GetRequiredService() + .CreateChatClient(backendName, options), + options.Site, + [.. options.Models]), + InferenceBackendKind.RestChatCompletions => new RestChatCompletionsBackend( + options, + CredentialResolverFor(options), + services.GetService>() ?? + NullLogger.Instance), + _ => throw new InvalidOperationException( + "Unsupported inference backend kind '" + options.Kind + "'.") + }; + } + + private static ICredentialResolver CredentialResolverFor(InferenceBackendOptions options) + { + return options.Authentication switch + { + BackendAuthentication.Anonymous => NullCredentialResolver.Instance, + BackendAuthentication.WorkloadIdentity => + new WorkloadIdentityCredentialResolver(options.TokenAudience), + _ => new FileCredentialResolver(options.CredentialDirectory) + }; + } + } +} diff --git a/src/Opc.Ua.AI.Server/Hosting/RestChatCompletionsChatClientFactory.cs b/src/Opc.Ua.AI.Server/Hosting/RestChatCompletionsChatClientFactory.cs new file mode 100644 index 0000000000..c5fc303771 --- /dev/null +++ b/src/Opc.Ua.AI.Server/Hosting/RestChatCompletionsChatClientFactory.cs @@ -0,0 +1,314 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Net.Http; +using System.Net.Http.Headers; +using System.Text.Json; +using System.Text.Json.Serialization; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Opc.Ua.AI.Inference; + +namespace Opc.Ua.AI.Server.Hosting +{ + /// + /// Service registration helpers for AI chat-client factories. + /// + public static class AIChatClientServiceCollectionExtensions + { + /// + /// Registers a chat-client factory over the OpenAI-compatible REST + /// chat-completions contract. + /// + /// The service collection to update. + /// is null. + public static IServiceCollection AddRestChatCompletionsAIChatClientFactory( + this IServiceCollection services) + { + if (services == null) + { + throw new ArgumentNullException(nameof(services)); + } + services.TryAddSingleton(); + return services; + } + } + + internal sealed class RestChatCompletionsChatClientFactory : IChatClientFactory + { + public IChatClient CreateChatClient(string backendName, InferenceBackendOptions options) + { + return new RestChatCompletionsChatClient(options, CredentialResolverFor(options)); + } + + private static ICredentialResolver CredentialResolverFor(InferenceBackendOptions options) + { + return options.Authentication switch + { + BackendAuthentication.Anonymous => NullCredentialResolver.Instance, + BackendAuthentication.WorkloadIdentity => + new WorkloadIdentityCredentialResolver(options.TokenAudience), + _ => new FileCredentialResolver(options.CredentialDirectory) + }; + } + } + + internal sealed class RestChatCompletionsChatClient : IChatClient + { + public RestChatCompletionsChatClient( + InferenceBackendOptions options, + ICredentialResolver credentials) + { + m_options = options ?? throw new ArgumentNullException(nameof(options)); + m_credentials = credentials ?? throw new ArgumentNullException(nameof(credentials)); + if (!string.IsNullOrEmpty(options.EndpointUri)) + { + m_http.BaseAddress = new Uri(options.EndpointUri, UriKind.Absolute); + } + } + + public async Task GetResponseAsync( + IEnumerable messages, + ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + using var message = new HttpRequestMessage( + HttpMethod.Post, + m_options.ChatCompletionsPath); + message.Content = JsonContentFor(messages, options); + await AuthenticateAsync(message, cancellationToken).ConfigureAwait(false); + + using HttpResponseMessage response = await m_http + .SendAsync(message, cancellationToken) + .ConfigureAwait(false); + response.EnsureSuccessStatusCode(); + + using Stream body = await response.Content + .ReadAsStreamAsync(cancellationToken) + .ConfigureAwait(false); + using JsonDocument document = await JsonDocument + .ParseAsync(body, cancellationToken: cancellationToken) + .ConfigureAwait(false); + + return ProjectResponse(document.RootElement, options?.ModelId ?? string.Empty); + } + + public IAsyncEnumerable GetStreamingResponseAsync( + IEnumerable messages, + ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + throw new NotSupportedException(); + } + + public object? GetService(Type serviceType, object? serviceKey = null) + { + return null; + } + + public void Dispose() + { + m_http.Dispose(); + } + + private async Task AuthenticateAsync(HttpRequestMessage message, CancellationToken ct) + { + if (string.IsNullOrEmpty(m_options.CredentialReference)) + { + return; + } + string? secret = await m_credentials + .ResolveAsync(m_options.CredentialReference, ct) + .ConfigureAwait(false); + if (string.IsNullOrEmpty(secret)) + { + return; + } + switch (m_options.Authentication) + { + case BackendAuthentication.ApiKey: + message.Headers.TryAddWithoutValidation(m_options.ApiKeyHeader, secret); + break; + case BackendAuthentication.BearerToken: + case BackendAuthentication.WorkloadIdentity: + message.Headers.Authorization = new AuthenticationHeaderValue("Bearer", secret); + break; + } + } + + private static ByteArrayContent JsonContentFor( + IEnumerable messages, + ChatOptions? options) + { + var request = new ChatCompletionRequest + { + Model = options?.ModelId, + Messages = ToMessages(messages), + Temperature = options?.Temperature, + MaxTokens = options?.MaxOutputTokens, + TopP = options?.TopP + }; + byte[] json = JsonSerializer.SerializeToUtf8Bytes( + request, + RestChatCompletionsChatClientJsonContext.Default.ChatCompletionRequest); + var content = new ByteArrayContent(json); + content.Headers.ContentType = new MediaTypeHeaderValue("application/json"); + return content; + } + + private static List ToMessages(IEnumerable messages) + { + var list = new List(); + foreach (ChatMessage message in messages) + { + list.Add(new ChatCompletionMessage + { + Role = ToRole(message.Role), + Content = message.Text ?? string.Empty + }); + } + return list; + } + + private static string ToRole(ChatRole role) + { + if (role == ChatRole.System) + { + return "system"; + } + if (role == ChatRole.Assistant) + { + return "assistant"; + } + if (role == ChatRole.Tool) + { + return "tool"; + } + return "user"; + } + + private static ChatResponse ProjectResponse(JsonElement root, string requestedModel) + { + string model = root.TryGetProperty("model", out JsonElement m) + ? m.GetString() ?? requestedModel + : requestedModel; + JsonElement choice = root.GetProperty("choices")[0]; + string content = choice + .GetProperty("message") + .GetProperty("content") + .GetString() ?? + string.Empty; + var response = new ChatResponse(new ChatMessage(ChatRole.Assistant, content)) + { + ModelId = model, + FinishReason = FinishReasonOf(choice) + }; + if (root.TryGetProperty("usage", out JsonElement usage)) + { + response.Usage = new UsageDetails + { + InputTokenCount = LongProperty(usage, "prompt_tokens"), + OutputTokenCount = LongProperty(usage, "completion_tokens"), + TotalTokenCount = LongProperty(usage, "total_tokens") + }; + } + return response; + } + + private static ChatFinishReason FinishReasonOf(JsonElement choice) + { + if (!choice.TryGetProperty("finish_reason", out JsonElement value)) + { + return ChatFinishReason.Stop; + } + string? reason = value.GetString(); + if (string.Equals(reason, "length", StringComparison.OrdinalIgnoreCase)) + { + return ChatFinishReason.Length; + } + if (string.Equals(reason, "content_filter", StringComparison.OrdinalIgnoreCase)) + { + return ChatFinishReason.ContentFilter; + } + if (string.Equals(reason, "tool_calls", StringComparison.OrdinalIgnoreCase)) + { + return ChatFinishReason.ToolCalls; + } + return ChatFinishReason.Stop; + } + + private static long LongProperty(JsonElement element, string name) + { + return element.TryGetProperty(name, out JsonElement value) && + value.TryGetInt64(out long result) + ? result + : 0L; + } + + private readonly HttpClient m_http = new(); + private readonly InferenceBackendOptions m_options; + private readonly ICredentialResolver m_credentials; + } + + internal sealed class ChatCompletionRequest + { + [JsonPropertyName("model")] + public string? Model { get; set; } + + [JsonPropertyName("messages")] + public List Messages { get; set; } = []; + + [JsonPropertyName("temperature")] + public float? Temperature { get; set; } + + [JsonPropertyName("max_tokens")] + public int? MaxTokens { get; set; } + + [JsonPropertyName("top_p")] + public float? TopP { get; set; } + } + + internal sealed class ChatCompletionMessage + { + [JsonPropertyName("role")] + public string Role { get; set; } = "user"; + + [JsonPropertyName("content")] + public string Content { get; set; } = string.Empty; + } + + [JsonSerializable(typeof(ChatCompletionRequest))] + internal sealed partial class RestChatCompletionsChatClientJsonContext : JsonSerializerContext; +} diff --git a/src/Opc.Ua.AI.Server/NugetREADME.md b/src/Opc.Ua.AI.Server/NugetREADME.md new file mode 100644 index 0000000000..5df23d752a --- /dev/null +++ b/src/Opc.Ua.AI.Server/NugetREADME.md @@ -0,0 +1,20 @@ +# Server support for OPC UA AI Model Management + +A node manager that publishes the model catalogue, datasets, deployments and inference endpoints, routes `Invoke` to an `IInferenceBackend`, runs learning jobs, and streams model artefacts through the standard file-transfer types. + +Part of the [OPC UA .NET Standard](https://github.com/OPCFoundation/UA-.NETStandard) stack. + +> **Draft.** The *OPC UA - AI Model Management and Inference* companion +> specification is a working draft. Its namespace URI and every NodeId are +> provisional, and every ObjectType and BrowseName can change when the working +> group publishes. + +## Documentation + +See the [AI Model Management guide](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/samples/AI/README.md) +for the example: `ModelManagementServer` publishes a catalogue and +routes inference, and `ModelManagementClient` walks it. + +## License + +MIT - see the [license](https://opcfoundation.org/license/mit.html). diff --git a/src/Opc.Ua.AI.Server/Opc.Ua.AI.Server.csproj b/src/Opc.Ua.AI.Server/Opc.Ua.AI.Server.csproj new file mode 100644 index 0000000000..71f75b01c7 --- /dev/null +++ b/src/Opc.Ua.AI.Server/Opc.Ua.AI.Server.csproj @@ -0,0 +1,39 @@ + + + $(AssemblyPrefix).AI.Server + + net10.0;net9.0;net8.0 + $(CustomTestTarget) + net10.0 + true + $(PackagePrefix).Opc.Ua.AI.Server + Opc.Ua.AI.Server + $(NoWarn);CS1591;CS0108 + enable + Server-side support for the OPC UA AI Model Management and Inference (draft) companion specification: a node manager that publishes the model catalogue, datasets, deployments and inference endpoints, routes Invoke to an IInferenceBackend, runs learning jobs, and streams model artefacts through the standard file-transfer types. + true + NugetREADME.md + true + false + false + + + $(PackageId).Debug + + + + + + + + + + + + + + + + + + diff --git a/src/Opc.Ua.AI.Server/StreamFileManager.cs b/src/Opc.Ua.AI.Server/StreamFileManager.cs new file mode 100644 index 0000000000..52b08ff1a4 --- /dev/null +++ b/src/Opc.Ua.AI.Server/StreamFileManager.cs @@ -0,0 +1,558 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using Opc.Ua; + +namespace Opc.Ua.AI.Server +{ + /// + /// Serves Part 5 FileType methods over in-memory buffers. + /// + /// + /// + /// The AI specification carries oversized inference payloads over + /// FileType rather than defining a transfer of its own, so this is the + /// piece that makes a large request work with a client that already knows how to + /// read a file. It is deliberately small: the interesting behaviour belongs to + /// the transfer, not to the plumbing that moves its bytes. + /// + /// + /// Buffers are in memory because an inference payload is transient by nature - + /// it exists between a caller assembling it and a model consuming it, and it has + /// no reason to reach a disk in between. + /// + /// + internal sealed class StreamFileManager : IDisposable + { + private readonly Lock m_lock = new(); + private readonly Dictionary m_files = []; + private readonly ulong m_maxSize; + private uint m_nextHandle; + + /// + /// Creates the manager. + /// + /// + /// Largest buffer a writer may produce, so that an unbounded write cannot + /// exhaust the Server. + /// + public StreamFileManager(ulong maxSize) + { + m_maxSize = maxSize; + } + + /// + /// Serves a file node from a buffer this manager owns. + /// + /// The file node to serve. + /// The buffer behind it. + /// + /// Whether a client may open the file for writing. A response buffer is + /// readable and not writable, which is not a restriction so much as a + /// statement: a client that could overwrite a model's answer could forge one. + /// + /// + /// The manager takes ownership of from here on. + /// Everything that touches it afterwards goes through + /// or , so the FileType methods + /// and whatever is producing the content serialise against each other. Two + /// components holding the same MemoryStream under two different locks is a + /// data race that shows up as a torn payload rather than an exception. + /// + public void Attach(FileState file, MemoryStream content, bool writable) + { + ArgumentNullException.ThrowIfNull(file); + ArgumentNullException.ThrowIfNull(content); + + var entry = new Entry(content, writable); + + lock (m_lock) + { + m_files[file.NodeId] = entry; + entry.Node = file; + } + + if (file.Writable is not null) + { + file.Writable.Value = writable; + } + + if (file.UserWritable is not null) + { + file.UserWritable.Value = writable; + } + + if (file.Size is not null) + { + file.Size.Value = (ulong)content.Length; + } + + if (file.Open is not null) + { + file.Open.OnCall = (ISystemContext context, + MethodState _, + NodeId objectId, + byte mode, + ref uint fileHandle) => Open(context, objectId, mode, ref fileHandle); + } + + if (file.Close is not null) + { + file.Close.OnCall = (ISystemContext context, + MethodState _, + NodeId objectId, + uint fileHandle) => Close(context, objectId, fileHandle); + } + + if (file.Read is not null) + { + file.Read.OnCall = (ISystemContext context, + MethodState _, + NodeId objectId, + uint fileHandle, + int length, + ref ByteString data) => Read(context, objectId, fileHandle, length, ref data); + } + + if (file.Write is not null) + { + file.Write.OnCall = (ISystemContext context, + MethodState _, + NodeId objectId, + uint fileHandle, + ByteString data) => Write(context, objectId, fileHandle, data); + } + + if (file.GetPosition is not null) + { + file.GetPosition.OnCall = (ISystemContext context, + MethodState _, + NodeId objectId, + uint fileHandle, + ref ulong position) => GetPosition(context, objectId, fileHandle, ref position); + } + + if (file.SetPosition is not null) + { + file.SetPosition.OnCall = (ISystemContext context, + MethodState _, + NodeId objectId, + uint fileHandle, + ulong position) => SetPosition(context, objectId, fileHandle, position); + } + } + + /// + /// Copies out the current contents of a file, under this manager's lock. + /// + /// + /// The copy is the point. A caller that read the MemoryStream directly would + /// be racing every concurrent Write, and Method calls are not serialised by + /// the Server - so a client is entirely free to keep uploading while + /// something else reads. + /// + public byte[] Snapshot(FileState file) + { + ArgumentNullException.ThrowIfNull(file); + + lock (m_lock) + { + return m_files.TryGetValue(file.NodeId, out Entry? entry) + ? entry.Content.ToArray() + : []; + } + } + + /// + /// Replaces the contents of a file, under this manager's lock. + /// + /// + /// Refreshes the published Size as well. A client sizes its reads from that + /// value, so leaving it stale does not merely mislead - it makes a correct + /// client read the wrong number of bytes, and a response written this way + /// would appear empty to anyone following Part 5 properly. + /// + public void Replace(FileState file, ReadOnlySpan content) + { + ArgumentNullException.ThrowIfNull(file); + + lock (m_lock) + { + if (!m_files.TryGetValue(file.NodeId, out Entry? entry)) + { + return; + } + + entry.Content.SetLength(0); + entry.Content.Write(content); + entry.Content.Position = 0; + + // Any handle open on the old contents now points into a buffer that + // no longer holds what it was reading, so they are closed rather + // than left to return bytes from two different answers. + entry.Handles.Clear(); + + RefreshSize(entry); + } + } + + /// + /// Stops serving every file under a node, closing whatever was open. + /// + public void Detach(NodeState parent) + { + ArgumentNullException.ThrowIfNull(parent); + + lock (m_lock) + { + var stale = new List(); + + foreach (KeyValuePair pair in m_files) + { + stale.Add(pair.Key); + } + + foreach (NodeId id in stale) + { + if (m_files.TryGetValue(id, out Entry? entry) && entry.Owner == parent.NodeId) + { + entry.Handles.Clear(); + m_files.Remove(id); + } + } + } + } + + /// + /// Associates already attached files with the object that owns them, so + /// discarding that object closes them. + /// + public void Own(NodeState owner, params FileState[] files) + { + ArgumentNullException.ThrowIfNull(owner); + ArgumentNullException.ThrowIfNull(files); + + lock (m_lock) + { + foreach (FileState file in files) + { + if (m_files.TryGetValue(file.NodeId, out Entry? entry)) + { + entry.Owner = owner.NodeId; + } + } + } + } + + /// + public void Dispose() + { + lock (m_lock) + { + m_files.Clear(); + } + } + + private ServiceResult Open(ISystemContext context, NodeId objectId, byte mode, ref uint fileHandle) + { + const byte read = 1; + const byte writeEraseExisting = 6; + + lock (m_lock) + { + if (!m_files.TryGetValue(objectId, out Entry? entry)) + { + return StatusCodes.BadNodeIdUnknown; + } + + bool writing = mode == writeEraseExisting; + + if (mode != read && !writing) + { + return ServiceResult.Create( + StatusCodes.BadNotSupported, + "Only Read (1) and Write+EraseExisting (6) are supported."); + } + + if (writing && !entry.Writable) + { + return ServiceResult.Create( + StatusCodes.BadInvalidState, + "This file is not writable."); + } + + if (writing) + { + entry.Content.SetLength(0); + RefreshSize(entry); + } + + fileHandle = ++m_nextHandle; + entry.Handles[fileHandle] = new Handle(writing, SessionIdOf(context)); + return ServiceResult.Good; + } + } + + private ServiceResult Close(ISystemContext context, NodeId objectId, uint fileHandle) + { + lock (m_lock) + { + if (!TryGet(context, objectId, fileHandle, out Entry? entry, out _)) + { + return StatusCodes.BadInvalidArgument; + } + + entry.Handles.Remove(fileHandle); + return ServiceResult.Good; + } + } + + private ServiceResult Read( + ISystemContext context, + NodeId objectId, + uint fileHandle, + int length, + ref ByteString data) + { + data = ByteString.Empty; + + lock (m_lock) + { + if (!TryGet(context, objectId, fileHandle, out Entry? entry, out Handle? handle)) + { + return StatusCodes.BadInvalidArgument; + } + + if (handle.Writing) + { + return ServiceResult.Create( + StatusCodes.BadInvalidState, + "File handle is open for writing."); + } + + if (length <= 0) + { + return ServiceResult.Good; + } + + long available = entry.Content.Length - handle.Position; + int take = (int)Math.Min(available, length); + + if (take <= 0) + { + return ServiceResult.Good; + } + + byte[] buffer = new byte[take]; + entry.Content.Position = handle.Position; + int read = entry.Content.Read(buffer, 0, take); + handle.Position += read; + + if (read != buffer.Length) + { + Array.Resize(ref buffer, read); + } + + data = ByteString.From(buffer); + return ServiceResult.Good; + } + } + + private ServiceResult Write(ISystemContext context, NodeId objectId, uint fileHandle, ByteString data) + { + lock (m_lock) + { + if (!TryGet(context, objectId, fileHandle, out Entry? entry, out Handle? handle)) + { + return StatusCodes.BadInvalidArgument; + } + + if (!handle.Writing) + { + return ServiceResult.Create( + StatusCodes.BadInvalidState, + "File handle is open for reading."); + } + + if (data.IsNull || data.Span.Length == 0) + { + return ServiceResult.Good; + } + + if ((ulong)(entry.Content.Length + data.Span.Length) > m_maxSize) + { + return ServiceResult.Create( + StatusCodes.BadOutOfMemory, + "Payload exceeds the configured maximum transfer size."); + } + + entry.Content.Position = handle.Position; + entry.Content.Write(data.Span); + handle.Position = entry.Content.Position; + RefreshSize(entry); + return ServiceResult.Good; + } + } + + private ServiceResult GetPosition(ISystemContext context, NodeId objectId, uint fileHandle, ref ulong position) + { + lock (m_lock) + { + if (!TryGet(context, objectId, fileHandle, out _, out Handle? handle)) + { + return StatusCodes.BadInvalidArgument; + } + + position = (ulong)handle.Position; + return ServiceResult.Good; + } + } + + private ServiceResult SetPosition(ISystemContext context, NodeId objectId, uint fileHandle, ulong position) + { + lock (m_lock) + { + if (!TryGet(context, objectId, fileHandle, out Entry? entry, out Handle? handle)) + { + return StatusCodes.BadInvalidArgument; + } + + if (position > (ulong)entry.Content.Length) + { + return StatusCodes.BadInvalidArgument; + } + + handle.Position = (long)position; + return ServiceResult.Good; + } + } + + /// + /// Keeps the published size in step with the buffer. + /// + /// + /// A client sizes its reads from this, so a stale value is not cosmetic: it + /// makes a correct client read the wrong number of bytes. + /// + private static void RefreshSize(Entry entry) + { + if (entry.Node?.Size is not null) + { + entry.Node.Size.Value = (ulong)entry.Content.Length; + } + } + + /// + /// Finds a handle, and refuses one that belongs to another Session. + /// + /// + /// Part 5 scopes a FileHandle to the Session that opened it, and the reason + /// is worth stating: handles here are small sequential integers, and a + /// transfer's NodeId is handed out by BeginTransfer. Without this + /// check any session could guess a handle and inject bytes into, reposition, + /// or close another session's in-flight upload - which for an inference + /// payload means altering what a model is asked, from outside the + /// conversation. + /// + private bool TryGet( + ISystemContext context, + NodeId objectId, + uint fileHandle, + [System.Diagnostics.CodeAnalysis.NotNullWhen(true)] out Entry? entry, + [System.Diagnostics.CodeAnalysis.NotNullWhen(true)] out Handle? handle) + { + handle = null; + + if (!m_files.TryGetValue(objectId, out entry)) + { + return false; + } + + if (!entry.Handles.TryGetValue(fileHandle, out handle)) + { + return false; + } + + if (handle.SessionId != SessionIdOf(context)) + { + handle = null; + return false; + } + + return true; + } + + /// + /// The Session a call arrived on, or a null NodeId outside a Session. + /// + private static NodeId SessionIdOf(ISystemContext context) + { + return (context as ISessionSystemContext)?.SessionId ?? NodeId.Null; + } + + private sealed class Entry + { + public Entry(MemoryStream content, bool writable) + { + Content = content; + Writable = writable; + } + + public MemoryStream Content { get; } + + public bool Writable { get; } + + public NodeId Owner { get; set; } = NodeId.Null; + + public FileState? Node { get; set; } + + public Dictionary Handles { get; } = []; + } + + private sealed class Handle + { + public Handle(bool writing, NodeId sessionId) + { + Writing = writing; + SessionId = sessionId; + } + + public bool Writing { get; } + + /// The Session that opened this handle. + public NodeId SessionId { get; } + + public long Position { get; set; } + } + } +} diff --git a/src/Opc.Ua.AI/AssemblyInfo.cs b/src/Opc.Ua.AI/AssemblyInfo.cs new file mode 100644 index 0000000000..be83f31ace --- /dev/null +++ b/src/Opc.Ua.AI/AssemblyInfo.cs @@ -0,0 +1,35 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +// The OPC UA stack surface this builds on is not CLS compliant (unsigned +// integers appear throughout the specification's own data types), so claiming +// compliance here would be false. +[assembly: CLSCompliant(false)] diff --git a/src/Opc.Ua.AI/Model/Opc.Ua.AI.NodeSet2.xml b/src/Opc.Ua.AI/Model/Opc.Ua.AI.NodeSet2.xml new file mode 100644 index 0000000000..c0a68ea4f0 --- /dev/null +++ b/src/Opc.Ua.AI/Model/Opc.Ua.AI.NodeSet2.xml @@ -0,0 +1,2458 @@ + + + + + http://opcfoundation.org/UA/xRegistry/ + http://opcfoundation.org/UA/AI/ + + + + + + + + + i=1 + i=6 + i=7 + i=9 + i=11 + i=12 + i=14 + i=15 + i=10 + i=8 + i=13 + i=17 + i=20 + i=21 + i=294 + i=290 + i=296 + i=887 + i=14533 + i=24 + i=47 + i=46 + i=45 + i=35 + i=40 + i=37 + i=17603 + i=38 + i=78 + i=80 + i=11508 + i=11510 + + + InferenceLocationEnum + Where inference executes. The result contract is identical in every case; this property exists so a client can reason about latency, availability and the trust boundary without changing how it reads results. + AiModelManagement DataTypes + + i=29 + ns=2;i=3901 + + In the OPC UA Server process or on its host.On a separate edge node reached over the network.In a remote or cloud service.Inside a simulator that also produces the input. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3001 + + OnServerEdgeOffServerCloudInSimulator + + + AcceleratorKindEnum + Compute device executing the model. + AiModelManagement DataTypes + + i=29 + ns=2;i=3902 + + + + + EnumStrings + + i=78 + i=68 + ns=2;i=3002 + + CpuGpuNpuFpgaTpuOther + + + DeploymentStateEnum + Runtime lifecycle state of a deployment. + AiModelManagement DataTypes + + i=29 + ns=2;i=3903 + + Declared but not serving.Able to serve; no work in progress.Serving at least one request.Serving below configured quality.Unable to serve. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3003 + + InactiveReadyActiveDegradedFaulted + + + DatasetSourceEnum + Provenance of the samples in a dataset. + AiModelManagement DataTypes + + i=29 + ns=2;i=3904 + + Captured from physical equipment.Generated or rendered by a simulator.Both, for example synthetic pre-training with real fine-tuning. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3004 + + RealSyntheticMixed + + + LearningJobStateEnum + State of a dataset-capture, retraining and promotion cycle. + AiModelManagement DataTypes + + i=29 + ns=2;i=3905 + + A candidate model is available for promotion. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3005 + + IdleCollectingLabellingTrainingValidatingReadyPromotedFailed + + + FinishReasonEnum + Why an inference call stopped producing output. A client that treats every non-error response as complete will silently accept a truncated one, which is why this is Mandatory on a response rather than a diagnostic. + AiModelManagement DataTypes + + i=29 + ns=2;i=3906 + + The model finished normally.Output was truncated by a length or budget limit. The result is incomplete and SHALL NOT be treated as final.The model requested a tool or function call and is waiting for its result.Output was withheld by a safety policy; see the SafetyAssessment.The caller or the Server cancelled the call.The call failed; the StatusCode carries the reason. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3006 + + StopLengthToolCallFilteredCancelledError + + + ApiDialectEnum + Wire contract a remote inference endpoint speaks. A Server needs this to call an endpoint it did not deploy; without it EndpointUri is a string nobody can act on. It describes the REMOTE endpoint and never affects how an OPC UA client calls this Server. + AiModelManagement DataTypes + + i=29 + ns=2;i=3907 + + Another OPC UA Server implementing this specification's Invoke Method.The de-facto REST contract for chat and embeddings that most serving runtimes expose, including ones that run on a single workstation.The Open Inference Protocol (KServe v2) predict contract.A tensor-oriented RPC contract such as those used by dedicated inference servers.An in-process runtime reached through a local library rather than a network protocol.A contract this specification does not name. EndpointDescriptionUri SHOULD then say where it is documented. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3007 + + OpcUaInferenceRestChatCompletionsOpenInferenceProtocolTensorRemoteProcedureEmbeddedRuntimeProprietary + + + AuthenticationKindEnum + How the Server authenticates ITSELF to a remote inference endpoint. This is not how a client authenticates to this Server, which is the ordinary OPC UA Session security and is unaffected. + AiModelManagement DataTypes + + i=29 + ns=2;i=3908 + + No credential. Permitted only where the endpoint is reachable solely from a trusted network segment.A shared secret presented as a key.A token obtained from an authorization service.An identity the hosting platform assigns to the Server, so no secret is stored at all. Preferred where the platform offers it.Both ends present certificates. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3008 + + AnonymousApiKeyBearerTokenWorkloadIdentityMutualTls + + + FallbackPolicyEnum + What the Server does when a deployment cannot serve. This is the question a plant asks that no cloud inference API answers, because a cloud API assumes the caller can simply wait. + AiModelManagement DataTypes + + i=29 + ns=2;i=3909 + + Report the failure to the caller and produce nothing. The safe default: a caller that is told nothing happened can decide for itself.Continue reporting the most recent successful result, marked stale. Legitimate only where a stale answer is safe, and the caller SHALL be able to see the staleness.Route to the deployment named by the FallsBackTo reference. The answer comes from a different model and the response SHALL say so. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3009 + + FailHoldLastFallBackTo + + + VersionBindingEnum + Whether a deployment is bound to one immutable model version or follows a moving pointer. Stated structurally rather than as an upgrade policy, because what a client needs to know is whether the artefact can change under it, not what schedule someone intends to change it on. + AiModelManagement DataTypes + + i=29 + ns=2;i=3910 + + Bound to one immutable version. The artefact behind this deployment cannot change without an observable change to the deployment.Bound to a mutable pointer such as a branch or channel. The artefact CAN change without any other change, which is why clause 12 requires the resulting promotion to be as authorized as an explicit one. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3010 + + PinnedFollowsRef + + + ImportModeEnum + Whether an import job brings the model's description or its bytes. + AiModelManagement DataTypes + + i=29 + ns=2;i=3911 + + Materialize the catalogue entry as a ModelType and leave the artefact where it is. Nothing is downloaded and inference runs at the source.Fetch the artefact, verify its Digest, and make it locally available so inference can run without the source.Federate, then stage if the target deployment's InferenceLocation is OnServer or EdgeOffServer - because those cannot reach the source at inference time. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3011 + + FederateStageAuto + + + SafetySeverityEnum + Severity of one safety finding. The scale is the convergent industry one; what each level means for a given category is the policy's business, not this specification's. + AiModelManagement DataTypes + + i=29 + ns=2;i=3912 + + + + + EnumStrings + + i=78 + i=68 + ns=2;i=3012 + + NoneLowMediumHigh + + + ReachabilityEnum + Whether the Server can currently reach a deployment's execution site. + AiModelManagement DataTypes + + i=29 + ns=2;i=3913 + + Never attempted, or the Server does not probe.The most recent attempt succeeded.The most recent attempt failed.Reachable, but the endpoint is refusing work for capacity reasons. RetryAfter SHOULD be populated. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3013 + + UnknownReachableUnreachableThrottled + + + TransferStateEnum + Stage of a chunked inference exchange. A client reads this rather than inferring progress from which Methods have succeeded, because a transfer that failed mid-write and one that has not started look alike from outside. + AiModelManagement DataTypes + + i=29 + ns=2;i=3914 + + The request is being written and is not yet complete.The request is complete and inference has not started.Inference is running.The response is readable.The exchange failed; LastError carries the reason.The Server reclaimed the transfer before it completed. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3014 + + BuildingReadyExecutingCompletedFailedExpired + + + DigestProvenanceEnum + Where a Digest came from, or why there is none. Digest is Mandatory so that its absence is uniform and browsable rather than indistinguishable from a Server that does not implement digests - but 'empty' then carries two different meanings, and a client that must decide whether to trust an artefact needs them apart. This member is what tells them apart, and it does the same job for a digest that IS present: a value the source asserted and a value this Server computed over bytes are not the same evidence, and only one of them survives a substituted artefact. + AiModelManagement DataTypes + + i=29 + ns=2;i=3915 + + There is no digest and the source does not publish one. Digest is empty. This is the honest answer for an endpoint that names models but never their content, and it is what most hosted inference APIs require.Digest carries what the source declared. No party this Server can speak for has hashed the artefact, so the value is an assertion forwarded rather than evidence held.This Server hashed the artefact it holds. The value is evidence, but nothing independent agrees with it - a substitution that happened before the Server obtained the bytes is not detected.This Server hashed the artefact during a staging import (clause 10.4) and it matched what the source declared. Two independent parties agree, which is the strongest statement this model can carry. + + + EnumStrings + + i=78 + i=68 + ns=2;i=3015 + + NotAvailableDeclaredBySourceComputedByServerVerifiedOnStage + + + TensorSignatureDataType + Shape and element type of one model input or output tensor. This is what lets a client check that what it intends to send matches what the model expects, before it sends it. + AiModelManagement DataTypes + + i=22 + ns=2;i=5001 + + Tensor name as declared by the model.Element type, for example float32, uint8 or int64.Dimensions; -1 marks a dynamic axis.Optional axis layout hint, for example NCHW or NHWC. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=2;i=3050 + + + + ModelReferenceDataType + Identity of a model as a publisher, name and version triple. Every model catalogue in practice identifies a model this way, which is why an import job takes this rather than a URL: a URL says where a copy is today, the triple says which artefact is meant. + AiModelManagement DataTypes + + i=22 + ns=2;i=5002 + + Organisation or namespace that published the model.Model name within that publisher.Immutable version identifier, or a mutable pointer such as a branch or channel name. Which one it is is stated by VersionBinding, not guessable from the string. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=2;i=3051 + + + + UsageDataType + What one inference call consumed. Deliberately NOT named in tokens: a token is one accounting unit among several, and a model that consumes images, samples or seconds of audio needs the same accounting. UnitKind says which unit the counts are in. + AiModelManagement DataTypes + + i=22 + ns=2;i=5003 + + Unit the counts are expressed in, for example 'tokens', 'images', 'samples' or 'seconds'.Units consumed by the input.Units produced as output.Total units billed or metered for the call, which is not always the sum: cached or deduplicated input may be counted once. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=2;i=3052 + + + + CapabilityDataType + One capability a deployment does or does not have. An open list rather than an enumeration because the set of things a model can do is not closed, and a client that cannot recognise a capability name is no worse off than one that cannot recognise an enumeration value it has never seen. + AiModelManagement DataTypes + + i=22 + ns=2;i=5004 + + Capability name, for example 'chat', 'embeddings', 'streaming', 'tool-call' or 'structured-output'.Whether this deployment supports it. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=2;i=3053 + + + + SafetyAssessmentDataType + One finding from a safety policy applied to an inference call. Category is a String and not an enumeration because harm categories are set by the policy an installation adopts, and an industrial taxonomy looks nothing like a consumer one. + AiModelManagement DataTypes + + i=22 + ns=2;i=5005 + + Category the policy assessed, for example 'out-of-distribution-input' or a policy-defined name.Severity of the finding.True when the content was withheld or altered rather than merely flagged.Human-readable explanation. For a human; SHALL NOT be parsed. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=2;i=3054 + + + + EvaluationMetricDataType + One measured metric from an evaluation run, with the threshold it was judged against. The threshold travels with the metric because a metric without its acceptance criterion cannot be acted on, and a reviewer reading it a year later has no way to recover what 'good' meant. + AiModelManagement DataTypes + + i=22 + ns=2;i=5006 + + Metric name, for example 'accuracy' or 'false-negative-rate'.Measured value.Unit of the value, or empty when dimensionless.Acceptance threshold applied.How Value was compared with Threshold: one of '>=', '<=', '>', '<' or '=='.Outcome of that comparison. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=2;i=3055 + + + + RateLimitDataType + Capacity a remote endpoint is currently granting. Surfaced so a client can distinguish 'the model said no' from 'the quota said no', which are different faults with different remedies. + AiModelManagement DataTypes + + i=22 + ns=2;i=5007 + + Unit the limit is expressed in, matching UsageDataType.UnitKind, or 'requests'.Units permitted per interval, or 0 when not published.Units still available in the current interval.Length of the interval the limit applies to.How long to wait before retrying. Zero when the endpoint gave no guidance. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=2;i=3056 + + + + UsesModel + Links a Deployment to the Model it executes. Clause 6.5 requires exactly one such reference per deployment; it is the only defined path from a result to the model artefact and its Digest, on which the provenance requirement of clause 12 depends. + AiModelManagement ReferenceTypes + IsUsedByDeployment + + i=32 + + + + TrainedOn + Links a Model to a Dataset it was trained or validated on. A model whose training data cannot be named is a model whose behaviour cannot be explained, which is why this reference exists rather than a string. + AiModelManagement ReferenceTypes + IsTrainingDataFor + + i=32 + + + + DerivedFrom + Links a Model to the Model it was fine-tuned, distilled or quantized from. Lineage is a chain, not a field: a model three derivations from its base is answerable for all three, and a string naming the immediate parent cannot be walked. + AiModelManagement ReferenceTypes + IsBaseOfModel + + i=32 + + + + FallsBackTo + Links a Deployment to the Deployment that serves in its place when it cannot. Clause 9 forbids a cycle, and requires the response to say which deployment actually answered. + AiModelManagement ReferenceTypes + IsFallbackFor + + i=32 + + + + ImportedFrom + Links a Model to the catalogue resource an import job materialized it from. This is what makes 'where did this model come from' answerable after the fact, rather than only at the moment of import. + AiModelManagement ReferenceTypes + WasImportedAs + + i=32 + + + + EvaluatedBy + Links a Model to an EvaluationRun that measured it. Optional and repeating: a model may be evaluated many times, and the run that gated its promotion is not necessarily the last one. + AiModelManagement ReferenceTypes + Evaluates + + i=32 + + + + AiRootType + Server-level entry point. A client that has just connected browses here to find every model, dataset, deployment and learning job the Server describes, without knowing its layout. + AiModelManagement + + i=58 + ns=2;i=6001 + ns=2;i=6002 + ns=2;i=6003 + ns=2;i=6004 + ns=2;i=6005 + ns=2;i=6114 + ns=2;i=6115 + ns=2;i=6116 + ns=2;i=6117 + + + + Models + ModelType instances. + + i=78 + i=61 + ns=2;i=1001 + + + + Datasets + DatasetType instances. + + i=80 + i=61 + ns=2;i=1001 + + + + Deployments + DeploymentType instances. + + i=78 + i=61 + ns=2;i=1001 + + + + LearningJobs + LearningJobType instances. + + i=80 + i=61 + ns=2;i=1001 + + + + SpecificationVersion + Release of this specification the Server implements, for example '0.1.0'. + + i=78 + i=68 + ns=2;i=1001 + + + + ModelType + Nameplate of a trained model. The member set is deliberately aligned with the IDTA 02060 AI Model Nameplate submodel template, which is currently the only standardised description of an industrial AI model, so an Asset Administration Shell can be populated from this node without loss. + AiModelManagement + + i=58 + ns=2;i=6006 + ns=2;i=6007 + ns=2;i=6008 + ns=2;i=6009 + ns=2;i=6010 + ns=2;i=6011 + ns=2;i=6012 + ns=2;i=6013 + ns=2;i=6014 + ns=2;i=6015 + ns=2;i=6016 + ns=2;i=6017 + ns=2;i=6018 + ns=2;i=6118 + ns=2;i=6119 + ns=2;i=6120 + ns=2;i=6121 + ns=2;i=6122 + ns=2;i=6165 + ns=2;i=6168 + ns=2;i=6169 + + + + ModelId + Identifier of the model. + + i=78 + i=68 + ns=2;i=1002 + + + + Name + Human-readable model name. Its Text SHALL be the name the source system uses for the model, carried across unchanged. A LocalizedText because the base model types names that way and retyping it would break every implementation, but the localizable part is the presentation: a Server MAY add a translation for display and SHALL NOT translate, reformat or prettify the Text itself. Two Servers that fetched one model from two mirrors are meant to produce the same string, and a name adjusted for house style is a name that no longer matches. + + i=78 + i=68 + ns=2;i=1002 + + + + Version + Model version. + + i=78 + i=68 + ns=2;i=1002 + + + + Framework + Producing framework, for example PyTorch, TensorFlow or scikit-learn. + + i=80 + i=68 + ns=2;i=1002 + + + + Format + Serialization format, for example ONNX, TensorRT or OpenVINO IR. + + i=80 + i=68 + ns=2;i=1002 + + + + TaskKind + What the model does, for example Detection2D, Classification, Segmentation, Forecasting or AnomalyDetection. Free text because the set of tasks is not closed and a closed enumeration would date faster than the model does. + + i=80 + i=68 + ns=2;i=1002 + + + + Digest + Cryptographic digest of the model artefact, for provenance and integrity. Mandatory: clause 12 requires it for every model whose artefact is obtainable through ArtifactUri, and it is the terminus of the provenance chain that UsesModel keeps intact. + + i=78 + i=68 + ns=2;i=1002 + + + + DigestAlgorithm + Hash function used for Digest. SHALL name a function with at least 256-bit output and no known collision weakness; SHA-256 is the default and is always acceptable. SHALL NOT be MD5, SHA-1 or a truncated variant - chosen-prefix collisions against those are practical, so a substituted artefact would pass verification. SHALL be non-empty where Digest is non-empty. See clause 12. + + i=78 + i=68 + ns=2;i=1002 + + + + ArtifactUri + Where the model artefact can be obtained. Treated as untrusted input. + + i=80 + i=68 + ns=2;i=1002 + + + + ProvenanceUri + Training provenance or model card location. + + i=80 + i=68 + ns=2;i=1002 + + + + LabelClasses + Ordered class label set, where the model produces classified output. The INDEX is what a consuming specification's class identifier refers to, so the order is part of the contract and a Server SHALL NOT reorder it in place. + + i=80 + i=68 + ns=2;i=1002 + + + + Inputs + Input tensor signatures. + + i=80 + i=63 + ns=2;i=1002 + + + + Outputs + Output tensor signatures. + + i=80 + i=63 + ns=2;i=1002 + + + + DatasetType + A dataset used to train or validate a model. Aligned with the IDTA 02058 AI Dataset submodel template. SourceKind distinguishes real capture from simulator output, which is the provenance a reviewer needs when synthetic data is involved. + AiModelManagement + + i=58 + ns=2;i=6019 + ns=2;i=6020 + ns=2;i=6021 + ns=2;i=6022 + ns=2;i=6023 + ns=2;i=6024 + ns=2;i=6025 + ns=2;i=6026 + ns=2;i=6027 + + + + DatasetId + Identifier of the dataset. + + i=78 + i=68 + ns=2;i=1003 + + + + Name + Human-readable dataset name. + + i=80 + i=68 + ns=2;i=1003 + + + + Version + Dataset version. + + i=80 + i=68 + ns=2;i=1003 + + + + SourceKind + Whether samples are real, synthetic or mixed. + + i=78 + i=68 + ns=2;i=1003 + + + + SampleCount + Number of samples. + + i=80 + i=68 + ns=2;i=1003 + + + + LabelClasses + Class labels present. + + i=80 + i=68 + ns=2;i=1003 + + + + CreatedAt + Creation time. + + i=80 + i=68 + ns=2;i=1003 + + + + ArtifactUri + Where the dataset can be obtained. Treated as untrusted input. + + i=80 + i=68 + ns=2;i=1003 + + + + Digest + Digest of the dataset artefact. + + i=80 + i=68 + ns=2;i=1003 + + + + DeploymentType + A model made executable somewhere. Aligned with the IDTA 02059 AI Deployment submodel template. InferenceLocation is the on-server versus off-server switch: it changes where the computation happens and therefore the trust boundary, and it changes nothing else. + AiModelManagement + + i=58 + ns=2;i=6028 + ns=2;i=6029 + ns=2;i=6030 + ns=2;i=6031 + ns=2;i=6032 + ns=2;i=6033 + ns=2;i=6034 + ns=2;i=6035 + ns=2;i=6123 + ns=2;i=6124 + ns=2;i=6125 + ns=2;i=6126 + ns=2;i=6127 + ns=2;i=6128 + ns=2;i=6129 + ns=2;i=6130 + ns=2;i=6131 + ns=2;i=6132 + ns=2;i=6133 + ns=2;i=6134 + ns=2;i=6135 + ns=2;i=6136 + ns=2;i=6139 + ns=2;i=6142 + ns=2;i=6146 + ns=2;i=6147 + ns=2;i=6172 + ns=2;i=6173 + ns=2;i=6174 + ns=2;i=6175 + + + + DeploymentId + Identifier of the deployment. + + i=78 + i=68 + ns=2;i=1004 + + + + InferenceLocation + Where inference executes. + + i=78 + i=68 + ns=2;i=1004 + + + + AcceleratorKind + Compute device executing the model. + + i=80 + i=68 + ns=2;i=1004 + + + + AcceleratorName + Free-text accelerator identification, for example an NPU or GPU part name. + + i=80 + i=68 + ns=2;i=1004 + + + + EndpointUri + Inference endpoint when InferenceLocation is not OnServer. Treated as untrusted input and subject to the resolver policy of clause 12. + + i=80 + i=68 + ns=2;i=1004 + + + + LatencyBudget + Latency the deployment is expected to meet. Set by whoever commissioned the deployment; ObservedLatency is what it actually achieved, and clause 6.4.3 compares the two. + + i=80 + i=68 + ns=2;i=1004 + + + + BatchSize + Configured inference batch size. + + i=80 + i=68 + ns=2;i=1004 + + + + State + Runtime state of the deployment. + + i=78 + i=68 + ns=2;i=1004 + + + + LearningJobType + One turn of the capture, label, train and promote loop. It exists so that corrections arriving from a consuming application have somewhere to accumulate and a defined path into a new model version. A Server may implement only the capture stages and leave training to an external MLOps system - the state machine is the same either way. + AiModelManagement + + ns=2;i=1006 + ns=2;i=6036 + ns=2;i=6037 + ns=2;i=6038 + ns=2;i=6039 + ns=2;i=6040 + ns=2;i=6041 + ns=2;i=6042 + ns=2;i=6043 + ns=2;i=6045 + + + + State + Current stage of the loop. This is the PHASE, not the program lifecycle: the inherited CurrentState says whether the job is running, this says what it is doing. Clause 7 requires the two to agree. + + i=78 + i=68 + ns=2;i=1005 + + + + Dataset + Dataset being accumulated or used. + + i=80 + i=68 + ns=2;i=1005 + + + + BaseModel + Model the job starts from. + + i=80 + i=68 + ns=2;i=1005 + + + + CandidateModel + Model produced by the job, awaiting promotion. + + i=80 + i=68 + ns=2;i=1005 + + + + SamplesCollected + Samples accumulated so far, including corrections fed back. + + i=80 + i=68 + ns=2;i=1005 + + + + StartCollection + Begin accumulating samples and corrections into the dataset. + + i=80 + ns=2;i=1005 + + + + StopCollection + Stop accumulating samples. + + i=80 + ns=2;i=1005 + + + + TriggerTraining + Request that a candidate model be trained from the collected dataset. + + i=80 + ns=2;i=1005 + ns=2;i=6044 + + + + OutputArguments + + i=78 + i=68 + ns=2;i=6043 + + i=297Acceptedi=1-1True when the request was queued. + + + PromoteModel + Promote the candidate model so that deployments begin using it. A Server SHALL require a distinct authorization for this Method: it changes what the equipment does without changing anything a reader of the address space would notice, which is precisely the change that needs a separate permission. + + i=80 + ns=2;i=1005 + ns=2;i=6046 + ns=2;i=6047 + + + + InputArguments + + i=78 + i=68 + ns=2;i=6045 + + i=297Deploymenti=17-1Deployment to update, or null for all. + + + OutputArguments + + i=78 + i=68 + ns=2;i=6045 + + i=297PromotedModeli=17-1The model now in use. + + + AiJobType + Abstract base of every long-running AI operation: learning, model import and asynchronous inference. It derives from the OPC 10000-10 ProgramStateMachineType, so the lifecycle - Ready, Running, Suspended, Halted - its transition events and its Start/Suspend/Resume/Halt Methods are inherited rather than reinvented, and every job in this model is auditable the same way. + AiModelManagement + + i=2391 + ns=2;i=6048 + ns=2;i=6049 + ns=2;i=6050 + ns=2;i=6051 + ns=2;i=6052 + ns=2;i=6053 + + + + JobId + Identifier of the job, unique within the Server. + + i=78 + i=68 + ns=2;i=1006 + + + + LastError + Diagnostic for the most recent failure. For a human; SHALL NOT be parsed. + + i=80 + i=68 + ns=2;i=1006 + + + + StartedAt + When the job last entered Running. + + i=80 + i=68 + ns=2;i=1006 + + + + FinishedAt + When the job last left Running, or null while it is running. + + i=80 + i=68 + ns=2;i=1006 + + + + Progress + Fraction complete, 0.0 to 1.0, or null where the job cannot estimate it. A Server SHALL NOT report a value it is guessing: null is informative, a fabricated 0.5 is not. + + i=80 + i=68 + ns=2;i=1006 + + + + RequestedBy + Identity that requested the job, recorded at the moment it started. Clause 12 requires this for any job that can promote a model. + + i=80 + i=68 + ns=2;i=1006 + + + + ModelImportJobType + Brings a model from a catalogue into this Server. It federates by default - materializing the catalogue entry as a ModelType whose artefact stays where it is - and stages the artefact when the target deployment could not otherwise reach it. Staging is the moment a substituted artefact would enter, which is why clause 10 requires the Digest to be verified there and nowhere else. + AiModelManagement + + ns=2;i=1006 + ns=2;i=6054 + ns=2;i=6055 + ns=2;i=6056 + ns=2;i=6057 + ns=2;i=6058 + ns=2;i=6059 + ns=2;i=6060 + ns=2;i=6061 + ns=2;i=6167 + + + + Source + ModelSourceType instance the model is pulled from, where the import calls an endpoint. Null where the import reads a catalogue instead, in which case Registry names it. Exactly one of the two is non-null. + + i=78 + i=68 + ns=2;i=1007 + + + + ModelReference + Publisher, name and version being imported. + + i=78 + i=68 + ns=2;i=1007 + + + + Mode + Whether to federate, stage, or decide from the target's InferenceLocation. + + i=78 + i=68 + ns=2;i=1007 + + + + TargetDeployment + Deployment to create or update on success, or null to import the model without deploying it. + + i=80 + i=68 + ns=2;i=1007 + + + + ImportedModel + ModelType instance the job produced. Null until the job succeeds. + + i=80 + i=68 + ns=2;i=1007 + + + + BytesTransferred + Artefact bytes fetched so far. Zero for a federating import, which moves none. + + i=80 + i=68 + ns=2;i=1007 + + + + DigestVerified + Whether the staged artefact's computed digest matched the one the catalogue declared. False on a staging import means the artefact SHALL NOT be deployed. + + i=80 + i=68 + ns=2;i=1007 + + + + Cancel + Abandon the import. A partially staged artefact SHALL be discarded rather than left where a later deployment could pick it up. + + i=80 + ns=2;i=1007 + + + + InferenceJobType + One asynchronous inference request. It exists because not every inference returns while the caller waits: a batch scored overnight and a long analysis over recorded data are ordinary industrial cases, and modelling them as a Method that blocks for hours is not. + AiModelManagement + + ns=2;i=1006 + ns=2;i=6062 + ns=2;i=6063 + ns=2;i=6064 + ns=2;i=6065 + ns=2;i=6066 + ns=2;i=6067 + ns=2;i=6068 + ns=2;i=6069 + ns=2;i=6070 + ns=2;i=6176 + ns=2;i=6177 + ns=2;i=6178 + ns=2;i=6179 + + + + Deployment + Deployment executing the request. + + i=78 + i=68 + ns=2;i=1008 + + + + RequestPayload + Request body, encoded as RequestContentType states. + + i=80 + i=68 + ns=2;i=1008 + + + + RequestContentType + Media type of RequestPayload. + + i=80 + i=68 + ns=2;i=1008 + + + + ResponsePayload + Response body once the job succeeds. + + i=80 + i=68 + ns=2;i=1008 + + + + ResponseContentType + Media type of ResponsePayload. + + i=80 + i=68 + ns=2;i=1008 + + + + ModelUsed + Model that ACTUALLY executed the request, which is not always the one the deployment named when the job was submitted - a fallback or a followed reference can change it in between. The provenance chain of clause 12 walks this, not the deployment's current model. + + i=80 + i=68 + ns=2;i=1008 + + + + Usage + What the call consumed. + + i=80 + i=68 + ns=2;i=1008 + + + + FinishReason + Why the call stopped producing output. + + i=80 + i=68 + ns=2;i=1008 + + + + SafetyAssessment + Findings from the safety policy, if any were applied. + + i=80 + i=68 + ns=2;i=1008 + + + + ModelSourceType + An externally hosted inference or catalogue endpoint this Server can reach. It carries everything needed to actually call something the Server did not deploy - the wire contract, how to authenticate, what the endpoint can do and whether it is answering - because a URI on its own is a string nobody can act on. + AiModelManagement + + i=58 + ns=2;i=6071 + ns=2;i=6072 + ns=2;i=6073 + ns=2;i=6074 + ns=2;i=6075 + ns=2;i=6076 + ns=2;i=6077 + ns=2;i=6078 + ns=2;i=6079 + ns=2;i=6080 + ns=2;i=6081 + ns=2;i=6082 + ns=2;i=6083 + ns=2;i=6085 + + + + SourceId + Identifier of the source. + + i=78 + i=68 + ns=2;i=1009 + + + + EndpointUri + Base URI of the endpoint. Untrusted input, subject to the resolver policy of clause 12. + + i=78 + i=68 + ns=2;i=1009 + + + + ApiDialect + Wire contract the endpoint speaks. + + i=78 + i=68 + ns=2;i=1009 + + + + EndpointDescriptionUri + Where the contract is documented. SHOULD be populated when ApiDialect is Proprietary, because otherwise nothing in the address space says how to call it. + + i=80 + i=68 + ns=2;i=1009 + + + + AuthenticationKind + How the Server authenticates itself to the endpoint. + + i=78 + i=68 + ns=2;i=1009 + + + + CredentialReference + Opaque handle naming the credential in whatever store the Server uses. It is a NAME, never a secret: clause 12 forbids a Server from exposing credential material through any Attribute of this model, and a client that can read this value learns only which credential is used, not what it is. + + i=80 + i=68 + ns=2;i=1009 + + + + TokenAudience + Audience or scope a bearer token is requested for, where AuthenticationKind is BearerToken. + + i=80 + i=68 + ns=2;i=1009 + + + + Reachability + Whether the Server can currently reach the endpoint. + + i=78 + i=68 + ns=2;i=1009 + + + + LastSuccessAt + When the endpoint last answered successfully. + + i=80 + i=68 + ns=2;i=1009 + + + + ConsecutiveFailures + Failures since the last success. Reset to zero on success. + + i=80 + i=68 + ns=2;i=1009 + + + + RateLimit + Capacity the endpoint is currently granting. + + i=80 + i=68 + ns=2;i=1009 + + + + Capabilities + What the endpoint reports it can do. + + i=80 + i=68 + ns=2;i=1009 + + + + TestConnection + Probe the endpoint and update Reachability. Defined so that a commissioning engineer can establish that credentials and network policy are right BEFORE a deployment depends on them, rather than discovering it from a failed inference. + + i=80 + ns=2;i=1009 + ns=2;i=6084 + + + + OutputArguments + + i=78 + i=68 + ns=2;i=6083 + + i=297Reachablei=1-1Whether the probe succeeded.i=297Detaili=21-1Diagnostic. For a human. + + + ListModels + Enumerate the models the source offers. + + i=80 + ns=2;i=1009 + ns=2;i=6086 + ns=2;i=6087 + + + + InputArguments + + i=78 + i=68 + ns=2;i=6085 + + i=297Filteri=12-1Optional substring or expression; empty for all.i=297MaxResultsi=7-1Upper bound on returned entries.i=297ContinuationPointi=15-1Empty on the first call; otherwise the value the previous call returned. A cap without a cursor bounds the response and puts every entry past it out of reach, which against a public catalogue means most of them. + + + OutputArguments + + i=78 + i=68 + ns=2;i=6085 + + i=297Modelsns=2;i=305110Publisher, name and version of each model offered.i=297ContinuationPointi=15-1Pass to the next call to continue. Empty when the enumeration is complete, which is how a client knows to stop rather than by comparing counts. + + + EvaluationRunType + One measurement of a model against a dataset. It is a first-class object and not a field on the model because the same model is evaluated many times, and because the run that gated a promotion has to remain readable afterwards to answer why the promotion was allowed. + AiModelManagement + + i=58 + ns=2;i=6088 + ns=2;i=6089 + ns=2;i=6090 + ns=2;i=6091 + ns=2;i=6092 + ns=2;i=6093 + ns=2;i=6094 + + + + RunId + Identifier of the run. + + i=78 + i=68 + ns=2;i=1014 + + + + EvaluatedModel + Model that was measured. + + i=78 + i=68 + ns=2;i=1014 + + + + Dataset + Dataset the model was measured against. + + i=80 + i=68 + ns=2;i=1014 + + + + CompletedAt + When the run finished. + + i=80 + i=68 + ns=2;i=1014 + + + + Metrics + Measured metrics, each with the threshold it was judged against. + + i=78 + i=68 + ns=2;i=1014 + + + + Passed + Whether every metric met its threshold. A Server SHALL NOT report true while any entry in Metrics has Passed false - a summary that disagrees with its own detail is worse than no summary. + + i=78 + i=68 + ns=2;i=1014 + + + + ReportUri + Where the full report lives. Untrusted input, subject to clause 12. + + i=80 + i=68 + ns=2;i=1014 + + + + ModelCardType + What a human needs to decide whether a model may be used here: what it is for, where it stops working, and under what terms. Separate from the nameplate because a nameplate answers 'which artefact is this' and a card answers 'should this be running on my line'. + AiModelManagement + + i=58 + ns=2;i=6095 + ns=2;i=6096 + ns=2;i=6097 + ns=2;i=6098 + ns=2;i=6099 + ns=2;i=6100 + ns=2;i=6101 + ns=2;i=6170 + ns=2;i=6171 + + + + IntendedUse + What the model is for. + + i=78 + i=68 + ns=2;i=1015 + + + + Limitations + Where it is known not to work. Mandatory because a card that lists only capabilities is marketing, and the failure modes are the half a commissioning engineer needs. + + i=78 + i=68 + ns=2;i=1015 + + + + OutOfScopeUse + Uses the supplier explicitly excludes. + + i=80 + i=68 + ns=2;i=1015 + + + + License + Licence identifier or URI governing use of the artefact. + + i=80 + i=68 + ns=2;i=1015 + + + + TrainingDataCutoff + Latest date represented in the training data. A model cannot know about anything after this, which is often the explanation for a field failure. + + i=80 + i=68 + ns=2;i=1015 + + + + EthicalConsiderations + Risks the supplier records. + + i=80 + i=68 + ns=2;i=1015 + + + + ContactUri + Where to report a problem with the model. + + i=80 + i=68 + ns=2;i=1015 + + + + ModelRegistryType + A catalogue of models and the datasets they were trained on. It narrows the abstract registry's group placeholder to model publishers, so that a client browsing it knows what it will find rather than discovering it. + AiModelManagement + + ns=1;i=63000 + ns=2;i=6144 + + + + ModelPublisherType + One publisher's namespace within a model registry: the organisation or project that released the models it contains. Publisher is the first element of the publisher/name/version triple by which every catalogue in practice identifies a model. + AiModelManagement + + ns=1;i=63001 + ns=2;i=6145 + + + + AiResourceType + Abstract base of everything a model registry holds. It exists so that the inherited <Resource> placeholder can be narrowed ONCE to something that admits models and datasets and nothing else - a publisher holds both, and a placeholder can be overridden only by one declaration. + AiModelManagement + + ns=1;i=63002 + + + + ModelResourceType + One model in a catalogue. Its versions are immutable and identified by content, so a version that has been seen cannot change meaning; mutable names such as a branch or a release channel are pointers AT versions, never versions themselves. Because the base type is a FileType, a Server that holds the artefact serves it through the inherited Open, Read and Close; one that only describes it leaves those unimplemented and points at the artefact instead. + AiModelManagement + + ns=2;i=1016 + ns=2;i=6102 + ns=2;i=6103 + ns=2;i=6104 + ns=2;i=6105 + ns=2;i=6106 + ns=2;i=6107 + ns=2;i=6108 + ns=2;i=6166 + + + + TaskKind + What the model does, for example 'object-detection' or 'anomaly-detection'. A String and not an enumeration, for the same reason it is one on ModelType: the set is not closed, and every catalogue in practice uses a free tag here. + + i=80 + i=68 + ns=2;i=1012 + + + + Framework + Runtime or library the artefact targets. + + i=80 + i=68 + ns=2;i=1012 + + + + Digest + Digest of the artefact this version names, as the catalogue declares it. A staging import compares its own computed digest with this and refuses on mismatch. + + i=80 + i=68 + ns=2;i=1012 + + + + DigestAlgorithm + Algorithm of Digest. Subject to the strength requirement of clause 12. + + i=80 + i=68 + ns=2;i=1012 + + + + SizeBytes + Artefact size, so a staging import can decide whether it has room before it starts rather than after it fails. + + i=80 + i=68 + ns=2;i=1012 + + + + Gated + Whether obtaining the artefact requires an acceptance or entitlement beyond ordinary authentication. A client that ignores this discovers it as a failure part-way through a staging import. + + i=80 + i=68 + ns=2;i=1012 + + + + MutableRefs + Mutable pointers this resource publishes - branches, tags or channels - that a deployment may follow instead of pinning. Naming them is what makes VersionBinding FollowsRef checkable. + + i=80 + i=68 + ns=2;i=1012 + + + + DatasetResourceType + One dataset in a catalogue, a sibling of ModelResourceType rather than something beneath it: a dataset outlives the models trained on it and is cited by several. + AiModelManagement + + ns=2;i=1016 + ns=2;i=6109 + ns=2;i=6110 + ns=2;i=6111 + ns=2;i=6112 + ns=2;i=6113 + + + + SourceKind + Whether the samples are real, synthetic or mixed. + + i=80 + i=68 + ns=2;i=1013 + + + + SampleCount + Samples in the dataset. + + i=80 + i=68 + ns=2;i=1013 + + + + Digest + Digest of the dataset artefact as the catalogue declares it. + + i=80 + i=68 + ns=2;i=1013 + + + + DigestAlgorithm + Algorithm of Digest. + + i=80 + i=68 + ns=2;i=1013 + + + + SizeBytes + Dataset size. + + i=80 + i=68 + ns=2;i=1013 + + + + Sources + ModelSourceType instances - the externally hosted endpoints and catalogues this Server can reach. + + i=80 + i=61 + ns=2;i=1001 + + + + Registries + ModelRegistryType instances this Server serves or mirrors. + + i=80 + i=61 + ns=2;i=1001 + + + + Evaluations + EvaluationRunType instances. + + i=80 + i=61 + ns=2;i=1001 + + + + Jobs + Import and asynchronous inference jobs. Learning jobs remain under LearningJobs. + + i=80 + i=61 + ns=2;i=1001 + + + + Card + What a human needs to decide whether this model may run here. + + i=80 + ns=2;i=1015 + ns=2;i=1002 + + + + Publisher + Organisation or namespace that published the model. With Name and Version this is the triple every catalogue identifies a model by, and it is what makes the same model recognisable across two installations that fetched it from different mirrors. + + i=80 + i=68 + ns=2;i=1002 + + + + ParameterCount + Parameters in the model, or 0 where not published. A crude but universally available proxy for what it will cost to run. + + i=80 + i=68 + ns=2;i=1002 + + + + Quantization + Numeric precision the artefact is stored in, for example 'fp32', 'int8' or 'fp8'. A quantized model is a DIFFERENT artefact with different behaviour, not a packaging detail, which is why it is stated rather than left to the format string. + + i=80 + i=68 + ns=2;i=1002 + + + + SafetyPolicyUri + Safety or content policy applied to this model's output, where one is. Untrusted input, subject to clause 12. + + i=80 + i=68 + ns=2;i=1002 + + + + Source + ModelSourceType instance this deployment executes through, where inference is not local. Null when InferenceLocation is OnServer. + + i=80 + i=68 + ns=2;i=1004 + + + + VersionBinding + Whether the deployment is pinned to an immutable model version or follows a mutable pointer. + + i=78 + i=68 + ns=2;i=1004 + + + + BoundRef + The mutable pointer being followed, where VersionBinding is FollowsRef. Empty when Pinned. + + i=80 + i=68 + ns=2;i=1004 + + + + FallbackPolicy + What the Server does when this deployment cannot serve. + + i=78 + i=68 + ns=2;i=1004 + + + + Reachability + Whether the execution site is currently reachable. Always Reachable for an OnServer deployment that is not Faulted. + + i=80 + i=68 + ns=2;i=1004 + + + + ConsecutiveFailures + Failed calls since the last success. + + i=80 + i=68 + ns=2;i=1004 + + + + LastSuccessAt + When this deployment last answered successfully. With FallbackPolicy HoldLast this is how a caller judges whether the held answer is still worth having. + + i=80 + i=68 + ns=2;i=1004 + + + + RateLimit + Capacity the execution site is currently granting. + + i=80 + i=68 + ns=2;i=1004 + + + + Capabilities + What this deployment can do. A client checks here before calling a typed profile rather than discovering the answer from a rejection. + + i=80 + i=68 + ns=2;i=1004 + + + + DataJurisdiction + Where input data is processed, named in whatever scheme the operator uses - a site, a legal jurisdiction, or a named zone. This is the question a plant actually asks, and no amount of latency or accuracy data answers it. + + i=78 + i=68 + ns=2;i=1004 + + + + EgressPermitted + Whether calling this deployment sends input data outside the operator's boundary. A Server SHALL set this true for every deployment whose InferenceLocation is Cloud, and SHALL NOT set it false merely because the channel is encrypted - the question is where the data goes, not who can read it in flight. + + i=78 + i=68 + ns=2;i=1004 + + + + RetainsInput + Whether the execution site retains input beyond serving the request, for example for provider-side logging or training. Unknown is not a value: a Server that cannot establish this SHALL report true, because the safe assumption is the one that keeps data in. + + i=80 + i=68 + ns=2;i=1004 + + + + EgressPolicyUri + Where the governing data policy is documented. + + i=80 + i=68 + ns=2;i=1004 + + + + Invoke + Run inference and return the result. The payload is opaque here: what goes in and comes out is the consuming specification's vocabulary, and an envelope that tried to type it would have to be extended for every domain. What this Method fixes is everything AROUND the payload - routing, parameters, accounting, why it stopped, and which model actually ran. + +The signature does not change with InferenceLocation. A deployment served from the Server's own process and one served from a remote service are called identically; the location changes the trust boundary and the latency, and nothing else. + + i=80 + ns=2;i=1004 + ns=2;i=6137 + ns=2;i=6138 + + + + InputArguments + + i=78 + i=68 + ns=2;i=6136 + + i=297Payloadi=15-1Request body.i=297PayloadUrii=12-1Location the request body is read from, where it is supplied by reference rather than carried. A Server SHALL accept exactly one of Payload and PayloadUri and SHALL reject a call supplying both or neither. Untrusted input subject to clause 12.2, and named data the execution site will read, so clause 9.5 applies to it.i=297ContentTypei=12-1Media type of Payload.i=297Parametersi=1453310Call parameters such as a sampling temperature or an output length bound. A Server SHALL reject a parameter it does not support rather than ignore it: a caller whose parameter was silently dropped believes it took effect.i=297Timeouti=290-1How long the caller will wait. Zero means the Server's default. + + + OutputArguments + + i=78 + i=68 + ns=2;i=6136 + + i=297ResponsePayloadi=15-1Response body.i=297ResponseContentTypei=12-1Media type of ResponsePayload.i=297ModelUsedi=17-1The model that ACTUALLY produced this response. Not necessarily the one the deployment names now: a fallback answered from a different deployment, and a FollowsRef binding may have moved. The provenance chain of clause 12 walks this.i=297Usagens=2;i=3052-1What the call consumed.i=297FinishReasonns=2;i=3006-1Why output stopped. A caller that ignores this will accept a truncated answer as a complete one.i=297SafetyAssessmentns=2;i=305410Findings from the safety policy, if any applied.i=297RetryAfteri=290-1How long to wait before retrying, where the failure was a capacity one. Zero when retrying immediately is as good as waiting, and meaningless when the failure was not retryable.i=297TransferRequiredi=1-1True when the deployment produced a response too large to return inline. ResponsePayload is then empty and the work is NOT lost - Transfer names where to read it. A client that ignores this reads an empty payload and concludes the model returned nothing.i=297Transferi=17-1InferenceTransferType instance holding the response, where TransferRequired is true. Null otherwise. + + + InvokeAsync + Submit inference to be completed later, returning immediately with the job that will carry the result. For work that does not finish while a caller waits - a batch scored overnight, an analysis over recorded data. + + i=80 + ns=2;i=1004 + ns=2;i=6140 + ns=2;i=6141 + + + + InputArguments + + i=78 + i=68 + ns=2;i=6139 + + i=297Payloadi=15-1Request body.i=297PayloadUrii=12-1Location the request body is read from, where it is supplied by reference rather than carried. Exactly one of Payload and PayloadUri on the same terms as Invoke. This is the argument that lets a batch already sitting in the plant's object store be scored without being copied through the Session first.i=297ContentTypei=12-1Media type of Payload.i=297Parametersi=1453310Call parameters. + + + OutputArguments + + i=78 + i=68 + ns=2;i=6139 + + i=297Jobi=17-1InferenceJobType instance tracking the request. The caller subscribes to it rather than polling. + + + GetCapabilities + Report what this deployment can do, refreshed from the execution site rather than from cache. Defined because a remote endpoint's capabilities change without anything in this address space changing. + + i=80 + ns=2;i=1004 + ns=2;i=6143 + + + + OutputArguments + + i=78 + i=68 + ns=2;i=6142 + + i=297Capabilitiesns=2;i=305310Current capabilities. + + + <Group> + A publisher namespace held by this registry. Narrows the inherited placeholder so a model registry admits ModelPublisherType and nothing else. + + i=11508 + ns=2;i=1011 + ns=2;i=1010 + + + + <Resource> + A model or dataset published in this namespace. Narrows the inherited placeholder to this model's own resource types. + + i=11508 + ns=2;i=1016 + ns=2;i=1011 + + + + MaxInlinePayloadSize + Largest request or response this deployment will carry inline through Invoke, in bytes. Zero means the deployment accepts no inline payload at all and BeginTransfer is the only way in. + +A client reads this BEFORE calling rather than discovering the bound from a rejection, and a Server SHALL NOT publish a value larger than its own MaxByteStringLength, the negotiated MaxMessageSize or the Session's MaxResponseMessageSize permit - the smallest of those is the real limit and a client cannot see all of them. + + i=78 + i=68 + ns=2;i=1004 + + + + BeginTransfer + Opens a chunked exchange for a payload that will not fit inline, returning the InferenceTransferType instance to write into. This is the general path: Invoke is the shortcut that happens to work when everything is small. + + i=80 + ns=2;i=1004 + ns=2;i=6148 + ns=2;i=6149 + + + + InputArguments + + i=78 + i=68 + ns=2;i=6147 + + i=297ContentTypei=12-1Media type of the request body.i=297RequestSizei=9-1Expected request size in bytes, or 0 when not known in advance. A Server that cannot accommodate the stated size refuses here rather than after the client has uploaded it. + + + OutputArguments + + i=78 + i=68 + ns=2;i=6147 + + i=297Transferi=17-1InferenceTransferType instance to write the request into.i=297Acceptedi=1-1False when the Server declined to open the exchange. + + + AiModelManagement + Entry point for the AI models this Server runs. A client browses Server/AiModelManagement/Models to find what this Server describes. + + ns=2;i=1001 + i=2253 + + + + InferenceTransferType + One chunked inference exchange. It exists because Invoke carries its payload as a ByteString, and a ByteString is bounded by MaxByteStringLength, the negotiated MaxMessageSize and the Session's MaxResponseMessageSize - none of which the model gets to choose. An image, a point cloud or a window of high-rate samples exceeds those routinely, and a call that cannot carry the input is not a call. + +Request and Response are Part 5 FileType objects: the client opens the request, writes it in chunks it selects, and closes it; after Execute the response is read the same way. Nothing here invents a transfer protocol, because OPC UA already has one and every client already implements it. + AiModelManagement + + i=58 + ns=2;i=6150 + ns=2;i=6151 + ns=2;i=6152 + ns=2;i=6153 + ns=2;i=6154 + ns=2;i=6155 + ns=2;i=6156 + ns=2;i=6157 + ns=2;i=6158 + ns=2;i=6159 + ns=2;i=6160 + ns=2;i=6161 + ns=2;i=6162 + ns=2;i=6164 + + + + TransferId + Identifier of this exchange. + + i=78 + i=68 + ns=2;i=1017 + + + + State + Stage the exchange has reached. + + i=78 + i=68 + ns=2;i=1017 + + + + Request + The request body, written by the client in chunks of its own choosing. Inference does not begin until Execute is called, so a partially written request is never acted on. + + i=78 + i=11575 + ns=2;i=1017 + + + + Response + The response body, readable once State is Completed. Empty before that. + + i=78 + i=11575 + ns=2;i=1017 + + + + ContentType + Media type of the request body. + + i=78 + i=68 + ns=2;i=1017 + + + + ResponseContentType + Media type of the response body. + + i=80 + i=68 + ns=2;i=1017 + + + + ModelUsed + The model that ACTUALLY produced the response, on the same terms as Invoke: a fallback or a followed reference can change it between the call and the read. + + i=80 + i=68 + ns=2;i=1017 + + + + Usage + What the call consumed. + + i=80 + i=68 + ns=2;i=1017 + + + + FinishReason + Why output stopped. + + i=80 + i=68 + ns=2;i=1017 + + + + SafetyAssessment + Findings from the safety policy, if any applied. + + i=80 + i=68 + ns=2;i=1017 + + + + LastError + Diagnostic for the Failed state. For a human; SHALL NOT be parsed. + + i=80 + i=68 + ns=2;i=1017 + + + + ExpiresAt + When the Server may reclaim this transfer if it has not completed. A client that abandons an exchange would otherwise hold Server resources until the Session ends, and a Server that never reclaimed them would be one denial of service away from unusable. + + i=80 + i=68 + ns=2;i=1017 + + + + Execute + Runs inference over the written request. The Method returns as soon as the request is accepted; State and the envelope members carry the outcome, which is what lets one exchange span a payload too large to have been a single call in the first place. + + i=78 + ns=2;i=1017 + ns=2;i=6163 + + + + OutputArguments + + i=78 + i=68 + ns=2;i=6162 + + i=297Acceptedi=1-1False when the request was incomplete or already executed. + + + Abort + Abandons the exchange and releases what it holds. A client that has stopped caring about a response SHOULD say so rather than leaving the Server to wait out ExpiresAt. + + i=80 + ns=2;i=1017 + + + + DigestProvenance + Where Digest came from, or why there is none. NotAvailable is the only value permitted with an empty Digest, and it SHALL be used rather than leaving a client to guess whether the source publishes no digest or this Server declined to carry one. + +A Server SHALL NOT put a non-content identifier in Digest to avoid saying NotAvailable. A response fingerprint, a resource name, a storage entity tag and a repository commit identifier are none of them digests of the artefact that ran, and a client that verified against one would believe it had checked something it had not. Where such an identifier is worth publishing it belongs in ArtifactUri or ProvenanceUri, which promise nothing about content. + + i=78 + i=68 + ns=2;i=1002 + + + + DigestProvenance + Where this resource's Digest came from, on the same terms as ModelType. A catalogue that declares a digest it did not compute is DeclaredBySource; one serving the artefact through the inherited Open, Read and Close can reach ComputedByServer. + + i=80 + i=68 + ns=2;i=1012 + + + + Registry + ModelRegistryType instance the model is imported from, where the import reads a catalogue rather than calling an endpoint. Null otherwise. + +A Server SHALL populate exactly one of Source and Registry, and SHALL leave the other null. The two name the two things an import can read from, and a job that named both would not say which one produced the artefact whose digest clause 10.4 verifies. + + i=80 + i=68 + ns=2;i=1007 + + + + PublishedAt + When the source first published this model, where the source states it. The same question DatasetType.CreatedAt answers for a dataset, and the same reason: a model trained before a process change may no longer represent the line it runs on, and Version is a vendor string that often cannot be ordered. + +This is the source's date, not when this Server learned of it - a Server SHALL NOT substitute its own acquisition time, which would make every model appear to date from the last restart. + + i=80 + i=68 + ns=2;i=1002 + + + + LastModifiedAt + When the artefact behind this model last changed at the source. + +It exists for the FollowsRef case of clause 9.3, where the artefact can change with nothing else changing. Clause 12.3.1 requires repointing to be treated as an authorization-bearing act and points at AiJobType.RequestedBy for the record - but a reference that moves AT THE SOURCE produces no job, so without this member the audit trail that clause demands cannot be constructed on the one path it exists to cover. A Server that follows a mutable reference SHALL populate it. + + i=80 + i=68 + ns=2;i=1002 + + + + DeprecatedFrom + When the source stops treating this model as current while continuing to serve it. The date that starts a requalification, not the one that ends production. + + i=80 + i=68 + ns=2;i=1015 + + + + SupportedUntil + When the source stops serving this model altogether. + +Its consequence is not degradation. On this date the deployment stops, Reachability goes Unreachable, and FallbackPolicy decides what happens next - which, where it is FallBackTo, means the line keeps producing and something outside the qualified configuration is answering. A date that was knowable a year in advance therefore becomes an unplanned change of model, and it is published by the serving system in machine-readable form. + + i=80 + i=68 + ns=2;i=1015 + + + + ApiDialect + The contract a client's Payload must satisfy when calling Invoke on this deployment. RestChatCompletions means the Payload is a chat-completions request body; OpenInferenceProtocol means it is an OIP inference body; EmbeddedRuntime and TensorRemoteProcedure name the tensor contracts described by Inputs and Outputs; Proprietary means the contract is named only by EndpointDescriptionUri. + +This does not type the payload - clause 8.2 keeps it opaque and that is unchanged. It names WHICH contract the opaque bytes are expected to satisfy, which is what a client browsing an unfamiliar deployment needs before it can send anything at all. + + i=80 + i=68 + ns=2;i=1004 + + + + EndpointDescriptionUri + Where the request and response contract for this deployment is documented. Untrusted input, subject to clause 12.2. Required in practice wherever ApiDialect is Proprietary, because nothing else then says what to send. + + i=80 + i=68 + ns=2;i=1004 + + + + RuntimeIdentity + Opaque identifier of the serving configuration currently behind this deployment - a serving-stack fingerprint, an engine profile, a container image digest. Compared for equality and never parsed, on the same terms as Digest. + +It is not the model. The same artefact served by two runtime builds can produce different numbers, and where the execution site publishes such an identity it is the only thing that records the difference. A change to it under a Pinned binding IS the observable change to the deployment that clause 9.3 says a pinned artefact cannot move without. + + i=80 + i=68 + ns=2;i=1004 + + + + ObservedLatency + Most recent inference latency this Server measured for this deployment. + +LatencyBudget states what the deployment is expected to meet, and clause 6.4.3 makes Degraded the state of a deployment that is answering but missing it. Without a measurement the comparison has no published input, so the state transition could not be checked against a Server that claimed it. A Server that reports Degraded on latency grounds SHALL populate this. + + i=80 + i=68 + ns=2;i=1004 + + + + RequestUri + Where the request body was read from, where it was supplied by reference rather than carried. Untrusted input under clause 12.2, and an egress path under clause 9.5. + + i=80 + i=68 + ns=2;i=1008 + + + + ResponseUri + Where the result was written, where the execution site returns a location rather than bytes. Empty when the response is carried inline or through Transfer. + + i=80 + i=68 + ns=2;i=1008 + + + + TransferRequired + True when the job produced a response too large to carry inline. ResponsePayload is then empty and the work is NOT lost - Transfer names where to read it. + + i=80 + i=68 + ns=2;i=1008 + + + + Transfer + InferenceTransferType instance holding the response, where TransferRequired is true. Null otherwise. + +Invoke carries the same pair, and the asymmetry would otherwise leave the jobs most likely to produce a large result - a batch scored overnight, an analysis over recorded data - bounded by exactly the three limits clause 8.2.4 says this model does not get to choose. + + i=80 + i=68 + ns=2;i=1008 + + + diff --git a/src/Opc.Ua.AI/NugetREADME.md b/src/Opc.Ua.AI/NugetREADME.md new file mode 100644 index 0000000000..f12f5420cd --- /dev/null +++ b/src/Opc.Ua.AI/NugetREADME.md @@ -0,0 +1,20 @@ +# OPC UA AI Model Management and Inference (draft) model + +Server- and client-independent contracts and the source-generated model for the draft *OPC UA - AI Model Management and Inference* companion specification: model catalogues, datasets, deployments, inference endpoints and learning jobs, over the abstract registry in *OPC UA - xRegistry*. + +Part of the [OPC UA .NET Standard](https://github.com/OPCFoundation/UA-.NETStandard) stack. + +> **Draft.** The *OPC UA - AI Model Management and Inference* companion +> specification is a working draft. Its namespace URI and every NodeId are +> provisional, and every ObjectType and BrowseName can change when the working +> group publishes. + +## Documentation + +See the [AI Model Management guide](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/samples/AI/README.md) +for the example: `ModelManagementServer` publishes a catalogue and +routes inference, and `ModelManagementClient` walks it. + +## License + +MIT - see the [license](https://opcfoundation.org/license/mit.html). diff --git a/src/Opc.Ua.AI/Opc.Ua.AI.csproj b/src/Opc.Ua.AI/Opc.Ua.AI.csproj new file mode 100644 index 0000000000..a1debf9545 --- /dev/null +++ b/src/Opc.Ua.AI/Opc.Ua.AI.csproj @@ -0,0 +1,55 @@ + + + $(AssemblyPrefix).AI + $(LibTargetFrameworks) + $(PackagePrefix).Opc.Ua.AI + Opc.Ua.AI + $(NoWarn);CS1591;CS0108 + enable + Server/client-independent OPC UA AI Model Management and Inference (draft) companion contracts and source-generated model, exposing model identifiers, typed states and proxies for model catalogues, datasets, deployments, inference endpoints and learning jobs over the OPC UA xRegistry base. + true + NugetREADME.md + true + true + + + $(PackageId).Debug + + + + + + + + + + + + + Analyzer + false + + + + + + + + + http://opcfoundation.org/UA/AI/ + Opc.Ua.AI + + + + + v105 + true + true + + + + + diff --git a/src/Opc.Ua.Client/Session/Subscription/DefaultSubscriptionEngine.cs b/src/Opc.Ua.Client/Session/Subscription/DefaultSubscriptionEngine.cs index 6aadf26290..d89f19fbb8 100644 --- a/src/Opc.Ua.Client/Session/Subscription/DefaultSubscriptionEngine.cs +++ b/src/Opc.Ua.Client/Session/Subscription/DefaultSubscriptionEngine.cs @@ -216,6 +216,27 @@ public EngineContextAdapter( m_timeProvider = timeProvider; } + /// + public int SessionSubscriptionCount + { + get + { +#if OPCUA_V1_CLIENT + int count = 0; + foreach (Subscription subscription in m_context.Subscriptions) + { + if (subscription.Created) + { + count++; + } + } + return count; +#else + return 0; +#endif + } + } + /// public IManagedSubscription CreateSubscription( ISubscriptionNotificationHandler handler, @@ -263,6 +284,19 @@ public ValueTask DeleteSubscriptionsAsync( requestHeader, subscriptionIds, ct); } + /// + public bool TryDispatchToSessionSubscription( + uint subscriptionId, + NotificationMessage message, + ArrayOf availableSequenceNumbers, + ArrayOf stringTable, + bool moreNotifications) + { + return m_context.TryDispatchToSessionSubscription( + subscriptionId, message, availableSequenceNumbers, + stringTable, moreNotifications); + } + private readonly ISubscriptionEngineContext m_context; private readonly TimeProvider m_timeProvider; } diff --git a/src/Opc.Ua.Client/Session/Subscription/ISubscriptionEngineContext.cs b/src/Opc.Ua.Client/Session/Subscription/ISubscriptionEngineContext.cs index 54cdefc45f..c6043bfe3c 100644 --- a/src/Opc.Ua.Client/Session/Subscription/ISubscriptionEngineContext.cs +++ b/src/Opc.Ua.Client/Session/Subscription/ISubscriptionEngineContext.cs @@ -259,6 +259,33 @@ void AsyncRequestCompleted( /// delete. ValueTask DeleteOrphanedSubscriptionAsync(uint subscriptionId); + /// + /// Delivers a publish response to a subscription the session holds outside + /// the subscription manager's registry, for example one created through the + /// classic Session.AddSubscription API. + /// + /// + /// Without this the V2 publish worker has no way to reach a classic + /// subscription: it cannot resolve the identifier, so the notification is + /// dropped and the subscription goes silent while still reporting itself as + /// created and publishing. + /// + /// The identifier from the publish response. + /// The notification message to deliver. + /// Sequence numbers still available + /// for republish on the server. + /// The response string table. + /// Whether the server has more notifications + /// queued for this subscription. + /// true when the session owned the identifier and the + /// message was delivered. + bool TryDispatchToSessionSubscription( + uint subscriptionId, + NotificationMessage message, + ArrayOf availableSequenceNumbers, + ArrayOf stringTable, + bool moreNotifications); + /// /// The number of good outstanding publish requests that /// are not defunct. diff --git a/src/Opc.Ua.Client/Session/Subscription/SessionEngineContext.cs b/src/Opc.Ua.Client/Session/Subscription/SessionEngineContext.cs index 9e91aa1037..f73127bbd2 100644 --- a/src/Opc.Ua.Client/Session/Subscription/SessionEngineContext.cs +++ b/src/Opc.Ua.Client/Session/Subscription/SessionEngineContext.cs @@ -301,6 +301,43 @@ public ValueTask DeleteOrphanedSubscriptionAsync( subscriptionId); } + /// + public bool TryDispatchToSessionSubscription( + uint subscriptionId, + NotificationMessage message, + ArrayOf availableSequenceNumbers, + ArrayOf stringTable, + bool moreNotifications) + { + if (subscriptionId == 0 || message == null) + { + return false; + } + + Subscription? target = null; + foreach (Subscription subscription in m_session.Subscriptions) + { + if (subscription.Id == subscriptionId) + { + target = subscription; + break; + } + } + if (target == null) + { + return false; + } + + message.MoreNotifications = moreNotifications; + message.StringTable = stringTable; + target.SaveMessageInCache(availableSequenceNumbers, message); + + OnPublishNotification( + target, + new NotificationEventArgs(target, message, stringTable)); + return true; + } + private void RaisePublishNotification( NotificationEventHandler callback, NotificationEventArgs args) @@ -345,5 +382,4 @@ public static partial void SessionUnexpectedErrorWhileRaisingNotification( this ILogger logger, Exception? exception); } - } diff --git a/src/Opc.Ua.Client/Subscription/ISubscriptionManagerContext.cs b/src/Opc.Ua.Client/Subscription/ISubscriptionManagerContext.cs index 73ba8c4622..d67b31a425 100644 --- a/src/Opc.Ua.Client/Subscription/ISubscriptionManagerContext.cs +++ b/src/Opc.Ua.Client/Subscription/ISubscriptionManagerContext.cs @@ -38,6 +38,18 @@ namespace Opc.Ua.Client.Subscriptions /// internal interface ISubscriptionManagerContext { + /// + /// Gets the number of created subscriptions the session owns outside this + /// manager's registry, for example through the classic + /// Session.AddSubscription API. + /// + /// + /// These subscriptions still need Publish workers. Without including them in + /// worker sizing, a session with only classic subscriptions never issues a + /// Publish request even though notifications can be dispatched to it. + /// + int SessionSubscriptionCount { get; } + /// /// Create a managed subscription /// @@ -96,5 +108,33 @@ ValueTask DeleteSubscriptionsAsync( RequestHeader? requestHeader, ArrayOf subscriptionIds, CancellationToken ct = default); + + /// + /// Delivers a publish response to a subscription the session holds outside + /// this manager's registry, for example one created through the classic + /// Session.AddSubscription API. + /// + /// + /// A session can carry subscriptions the manager never created. Their + /// identifiers do not resolve here, and dropping the notification would + /// leave the subscription silent while it still reports itself as created + /// and publishing; deleting it as abandoned would destroy a live + /// subscription the application owns. + /// + /// The identifier from the publish response. + /// The notification message to deliver. + /// Sequence numbers still available + /// for republish on the server. + /// The response string table. + /// Whether the server has more notifications + /// queued for this subscription. + /// true when the session owned the identifier and the + /// message was delivered. + bool TryDispatchToSessionSubscription( + uint subscriptionId, + NotificationMessage message, + ArrayOf availableSequenceNumbers, + ArrayOf stringTable, + bool moreNotifications); } } diff --git a/src/Opc.Ua.Client/Subscription/SubscriptionManager.cs b/src/Opc.Ua.Client/Subscription/SubscriptionManager.cs index 2df6c216bc..125279cfc2 100644 --- a/src/Opc.Ua.Client/Subscription/SubscriptionManager.cs +++ b/src/Opc.Ua.Client/Subscription/SubscriptionManager.cs @@ -1515,7 +1515,7 @@ private async Task PublishControllerAsync(CancellationToken ct) int GetDesiredPublishWorkerCount() { - int publishCount = CreatedCount; + int publishCount = CreatedCount + m_session.SessionSubscriptionCount; if (publishCount != 0) { // @@ -1752,6 +1752,21 @@ await DelayUnresolvedSubscriptionAsync(ct) await DelayUnresolvedSubscriptionAsync(ct) .ConfigureAwait(false); } + else if (m_outer.m_session.TryDispatchToSessionSubscription( + subscriptionId, + notificationMessage, + availableSequenceNumbers, + response.ResponseHeader.StringTable, + moreNotifications)) + { + // The session holds this subscription through the classic + // API. It is live and owned by the application, so the + // notification belongs to it: deliver rather than drop it, + // and never delete it as abandoned. + Interlocked.Increment(ref m_outer.m_goodPublishRequestCount); + m_lastUnknownSubscriptionId = 0; + m_consecutiveUnresolvedResponses = 0; + } else { m_logger.PublishWorkerReceivedUnknownSubscription( diff --git a/src/Opc.Ua.Client/Utils/OptionsReader.cs b/src/Opc.Ua.Client/Utils/OptionsReader.cs index d33dd4fe31..1181af2921 100644 --- a/src/Opc.Ua.Client/Utils/OptionsReader.cs +++ b/src/Opc.Ua.Client/Utils/OptionsReader.cs @@ -42,7 +42,7 @@ namespace Opc.Ua /// /// internal sealed class OptionsReader + DynamicallyAccessedMemberTypes.PublicParameterlessConstructor)] TOptions> : IDisposable where TChange : struct where TOptions : class { @@ -59,7 +59,7 @@ public OptionsReader(IOptionsMonitor options, { SingleReader = true }); - options.OnChange((o, n) => + m_subscription = options.OnChange((o, n) => { TChange? change = convert(o); if (change != null) @@ -69,6 +69,20 @@ public OptionsReader(IOptionsMonitor options, }); } + /// + /// Stops observing the options monitor. + /// + /// + /// An is normally a singleton, so a + /// reader that never released its registration would be held by it for the + /// life of the process and every options change would fan out to readers + /// nobody is draining. + /// + public void Dispose() + { + m_subscription?.Dispose(); + } + /// /// Wait for changes /// @@ -90,6 +104,7 @@ public bool TryGetNextChange(out TChange change) } private readonly Channel m_changes; + private readonly IDisposable? m_subscription; } /// @@ -97,7 +112,7 @@ public bool TryGetNextChange(out TChange change) /// /// internal sealed class OptionsReader<[DynamicallyAccessedMembers( - DynamicallyAccessedMemberTypes.PublicParameterlessConstructor)] TOptions> + DynamicallyAccessedMemberTypes.PublicParameterlessConstructor)] TOptions> : IDisposable where TOptions : class { /// @@ -114,7 +129,7 @@ public OptionsReader(IOptionsMonitor options, int capacity = 5) FullMode = BoundedChannelFullMode.DropOldest }); - options.OnChange((o, n) => + m_subscription = options.OnChange((o, n) => { if (o == null) { @@ -129,6 +144,20 @@ public OptionsReader(IOptionsMonitor options, int capacity = 5) }); } + /// + /// Stops observing the options monitor. + /// + /// + /// An is normally a singleton, so a + /// reader that never released its registration would be held by it for the + /// life of the process and every options change would fan out to readers + /// nobody is draining. + /// + public void Dispose() + { + m_subscription?.Dispose(); + } + /// /// Wait for changes /// @@ -150,6 +179,7 @@ public bool TryGet([MaybeNullWhen(false)] out TOptions? change) } private readonly Channel m_changes; + private readonly IDisposable? m_subscription; private TOptions? m_current; } } diff --git a/src/Opc.Ua.Core/Stack/Client/ObjectTypeClient.cs b/src/Opc.Ua.Core/Stack/Client/ObjectTypeClient.cs index 68eae0d7eb..a1ecfe57d4 100644 --- a/src/Opc.Ua.Core/Stack/Client/ObjectTypeClient.cs +++ b/src/Opc.Ua.Core/Stack/Client/ObjectTypeClient.cs @@ -226,6 +226,7 @@ protected async ValueTask ResolveChildNodeIdAsync( new RelativePathElement { ReferenceTypeId = ReferenceTypeIds.HasComponent, + IsInverse = false, IncludeSubtypes = true, TargetName = new QualifiedName(browseName, (ushort)nsIdx) } diff --git a/src/Opc.Ua.OpenUsd.Client/OpenUsdConnector.cs b/src/Opc.Ua.OpenUsd.Client/OpenUsdConnector.cs index 6fbc0fb40f..dcfc91b3a2 100644 --- a/src/Opc.Ua.OpenUsd.Client/OpenUsdConnector.cs +++ b/src/Opc.Ua.OpenUsd.Client/OpenUsdConnector.cs @@ -350,8 +350,8 @@ public async Task> DiscoverAllRepresentationsAsync(Canc { continue; } - if (!typeDef.IsNull - && m_bindingTypeIntents.TryGetValue(typeDef, out OpenUsdIntentProfile intent)) + if (!typeDef.IsNull && + m_bindingTypeIntents.TryGetValue(typeDef, out OpenUsdIntentProfile intent)) { Dictionary bp = await ChildrenByNameAsync(childId, ct) .ConfigureAwait(false); @@ -509,7 +509,8 @@ private async Task> ResolveBySemanticIdAsync( { string? name = refs[i].BrowseName.Name; string target = ExpandedNodeId.ToNodeId(refs[i].NodeId, m_session.NamespaceUris) - .ToString() ?? string.Empty; + .ToString() ?? + string.Empty; if (string.Equals(name, semanticId, StringComparison.Ordinal) || string.Equals(target, semanticId, StringComparison.Ordinal) || string.Equals(refs[i].NodeId.ToString(), semanticId, StringComparison.Ordinal)) @@ -576,8 +577,10 @@ private async Task> VariablesInSubtreeAsync(NodeId root, Cancellati seen.Add(root); // The represented Object's subtree is bounded; two levels of nesting cover the // Object -> (Folder) -> Variable shapes the binding model uses. - int depth = 0; - while (queue.Count > 0 && depth < 4) + for ( + // The represented Object's subtree is bounded; two levels of nesting cover the + // Object -> (Folder) -> Variable shapes the binding model uses. + int depth = 0; queue.Count > 0 && depth < 4; depth++) { int level = queue.Count; for (int i = 0; i < level; i++) @@ -600,7 +603,6 @@ private async Task> VariablesInSubtreeAsync(NodeId root, Cancellati } } } - depth++; } return found; } @@ -656,20 +658,24 @@ public async Task StartAsync(CancellationToken ct) m_subscription = subscription; m_session.AddSubscription(subscription); await subscription.CreateAsync(ct).ConfigureAwait(false); + int bindingCount = 0; + int monitoredCount = 0; foreach (RepresentationInfo rep in reps) { foreach (BindingInfo b in rep.Bindings) { + bindingCount++; + // Command bindings are actuated on demand (IssueCommandAsync), and // history bindings are replayed via ReplayHistoryAsync — neither is a // live MonitoredItem. Telemetry and alarm bindings subscribe here. // §5.4: Enabled = false is a tombstone — a suppressed binding is not // subscribed at all. - if (!b.Enabled - || b.SourceNodeId.IsNull - || b.Intent == OpenUsdIntentProfile.UsdToUaCommand - || b.Intent == OpenUsdIntentProfile.UaHistoryToUsd) + if (!b.Enabled || + b.SourceNodeId.IsNull || + b.Intent == OpenUsdIntentProfile.UsdToUaCommand || + b.Intent == OpenUsdIntentProfile.UaHistoryToUsd) { continue; } @@ -684,9 +690,11 @@ public async Task StartAsync(CancellationToken ct) }; item.Notification += OnNotification; subscription.AddItem(item); + monitoredCount++; } } await subscription.ApplyChangesAsync(ct).ConfigureAwait(false); + m_logger.LiveBindingsSubscribed(bindingCount, reps.Count, monitoredCount); // Compose each representation's components into the USD prim tree (§5.12): // author child/reference/instance prims and federate to remote servers @@ -819,6 +827,22 @@ private void OnNotification(MonitoredItem item, MonitoredItemNotificationEventAr if (!usdValue.IsNull) { m_sink.SetAttribute(b.PrimPath!, b.PropertyName!, usdValue); + if (m_logger.IsEnabled(LogLevel.Debug)) + { + m_logger.LiveUpdateApplied( + b.PrimPath ?? string.Empty, + b.PropertyName ?? string.Empty, + b.SourceNodeId.ToString()); + } + } + else + { + // Silence here is what makes an unresolved target so hard to find: the + // prim simply never moves, while every subscription counter says the + // data is arriving. Say so instead. + m_logger.LiveUpdateUnresolved( + b.PrimPath ?? string.Empty, b.PropertyName ?? string.Empty, + b.SourceNodeId.ToString(), b.Kind.ToString()); } } } @@ -880,7 +904,6 @@ public static Variant Convert(BindingInfo b, Variant raw) return TryToDouble(raw, out double v) ? new Variant(v != 0.0 ? "inherited" : "invisible") : default; - case OpenUsdRenderTargetKind.Transform: // A matrix4d/quaternion target requires the full §5.8 matrix profile // (row-major, row-vector, translation in the 4th row; quaternions // reordered (x,y,z,w) -> (w,x,y,z) and normalised). That profile is not @@ -997,8 +1020,7 @@ private static GeoComponent GeoComponentOf(string? propertyName) } string name = propertyName!; int sep = name.LastIndexOfAny([':', '.', '/']); - string leaf = (sep >= 0 ? name.Substring(sep + 1) : name).ToLowerInvariant(); - switch (leaf) + switch ((sep >= 0 ? name.Substring(sep + 1) : name).ToLowerInvariant()) { case "latitude": case "longitude": @@ -1114,7 +1136,9 @@ private static bool TryToDouble(Variant v, out double result) return VariantConversions.TryGetDouble(v, out result); } - // UNECE common codes used by the §5.8 unit profiles. + /// + /// UNECE common codes used by the §5.8 unit profiles. + /// private const string kUneceRadian = "C81"; private const string kUneceDegree = "DD"; @@ -1183,7 +1207,8 @@ private static double UnitFactor(BindingInfo b) { string source = UnitCode(b.SourceEngineeringUnits); string target = UnitCode(b.TargetEngineeringUnits); - if (source.Length == 0 || target.Length == 0 || + if (source.Length == 0 || + target.Length == 0 || string.Equals(source, target, StringComparison.Ordinal)) { return 1.0; @@ -1228,8 +1253,7 @@ internal static string UnitCode(EUInformation? units) return new string(chars, 0, n); } } - string display = units.DisplayName.Text ?? string.Empty; - return display; + return units.DisplayName.Text ?? string.Empty; } /// @@ -1351,10 +1375,10 @@ await m_session.CallAsync(methodOwner, cmd.CommandMethodId, ct, { foreach (BindingInfo b in r.Bindings) { - if (b.Enabled - && b.Intent == OpenUsdIntentProfile.UsdToUaCommand - && b.SignalRole == OpenUsdSignalRole.Controllable - && (!b.CommandTargetNodeId.IsNull || !b.CommandMethodId.IsNull)) + if (b.Enabled && + b.Intent == OpenUsdIntentProfile.UsdToUaCommand && + b.SignalRole == OpenUsdSignalRole.Controllable && + (!b.CommandTargetNodeId.IsNull || !b.CommandMethodId.IsNull)) { return b; } @@ -1507,10 +1531,10 @@ public async Task ReplayHistoryAsync(DateTime startTime, DateTime endTime, { // §5.4: Enabled = false is a tombstone — a suppressed history binding // is not replayed. - if (!b.Enabled - || b.Intent != OpenUsdIntentProfile.UaHistoryToUsd - || b.SourceNodeId.IsNull - || !b.TimeSampled) + if (!b.Enabled || + b.Intent != OpenUsdIntentProfile.UaHistoryToUsd || + b.SourceNodeId.IsNull || + !b.TimeSampled) { continue; } diff --git a/src/Opc.Ua.OpenUsd.Client/OpenUsdConnectorLog.cs b/src/Opc.Ua.OpenUsd.Client/OpenUsdConnectorLog.cs index dae3d708aa..5e6b5aae74 100644 --- a/src/Opc.Ua.OpenUsd.Client/OpenUsdConnectorLog.cs +++ b/src/Opc.Ua.OpenUsd.Client/OpenUsdConnectorLog.cs @@ -67,5 +67,22 @@ public static partial void CrossServerFederationFailed( Message = "Refusing OpenUSD command: the session does not hold the write/Call " + "authorization required by {TargetNodeId}.")] public static partial void CommandRefusedUnauthorized(this ILogger logger, string targetNodeId); + + [LoggerMessage(EventId = OpenUsdEventIds.Connector + 7, Level = LogLevel.Information, + Message = "OpenUSD live stream bound {BindingCount} binding(s) across " + + "{RepresentationCount} representation(s) and is monitoring {MonitoredCount} item(s).")] + public static partial void LiveBindingsSubscribed( + this ILogger logger, int bindingCount, int representationCount, int monitoredCount); + + [LoggerMessage(EventId = OpenUsdEventIds.Connector + 8, Level = LogLevel.Debug, + Message = "OpenUSD live update: {PrimPath}.{PropertyName} resolved from {SourceNodeId}.")] + public static partial void LiveUpdateApplied( + this ILogger logger, string primPath, string propertyName, string sourceNodeId); + + [LoggerMessage(EventId = OpenUsdEventIds.Connector + 9, Level = LogLevel.Warning, + Message = "OpenUSD live update left {PrimPath}.{PropertyName} unresolved: the value from " + + "{SourceNodeId} is not one the {Kind} profile accepts, so the prim will not follow it.")] + public static partial void LiveUpdateUnresolved( + this ILogger logger, string primPath, string propertyName, string sourceNodeId, string kind); } } diff --git a/src/Opc.Ua.Robotics.Client/Intent/MissionHandle.cs b/src/Opc.Ua.Robotics.Client/Intent/MissionHandle.cs new file mode 100644 index 0000000000..551ca75d56 --- /dev/null +++ b/src/Opc.Ua.Robotics.Client/Intent/MissionHandle.cs @@ -0,0 +1,340 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua.RobotIntent; + +namespace Opc.Ua.Robotics.Client.Intent +{ + /// + /// Delegate invoked when a mission snapshot changes. + /// + public delegate void MissionChangedHandler(MissionSnapshot snapshot); + + /// + /// Awaitable handle for a MissionType instance. Reconnect-safe: re-reads the + /// mission state after a session reconnect so the caller never stalls on a + /// completion task that will never fire. + /// + public sealed class MissionHandle : IAsyncDisposable + { + /// + /// Creates a handle for a mission. + /// + public MissionHandle(RobotIntentControllerClient controller, string missionId, NodeId missionNode) + { + m_controller = controller ?? throw new ArgumentNullException(nameof(controller)); + MissionId = missionId; + MissionNode = missionNode; + m_controller.Transport.Reconnected += OnReconnected; + } + + /// + /// Raised when an observed value changes or the state is re-read after reconnect. + /// + public event MissionChangedHandler? Changed; + + /// + /// Gets the mission id. + /// + public string MissionId { get; } + + /// + /// Gets the mission node. + /// + public NodeId MissionNode { get; } + + /// + /// Gets the last known snapshot. + /// + public MissionSnapshot Current + { + get + { + lock (m_lock) + { + return m_current; + } + } + private set + { + lock (m_lock) + { + m_current = value; + } + } + } + + /// + /// Gets a task that completes when the mission reaches a terminal execution state. + /// + public Task Completion => m_completion.Task; + + /// + /// Starts observation by reading the initial state. Subscribe before reading + /// to close the fast-completion race. + /// + public async ValueTask StartAsync(CancellationToken cancellationToken = default) + { + m_executionStateNode = await m_controller.Transport.ResolveChildAsync( + MissionNode, + "ExecutionState", + cancellationToken).ConfigureAwait(false); + ArrayOf nodes = [m_executionStateNode]; + m_pumpTask = PumpAsync(nodes, m_disposeCts.Token); + await RefreshAsync(cancellationToken).ConfigureAwait(false); + } + + /// + /// Re-reads the mission snapshot after reconnect or on demand. + /// + public async ValueTask RefreshAsync(CancellationToken cancellationToken = default) + { + MissionSnapshot snapshot = await m_controller.Transport.ReadMissionSnapshotAsync( + MissionNode, + cancellationToken).ConfigureAwait(false); + Apply(snapshot, fullyObserved: true); + } + + /// + /// Requests cancellation of the mission. + /// + public ValueTask CancelAsync( + StopModeEnum stopMode, + CancellationToken cancellationToken = default) + { + return m_controller.Transport.CancelMissionAsync(MissionId, stopMode, cancellationToken); + } + + /// + /// Waits up to the timeout for the mission to complete, returning the current state on timeout. + /// + /// + public async ValueTask WaitForCompletionAsync( + TimeSpan timeout, + CancellationToken cancellationToken = default) + { + if (timeout < TimeSpan.Zero && timeout != Timeout.InfiniteTimeSpan) + { + throw new ArgumentOutOfRangeException(nameof(timeout)); + } + + if (Completion.IsCompleted) + { + MissionSnapshot terminal = await Completion.ConfigureAwait(false); + return new MissionWaitResult + { + Completed = true, + TerminalState = terminal.ExecutionState, + Failure = terminal.Failure, + FailureMessage = terminal.FailureMessage, + Current = terminal + }; + } + + Task delay = timeout == Timeout.InfiniteTimeSpan + ? Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken) + : Task.Delay(timeout, cancellationToken); + Task completed = await Task.WhenAny(Completion, delay).ConfigureAwait(false); + if (ReferenceEquals(completed, Completion)) + { + MissionSnapshot terminal = await Completion.ConfigureAwait(false); + return new MissionWaitResult + { + Completed = true, + TerminalState = terminal.ExecutionState, + Failure = terminal.Failure, + FailureMessage = terminal.FailureMessage, + Current = terminal + }; + } + + cancellationToken.ThrowIfCancellationRequested(); + await RefreshAsync(cancellationToken).ConfigureAwait(false); + MissionSnapshot current = Current; + return new MissionWaitResult + { + Completed = false, + TerminalState = current.ExecutionState, + Failure = current.Failure, + FailureMessage = current.FailureMessage, + Current = current + }; + } + + /// + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref m_disposed, 1) != 0) + { + return; + } + + try + { + await m_disposeCts.CancelAsync().ConfigureAwait(false); + m_controller.Transport.Reconnected -= OnReconnected; + if (m_pumpTask != null) + { + try + { + await m_pumpTask.ConfigureAwait(false); + } + catch (OperationCanceledException) + { + } + catch (Exception exception) + { + m_controller.Transport.Logger.MissionSubscriptionFailed(exception, MissionId, MissionNode); + } + } + } + finally + { + m_disposeCts.Dispose(); + } + } + + private async Task PumpAsync(ArrayOf nodes, CancellationToken cancellationToken) + { + try + { + await foreach (RobotIntentDataChange change in m_controller.Transport + .SubscribeDataChangesAsync(nodes, cancellationToken).ConfigureAwait(false)) + { + if (Matches(change.NodeId, m_executionStateNode) && + TryGetEnumValue(change.Value, out ExecutionStateEnum state)) + { + MissionSnapshot snapshot = Current with { ExecutionState = state }; + if (Apply(snapshot, fullyObserved: false) && RobotIntentRules.IsTerminal(state)) + { + await RefreshAsync(cancellationToken).ConfigureAwait(false); + } + } + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + } + catch (Exception exception) + { + m_controller.Transport.Logger.MissionSubscriptionFailed(exception, MissionId, MissionNode); + } + } + + private bool Apply(MissionSnapshot snapshot, bool fullyObserved) + { + bool completed = false; + lock (m_lock) + { + m_current = snapshot; + if (RobotIntentRules.IsTerminal(snapshot.ExecutionState) && fullyObserved) + { + completed = m_completion.TrySetResult(snapshot); + } + } + Changed?.Invoke(snapshot); + return !fullyObserved && RobotIntentRules.IsTerminal(snapshot.ExecutionState); + } + + private void OnReconnected() + { + if (Volatile.Read(ref m_disposed) != 0) + { + return; + } + _ = RefreshAfterReconnectAsync(); + } + + private async Task RefreshAfterReconnectAsync() + { + try + { + await RefreshAsync(m_disposeCts.Token).ConfigureAwait(false); + } + catch (OperationCanceledException) when (m_disposeCts.IsCancellationRequested) + { + } + catch (Exception exception) + { + m_controller.Transport.Logger.MissionSubscriptionFailed(exception, MissionId, MissionNode); + } + } + + private static bool Matches(NodeId observed, NodeId expected) + { + return !expected.IsNull && observed == expected; + } + + private static bool TryGetEnumValue( + Variant value, out TEnum result) where TEnum : struct, Enum + { + if (value.TryGetValue(out int intValue)) + { + result = EnumHelper.Int32ToEnum(intValue); + return true; + } + if (value.TryGetValue(out TEnum enumValue)) + { + result = enumValue; + return true; + } + result = default; + return false; + } + + private readonly RobotIntentControllerClient m_controller; + private readonly Lock m_lock = new(); + private readonly CancellationTokenSource m_disposeCts = new(); + + private readonly TaskCompletionSource m_completion = new( + TaskCreationOptions.RunContinuationsAsynchronously); + + private MissionSnapshot m_current = new(); + private NodeId m_executionStateNode = NodeId.Null; + private Task? m_pumpTask; + private int m_disposed; + } + + internal static partial class MissionHandleLog + { + [LoggerMessage( + EventId = RobotIntentClientEventIds.MissionSubscriptionFailed, + Level = LogLevel.Warning, + Message = "Robot Intent mission subscription failed. MissionId={MissionId}, MissionNode={MissionNode}.")] + public static partial void MissionSubscriptionFailed( + this ILogger logger, + Exception exception, + string missionId, + NodeId missionNode); + } +} diff --git a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentBuilders.cs b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentBuilders.cs index 9adbfd9664..8b181c414f 100644 --- a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentBuilders.cs +++ b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentBuilders.cs @@ -176,9 +176,19 @@ public static SimpleIntentBuilder Release(NodeId tool) /// /// Creates a Pick builder. /// - public static SimpleIntentBuilder Pick(NodeId source, NodeId tool) + /// The Location to pick from. + /// The Tool to acquire the object with. + /// + /// What to pick, for a Location that can hold more than one kind of object. Empty + /// means whatever is there, which is only unambiguous for a single-kind Location. + /// + public static SimpleIntentBuilder Pick( + NodeId source, + NodeId tool, + string objectClass = "") { - return new SimpleIntentBuilder(new PickIntentDataType { Source = source, Tool = tool }); + return new SimpleIntentBuilder( + new PickIntentDataType { Source = source, Tool = tool, ObjectClass = objectClass }); } /// @@ -427,6 +437,7 @@ public JointMoveIntentBuilder(uint axisCount) /// /// Sets joint targets and validates the axis count. /// + /// public JointMoveIntentBuilder ToJoints(ArrayOf jointTargets) { if (m_axisCount > 0 && jointTargets.Count != m_axisCount) @@ -444,6 +455,7 @@ public JointMoveIntentBuilder ToJoints(ArrayOf jointTargets) /// /// Sets a target pose. /// + /// public JointMoveIntentBuilder ToPose(Pose3DDataType pose) { Intent.HasJointTargets = false; @@ -471,6 +483,7 @@ public LinearMoveIntentBuilder() /// /// Sets the target pose. /// + /// public LinearMoveIntentBuilder To(Pose3DDataType target) { Intent.Target = target ?? throw new ArgumentNullException(nameof(target)); @@ -495,6 +508,7 @@ public CircularMoveIntentBuilder() /// /// Sets the via point. /// + /// public CircularMoveIntentBuilder Via(Pose3DDataType viaPoint) { Intent.ViaPoint = viaPoint ?? throw new ArgumentNullException(nameof(viaPoint)); @@ -504,6 +518,7 @@ public CircularMoveIntentBuilder Via(Pose3DDataType viaPoint) /// /// Sets the target pose. /// + /// public CircularMoveIntentBuilder To(Pose3DDataType target) { Intent.Target = target ?? throw new ArgumentNullException(nameof(target)); @@ -853,6 +868,7 @@ public static ContentFilter Always() /// /// Creates an equality condition comparing an attribute operand with a literal value. /// + /// public static ContentFilter Equals(SimpleAttributeOperand operand, Variant value) { if (operand is null) diff --git a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentClient.cs b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentClient.cs index e65741b79b..1c4fd173ab 100644 --- a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentClient.cs +++ b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentClient.cs @@ -150,6 +150,7 @@ public ValueTask ReadStateAsync(CancellationToken ca /// /// Submits an intent and returns an awaitable operation handle when accepted. /// + /// public async ValueTask SubmitIntentAsync( IntentDataType intent, CancellationToken cancellationToken = default) @@ -288,6 +289,7 @@ public async ValueTask RequestAuthorityAsync( /// Use when refusal is expected and the caller wants to branch on the /// lease. /// + /// public async ValueTask RequireAuthorityAsync( CancellationToken cancellationToken = default) { @@ -315,6 +317,7 @@ public async ValueTask RequireAuthorityAsync( /// /// Submits a mission and records its update id for local stale-update checks. /// + /// public async ValueTask SubmitMissionAsync( MissionDataType mission, CancellationToken cancellationToken = default) @@ -336,6 +339,43 @@ public async ValueTask SubmitMissionAsync( return result; } + /// + /// Submits a mission and returns an awaitable handle when accepted. + /// + /// + public async ValueTask SubmitAndTrackMissionAsync( + MissionDataType mission, + CancellationToken cancellationToken = default) + { + MissionSubmissionResult result = await SubmitMissionAsync(mission, cancellationToken) + .ConfigureAwait(false); + if (!result.Accepted) + { + throw ServiceResultException.Create( + StatusCodes.BadRequestNotAllowed, + "Mission refused: {0} {1}", + result.Failure, + result.Message.Text ?? string.Empty); + } + string id = result.MissionId.Length == 0 ? mission.MissionId ?? string.Empty : result.MissionId; + MissionHandle handle = new(this, id, result.Operation); + await handle.StartAsync(cancellationToken).ConfigureAwait(false); + return handle; + } + + /// + /// Opens an awaitable mission handle for an existing mission node. + /// + public async ValueTask TrackMissionAsync( + string missionId, + NodeId missionNode, + CancellationToken cancellationToken = default) + { + MissionHandle handle = new(this, missionId, missionNode); + await handle.StartAsync(cancellationToken).ConfigureAwait(false); + return handle; + } + /// /// Replaces the mission horizon after locally rejecting non-increasing update ids. /// @@ -357,7 +397,8 @@ public async ValueTask UpdateMissionAsync( "MissionUpdateId shall be strictly greater than the mission's current value."); return new MissionUpdateOutcome( MissionUpdateResultEnum.Outdated, - new LocalizedText("MissionUpdateId shall be strictly greater than the mission's current value.")); + new LocalizedText( + "MissionUpdateId shall be strictly greater than the mission's current value.")); } } MissionUpdateOutcome outcome = await Transport.UpdateMissionAsync( @@ -403,6 +444,9 @@ public async ValueTask OpenRealTimeChannelAsync( private readonly Dictionary m_lastMissionUpdateIds = new(StringComparer.Ordinal); } + // LoggerMessage source generation requires a partial containing type. + // TODO: Remove when RCS1043 recognizes source-generated logging containers. +#pragma warning disable RCS1043 internal static partial class RobotIntentControllerClientLog { [LoggerMessage( @@ -417,4 +461,5 @@ public static partial void MissionUpdateRefusedLocal( MissionUpdateResultEnum result, string message); } +#pragma warning restore RCS1043 } diff --git a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentClientEventIds.cs b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentClientEventIds.cs index 6f2aa97804..142942f64e 100644 --- a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentClientEventIds.cs +++ b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentClientEventIds.cs @@ -46,5 +46,6 @@ internal static class RobotIntentClientEventIds public const int ChannelLeaseRenewalFailed = 7212; public const int AuthorityReleaseFailed = 7213; public const int OperationSubscriptionFailed = 7214; + public const int MissionSubscriptionFailed = 7215; } } diff --git a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentTransport.cs b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentTransport.cs index 18c8f39734..d7e8b22441 100644 --- a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentTransport.cs +++ b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentTransport.cs @@ -96,6 +96,11 @@ public interface IRobotIntentTransport /// ValueTask> ListMissionsAsync(CancellationToken ct = default); + /// + /// Reads a single mission snapshot by node id. + /// + ValueTask ReadMissionSnapshotAsync(NodeId mission, CancellationToken ct = default); + /// /// Submits an intent. /// @@ -218,14 +223,55 @@ public UaRobotIntentTransport( Logger = telemetry.CreateLogger(); m_streaming = streaming ?? RoboticsClient.GetDefaultStreaming(session); m_proxy = new IntentControllerTypeClient(session, controllerId, telemetry); - if (observeReconnect && session is ManagedSession managedSession) - { - managedSession.ConnectionStateChanged += OnConnectionStateChanged; - } + m_reconnectSource = observeReconnect ? session as ManagedSession : null; } /// - public event RobotIntentReconnectHandler? Reconnected; + /// + /// The underlying session notification is attached only while somebody is + /// listening here, and detached again when the last listener leaves. + /// + /// A transport is short-lived — a new one is created per controller + /// lookup, and an MCP tool call creates one per invocation — while the + /// it observes lives for the whole + /// connection. Subscribing in the constructor therefore handed the + /// session a strong reference to every transport ever created: memory + /// grew without bound, and each reconnect had to fan out to a longer and + /// longer list of transports nobody was using. Attaching on demand means + /// the common path, where no caller wants reconnect notifications at all, + /// costs nothing and retains nothing. + /// + /// + public event RobotIntentReconnectHandler? Reconnected + { + add + { + lock (m_reconnectLock) + { + bool attach = m_reconnected == null; + m_reconnected += value; + if (attach && m_reconnected != null && m_reconnectSource != null) + { + m_reconnectSource.ConnectionStateChanged += OnConnectionStateChanged; + } + } + } + remove + { + lock (m_reconnectLock) + { + if (m_reconnected == null) + { + return; + } + m_reconnected -= value; + if (m_reconnected == null && m_reconnectSource != null) + { + m_reconnectSource.ConnectionStateChanged -= OnConnectionStateChanged; + } + } + } + } /// public ILogger Logger { get; } @@ -281,16 +327,18 @@ public async ValueTask ReadControllerAsync(Cancellati uint maxQueueDepth = await ReadChildValueOrDefaultAsync(ControllerId, ["MaxQueueDepth"], 0u, ct) .ConfigureAwait(false); + ArrayOf frames = + await BrowseOptionalFolderAsync("Frames", ct).ConfigureAwait(false); RobotIntentLookups lookups = new() { - Frames = await BrowseOptionalFolderAsync("Frames", ct).ConfigureAwait(false), + Frames = frames, + FramesByFrameId = await ReadFrameIdLookupsAsync(frames, ct).ConfigureAwait(false), Tools = await BrowseOptionalFolderAsync("Tools", ct).ConfigureAwait(false), Locations = await BrowseOptionalFolderAsync("Locations", ct).ConfigureAwait(false), Axes = await BrowseOptionalFolderAsync("Axes", ct).ConfigureAwait(false), Outputs = await BrowseOptionalFolderAsync("Outputs", ct).ConfigureAwait(false), Programs = await BrowseOptionalFolderAsync("Programs", ct).ConfigureAwait(false) }; - lookups = lookups with { FramesByFrameId = lookups.Frames }; RobotIntentControllerInfo info = new() { @@ -409,7 +457,7 @@ public async ValueTask> ListMissionsAsync(CancellationT var snapshots = new List(missionIds.Count); for (int ii = 0; ii < missionIds.Count; ii++) { - snapshots.Add(await ReadMissionSnapshotAsync(missionIds[ii], ct).ConfigureAwait(false)); + snapshots.Add(await ReadMissionSnapshotInternalAsync(missionIds[ii], ct).ConfigureAwait(false)); } return [.. snapshots]; } @@ -693,28 +741,159 @@ private async ValueTask ReadSafetyStateAsync(Can }; } - private async ValueTask ReadMissionSnapshotAsync(NodeId mission, CancellationToken ct) + private async ValueTask ReadMissionSnapshotInternalAsync( + NodeId mission, CancellationToken ct) { NodeId stateNode = await TranslateAsync(mission, ["ExecutionState"], ct).ConfigureAwait(false); + ExecutionStateEnum executionState = stateNode.IsNull + ? ExecutionStateEnum.Accepted + : await ReadEnumValueAsync(stateNode, ct).ConfigureAwait(false); + IntentFailureEnum failure = IntentFailureEnum.None; + LocalizedText failureMessage = LocalizedText.Null; + if (RobotIntentRules.IsTerminal(executionState)) + { + failure = await ReadFinalResultEnumAsync(mission, ct).ConfigureAwait(false); + failureMessage = await ReadFinalResultMessageAsync(mission, ct).ConfigureAwait(false); + } + MissionDataType missionData = await ReadChildValueOrDefaultAsync( + mission, ["Mission"], new MissionDataType(), ct).ConfigureAwait(false); + string currentStepId = await ReadChildValueOrDefaultAsync( + mission, ["CurrentStepId"], string.Empty, ct).ConfigureAwait(false); return new MissionSnapshot { MissionNode = mission, - MissionId = await ReadChildValueOrDefaultAsync(mission, ["MissionId"], string.Empty, ct) - .ConfigureAwait(false), - MissionUpdateId = await ReadChildValueOrDefaultAsync(mission, ["MissionUpdateId"], 0u, ct) - .ConfigureAwait(false), - Mission = await ReadChildValueOrDefaultAsync(mission, ["Mission"], new MissionDataType(), ct) - .ConfigureAwait(false), - ExecutionState = stateNode.IsNull - ? ExecutionStateEnum.Accepted - : await ReadEnumValueAsync(stateNode, ct).ConfigureAwait(false), - CurrentStepId = await ReadChildValueOrDefaultAsync(mission, ["CurrentStepId"], string.Empty, ct) - .ConfigureAwait(false), - ReleasedStepCount = await ReadChildValueOrDefaultAsync(mission, ["ReleasedStepCount"], 0u, ct) - .ConfigureAwait(false) + MissionId = await ReadChildValueOrDefaultAsync( + mission, ["MissionId"], string.Empty, ct).ConfigureAwait(false), + MissionUpdateId = await ReadChildValueOrDefaultAsync( + mission, ["MissionUpdateId"], 0u, ct).ConfigureAwait(false), + Mission = missionData, + ExecutionState = executionState, + CurrentStepId = currentStepId, + CurrentIntentId = DeriveCurrentIntentId(missionData, currentStepId), + ReleasedStepCount = await ReadChildValueOrDefaultAsync( + mission, ["ReleasedStepCount"], 0u, ct).ConfigureAwait(false), + Failure = failure, + FailureMessage = failureMessage, + Steps = DeriveStepOperations(missionData) }; } + private async ValueTask> ReadFrameIdLookupsAsync( + ArrayOf frames, + CancellationToken ct) + { + var frameIds = new List(frames.Count); + for (int ii = 0; ii < frames.Count; ii++) + { + RobotIntentNodeLookupEntry frame = frames[ii]; + string frameId = await ReadChildValueOrDefaultAsync( + frame.NodeId, + ["FrameId"], + string.Empty, + ct).ConfigureAwait(false); + if (!string.IsNullOrEmpty(frameId)) + { + frameIds.Add(frame with { Name = frameId }); + } + } + return [.. frameIds]; + } + + /// + public ValueTask ReadMissionSnapshotAsync(NodeId mission, CancellationToken ct = default) + { + return ReadMissionSnapshotInternalAsync(mission, ct); + } + + private static ArrayOf DeriveStepOperations(MissionDataType missionData) + { + if (missionData.Steps.IsNull || missionData.Steps.IsEmpty) + { + return []; + } + var ops = new List(missionData.Steps.Count); + for (int ii = 0; ii < missionData.Steps.Count; ii++) + { + MissionStepDataType step = missionData.Steps[ii]; + if (step == null) + { + continue; + } + ops.Add(new MissionStepOperation + { + StepId = step.StepId ?? string.Empty, + IntentId = step.Intent?.IntentId ?? string.Empty, + OperationNodeId = step.Operation.IsNull + ? NodeId.Null : step.Operation, + State = step.Status + }); + } + return [.. ops]; + } + + private static string DeriveCurrentIntentId( + MissionDataType missionData, + string currentStepId) + { + if (string.IsNullOrEmpty(currentStepId) || + missionData.Steps.IsNull || + missionData.Steps.IsEmpty) + { + return string.Empty; + } + for (int ii = 0; ii < missionData.Steps.Count; ii++) + { + MissionStepDataType step = missionData.Steps[ii]; + if (step != null && + string.Equals( + step.StepId, currentStepId, StringComparison.Ordinal)) + { + return step.Intent?.IntentId ?? string.Empty; + } + } + return string.Empty; + } + + private async ValueTask ReadFinalResultEnumAsync(NodeId parent, CancellationToken ct) + { + NodeId finalResult = await ResolveHierarchicalChildByNameAsync( + parent, + "FinalResultData", + ct).ConfigureAwait(false); + if (finalResult.IsNull) + { + return IntentFailureEnum.None; + } + NodeId failureNode = await ResolveHierarchicalChildByNameAsync( + finalResult, + "Failure", + ct).ConfigureAwait(false); + if (failureNode.IsNull) + { + return IntentFailureEnum.None; + } + return await ReadEnumValueAsync(failureNode, ct).ConfigureAwait(false); + } + + private async ValueTask ReadFinalResultMessageAsync(NodeId parent, CancellationToken ct) + { + NodeId finalResult = await ResolveHierarchicalChildByNameAsync( + parent, + "FinalResultData", + ct).ConfigureAwait(false); + if (finalResult.IsNull) + { + return LocalizedText.Null; + } + NodeId messageNode = await ResolveHierarchicalChildByNameAsync( + finalResult, + "Message", + ct).ConfigureAwait(false); + return messageNode.IsNull + ? LocalizedText.Null + : await m_session.ReadValueAsync(messageNode, ct).ConfigureAwait(false); + } + private async ValueTask> BrowseOptionalFolderAsync( string browseName, CancellationToken ct) @@ -769,6 +948,49 @@ private async ValueTask> BrowseObjectReferencesAsy return results[0]; } + private async ValueTask ResolveHierarchicalChildByNameAsync( + NodeId parent, + string browseName, + CancellationToken ct) + { + (ArrayOf> results, _) = await m_session.ManagedBrowseAsync( + null, + null, + [parent], + 0, + BrowseDirection.Forward, + global::Opc.Ua.ReferenceTypeIds.HierarchicalReferences, + includeSubtypes: true, + nodeClassMask: 0, + ct).ConfigureAwait(false); + if (results.Count == 0) + { + return NodeId.Null; + } + + NodeId match = NodeId.Null; + for (int ii = 0; ii < results[0].Count; ii++) + { + ReferenceDescription reference = results[0][ii]; + if (!string.Equals(reference.BrowseName.Name, browseName, StringComparison.Ordinal)) + { + continue; + } + NodeId nodeId = ExpandedNodeId.ToNodeId(reference.NodeId, m_session.NamespaceUris); + if (nodeId.IsNull) + { + continue; + } + if (!match.IsNull) + { + throw new InvalidOperationException( + $"Node '{parent}' has more than one hierarchical child named '{browseName}'."); + } + match = nodeId; + } + return match; + } + private async ValueTask ReadChildValueOrDefaultAsync( NodeId root, IReadOnlyList path, @@ -900,7 +1122,12 @@ private void OnConnectionStateChanged(object? sender, ConnectionStateChangedEven if (e.NewState == ConnectionState.Connected && e.PreviousState is ConnectionState.Reconnecting or ConnectionState.Failover) { - Reconnected?.Invoke(); + RobotIntentReconnectHandler? handler; + lock (m_reconnectLock) + { + handler = m_reconnected; + } + handler?.Invoke(); } } @@ -909,6 +1136,9 @@ private void OnConnectionStateChanged(object? sender, ConnectionStateChangedEven private readonly ISession m_session; private readonly IStreamingSubscription m_streaming; private readonly IntentControllerTypeClient m_proxy; + private readonly ManagedSession? m_reconnectSource; + private readonly Lock m_reconnectLock = new(); + private RobotIntentReconnectHandler? m_reconnected; } internal static partial class UaRobotIntentTransportLog diff --git a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentTypes.cs b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentTypes.cs index 615a40ffe6..8ba5d1e9e3 100644 --- a/src/Opc.Ua.Robotics.Client/Intent/RobotIntentTypes.cs +++ b/src/Opc.Ua.Robotics.Client/Intent/RobotIntentTypes.cs @@ -456,6 +456,32 @@ public sealed record IntentCommandOutcome(bool Accepted); /// public sealed record CommandAuthorityOutcome(bool Granted, NodeId CurrentOwner); + /// + /// Per-step correlation between the mission step and its operation instance. + /// + public sealed record MissionStepOperation + { + /// + /// Gets the step id within the mission. + /// + public string StepId { get; init; } = string.Empty; + + /// + /// Gets the intent id admitted for this step. + /// + public string IntentId { get; init; } = string.Empty; + + /// + /// Gets the operation node tracking this step, or if it has not executed yet. + /// + public NodeId OperationNodeId { get; init; } = NodeId.Null; + + /// + /// Gets the execution state of this step. + /// + public ExecutionStateEnum State { get; init; } = ExecutionStateEnum.Accepted; + } + /// /// Current observable state of a mission. /// @@ -491,10 +517,63 @@ public sealed record MissionSnapshot /// public string CurrentStepId { get; init; } = string.Empty; + /// + /// Gets the intent id of the currently executing step, or an empty string. + /// + public string CurrentIntentId { get; init; } = string.Empty; + /// /// Gets the number of committed steps in the mission base. /// public uint ReleasedStepCount { get; init; } + + /// + /// Gets the failure classification when the mission failed. + /// + public IntentFailureEnum Failure { get; init; } = IntentFailureEnum.None; + + /// + /// Gets the human-readable failure message when the mission failed. + /// + public LocalizedText FailureMessage { get; init; } = LocalizedText.Null; + + /// + /// Gets per-step operation correlation for the mission. A MissionStep can + /// retain one operation, so a retried or revisited step reports its latest + /// admitted attempt. + /// + public ArrayOf Steps { get; init; } = []; + } + + /// + /// Result of a bounded wait for a mission to reach a terminal state. + /// + public sealed record MissionWaitResult + { + /// + /// Gets a value indicating whether the mission reached a terminal state before the timeout. + /// + public bool Completed { get; init; } + + /// + /// Gets the terminal execution state when is true. + /// + public ExecutionStateEnum TerminalState { get; init; } = ExecutionStateEnum.Accepted; + + /// + /// Gets the failure classification when the mission failed. + /// + public IntentFailureEnum Failure { get; init; } = IntentFailureEnum.None; + + /// + /// Gets the human-readable failure message when the mission failed. + /// + public LocalizedText FailureMessage { get; init; } = LocalizedText.Null; + + /// + /// Gets the current mission snapshot, refreshed on timeout. + /// + public MissionSnapshot Current { get; init; } = new(); } /// @@ -634,6 +713,7 @@ or ExecutionStateEnum.Cancelled /// /// Derives the client-facing facet snapshot from a capability declaration. /// + /// public static RobotIntentFacets DeriveFacets(RobotIntentControllerInfo controller) { if (controller is null) diff --git a/src/Opc.Ua.Robotics.Server/Builders/IntentBuilders.cs b/src/Opc.Ua.Robotics.Server/Builders/IntentBuilders.cs index c38c91ee40..d5e6b937b3 100644 --- a/src/Opc.Ua.Robotics.Server/Builders/IntentBuilders.cs +++ b/src/Opc.Ua.Robotics.Server/Builders/IntentBuilders.cs @@ -42,6 +42,9 @@ namespace Opc.Ua.Robotics.Server.Builders { internal sealed class IntentControllerBuilder : IIntentControllerBuilder { + // Explicit calls select RobotIntent generated extensions over same-named Robotics extensions. + // TODO: Remove when RCS1196 recognizes deliberate extension-method disambiguation. +#pragma warning disable RCS1196 public IntentControllerBuilder(IRobotIntentBuildContext context, string browseName) { m_context = context ?? throw new ArgumentNullException(nameof(context)); @@ -319,6 +322,7 @@ public IIntentRealTimeChannelBuilder AddRealTimeChannel( m_realTimeChannels.Add(builder); return builder; } +#pragma warning restore RCS1196 public IIntentControllerBuilder Accepts( bool cancelSupported = true, @@ -394,8 +398,8 @@ public IIntentControllerBuilder Accepts( } if (pauseSupported) { - State.AddPause(m_context.Context); - State.AddResume(m_context.Context); + State.AddPause(m_context.Context) + .AddResume(m_context.Context); MarkCommandMethod(State.Pause!); MarkCommandMethod(State.Resume!); } @@ -431,7 +435,6 @@ public ArrayOf ComputeFacets() return RobotIntentFacetCalculator.Compute(State); } - internal async ValueTask RegisterAsync(CancellationToken cancellationToken) { EnsureMutable(); @@ -498,12 +501,22 @@ m_context is RobotIntentBuildContext buildContextWithServices && throw new InvalidOperationException( global::Opc.Ua.Robotics.Server.RobotIntentBuildServiceProvider.MissingExecutorMessage); } - return new IntentControllerHost( + var host = new IntentControllerHost( State, executor, m_context.Manager.AddPredefinedNodeAsync, m_hostOptions, RemovePredefinedNodeAsync); + + // Command authority is held per Session, so it has to be given back when the + // Session goes away. Without this subscription a client that crashes or is + // killed keeps the robot locked, and the next client is refused with no way + // to recover short of restarting the Server. + if (m_context.Manager.Server?.SessionManager is { } sessionManager) + { + host.AttachSessionManager(m_context.Context, sessionManager); + } + return host; } private async ValueTask RemovePredefinedNodeAsync(NodeState node, CancellationToken cancellationToken) @@ -552,8 +565,8 @@ private void EnsureOptionalMethods() { if (m_hostOptions.MissionsSupported) { - State.AddSubmitMission(m_context.Context); - State.AddCancelMission(m_context.Context); + State.AddSubmitMission(m_context.Context) + .AddCancelMission(m_context.Context); MarkCommandMethod(State.SubmitMission!); MarkCommandMethod(State.CancelMission!); } @@ -564,8 +577,8 @@ private void EnsureOptionalMethods() } if (m_hostOptions.RealTimeChannelsSupported) { - State.AddOpenRealTimeChannel(m_context.Context); - State.AddCloseRealTimeChannel(m_context.Context); + State.AddOpenRealTimeChannel(m_context.Context) + .AddCloseRealTimeChannel(m_context.Context); EnsureOpenRealTimeChannelArguments(State.OpenRealTimeChannel!); EnsureCloseRealTimeChannelArguments(State.CloseRealTimeChannel!); MarkCommandMethod(State.OpenRealTimeChannel!); @@ -573,8 +586,8 @@ private void EnsureOptionalMethods() } if (m_capabilities.Any(static capability => capability.PauseSupported)) { - State.AddPause(m_context.Context); - State.AddResume(m_context.Context); + State.AddPause(m_context.Context) + .AddResume(m_context.Context); MarkCommandMethod(State.Pause!); MarkCommandMethod(State.Resume!); } @@ -1314,9 +1327,7 @@ public static bool TryGetInterop40010Binding( return false; } - private sealed class SafetyAdmissionGateBinding - { - } + private sealed class SafetyAdmissionGateBinding; private sealed class Interop40010Binding { @@ -1330,6 +1341,7 @@ public Interop40010Binding(MotionDeviceSystemState motionDeviceSystem) private static readonly ConditionalWeakTable s_safetyAdmissionGated = new(); + private static readonly ConditionalWeakTable s_interop40010Bindings = new(); } diff --git a/src/Opc.Ua.Robotics.Server/Hosting/OpcUaServerRobotIntentBuilderExtensions.cs b/src/Opc.Ua.Robotics.Server/Hosting/OpcUaServerRobotIntentBuilderExtensions.cs index 2cb7607788..f164fea589 100644 --- a/src/Opc.Ua.Robotics.Server/Hosting/OpcUaServerRobotIntentBuilderExtensions.cs +++ b/src/Opc.Ua.Robotics.Server/Hosting/OpcUaServerRobotIntentBuilderExtensions.cs @@ -49,6 +49,7 @@ public static class OpcUaServerRobotIntentBuilderExtensions /// /// Registers the standalone Robot Intent node manager. /// + /// is null. public static IOpcUaServerBuilder AddRobotIntent( this IOpcUaServerBuilder builder, Action? configure = null) @@ -103,6 +104,7 @@ public static IOpcUaServerBuilder AddRobotIntent( /// /// The executor implementation type. /// + /// is null. public static IOpcUaServerBuilder AddRobotIntentExecutor< [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] TExecutor>( this IOpcUaServerBuilder builder) @@ -112,7 +114,15 @@ public static IOpcUaServerBuilder AddRobotIntentExecutor< { throw new ArgumentNullException(nameof(builder)); } - builder.Services.AddSingleton(); + + // Register the concrete type and resolve the interface from it, so an + // application that also injects TExecutor directly - to observe the arm + // it is driving, for example - shares the instance the intents run on. + // Registering IIntentExecutor against the type would construct a second + // executor, leaving the application watching a device that never moves. + builder.Services.AddSingleton(); + builder.Services.AddSingleton( + services => services.GetRequiredService()); return builder; } @@ -122,6 +132,7 @@ public static IOpcUaServerBuilder AddRobotIntentExecutor< /// /// The executor implementation type. /// + /// is null. public static IOpcUaServerBuilder AddRobotIntentExecutor< [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] TExecutor>( this IOpcUaServerBuilder builder, @@ -142,6 +153,7 @@ public static IOpcUaServerBuilder AddRobotIntentExecutor< /// /// Registers an executor instance for one Robot Intent controller browse name. /// + /// is null. public static IOpcUaServerBuilder AddRobotIntentExecutor( this IOpcUaServerBuilder builder, string controllerBrowseName, @@ -169,6 +181,7 @@ public static IOpcUaServerBuilder ConfigureRobotIntent( /// /// Registers a Robot Intent configurator for the standalone manager. /// + /// is null. public static IOpcUaServerBuilder ConfigureRobotIntent( this IOpcUaServerBuilder builder, Action configure) @@ -190,6 +203,8 @@ public static IOpcUaServerBuilder ConfigureRobotIntent( /// /// The standalone Robot Intent node manager type. /// + /// is null. + /// public static IOpcUaServerBuilder ConfigureRobotIntentFor( this IOpcUaServerBuilder builder, Func configure) diff --git a/src/Opc.Ua.Robotics.Server/Intent/IntentControllerHost.cs b/src/Opc.Ua.Robotics.Server/Intent/IntentControllerHost.cs index e230ac804f..9d6ccea764 100644 --- a/src/Opc.Ua.Robotics.Server/Intent/IntentControllerHost.cs +++ b/src/Opc.Ua.Robotics.Server/Intent/IntentControllerHost.cs @@ -71,7 +71,7 @@ public IntentControllerHost( IntentControllerHostOptions? options = null, Func? removeNode = null) { - m_controller = controller ?? throw new ArgumentNullException(nameof(controller)); + Controller = controller ?? throw new ArgumentNullException(nameof(controller)); m_executor = executor ?? throw new ArgumentNullException(nameof(executor)); m_addNode = addNode ?? throw new ArgumentNullException(nameof(addNode)); m_options = options ?? new IntentControllerHostOptions(); @@ -81,7 +81,7 @@ public IntentControllerHost( /// /// The controller node this host drives. /// - public IntentControllerState Controller => m_controller; + public IntentControllerState Controller { get; } /// /// The Session that currently holds command authority, or null. @@ -123,6 +123,7 @@ public SafetyStatus SafetyState /// /// Starts the execution pump and wires the controller's Methods. /// + /// is null. public void Start(ISystemContext context) { if (context == null) @@ -140,10 +141,10 @@ public void Start(ISystemContext context) } m_logger = context.Telemetry.CreateLogger(); m_namespaceUris = context.NamespaceUris; - m_intentsFolder = EnsureFolder(context, m_controller.Intents, BrowseNames.Intents); + m_intentsFolder = EnsureFolder(context, Controller.Intents, BrowseNames.Intents); if (m_options.MissionsSupported) { - m_missionsFolder = EnsureFolder(context, m_controller.Missions, BrowseNames.Missions); + m_missionsFolder = EnsureFolder(context, Controller.Missions, BrowseNames.Missions); } ResolveCapabilities(context); BuildReferenceIndexes(context); @@ -165,11 +166,11 @@ public void Start(ISystemContext context) private void WireVariableReads(ISystemContext context) { - if (m_options.SafetyStatusReader == null || m_controller.Ready == null) + if (m_options.SafetyStatusReader == null || Controller.Ready == null) { return; } - m_controller.Ready.OnReadValueAsync = async ( + Controller.Ready.OnReadValueAsync = async ( readContext, variable, indexRange, @@ -204,6 +205,7 @@ private void WireVariableReads(ISystemContext context) /// parameters are wrong. A refusal creates no operation instance and moves /// nothing. /// + /// is null. public IntentAdmission SubmitIntent( ISystemContext context, NodeId? sessionId, @@ -221,12 +223,13 @@ public IntentAdmission SubmitIntent( "A current safety snapshot is required before admission."); } return SubmitCore( - context, sessionId, ClientNameOf(context), intent, missionId, forceNewId: false); + context, sessionId, ClientNameOf(context), intent, missionId); } /// /// Admits one intent after refreshing the safety status, per OPC UA - Robot Intent clause 10.4. /// + /// is null. public async ValueTask SubmitIntentAsync( ISystemContext context, NodeId? sessionId, @@ -243,7 +246,7 @@ public async ValueTask SubmitIntentAsync( await RefreshSafetyStateAsync(context, cancellationToken).ConfigureAwait(false); } return SubmitCore( - context, sessionId, ClientNameOf(context), intent, missionId, forceNewId: false); + context, sessionId, ClientNameOf(context), intent, missionId); } /// @@ -255,6 +258,7 @@ public async ValueTask SubmitIntentAsync( /// cannot be abandoned part-way without leaving the cell in a worse state than /// completing them. /// + /// is null. public bool CancelIntent( ISystemContext context, NodeId? sessionId, @@ -316,6 +320,7 @@ public bool CancelIntent( /// /// Asks the Server to end every outstanding intent and mission. /// + /// is null. public uint CancelAll( ISystemContext context, NodeId? sessionId, @@ -378,6 +383,7 @@ public uint CancelAll( /// Pauses queue dispatch. The executing intent keeps running because the /// executor interface has no pause acknowledgement channel. /// + /// is null. public bool Pause(ISystemContext context, NodeId? sessionId) { if (context == null) @@ -407,6 +413,7 @@ public bool Pause(ISystemContext context, NodeId? sessionId) /// /// Continues execution suspended by . /// + /// is null. public bool Resume(ISystemContext context, NodeId? sessionId) { if (context == null) @@ -441,6 +448,7 @@ public bool Resume(ISystemContext context, NodeId? sessionId) /// The new attempt is a NEW operation instance. The original stays where it is, /// terminal, with its own result, so the history of what was tried survives. /// + /// is null. public IntentAdmission Retry(ISystemContext context, NodeId? sessionId, string intentId) { if (context == null) @@ -454,6 +462,8 @@ public IntentAdmission Retry(ISystemContext context, NodeId? sessionId, string i } IntentDataType? intent; string missionId; + string retryIntentId; + string baseIntentId; lock (m_lock) { if (m_options.RequireControlAuthority && !HasAuthority(sessionId)) @@ -483,9 +493,20 @@ public IntentAdmission Retry(ISystemContext context, NodeId? sessionId, string i } intent = entry.Intent; missionId = entry.MissionId; + baseIntentId = entry.BaseIntentId; + retryIntentId = NextRetryIntentIdLocked(baseIntentId); } - return SubmitCore(context, sessionId, ClientNameOf(context), intent!, missionId, forceNewId: true); + IntentDataType retryIntent = CloneIntent(intent!); + retryIntent.IntentId = retryIntentId; + return SubmitCore( + context, + sessionId, + ClientNameOf(context), + retryIntent, + missionId, + retryIntentId, + baseIntentId); } /// @@ -497,6 +518,7 @@ public IntentAdmission Retry(ISystemContext context, NodeId? sessionId, string i /// which concerns remote command against local manual control and is enforced /// by safety-rated means outside this interface. /// + /// is null. public bool RequestControl(ISystemContext context, NodeId? sessionId, out NodeId? owner) { if (context == null) @@ -526,6 +548,7 @@ public bool RequestControl(ISystemContext context, NodeId? sessionId, out NodeId /// /// Gives up command authority. Outstanding intents are unaffected. /// + /// is null. public void ReleaseControl(ISystemContext context, NodeId? sessionId) { if (context == null) @@ -555,6 +578,7 @@ public void ReleaseControl(ISystemContext context, NodeId? sessionId) /// then refuses on the same values a client can read, so the refusal is /// explainable from the address space rather than from Server-internal state. /// + /// is null. public void UpdateSafetyState(ISystemContext context, SafetyStatus status) { if (context == null) @@ -583,6 +607,7 @@ public void UpdateSafetyState(ISystemContext context, SafetyStatus status) /// /// Without this a crashed client locks the robot for good. /// + /// is null. public void OnSessionClosed(ISystemContext context, NodeId sessionId) { if (context == null) @@ -607,6 +632,7 @@ public void OnSessionClosed(ISystemContext context, NodeId sessionId) /// /// Subscribes this host to the Server session lifetime notifications. /// + /// is null. public void AttachSessionManager( ISystemContext context, global::Opc.Ua.Server.ISessionManager sessionManager) { @@ -640,6 +666,7 @@ public void AttachSessionManager( /// travel here. A lease that is not renewed lapses, so a client that dies does /// not hold the channel for good - the same reasoning as command authority. /// + /// is null. public RealTimeLease OpenRealTimeChannel( ISystemContext context, NodeId? sessionId, string channelId, double requestedLeaseMs) { @@ -703,6 +730,7 @@ public RealTimeLease OpenRealTimeChannel( /// /// Gives up a lease on a brokered channel. /// + /// is null. public bool CloseRealTimeChannel(ISystemContext context, NodeId? sessionId, string channelId) { if (context == null) @@ -733,6 +761,7 @@ public bool CloseRealTimeChannel(ISystemContext context, NodeId? sessionId, stri /// /// Submits an ordered sequence of intents tracked as one unit. /// + /// is null. public MissionAdmission SubmitMission( ISystemContext context, NodeId? sessionId, @@ -754,6 +783,7 @@ public MissionAdmission SubmitMission( /// /// Submits a mission after refreshing the safety status, per OPC UA - Robot Intent clause 10.4. /// + /// is null. public async ValueTask SubmitMissionAsync( ISystemContext context, NodeId? sessionId, @@ -862,6 +892,14 @@ private MissionAdmission SubmitMissionCore( return MissionAdmission.Refused(IntentFailureEnum.ParameterInvalid, $"MissionId '{id}' is already outstanding."); } + + Check intentIds = PreflightStepIntentIdsLocked(mission, id); + if (!intentIds.Ok) + { + return MissionAdmission.Refused(IntentFailureEnum.ParameterInvalid, + intentIds.Message ?? "A step IntentId is invalid."); + } + if (mission.Steps[0]?.Intent is { BufferMode: not BufferModeEnum.Aborting } && m_queue.Count >= m_options.MaxQueueDepth) { @@ -869,9 +907,10 @@ private MissionAdmission SubmitMissionCore( "The queue is at MaxQueueDepth."); } - var entry = new MissionEntry(id, mission); + var entry = new MissionEntry(id, mission, Interlocked.Increment(ref m_nextId)); m_missions[id] = entry; CreateMissionNode(context, entry); + m_missionHistory.Add(entry.Node!.NodeId, entry); SetMissionStateLocked(context, entry, ExecutionStateEnum.Executing); StartNextStepLocked(context, entry, sessionId); return MissionAdmission.Admitted(id, entry.Node!.NodeId); @@ -886,6 +925,7 @@ private MissionAdmission SubmitMissionCore( /// executed, so an update that would alter a released step is refused rather /// than partly applied, and the whole update is applied atomically. /// + /// is null. public MissionUpdateOutcome UpdateMission( ISystemContext context, NodeId? sessionId, @@ -927,6 +967,7 @@ public MissionUpdateOutcome UpdateMission( "MissionUpdateId must be greater than the mission's current value."); } + uint released = MissionRules.ReleasedCount(entry.Mission.Steps); Check conflict = MissionRules.ValidateBasePreserved(entry.Mission.Steps, steps); if (!conflict.Ok) { @@ -946,7 +987,37 @@ public MissionUpdateOutcome UpdateMission( graph.Message ?? "The mission graph is not valid."); } - entry.Mission.Steps = steps; + var reservedIds = new HashSet(StringComparer.Ordinal); + for (int ii = 0; ii < released; ii++) + { + string baseId = entry.GetBaseIntentId(entry.Mission.Steps[ii], ii); + if (!string.IsNullOrEmpty(baseId)) + { + reservedIds.Add(baseId); + } + } + Check intentIds = PreflightStepIntentIdsLocked( + steps, + entry.MissionId, + (int)released, + reservedIds); + if (!intentIds.Ok) + { + return new MissionUpdateOutcome( + MissionUpdateResultEnum.Rejected, + intentIds.Message ?? "A horizon step IntentId is invalid."); + } + + var merged = new List(steps.Count); + for (int ii = 0; ii < released; ii++) + { + merged.Add(entry.Mission.Steps[ii]); + } + for (int ii = (int)released; ii < steps.Count; ii++) + { + merged.Add(steps[ii]); + } + entry.ReplaceSteps([.. merged], (int)released); entry.Mission.MissionUpdateId = missionUpdateId; PublishMissionLocked(context, entry); return new MissionUpdateOutcome(MissionUpdateResultEnum.Accepted, null); @@ -956,6 +1027,7 @@ public MissionUpdateOutcome UpdateMission( /// /// Ends a mission and every intent belonging to it. /// + /// is null. public bool CancelMission( ISystemContext context, NodeId? sessionId, @@ -1036,8 +1108,7 @@ public void Dispose() m_disposed = true; UnhookSessionManager(); m_shutdown.Cancel(); - IntentEntry[] entries = SnapshotIntents(); - foreach (IntentEntry entry in entries) + foreach (IntentEntry entry in SnapshotIntents()) { entry.RequestCancel(IntentFailureEnum.Other, StopModeEnum.QuickStop); } @@ -1060,6 +1131,7 @@ public async ValueTask DisposeAsync() /// The clause 6.3 table, in code. A pairing not listed there is not legal, so /// this mapping is total and has no default arm that guesses. /// + /// internal static uint MapToProgramState(ExecutionStateEnum state) { return state switch @@ -1084,7 +1156,8 @@ private IntentAdmission SubmitCore( string clientName, IntentDataType? intent, string missionId, - bool forceNewId) + string? admittedIntentId = null, + string? baseIntentId = null) { if (context == null) { @@ -1184,7 +1257,7 @@ private IntentAdmission SubmitCore( scope.Message ?? "The intent references an invalid node."); } - string id = forceNewId ? string.Empty : intent.IntentId ?? string.Empty; + string id = admittedIntentId ?? intent.IntentId ?? string.Empty; if (string.IsNullOrEmpty(id)) { id = FormattableString.Invariant( @@ -1205,6 +1278,7 @@ private IntentAdmission SubmitCore( var entry = new IntentEntry( id, + baseIntentId ?? id, intent, missionId, Interlocked.Increment(ref m_nextAdmissionSequence)) @@ -1304,7 +1378,7 @@ motion.Constraints is { } constraints && private void PublishSafetyLocked(ISystemContext context) { - if (m_controller.SafetyState is not { } node) + if (Controller.SafetyState is not { } node) { return; } @@ -1344,7 +1418,7 @@ private void ResolveCapabilities(ISystemContext context) published.Add(resolved); } - if (m_controller.Capabilities is { } capabilities) + if (Controller.Capabilities is { } capabilities) { SetValue(capabilities.SupportedIntents, new ArrayOf(published.ToArray())); SetValue(capabilities.MissionsSupported, m_options.MissionsSupported); @@ -1726,8 +1800,8 @@ private async Task PumpAsync(ISystemContext context, CancellationToken shutdown) next.IntentId, next.Intent, progress, - m_controller.NodeId, - m_controller.BrowseName.Name ?? string.Empty) + Controller.NodeId, + Controller.BrowseName.Name ?? string.Empty) { MissionId = next.MissionId }; @@ -1906,11 +1980,11 @@ private void BuildReferenceIndexes(ISystemContext context) m_programs.Clear(); m_outputDataTypes.Clear(); m_frameIds.Clear(); - IndexFolder(context, m_controller.Locations, m_locations); - IndexFolder(context, m_controller.Tools, m_tools); - IndexFolder(context, m_controller.Programs, m_programs); - IndexFrames(context, m_controller.Frames); - IndexOutputs(context, m_controller.Outputs); + IndexFolder(context, Controller.Locations, m_locations); + IndexFolder(context, Controller.Tools, m_tools); + IndexFolder(context, Controller.Programs, m_programs); + IndexFrames(context, Controller.Frames); + IndexOutputs(context, Controller.Outputs); } private void IndexFolder(ISystemContext context, NodeState? folder, HashSet index) @@ -1990,7 +2064,7 @@ private void IndexOutputs(ISystemContext context, NodeState? folder) private void CreateChannels(ISystemContext context) { FolderState folder = EnsureFolder( - context, m_controller.RealTimeChannels, BrowseNames.RealTimeChannels); + context, Controller.RealTimeChannels, BrowseNames.RealTimeChannels); foreach (DeclaredChannel declared in m_options.Channels) { var node = new RealTimeChannelState(folder) @@ -2042,12 +2116,131 @@ private void PublishChannelLocked(ISystemContext context, ChannelEntry channel) node.ClearChangeMasks(context, true); } + private Check PreflightStepIntentIdsLocked(MissionDataType mission, string missionId) + { + return PreflightStepIntentIdsLocked( + mission.Steps, + missionId, + startIndex: 0, + reservedIds: null); + } + + private Check PreflightStepIntentIdsLocked( + ArrayOf steps, + string missionId, + int startIndex, + HashSet? reservedIds) + { + var assigned = reservedIds == null + ? new HashSet(StringComparer.Ordinal) + : new HashSet(reservedIds, StringComparer.Ordinal); + var generatedIds = new string[steps.Count]; + + for (int ii = startIndex; ii < steps.Count; ii++) + { + MissionStepDataType step = steps[ii]; + IntentDataType? intent = step?.Intent; + if (intent == null) + { + continue; + } + + string suppliedId = intent.IntentId ?? string.Empty; + if (!string.IsNullOrEmpty(suppliedId)) + { + if (string.IsNullOrWhiteSpace(suppliedId)) + { + return Check.Fail( + $"Step '{step!.StepId}' has a whitespace-only IntentId."); + } + if (!assigned.Add(suppliedId)) + { + return Check.Fail( + $"IntentId '{suppliedId}' appears on more than one step."); + } + if (m_intents.ContainsKey(suppliedId)) + { + return Check.Fail( + $"IntentId '{suppliedId}' collides with a retained operation."); + } + } + else + { + string stepId = step!.StepId ?? string.Empty; + string generatedId = string.IsNullOrEmpty(stepId) + ? FormattableString.Invariant($"{missionId}/step-{ii}") + : FormattableString.Invariant($"{missionId}/{stepId}"); + if (assigned.Contains(generatedId)) + { + return Check.Fail( + $"Generated IntentId '{generatedId}' appears on more than one step."); + } + generatedId = NextAvailableGeneratedIntentIdLocked(generatedId, assigned); + assigned.Add(generatedId); + generatedIds[ii] = generatedId; + } + } + + for (int ii = startIndex; ii < steps.Count; ii++) + { + if (!string.IsNullOrEmpty(generatedIds[ii]) && + steps[ii]?.Intent is { } generatedIntent) + { + generatedIntent.IntentId = generatedIds[ii]; + } + } + + return Check.Pass; + } + + private string NextAvailableGeneratedIntentIdLocked( + string baseIntentId, + HashSet assigned) + { + if (!assigned.Contains(baseIntentId) && !m_intents.ContainsKey(baseIntentId)) + { + return baseIntentId; + } + + for (int run = 2; ; run++) + { + string candidate = FormattableString.Invariant($"{baseIntentId}#run-{run}"); + if (!assigned.Contains(candidate) && !m_intents.ContainsKey(candidate)) + { + return candidate; + } + } + } + + private string NextRetryIntentIdLocked(string baseIntentId) + { + for (int attempt = 2; ; attempt++) + { + string candidate = FormattableString.Invariant($"{baseIntentId}#attempt-{attempt}"); + if (!m_intents.ContainsKey(candidate)) + { + return candidate; + } + } + } + + private static IntentDataType CloneIntent(IntentDataType intent) + { + if (intent.Clone() is IntentDataType clone) + { + return clone; + } + throw new InvalidOperationException( + $"Intent type '{intent.GetType().FullName}' did not clone as {nameof(IntentDataType)}."); + } + private MissionAdvanceResult StartNextStepLocked( ISystemContext context, MissionEntry mission, NodeId? sessionId) { - MissionStepDataType? step = MissionRules.NextPending(mission.Mission.Steps, mission.NextIndex); + MissionStepDataType? step = + MissionRules.NextPending(mission.Mission.Steps, mission.NextIndex); if (step == null) { FinishMissionLocked(context, mission, ExecutionStateEnum.Succeeded); @@ -2055,15 +2248,56 @@ private MissionAdvanceResult StartNextStepLocked( } mission.CurrentStepId = step.StepId ?? string.Empty; - IntentAdmission admission = - SubmitCore( - context, sessionId, ClientNameOf(context), step.Intent, mission.MissionId, forceNewId: true); + IntentDataType? stepIntent = step.Intent; + if (stepIntent == null) + { + FinishMissionLocked( + context, + mission, + ExecutionStateEnum.Failed, + IntentFailureEnum.ParameterInvalid, + $"Mission step '{mission.CurrentStepId}' has no intent."); + return MissionAdvanceResult.Refused; + } + + string baseId = mission.GetBaseIntentId(step, mission.NextIndex); + int attempt = mission.IncrementAttempt(mission.CurrentStepId); + string admittedId = attempt == 1 + ? baseId + : FormattableString.Invariant($"{baseId}#attempt-{attempt}"); + mission.CurrentIntentId = admittedId; + stepIntent.IntentId = admittedId; + IntentDataType admittedIntent = CloneIntent(stepIntent); + + IntentAdmission admission = SubmitCore( + context, + sessionId, + ClientNameOf(context), + admittedIntent, + mission.MissionId, + admittedId, + baseId); if (!admission.Accepted) { - FinishMissionLocked(context, mission, ExecutionStateEnum.Failed); + FinishMissionLocked( + context, + mission, + ExecutionStateEnum.Failed, + admission.Failure, + admission.Message ?? "A mission step was refused."); return MissionAdvanceResult.Refused; } + mission.CurrentIntentId = admission.IntentId; + stepIntent.IntentId = admission.IntentId; + if (m_intents.TryGetValue(admission.IntentId, out IntentEntry? entry)) + { + MissionRules.SetStatus( + mission.Mission.Steps, + mission.NextIndex, + entry.State, + entry.Node?.NodeId); + } PublishMissionLocked(context, mission); return MissionAdvanceResult.Started; } @@ -2090,7 +2324,10 @@ private void AdvanceMissionLocked(ISystemContext context, IntentEntry entry, Int { // The compensation ran; the mission still ends, because that is // what distinguishes Compensate from Fallback. - FinishMissionLocked(context, mission, ExecutionStateEnum.Failed); + FinishMissionLocked( + context, mission, ExecutionStateEnum.Failed, + IntentFailureEnum.Other, + "Compensation completed; the mission is still failed."); return; } mission.RetriesUsed = 0; @@ -2107,7 +2344,7 @@ private void AdvanceMissionLocked(ISystemContext context, IntentEntry entry, Int return; } - ApplyErrorPolicyLocked(context, mission); + ApplyErrorPolicyLocked(context, mission, outcome); } /// @@ -2189,7 +2426,10 @@ private MissionTransitionSelection SelectTransitionUnlocked( /// /// Applies a failed step's error policy, per clause 7.4. /// - private void ApplyErrorPolicyLocked(ISystemContext context, MissionEntry mission) + private void ApplyErrorPolicyLocked( + ISystemContext context, + MissionEntry mission, + IntentOutcome stepOutcome) { MissionStepDataType? step = MissionRules.NextPending(mission.Mission.Steps, mission.NextIndex); @@ -2204,7 +2444,10 @@ private void ApplyErrorPolicyLocked(ISystemContext context, MissionEntry mission StartNextStepLocked(context, mission, ControlOwner); return; } - FinishMissionLocked(context, mission, ExecutionStateEnum.Failed); + FinishMissionLocked( + context, mission, ExecutionStateEnum.Failed, + stepOutcome.Failure, + stepOutcome.Message ?? "Retries exhausted."); return; case ErrorPolicyEnum.Skip: mission.RetriesUsed = 0; @@ -2219,7 +2462,10 @@ private void ApplyErrorPolicyLocked(ISystemContext context, MissionEntry mission mission.Mission.Steps, step?.FallbackStepId ?? string.Empty); if (target < 0) { - FinishMissionLocked(context, mission, ExecutionStateEnum.Failed); + FinishMissionLocked( + context, mission, ExecutionStateEnum.Failed, + stepOutcome.Failure, + stepOutcome.Message ?? "Fallback step not found."); return; } mission.RetriesUsed = 0; @@ -2228,7 +2474,10 @@ private void ApplyErrorPolicyLocked(ISystemContext context, MissionEntry mission StartNextStepLocked(context, mission, ControlOwner); return; default: - FinishMissionLocked(context, mission, ExecutionStateEnum.Failed); + FinishMissionLocked( + context, mission, ExecutionStateEnum.Failed, + stepOutcome.Failure, + stepOutcome.Message ?? "The step failed."); return; } } @@ -2237,17 +2486,42 @@ private void FinishMissionLocked( ISystemContext context, MissionEntry mission, ExecutionStateEnum state, - IntentFailureEnum failure = IntentFailureEnum.None) + IntentFailureEnum failure = IntentFailureEnum.None, + string failureMessage = "") { if (IntentOutcome.IsTerminal(mission.State)) { return; } + if (state == ExecutionStateEnum.Failed && failure == IntentFailureEnum.None) + { + failure = IntentFailureEnum.Other; + if (string.IsNullOrWhiteSpace(failureMessage)) + { + failureMessage = "The mission failed without a specific failure classification."; + } + } mission.Failure = state == ExecutionStateEnum.Failed ? failure : IntentFailureEnum.None; - PublishMissionFinalResultLocked(context, mission); - SetMissionStateLocked(context, mission, state); + mission.FailureMessage = state == ExecutionStateEnum.Failed + ? failureMessage ?? string.Empty + : string.Empty; + if (!string.IsNullOrEmpty(mission.CurrentStepId) && + mission.NextIndex >= 0 && + mission.NextIndex < mission.Mission.Steps.Count && + mission.Mission.Steps[mission.NextIndex] is { } currentStep && + !IntentOutcome.IsTerminal(currentStep.Status)) + { + NodeId? operation = m_intents.TryGetValue(mission.CurrentIntentId, out IntentEntry? entry) + ? entry.Node?.NodeId + : null; + MissionRules.SetStatus(mission.Mission.Steps, mission.NextIndex, state, operation); + } mission.CurrentStepId = string.Empty; + mission.CurrentIntentId = string.Empty; + PublishMissionFinalResultLocked(context, mission); PublishMissionLocked(context, mission); + SetMissionStateLocked(context, mission, state); + PruneTerminalMissionsLocked(); } private void CreateOperationNode(ISystemContext context, IntentEntry entry) @@ -2257,7 +2531,7 @@ private void CreateOperationNode(ISystemContext context, IntentEntry entry) var node = new IntentOperationState(folder) { NodeId = ChildNodeId(folder.NodeId, entry.OperationNodeName), - BrowseName = new QualifiedName(entry.IntentId, folder.BrowseName.NamespaceIndex), + BrowseName = new QualifiedName(entry.IntentId, Controller.BrowseName.NamespaceIndex), DisplayName = new LocalizedText(entry.IntentId), SymbolicName = entry.IntentId, ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent, @@ -2266,9 +2540,9 @@ private void CreateOperationNode(ISystemContext context, IntentEntry entry) EventNotifier = global::Opc.Ua.EventNotifiers.SubscribeToEvents }; node.Create(context, node.NodeId, node.BrowseName, node.DisplayName, false); - node.AddProgress(context); - node.AddQueuePosition(context); - node.AddCurrentPose(context); + node.AddProgress(context) + .AddQueuePosition(context) + .AddCurrentPose(context); EnsureFinalResultVariable(context, node).Value = Variant.Null; node.AddReference(global::Opc.Ua.ReferenceTypeIds.HasComponent, true, folder.NodeId); folder.AddReference(global::Opc.Ua.ReferenceTypeIds.HasComponent, false, node.NodeId); @@ -2294,7 +2568,7 @@ private void CreateMissionNode(ISystemContext context, MissionEntry entry) var node = new MissionObjectState(folder) { NodeId = ChildNodeId(folder.NodeId, entry.MissionNodeName), - BrowseName = new QualifiedName(entry.MissionId, folder.BrowseName.NamespaceIndex), + BrowseName = new QualifiedName(entry.MissionId, Controller.BrowseName.NamespaceIndex), DisplayName = new LocalizedText(entry.MissionId), SymbolicName = entry.MissionId, ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent, @@ -2303,6 +2577,16 @@ private void CreateMissionNode(ISystemContext context, MissionEntry entry) EventNotifier = global::Opc.Ua.EventNotifiers.SubscribeToEvents }; node.Create(context, node.NodeId, node.BrowseName, node.DisplayName, false); + EnsureFinalResultVariable( + context, + node, + nameof(IntentResultDataType.Failure), + DataTypeIds.IntentFailureEnum).Value = Variant.From((int)IntentFailureEnum.None); + EnsureFinalResultVariable( + context, + node, + nameof(IntentResultDataType.Message), + global::Opc.Ua.DataTypeIds.LocalizedText).Value = Variant.From(LocalizedText.Null); node.AddReference(global::Opc.Ua.ReferenceTypeIds.HasComponent, true, folder.NodeId); folder.AddReference(global::Opc.Ua.ReferenceTypeIds.HasComponent, false, node.NodeId); @@ -2323,6 +2607,26 @@ private void SetExecutionStateLocked( ExecutionStateEnum previous = entry.State; entry.State = state; PublishExecutionStateLocked(context, entry, previous, state); + UpdateMissionStepStateLocked(context, entry, state); + } + + private void UpdateMissionStepStateLocked( + ISystemContext context, + IntentEntry entry, + ExecutionStateEnum state) + { + if (string.IsNullOrEmpty(entry.MissionId) || + !m_missions.TryGetValue(entry.MissionId, out MissionEntry? mission) || + mission.CurrentIntentId != entry.IntentId) + { + return; + } + MissionRules.SetStatus( + mission.Mission.Steps, + mission.NextIndex, + state, + entry.Node?.NodeId); + PublishMissionLocked(context, mission); } private void PublishExecutionStateLocked( @@ -2445,6 +2749,33 @@ private void PruneTerminalOperationsLocked() } } + private void PruneTerminalMissionsLocked() + { + uint keep = m_options.RetainedTerminalMissions; + if (keep == 0) + { + return; + } + var terminal = m_missionHistory + .Where(kv => IntentOutcome.IsTerminal(kv.Value.State)) + .OrderBy(kv => kv.Value.AdmissionSequence) + .ToList(); + for (int i = 0; i < terminal.Count - (int)keep; i++) + { + KeyValuePair victim = terminal[i]; + if (m_removeNode != null && victim.Value.Node is { } stale) + { + RemoveNode(stale); + } + m_missionHistory.Remove(victim.Key); + if (m_missions.TryGetValue(victim.Value.MissionId, out MissionEntry? current) && + ReferenceEquals(current, victim.Value)) + { + m_missions.Remove(victim.Value.MissionId); + } + } + } + /// /// Retires a per-invocation node. Mirrors : the removal /// usually completes synchronously, and when it does not the task is observed @@ -2487,6 +2818,17 @@ private void CompleteLocked(ISystemContext context, IntentEntry entry, IntentOut IntentFailureEnum.Other, $"Executor returned non-terminal outcome {outcome.State}."); } + else if (outcome.State == ExecutionStateEnum.Failed && + outcome.Failure == IntentFailureEnum.None) + { + outcome = outcome with + { + Failure = IntentFailureEnum.Other, + Message = string.IsNullOrWhiteSpace(outcome.Message) + ? "Executor reported failure without a failure classification." + : outcome.Message + }; + } ExecutionStateEnum previous = entry.State; entry.State = outcome.State; @@ -2572,6 +2914,13 @@ private static void PublishMissionFinalResultLocked(ISystemContext context, Miss DataTypeIds.IntentFailureEnum); failure.Value = Variant.From((int)entry.Failure); failure.ClearChangeMasks(context, false); + BaseDataVariableState message = EnsureFinalResultVariable( + context, + node, + nameof(IntentResultDataType.Message), + global::Opc.Ua.DataTypeIds.LocalizedText); + message.Value = Variant.From(new LocalizedText(entry.FailureMessage)); + message.ClearChangeMasks(context, false); node.ClearChangeMasks(context, true); } @@ -2646,13 +2995,13 @@ private void PublishMissionLocked(ISystemContext context, MissionEntry entry) private void PublishControllerState(ISystemContext context) { - SetValue(m_controller.OperationalMode, m_options.OperationalMode); - SetValue(m_controller.Ready, IsReadyLocked()); - SetValue(m_controller.ControlOwner, ControlOwner ?? global::Opc.Ua.NodeId.Null); - SetValue(m_controller.MaxQueueDepth, m_options.MaxQueueDepth); - SetValue(m_controller.ActiveIntent, m_current?.Node?.NodeId ?? NodeId.Null); + SetValue(Controller.OperationalMode, m_options.OperationalMode); + SetValue(Controller.Ready, IsReadyLocked()); + SetValue(Controller.ControlOwner, ControlOwner ?? global::Opc.Ua.NodeId.Null); + SetValue(Controller.MaxQueueDepth, m_options.MaxQueueDepth); + SetValue(Controller.ActiveIntent, m_current?.Node?.NodeId ?? NodeId.Null); SetActiveMission(context, ActiveMissionNodeId()); - m_controller.ClearChangeMasks(context, true); + Controller.ClearChangeMasks(context, true); } private bool IsReadyLocked() @@ -2688,20 +3037,20 @@ private NodeId ActiveMissionNodeId() private void SetActiveMission(ISystemContext context, NodeId value) { - var browseName = new QualifiedName("ActiveMission", m_controller.BrowseName.NamespaceIndex); - if (m_controller.FindChild(context, browseName) is BaseDataVariableState typed) + var browseName = new QualifiedName("ActiveMission", Controller.BrowseName.NamespaceIndex); + if (Controller.FindChild(context, browseName) is BaseDataVariableState typed) { SetValue(typed, value); typed.ClearChangeMasks(context, false); return; } - if (m_controller.FindChild(context, browseName) is PropertyState property) + if (Controller.FindChild(context, browseName) is PropertyState property) { SetValue(property, value); property.ClearChangeMasks(context, false); return; } - if (m_controller.FindChild(context, browseName) is BaseDataVariableState variable) + if (Controller.FindChild(context, browseName) is BaseDataVariableState variable) { variable.Value = value; variable.ClearChangeMasks(context, false); @@ -2748,23 +3097,32 @@ private FolderState EnsureFolder(ISystemContext context, FolderState? declared, { return declared; } - var folder = new FolderState(m_controller) + ushort riNs = RobotIntentNamespaceIndex(context); + var folder = new FolderState(Controller) { - NodeId = ChildNodeId(m_controller.NodeId, browseName), - BrowseName = new QualifiedName(browseName, m_controller.BrowseName.NamespaceIndex), + NodeId = ChildNodeId(Controller.NodeId, browseName), + BrowseName = new QualifiedName(browseName, riNs), DisplayName = new LocalizedText(browseName), SymbolicName = browseName, ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent, TypeDefinitionId = global::Opc.Ua.ObjectTypeIds.FolderType, EventNotifier = global::Opc.Ua.EventNotifiers.None }; - m_controller.AddChild(folder); - folder.AddReference(global::Opc.Ua.ReferenceTypeIds.HasComponent, true, m_controller.NodeId); - m_controller.AddReference(global::Opc.Ua.ReferenceTypeIds.HasComponent, false, folder.NodeId); + Controller.AddChild(folder); + folder.AddReference( + global::Opc.Ua.ReferenceTypeIds.HasComponent, true, Controller.NodeId); + Controller.AddReference( + global::Opc.Ua.ReferenceTypeIds.HasComponent, false, folder.NodeId); AddNode(folder); return folder; } + private static ushort RobotIntentNamespaceIndex(ISystemContext context) + { + int idx = context.NamespaceUris.GetIndex(Namespaces.RobotIntent); + return idx < 0 ? (ushort)0 : (ushort)idx; + } + /// /// Publishes a per-invocation node. The add usually completes synchronously; /// when it does not, the task is observed so a failure surfaces instead of @@ -2791,21 +3149,21 @@ private static NodeId ChildNodeId(NodeId parent, string name) private void WireMethods(ISystemContext context) { - SetMethodExecutable(m_controller.RequestControl); - SetMethodExecutable(m_controller.ReleaseControl); - SetMethodExecutable(m_controller.SubmitIntent); - SetMethodExecutable(m_controller.CancelIntent); - SetMethodExecutable(m_controller.CancelAll); - SetMethodExecutable(m_controller.Pause); - SetMethodExecutable(m_controller.Resume); - SetMethodExecutable(m_controller.Retry); - SetMethodExecutable(m_controller.SubmitMission); - SetMethodExecutable(m_controller.UpdateMission); - SetMethodExecutable(m_controller.CancelMission); - SetMethodExecutable(m_controller.OpenRealTimeChannel); - SetMethodExecutable(m_controller.CloseRealTimeChannel); - - if (m_controller.RequestControl is { } requestControl) + SetMethodExecutable(Controller.RequestControl); + SetMethodExecutable(Controller.ReleaseControl); + SetMethodExecutable(Controller.SubmitIntent); + SetMethodExecutable(Controller.CancelIntent); + SetMethodExecutable(Controller.CancelAll); + SetMethodExecutable(Controller.Pause); + SetMethodExecutable(Controller.Resume); + SetMethodExecutable(Controller.Retry); + SetMethodExecutable(Controller.SubmitMission); + SetMethodExecutable(Controller.UpdateMission); + SetMethodExecutable(Controller.CancelMission); + SetMethodExecutable(Controller.OpenRealTimeChannel); + SetMethodExecutable(Controller.CloseRealTimeChannel); + + if (Controller.RequestControl is { } requestControl) { requestControl.OnCallAsync = (ctx, method, objectId, ct) => { @@ -2818,7 +3176,7 @@ private void WireMethods(ISystemContext context) }); }; } - if (m_controller.ReleaseControl is { } releaseControl) + if (Controller.ReleaseControl is { } releaseControl) { releaseControl.OnCallMethod2Async = (ctx, method, objectId, inputArguments, outputArguments, ct) => { @@ -2826,17 +3184,17 @@ private void WireMethods(ISystemContext context) return new ValueTask(ServiceResult.Good); }; } - if (m_controller.SubmitIntent is { } submit) + if (Controller.SubmitIntent is { } submit) { submit.OnCallAsync = async (ctx, method, objectId, intent, ct) => { await RefreshSafetyStateAsync(context, ct).ConfigureAwait(false); IntentAdmission admission = SubmitCore( - context, SessionOf(ctx), ClientNameOf(ctx), intent, string.Empty, forceNewId: false); + context, SessionOf(ctx), ClientNameOf(ctx), intent, string.Empty); return ToSubmitIntentResult(admission); }; } - if (m_controller.CancelIntent is { } cancelIntent) + if (Controller.CancelIntent is { } cancelIntent) { cancelIntent.OnCallAsync = (ctx, method, objectId, intentId, stopMode, ct) => new ValueTask(new CancelIntentMethodStateResult @@ -2845,7 +3203,7 @@ private void WireMethods(ISystemContext context) Accepted = CancelIntent(context, SessionOf(ctx), intentId, stopMode) }); } - if (m_controller.CancelAll is { } cancelAll) + if (Controller.CancelAll is { } cancelAll) { cancelAll.OnCallAsync = (ctx, method, objectId, stopMode, ct) => new ValueTask(new CancelAllMethodStateResult @@ -2854,7 +3212,7 @@ private void WireMethods(ISystemContext context) Cancelled = CancelAll(context, SessionOf(ctx), stopMode) }); } - if (m_controller.Pause is { } pause) + if (Controller.Pause is { } pause) { pause.OnCallAsync = (ctx, method, objectId, ct) => new ValueTask(new PauseMethodStateResult @@ -2863,7 +3221,7 @@ private void WireMethods(ISystemContext context) Accepted = Pause(context, SessionOf(ctx)) }); } - if (m_controller.Resume is { } resume) + if (Controller.Resume is { } resume) { resume.OnCallAsync = (ctx, method, objectId, ct) => new ValueTask(new ResumeMethodStateResult @@ -2872,7 +3230,7 @@ private void WireMethods(ISystemContext context) Accepted = Resume(context, SessionOf(ctx)) }); } - if (m_controller.Retry is { } retry) + if (Controller.Retry is { } retry) { retry.OnCallAsync = (ctx, method, objectId, intentId, ct) => { @@ -2887,7 +3245,7 @@ private void WireMethods(ISystemContext context) }); }; } - if (m_controller.SubmitMission is { } submitMission) + if (Controller.SubmitMission is { } submitMission) { submitMission.OnCallAsync = async (ctx, method, objectId, mission, ct) => { @@ -2907,7 +3265,7 @@ private void WireMethods(ISystemContext context) }; }; } - if (m_controller.UpdateMission is { } updateMission) + if (Controller.UpdateMission is { } updateMission) { updateMission.OnCallAsync = (ctx, method, objectId, missionId, updateId, steps, ct) => { @@ -2920,7 +3278,7 @@ private void WireMethods(ISystemContext context) }); }; } - if (m_controller.CancelMission is { } cancelMission) + if (Controller.CancelMission is { } cancelMission) { cancelMission.OnCallAsync = (ctx, method, objectId, missionId, stopMode, ct) => new ValueTask(new CancelMissionMethodStateResult @@ -2929,7 +3287,7 @@ private void WireMethods(ISystemContext context) Accepted = CancelMission(context, SessionOf(ctx), missionId, stopMode) }); } - if (m_controller.OpenRealTimeChannel is { } openChannel) + if (Controller.OpenRealTimeChannel is { } openChannel) { openChannel.OnCallAsync = (ctx, method, objectId, channelId, requestedLease, ct) => { @@ -2947,7 +3305,7 @@ private void WireMethods(ISystemContext context) }); }; } - if (m_controller.CloseRealTimeChannel is { } closeChannel) + if (Controller.CloseRealTimeChannel is { } closeChannel) { closeChannel.OnCallAsync = (ctx, method, objectId, channelId, ct) => new ValueTask( @@ -3193,8 +3551,7 @@ private void DisposeResources() m_shutdown.Cancel(); m_shutdown.Dispose(); m_pump.Dispose(); - IntentEntry[] entries = SnapshotIntents(); - foreach (IntentEntry entry in entries) + foreach (IntentEntry entry in SnapshotIntents()) { entry.Dispose(); } @@ -3283,18 +3640,19 @@ private async Task WaitForPumpShutdownAsync(Task pumpTask) private const uint StateSuspended = 3; private const uint StateHalted = 4; private const StopModeEnum SupersededStopMode = StopModeEnum.QuickStop; + private const string UnsupportedFastenJointMessage = "FastenIntent.Joint references from OPC 40450/40451 are not supported by this controller model; " + "omit Joint and provide the fastening parameters directly."; private readonly Lock m_lock = new(); - private readonly IntentControllerState m_controller; private readonly IIntentExecutor m_executor; private readonly IntentControllerHostOptions m_options; private readonly Func m_addNode; private readonly Func? m_removeNode; private readonly Dictionary m_intents = []; private readonly Dictionary m_missions = []; + private readonly Dictionary m_missionHistory = []; private readonly LinkedList m_queue = new(); private readonly Dictionary m_capabilities = []; private NamespaceTable m_namespaceUris = new(); @@ -3388,12 +3746,14 @@ public void ReportBlendBegin(Pose3DDataType pose) private sealed class IntentEntry( string intentId, + string baseIntentId, IntentDataType intent, string missionId, long admissionSequence) : IDisposable { public string IntentId { get; } = intentId; + public string BaseIntentId { get; } = baseIntentId; public string OperationNodeName { get; } = $"{intentId}-{Guid.NewGuid():N}"; public IntentDataType Intent { get; } = intent; public string MissionId { get; } = missionId; @@ -3453,19 +3813,92 @@ private sealed class ChannelEntry public DateTime Expiry { get; set; } = DateTime.MinValue; } - private sealed class MissionEntry(string missionId, MissionDataType mission) + private sealed class MissionEntry(string missionId, MissionDataType mission, long sequence) { public string MissionId { get; } = missionId; public string MissionNodeName { get; } = $"{missionId}-{Guid.NewGuid():N}"; public MissionDataType Mission { get; } = mission; + public long AdmissionSequence { get; } = sequence; public MissionObjectState? Node { get; set; } public ExecutionStateEnum State { get; set; } = ExecutionStateEnum.Accepted; public IntentFailureEnum Failure { get; set; } + public string FailureMessage { get; set; } = string.Empty; public int NextIndex { get; set; } public uint RetriesUsed { get; set; } public bool Compensating { get; set; } public string CurrentStepId { get; set; } = string.Empty; public string CurrentIntentId { get; set; } = string.Empty; + + public Dictionary StepAttempts { get; } = + new(StringComparer.Ordinal); + + public Dictionary StepBaseIntentIds { get; } = + CreateStepBaseIntentIds(mission.Steps); + + public int IncrementAttempt(string stepId) + { + if (!StepAttempts.TryGetValue(stepId, out int count)) + { + count = 0; + } + count++; + StepAttempts[stepId] = count; + return count; + } + + public string GetBaseIntentId(MissionStepDataType step, int index) + { + string stepId = step.StepId ?? string.Empty; + if (StepBaseIntentIds.TryGetValue(stepId, out string? intentId) && + !string.IsNullOrEmpty(intentId)) + { + return intentId; + } + intentId = step.Intent?.IntentId ?? string.Empty; + if (string.IsNullOrEmpty(intentId)) + { + intentId = string.IsNullOrEmpty(stepId) + ? FormattableString.Invariant($"{MissionId}/step-{index}") + : FormattableString.Invariant($"{MissionId}/{stepId}"); + if (step.Intent != null) + { + step.Intent.IntentId = intentId; + } + } + StepBaseIntentIds[stepId] = intentId; + return intentId; + } + + public void ReplaceSteps(ArrayOf steps, int preservedCount) + { + for (int ii = preservedCount; ii < Mission.Steps.Count; ii++) + { + string oldStepId = Mission.Steps[ii].StepId ?? string.Empty; + StepAttempts.Remove(oldStepId); + StepBaseIntentIds.Remove(oldStepId); + } + + Mission.Steps = steps; + for (int ii = preservedCount; ii < steps.Count; ii++) + { + MissionStepDataType step = steps[ii]; + string stepId = step.StepId ?? string.Empty; + StepAttempts.Remove(stepId); + StepBaseIntentIds[stepId] = step.Intent?.IntentId ?? string.Empty; + } + } + + private static Dictionary CreateStepBaseIntentIds( + ArrayOf steps) + { + var ids = new Dictionary(StringComparer.Ordinal); + for (int ii = 0; ii < steps.Count; ii++) + { + MissionStepDataType step = steps[ii]; + ids[step.StepId ?? string.Empty] = step.Intent?.IntentId ?? string.Empty; + } + return ids; + } } } diff --git a/src/Opc.Ua.Robotics.Server/Intent/IntentControllerHostOptions.cs b/src/Opc.Ua.Robotics.Server/Intent/IntentControllerHostOptions.cs index 355ae79c9c..42fdd56dc0 100644 --- a/src/Opc.Ua.Robotics.Server/Intent/IntentControllerHostOptions.cs +++ b/src/Opc.Ua.Robotics.Server/Intent/IntentControllerHostOptions.cs @@ -184,6 +184,13 @@ public sealed class IntentControllerHostOptions /// public uint RetainedTerminalOperations { get; set; } = 128; + /// + /// How many terminal missions to keep browsable per controller, or zero to keep + /// every one of them. Mirrors for + /// mission nodes. The default keeps the latest 32 terminal missions. + /// + public uint RetainedTerminalMissions { get; set; } = 32; + /// /// How long asynchronous disposal waits for an executing intent to observe /// shutdown cancellation and let the pump drain, in milliseconds. @@ -320,6 +327,7 @@ public sealed record DeclaredCapability /// /// Resolves this declaration into the value published in the address space. /// + /// public IntentCapabilityDataType Resolve(NamespaceTable namespaceUris) { ArrayOf buffers = SupportedBufferModes.IsNull || SupportedBufferModes.IsEmpty @@ -1170,7 +1178,14 @@ private static bool IntentEqual(IntentDataType? left, IntentDataType? right) { return left == right; } - return left.IsEqual(right); + if (left.Clone() is not IntentDataType leftComparable || + right.Clone() is not IntentDataType rightComparable) + { + return false; + } + leftComparable.IntentId = string.Empty; + rightComparable.IntentId = string.Empty; + return leftComparable.IsEqual(rightComparable); } } diff --git a/src/Opc.Ua.Robotics/Model/Opc.Ua.RobotIntent.NodeSet2.xml b/src/Opc.Ua.Robotics/Model/Opc.Ua.RobotIntent.NodeSet2.xml index 1bd59f253d..749397db5f 100644 --- a/src/Opc.Ua.Robotics/Model/Opc.Ua.RobotIntent.NodeSet2.xml +++ b/src/Opc.Ua.Robotics/Model/Opc.Ua.RobotIntent.NodeSet2.xml @@ -50,7 +50,7 @@ Admitted and validated, not yet queued or executing.Waiting behind another intent because BufferMode is Buffered or a blending mode. Corresponds to PLCopen Busy without Active.Commanding the robot now.Paused by request; position is retained and execution can resume.A cancel was accepted and the Server is bringing the motion to a controlled end. Not yet terminal.Terminal. Completed as requested.Terminal. Did not complete; Failure carries the reason.Terminal. Ended early because a cancel was accepted.Terminal for now, but the Server can re-attempt it on Retry. A Server that does not offer Retry never enters this state and reports Failed instead. - + EnumStrings i=78 @@ -69,7 +69,7 @@ Abort what is executing and start immediately. The aborted intent terminates as Cancelled. This is the default.Queue; start when the predecessor succeeds.Blend at the lower of the two boundary speeds.Blend at the predecessor's boundary speed.Blend at the successor's boundary speed.Blend at the higher of the two boundary speeds. - + EnumStrings i=78 @@ -88,7 +88,7 @@ Runs in the background; motion may continue and other intents may run concurrently.Motion stops for the duration; other intents may still run.Motion may continue; no other intent may run concurrently.Motion stops and no other intent may run concurrently. The intent has exclusive use of the robot. - + EnumStrings i=78 @@ -107,7 +107,7 @@ Come to rest on the target before the next motion begins. ABB fine, FANUC FINE, Yaskawa PL=0, KUKA no approximation.Round the corner into the next motion without stopping. - + EnumStrings i=78 @@ -126,7 +126,7 @@ Open the end effector where it is, without placing.Set the object down under control at the target.Retain the object until the Server judges a receiving party has taken it. - + EnumStrings i=78 @@ -145,7 +145,7 @@ The Server chooses.Along the tool's own Z axis.From above, in the frame the target is expressed in.Laterally, in the frame the target is expressed in. - + EnumStrings i=78 @@ -164,7 +164,7 @@ The cell-level reference frame.The robot base frame.The flange at the end of the last link, to which an end effector is fitted.A tool frame, whose origin is a tool centre point.A workpiece or work-object frame.A frame whose role is none of the above. - + EnumStrings i=78 @@ -183,7 +183,7 @@ Booting, uncalibrated, or a safety system fault.Teaching mode with a speed ceiling and a held enabling device.Program verification with an enabling device.Automatic operation with the safeguarded space secured.Automatic operation commanded by an external system. This is the mode this specification is written for. - + EnumStrings i=78 @@ -202,7 +202,7 @@ No failure. Reported on a successful outcome.The target lies outside the reachable workspace.No kinematic solution, or a singularity on the path.A collision was predicted or detected.A joint limit would be or was exceeded.The requested speed or acceleration is not permitted in the active mode.The required tool is not fitted or not identified.The object to act on was not present.The object was not acquired, or was lost in transit.The intent did not complete within its permitted time.Refused because the operational mode does not permit it. See clause 10.Refused because the caller does not hold command authority. See clause 8.The Server does not implement this intent type, or this combination of options.A parameter was missing, malformed or out of range.The queue is at MaxQueueDepth.An Aborting submission or a mission update replaced it before it could run.A fault in the robot, the end effector or the controller.A safety function acted. The safety system, not this interface, decided this.A reason none of the above describes; see Message.Refused because the request would exceed a limit the safety system is enforcing. See clause 10.3.A mission branch point had no true outgoing transition, or the selected transition target did not resolve. See clause 7.4. - + EnumStrings i=78 @@ -221,7 +221,7 @@ Decelerate along the programmed path.Stop when the current cycle completes.Stop at a point the process defines as safe.Decelerate as quickly as the drives allow.Stop when the current instruction completes. - + EnumStrings i=78 @@ -240,7 +240,7 @@ Rotates. Its joint target is in radians.Translates. Its joint target is in metres. - + EnumStrings i=78 @@ -259,7 +259,7 @@ The horizon was replaced as requested.MissionUpdateId was not greater than the current one.The update would have altered a released step.No mission with that MissionId is held.Refused for a reason the Server states in a message. - + EnumStrings i=78 @@ -839,7 +839,7 @@ ns=1;i=6019 - + OutputArguments i=78 @@ -866,7 +866,7 @@ ns=1;i=6023 - + InputArguments i=78 @@ -875,7 +875,7 @@ i=297Intentns=1;i=3053-1The intent to execute. - + OutputArguments i=78 @@ -894,7 +894,7 @@ ns=1;i=6026 - + InputArguments i=78 @@ -903,7 +903,7 @@ i=297IntentIdi=12-1The intent to cancel.i=297StopModens=1;i=3010-1How urgently to stop. - + OutputArguments i=78 @@ -922,7 +922,7 @@ ns=1;i=6029 - + InputArguments i=78 @@ -931,7 +931,7 @@ i=297StopModens=1;i=3010-1How urgently to stop. - + OutputArguments i=78 @@ -949,7 +949,7 @@ ns=1;i=6031 - + OutputArguments i=78 @@ -967,7 +967,7 @@ ns=1;i=6033 - + OutputArguments i=78 @@ -986,7 +986,7 @@ ns=1;i=6036 - + InputArguments i=78 @@ -995,7 +995,7 @@ i=297IntentIdi=12-1The intent to re-attempt. - + OutputArguments i=78 @@ -1014,7 +1014,7 @@ ns=1;i=6039 - + InputArguments i=78 @@ -1023,7 +1023,7 @@ i=297Missionns=1;i=3068-1The mission to execute. - + OutputArguments i=78 @@ -1042,7 +1042,7 @@ ns=1;i=6042 - + InputArguments i=78 @@ -1051,7 +1051,7 @@ i=297MissionIdi=12-1The mission to update.i=297MissionUpdateIdi=7-1Revision of the update. Must be greater than the mission's current value.i=297Stepsns=1;i=306710The steps that replace the horizon. - + OutputArguments i=78 @@ -1070,7 +1070,7 @@ ns=1;i=6045 - + InputArguments i=78 @@ -1079,7 +1079,7 @@ i=297MissionIdi=12-1The mission to cancel.i=297StopModens=1;i=3010-1How urgently to stop. - + OutputArguments i=78 @@ -1706,7 +1706,7 @@ No safe motion function is active.Safe Torque Off: torque is removed.Safe Stop 1: a controlled ramp to standstill, then Safe Torque Off.Safe Stop 2: a controlled ramp to standstill, which is then held under power.Safe Operating Stop: standstill is monitored while the drive remains energised.Safely Limited Speed: speed is monitored against a limit.Safely Limited Position: position is monitored against a limit.Safe Direction: motion is permitted in one direction only.Safe Brake Control: a brake is commanded safely. - + EnumStrings i=78 @@ -1725,7 +1725,7 @@ Universal Robots Real-Time Data Exchange.ABB Externally Guided Motion.KUKA Fast Research Interface.KUKA Robot Sensor Interface.Yaskawa MotoROS2.OPC UA FX (OPC 10000-80 to -84). The open path, and the only one in this list that is an OPC Foundation specification.A transport identified by the channel's own descriptor. - + EnumStrings i=78 @@ -1744,7 +1744,7 @@ The Server connects to an endpoint the client is listening on.The client connects to the endpoint the Server publishes. - + EnumStrings i=78 @@ -1763,7 +1763,7 @@ End the mission. This is the default and the behaviour of a mission that declares no policy.Re-attempt the step. A Server bounds the attempts and reports Failed when they are exhausted.Record the failure and continue with the next step.Continue at FallbackStepId instead.Run the fallback step to undo the work already done, then end the mission. - + EnumStrings i=78 @@ -1782,7 +1782,7 @@ Exactly one transition is taken - the first whose condition holds. An OR divergence.Every transition is taken and the branches run concurrently. An AND divergence. - + EnumStrings i=78 @@ -1801,7 +1801,7 @@ No weave.Sinusoidal oscillation.Triangular oscillation.Trapezoidal oscillation, with a dwell at each edge. - + EnumStrings i=78 @@ -2382,7 +2382,7 @@ ns=1;i=6128 - + InputArguments i=78 @@ -2391,7 +2391,7 @@ i=297ChannelIdi=12-1The channel to open.i=297RequestedLeasei=290-1How long the lease is wanted for, in milliseconds. - + OutputArguments i=78 @@ -2410,7 +2410,7 @@ ns=1;i=6131 - + InputArguments i=78 @@ -2419,7 +2419,7 @@ i=297ChannelIdi=12-1The channel to release. - + OutputArguments i=78 diff --git a/src/Opc.Ua.Server/Hosting/OpcUaServerApplicationConfigurationFeature.cs b/src/Opc.Ua.Server/Hosting/OpcUaServerApplicationConfigurationFeature.cs index 573bdc8965..bb7bec1b66 100644 --- a/src/Opc.Ua.Server/Hosting/OpcUaServerApplicationConfigurationFeature.cs +++ b/src/Opc.Ua.Server/Hosting/OpcUaServerApplicationConfigurationFeature.cs @@ -66,6 +66,7 @@ internal static IApplicationConfigurationBuilderSecurity Configure( { IApplicationConfigurationBuilderTransportQuotas quotasBuilder = builder .SetMaxByteStringLength((int)options.MaxByteStringLength) + .SetMaxStringLength((int)options.MaxStringLength) .SetMaxArrayLength((int)options.MaxArrayLength); if (options.MaxMessageSize is int maxMessageSize) diff --git a/src/Opc.Ua.Server/Hosting/OpcUaServerOptions.cs b/src/Opc.Ua.Server/Hosting/OpcUaServerOptions.cs index 2fb60b0551..a436124889 100644 --- a/src/Opc.Ua.Server/Hosting/OpcUaServerOptions.cs +++ b/src/Opc.Ua.Server/Hosting/OpcUaServerOptions.cs @@ -97,6 +97,19 @@ public sealed class OpcUaServerOptions /// public uint MaxByteStringLength { get; set; } = 4 * 1024 * 1024; + /// + /// Maximum string length advertised on the transport. Defaults to + /// . + /// + /// + /// This is a separate quota from and + /// : none of those govern a String, so a + /// Server whose address space legitimately carries a long string - a serialised + /// document, a URI carrying an inline payload - could not raise the limit through + /// the hosting API at all before this option existed. + /// + public uint MaxStringLength { get; set; } = (uint)DefaultEncodingLimits.MaxStringLength; + /// /// Maximum array length advertised on the transport. Defaults to 1 Mi /// elements. diff --git a/src/Opc.Ua.Types/BuiltIn/RelativePathElement.cs b/src/Opc.Ua.Types/BuiltIn/RelativePathElement.cs index 4d66e9ce16..54729a3b75 100644 --- a/src/Opc.Ua.Types/BuiltIn/RelativePathElement.cs +++ b/src/Opc.Ua.Types/BuiltIn/RelativePathElement.cs @@ -55,7 +55,7 @@ private void Initialize(StreamingContext context) private void Initialize() { - IsInverse = true; + IsInverse = false; IncludeSubtypes = true; } diff --git a/src/Opc.Ua.Vision.Client/Hosting/OpcUaVisionClientBuilderExtensions.cs b/src/Opc.Ua.Vision.Client/Hosting/OpcUaVisionClientBuilderExtensions.cs new file mode 100644 index 0000000000..9e4687ac7a --- /dev/null +++ b/src/Opc.Ua.Vision.Client/Hosting/OpcUaVisionClientBuilderExtensions.cs @@ -0,0 +1,80 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Opc.Ua; +using Opc.Ua.Client; +using Opc.Ua.Vision.Client; + +namespace Microsoft.Extensions.DependencyInjection +{ + /// + /// Registers the Vision client over the managed OPC UA session. + /// + public static class OpcUaVisionClientBuilderExtensions + { + /// + /// Registers a and a + /// Func<CancellationToken, Task<VisionClient>> so + /// downstream services can request Vision clients. + /// + /// + /// The client builder returned by AddClient. + /// + public static IOpcUaClientBuilder AddVisionClient( + this IOpcUaClientBuilder builder) + { + builder.ThrowIfNull(nameof(builder)); + + builder.Services.TryAddSingleton(sp => + { + Func> sessionFactory = + sp.GetService>>() + ?? throw new InvalidOperationException( + "AddVisionClient requires AddClient to be called first."); + ITelemetryContext telemetry = + sp.GetRequiredService(); + return new VisionClientFactory(sessionFactory, telemetry); + }); + + builder.Services.TryAddSingleton< + Func>>(sp => + { + VisionClientFactory factory = + sp.GetRequiredService(); + return factory.CreateAsync; + }); + + return builder; + } + } +} diff --git a/src/Opc.Ua.Vision.Client/NugetREADME.md b/src/Opc.Ua.Vision.Client/NugetREADME.md new file mode 100644 index 0000000000..dfbb2f1c70 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/NugetREADME.md @@ -0,0 +1,87 @@ +# Opc.Ua.Vision.Client + +Client-side helpers for the **OPC UA — Vision** companion model (working-group +draft). + +Built on **Opc.Ua.Vision** and **Opc.Ua.Client**, this package exposes a +high-level surface over the source-generated proxies so a client can drive a +Vision server without knowing NodeIds or BrowseNames: + +- `VisionClient` — resolves the well-known `Vision` object (§4.2) and + enumerates sensors, inference pipelines, and coordinate frames; +- `VisionSensorClient` — reads a sensor's identity, imaging members, optics, + illumination, mounted frame and calibrations (intrinsic and hand-eye + extrinsic); +- `VisionFrameGraph` — walks the `CoordinateFrameType` tree and composes + transforms between any two named frames (camera → flange → base, camera → + flange → tool centre point). Follows the §5.12 conventions exactly — + right-handed frames, quaternion order (x, y, z, w), corner-datum principal + point, and refuses a non-unit quaternion within tolerance 1e-6; +- `VisionMediaClient` — `GetClip` by reference (default), inline + `LatestClip`/`LatestClipMetadata`, and honest surfacing of the §6.4 case + where inline delivery is disabled (`Bad_NotSupported`); +- `VisionPipelineClient` — `RunInference`, `StartContinuous`, `Stop`, and + reading pipeline members including the deployment inference location; +- `VisionResultReader` — reads `DetectionResultType`, `InspectionResultType`, + `SegmentationResultType` and streams result changes over an + `IStreamingSubscription`; +- `VisionFeedbackClient` — `SubmitDetections`, `SubmitInspectionResult`, + `SubmitCorrection`, `SubmitImageReference`, the headline path for an + off-server vision-language model publishing results the Server did not + compute; +- `AddVisionClient()` — DI registration for `IOpcUaClientBuilder`; +- `session.Vision(telemetry)` — non-DI extension for creating a client over + any connected `ISession`. + +## Example + +```csharp +using Opc.Ua.Client; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +// From any connected session: +VisionClient vision = session.Vision(telemetry); +if (!vision.IsVisionNamespaceAvailable) +{ + return; // Server does not implement Vision. +} + +await foreach (VisionNodeEntry sensor in vision.EnumerateSensorsAsync(ct)) +{ + VisionSensorClient s = vision.Sensor(sensor.NodeId); + VisionSensorIdentity identity = await s.ReadIdentityAsync(ct); + // Read intrinsic / hand-eye calibrations, media endpoints, etc. +} + +// Read the latest detection result for a pipeline, then compose the +// first detection's pose from the camera frame into the world frame. +VisionFrameGraph frames = vision.Frames(); +VisionDetectionResultSnapshot det = + await vision.Result(resultNodeId).ReadDetectionAsync(ct); +if (det.Detections.Count > 0 && det.Detections[0].HasPose) +{ + VisionPose3DDataType inWorld = await frames.ComposeAsync( + det.Detections[0].Pose, cameraFrameNodeId, worldFrameNodeId, ct); +} +``` + +See the [Vision developer guide](https://github.com/OPCFoundation/UA-.NETStandard/blob/main/docs/Vision.md) +for discovery, streaming, feedback and the frame-graph composition +example. + +> The namespace `http://opcfoundation.org/UA/Vision/` and every NodeId in it +> are **provisional**. The model is a working-group draft and is neither +> official nor endorsed by the OPC Foundation. + +## Related packages + +| Package | Adds | +|---|---| +| `OPCFoundation.NetStandard.Opc.Ua.Vision` | Source-generated Vision model (required) | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Server` | Hosting a Vision server | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Vision` | MCP tools that let a language model use `VisionClient` from an agent | + +## License + +OPC Foundation MIT License 1.00 — diff --git a/src/Opc.Ua.Vision.Client/Opc.Ua.Vision.Client.csproj b/src/Opc.Ua.Vision.Client/Opc.Ua.Vision.Client.csproj new file mode 100644 index 0000000000..a6d11ee5e5 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/Opc.Ua.Vision.Client.csproj @@ -0,0 +1,28 @@ + + + $(AssemblyPrefix).Vision.Client + $(LibTargetFrameworks) + $(PackagePrefix).Opc.Ua.Vision.Client + Opc.Ua.Vision.Client + $(NoWarn);CS1591 + enable + Client-side helpers for the OPC UA Vision (draft) companion model: discover the well-known Vision root, browse sensors, calibrations and coordinate frames, compose transforms across the frame tree, drive inference pipelines, read detection, inspection and segmentation results, and submit feedback (detections, corrections, image references) to a Vision server. + true + NugetREADME.md + true + true + + + $(PackageId).Debug + + + + + + + + + + + + diff --git a/src/Opc.Ua.Vision.Client/Properties/AssemblyInfo.cs b/src/Opc.Ua.Vision.Client/Properties/AssemblyInfo.cs new file mode 100644 index 0000000000..7798c9bd57 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/Properties/AssemblyInfo.cs @@ -0,0 +1,32 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +[assembly: CLSCompliant(false)] diff --git a/src/Opc.Ua.Vision.Client/SessionVisionExtensions.cs b/src/Opc.Ua.Vision.Client/SessionVisionExtensions.cs new file mode 100644 index 0000000000..3ff35d1e8c --- /dev/null +++ b/src/Opc.Ua.Vision.Client/SessionVisionExtensions.cs @@ -0,0 +1,66 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Opc.Ua.Client; + +namespace Opc.Ua.Vision.Client +{ + /// + /// Convenience extensions for opening a high-level Vision client from an + /// existing session, without DI. + /// + public static class SessionVisionExtensions + { + /// + /// Creates a high-level Vision client over the supplied session. + /// + /// + /// The connected session. + /// + /// + /// The telemetry context used by generated proxies. + /// + /// + public static VisionClient Vision( + this ISession session, + ITelemetryContext telemetry) + { + if (session is null) + { + throw new ArgumentNullException(nameof(session)); + } + if (telemetry is null) + { + throw new ArgumentNullException(nameof(telemetry)); + } + return new VisionClient(session, telemetry); + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionClient.cs b/src/Opc.Ua.Vision.Client/VisionClient.cs new file mode 100644 index 0000000000..e56d73638a --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionClient.cs @@ -0,0 +1,519 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client; + +namespace Opc.Ua.Vision.Client +{ + /// + /// High-level client for the OPC UA Vision (draft) companion model. It resolves + /// the well-known Vision object under the Server (§4.2), enumerates + /// sensors, inference pipelines and coordinate frames, and hands out focused + /// sub-clients for sensors, frames, media management, pipelines and feedback. + /// + /// + /// Every enumeration operation is subtype aware: a Server that specialises the + /// abstract Vision types (ImageSensorType vs a vendor-derived + /// AcmeCameraType, or one of the concrete result subtypes) is discovered + /// as an instance of the closest declared Vision base type. + /// + public sealed class VisionClient + { + /// + /// Creates a Vision client over a connected session. + /// + /// + /// The connected session. + /// + /// + /// The telemetry context used by generated proxies. + /// + public VisionClient(ISession session, ITelemetryContext telemetry) + { + Operations = new VisionClientOperations(session, telemetry); + } + + /// + /// Gets the connected session. + /// + public ISession Session => Operations.Session; + + /// + /// Gets the telemetry context. + /// + public ITelemetryContext Telemetry => Operations.Telemetry; + + /// + /// Gets whether the Server exposes the Vision namespace at all. Where + /// false, every enumeration on this client returns an empty result. + /// + public bool IsVisionNamespaceAvailable + => Operations.TryGetVisionNamespaceIndex(out _); + + /// + /// Resolves the NodeId of the well-known Vision object (§4.2). Returns + /// a null NodeId when the Server does not expose the Vision namespace. + /// + public NodeId VisionRootId + { + get + { + if (!Operations.TryGetVisionNamespaceIndex(out ushort _)) + { + return NodeId.Null; + } + return NodeId.Create( + Objects.Vision, Namespaces.Vision, Session.NamespaceUris); + } + } + + /// + /// Resolves the NodeId of the mandatory Vision/Sensors folder. Returns + /// a null NodeId when the Server does not expose the Vision namespace. + /// + public NodeId SensorsFolderId + { + get + { + if (!Operations.TryGetVisionNamespaceIndex(out ushort _)) + { + return NodeId.Null; + } + return NodeId.Create( + Objects.Vision_Sensors, Namespaces.Vision, Session.NamespaceUris); + } + } + + /// + /// Resolves the NodeId of the mandatory Vision/Sensors folder by browse-path + /// from the Vision root, falling back to the well-known identifier. + /// + /// + /// The well-known identifier only holds for a Server that materialises the Vision + /// tree in the Vision namespace itself. A Server that builds it as instances in its + /// own namespace - which is what the fluent builder produces - has a Sensors folder + /// whose NodeId is its own, so the well-known one resolves to nothing and every + /// sensor is invisible. Pipelines and Frames already resolve by browse path; this + /// makes Sensors behave the same way. + /// + /// + /// Cancels the operation. + /// + public async ValueTask GetSensorsFolderIdAsync( + CancellationToken cancellationToken = default) + { + NodeId resolved = await ResolveOptionalRootChildAsync( + BrowseNames.Sensors, cancellationToken).ConfigureAwait(false); + return resolved.IsNull ? SensorsFolderId : resolved; + } + + /// + /// Resolves the NodeId of the optional Vision/Pipelines folder by + /// browse-path from the Vision root. Returns a null NodeId when the folder + /// is not present. + /// + /// + /// Cancels the operation. + /// + public ValueTask GetPipelinesFolderIdAsync( + CancellationToken cancellationToken = default) + { + return ResolveOptionalRootChildAsync( + BrowseNames.Pipelines, cancellationToken); + } + + /// + /// Resolves the NodeId of the optional Vision/Frames folder by + /// browse-path from the Vision root. Returns a null NodeId when the folder + /// is not present. + /// + /// + /// Cancels the operation. + /// + public ValueTask GetFramesFolderIdAsync( + CancellationToken cancellationToken = default) + { + return ResolveOptionalRootChildAsync( + BrowseNames.Frames, cancellationToken); + } + + /// + /// Discovers every sensor exposed under the Vision root, including instances + /// of vendor subtypes derived from VisionSensorType. + /// + /// + /// Cancels the operation. + /// + public async ValueTask> DiscoverSensorsAsync( + CancellationToken cancellationToken = default) + { + NodeId sensors = await GetSensorsFolderIdAsync(cancellationToken).ConfigureAwait(false); + if (sensors.IsNull) + { + return ArrayOf.Empty; + } + return await Operations.DiscoverInstancesAsync( + sensors, + Operations.VisionNamespaceType(ObjectTypes.VisionSensorType), + cancellationToken).ConfigureAwait(false); + } + + /// + /// Discovers every inference pipeline exposed under the Vision root. + /// + /// + /// Cancels the operation. + /// + public async ValueTask> DiscoverPipelinesAsync( + CancellationToken cancellationToken = default) + { + NodeId pipelines = await GetPipelinesFolderIdAsync(cancellationToken) + .ConfigureAwait(false); + if (pipelines.IsNull) + { + return ArrayOf.Empty; + } + return await Operations.DiscoverInstancesAsync( + pipelines, + Operations.VisionNamespaceType(ObjectTypes.InferencePipelineType), + cancellationToken).ConfigureAwait(false); + } + + /// + /// Discovers every coordinate frame exposed under the Vision root. + /// + /// + /// Cancels the operation. + /// + public async ValueTask> DiscoverFramesAsync( + CancellationToken cancellationToken = default) + { + NodeId frames = await GetFramesFolderIdAsync(cancellationToken) + .ConfigureAwait(false); + if (frames.IsNull) + { + return ArrayOf.Empty; + } + return await Operations.DiscoverInstancesAsync( + frames, + Operations.VisionNamespaceType(ObjectTypes.CoordinateFrameType), + cancellationToken).ConfigureAwait(false); + } + + /// + /// Enumerates the sensors under the Vision root along with their BrowseName, + /// DisplayName and TypeDefinition, so a client can render a picker without a + /// second round-trip. + /// + /// + /// Cancels the operation. + /// + public async IAsyncEnumerable EnumerateSensorsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId parent = await GetSensorsFolderIdAsync(cancellationToken) + .ConfigureAwait(false); + await foreach (VisionNodeEntry entry in EnumerateInstancesAsync( + parent, ObjectTypes.VisionSensorType, cancellationToken) + .ConfigureAwait(false)) + { + yield return entry; + } + } + + /// + /// Enumerates the inference pipelines under the Vision root along with their + /// BrowseName, DisplayName and TypeDefinition. + /// + /// + /// Cancels the operation. + /// + public async IAsyncEnumerable EnumeratePipelinesAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId parent = await GetPipelinesFolderIdAsync(cancellationToken) + .ConfigureAwait(false); + await foreach (VisionNodeEntry entry in EnumerateInstancesAsync( + parent, ObjectTypes.InferencePipelineType, cancellationToken) + .ConfigureAwait(false)) + { + yield return entry; + } + } + + /// + /// Enumerates the coordinate frames under the Vision root along with their + /// BrowseName, DisplayName and TypeDefinition. + /// + /// + /// Cancels the operation. + /// + public async IAsyncEnumerable EnumerateFramesAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId parent = await GetFramesFolderIdAsync(cancellationToken) + .ConfigureAwait(false); + await foreach (VisionNodeEntry entry in EnumerateInstancesAsync( + parent, ObjectTypes.CoordinateFrameType, cancellationToken) + .ConfigureAwait(false)) + { + yield return entry; + } + } + + /// + /// Opens a focused sensor client over . + /// + /// + /// The sensor object NodeId; typically obtained from + /// . + /// + public VisionSensorClient Sensor(NodeId sensorNodeId) + { + return new VisionSensorClient(Operations, sensorNodeId); + } + + /// + /// Opens a focused inference pipeline client over + /// . + /// + /// + /// The pipeline object NodeId; typically obtained from + /// . + /// + public VisionPipelineClient Pipeline(NodeId pipelineNodeId) + { + return new VisionPipelineClient(Operations, pipelineNodeId); + } + + /// + /// Opens a focused feedback client over the feedback object of a pipeline. + /// + /// + /// The VisionFeedbackType object NodeId — the value of + /// InferencePipelineType.Feedback. + /// + public VisionFeedbackClient Feedback(NodeId feedbackNodeId) + { + return new VisionFeedbackClient(Operations, feedbackNodeId); + } + + /// + /// Opens a focused media-management client over the media object of a sensor. + /// + /// + /// The VisionMediaManagementType object NodeId — the value of + /// VisionSensorType.Media. + /// + public VisionMediaClient Media(NodeId mediaNodeId) + { + return new VisionMediaClient(Operations, mediaNodeId); + } + + /// + /// Opens a focused result reader over an InspectionResultType, + /// DetectionResultType or SegmentationResultType instance. + /// + /// + /// The result object NodeId. + /// + public VisionResultReader Result(NodeId resultNodeId) + { + return new VisionResultReader(Operations, resultNodeId); + } + + /// + /// Opens a focused frame graph over the Server's coordinate-frame tree. The + /// graph resolves instances and composes + /// transforms between two named frames per the §5.12 conventions + /// (right-handed frames, quaternion order (x, y, z, w), metres). + /// + public VisionFrameGraph Frames() + { + return new VisionFrameGraph(Operations); + } + + /// + /// Creates the one-shot inference service for running inference, + /// determining the result kind, and reading a bounded concise summary. + /// + public VisionInferenceService Inference() + { + return new VisionInferenceService(Operations); + } + + /// + /// Resolves a pipeline by exact unique name (BrowseName.Name or + /// DisplayName.Text, trimmed) or by NodeId string. Returns the + /// matching . + /// + /// + /// A NodeId string (e.g. ns=2;s=Vision/Pipelines/Abc) or an + /// exact published name. + /// + /// + /// Cancels the operation. + /// + /// + /// is null or empty. + /// + /// + /// No match found, or multiple pipelines matched the name. + /// + public async Task ResolvePipelineAsync( + string pipelineSelector, + CancellationToken cancellationToken = default) + { + if (string.IsNullOrWhiteSpace(pipelineSelector)) + { + throw new ArgumentException( + "Pipeline selector must not be null, empty, or whitespace.", + nameof(pipelineSelector)); + } + + string trimmed = pipelineSelector.Trim(); + + bool isNodeId = NodeId.TryParse(trimmed, out NodeId parsed) && + !parsed.IsNull; + + var candidates = new List(); + var all = new List(); + + await foreach (VisionNodeEntry entry in EnumeratePipelinesAsync( + cancellationToken).ConfigureAwait(false)) + { + all.Add(FormatPipelineEntry(entry)); + + if (isNodeId && entry.NodeId == parsed) + { + return entry; + } + + string? browseName = entry.BrowseName.Name?.Trim(); + string? displayName = entry.DisplayName.Text?.Trim(); + + if (string.Equals(trimmed, browseName, StringComparison.Ordinal) || + string.Equals(trimmed, displayName, StringComparison.Ordinal)) + { + candidates.Add(entry); + } + } + + if (candidates.Count == 1) + { + return candidates[0]; + } + + if (candidates.Count > 1) + { + var names = new List(candidates.Count); + for (int i = 0; i < candidates.Count; i++) + { + names.Add(FormatPipelineEntry(candidates[i])); + } + + throw new InvalidOperationException( + $"Ambiguous pipeline name '{trimmed}': {string.Join(", ", names)}."); + } + + string available = all.Count > 0 + ? string.Join(", ", all) + : "(none)"; + throw new InvalidOperationException( + $"Pipeline '{trimmed}' not found. Available: {available}."); + } + + internal VisionClientOperations Operations { get; } + + private static string FormatPipelineEntry(VisionNodeEntry entry) + { + return $"BrowseName='{entry.BrowseName.Name}', " + + $"DisplayName='{entry.DisplayName.Text}', NodeId='{entry.NodeId}'"; + } + + private async ValueTask ResolveOptionalRootChildAsync( + string browseName, + CancellationToken cancellationToken) + { + NodeId root = VisionRootId; + if (root.IsNull) + { + return NodeId.Null; + } + return await Operations.ResolveChildAsync( + root, browseName, cancellationToken).ConfigureAwait(false); + } + + private async IAsyncEnumerable EnumerateInstancesAsync( + NodeId root, + uint typeIdentifier, + [EnumeratorCancellation] CancellationToken cancellationToken) + { + if (root.IsNull) + { + yield break; + } + NodeId typeDefinition = Operations.VisionNamespaceType(typeIdentifier); + if (typeDefinition.IsNull) + { + yield break; + } + ArrayOf references = await Operations + .BrowseHierarchicalObjectsAsync(root, cancellationToken).ConfigureAwait(false); + var matches = new List(); + for (int ii = 0; ii < references.Count; ii++) + { + ReferenceDescription reference = references[ii]; + NodeId typeDef = ExpandedNodeId.ToNodeId( + reference.TypeDefinition, Session.NamespaceUris); + NodeId nodeId = ExpandedNodeId.ToNodeId( + reference.NodeId, Session.NamespaceUris); + if (typeDef.IsNull || nodeId.IsNull) + { + continue; + } + if (await Session.NodeCache.IsTypeOfAsync( + typeDef, typeDefinition, cancellationToken).ConfigureAwait(false)) + { + matches.Add(new VisionNodeEntry( + nodeId, reference.BrowseName, reference.DisplayName, typeDef)); + } + } + for (int ii = 0; ii < matches.Count; ii++) + { + yield return matches[ii]; + } + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionClientEventIds.cs b/src/Opc.Ua.Vision.Client/VisionClientEventIds.cs new file mode 100644 index 0000000000..e6a1565f6c --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionClientEventIds.cs @@ -0,0 +1,42 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.Client +{ + internal static class VisionClientEventIds + { + public const int SensorReadFailed = 7300; + public const int FrameChainNotResolvable = 7301; + public const int FrameChainStale = 7302; + public const int InlineClipUnsupported = 7303; + public const int InlineClipOverflow = 7304; + public const int InlineClipUnavailable = 7305; + public const int FeedbackRefused = 7306; + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionClientFactory.cs b/src/Opc.Ua.Vision.Client/VisionClientFactory.cs new file mode 100644 index 0000000000..d2c3cbab7d --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionClientFactory.cs @@ -0,0 +1,78 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client; + +namespace Opc.Ua.Vision.Client +{ + /// + /// Session-bound factory for creating Vision clients through DI. + /// + public sealed class VisionClientFactory + { + private readonly Func> m_sessionFactory; + private readonly ITelemetryContext m_telemetry; + + /// + /// Creates a Vision client factory. + /// + /// + /// The managed-session factory registered by AddClient. + /// + /// + /// The telemetry context used by generated proxies. + /// + public VisionClientFactory( + Func> sessionFactory, + ITelemetryContext telemetry) + { + m_sessionFactory = sessionFactory + ?? throw new ArgumentNullException(nameof(sessionFactory)); + m_telemetry = telemetry + ?? throw new ArgumentNullException(nameof(telemetry)); + } + + /// + /// Creates a Vision client over the current managed session. + /// + /// + /// Cancels session acquisition. + /// + public async Task CreateAsync( + CancellationToken cancellationToken = default) + { + ManagedSession session = await m_sessionFactory(cancellationToken) + .ConfigureAwait(false); + return new VisionClient(session, m_telemetry); + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionClientOperations.cs b/src/Opc.Ua.Vision.Client/VisionClientOperations.cs new file mode 100644 index 0000000000..20452adf8b --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionClientOperations.cs @@ -0,0 +1,426 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client; + +namespace Opc.Ua.Vision.Client +{ + /// + /// Shared low-level plumbing used by every high-level Vision client: + /// namespace-index resolution, subtype-aware browsing, browse-path + /// resolution, and typed value reads over the connected session. + /// + internal sealed class VisionClientOperations + { + public VisionClientOperations(ISession session, ITelemetryContext telemetry) + { + Session = session ?? throw new ArgumentNullException(nameof(session)); + Telemetry = telemetry ?? throw new ArgumentNullException(nameof(telemetry)); + RegisterEncodeableTypes(session); + } + + public ISession Session { get; } + + public ITelemetryContext Telemetry { get; } + + public bool TryGetVisionNamespaceIndex(out ushort namespaceIndex) + { + int index = Session.NamespaceUris.GetIndex(Namespaces.Vision); + if (index < 0) + { + namespaceIndex = 0; + return false; + } + namespaceIndex = (ushort)index; + return true; + } + + public NodeId VisionNamespaceType(uint identifier) + { + return TryGetVisionNamespaceIndex(out ushort ns) + ? new NodeId(identifier, ns) + : NodeId.Null; + } + + public NodeId VisionReference(uint identifier) + { + return VisionNamespaceType(identifier); + } + + public async ValueTask> BrowseAsync( + NodeId nodeId, + NodeId referenceTypeId, + BrowseDirection direction, + uint nodeClassMask, + CancellationToken cancellationToken) + { + (ArrayOf> results, ArrayOf errors) = + await Session.ManagedBrowseAsync( + requestHeader: null, + view: null, + nodesToBrowse: [nodeId], + maxResultsToReturn: 0, + browseDirection: direction, + referenceTypeId: referenceTypeId, + includeSubtypes: true, + nodeClassMask: nodeClassMask, + ct: cancellationToken).ConfigureAwait(false); + if (errors.Count > 0 && ServiceResult.IsBad(errors[0])) + { + return ArrayOf.Empty; + } + return results.Count > 0 ? results[0] : ArrayOf.Empty; + } + + public ValueTask> BrowseHierarchicalObjectsAsync( + NodeId nodeId, CancellationToken cancellationToken) + { + return BrowseAsync( + nodeId, + Opc.Ua.ReferenceTypeIds.HierarchicalReferences, + BrowseDirection.Forward, + (uint)NodeClass.Object, + cancellationToken); + } + + public async ValueTask> DiscoverInstancesAsync( + NodeId root, NodeId typeDefinition, CancellationToken cancellationToken) + { + if (typeDefinition.IsNull) + { + return ArrayOf.Empty; + } + ArrayOf references = await BrowseHierarchicalObjectsAsync( + root, cancellationToken).ConfigureAwait(false); + var matches = new List(); + for (int ii = 0; ii < references.Count; ii++) + { + ReferenceDescription reference = references[ii]; + NodeId typeDef = ExpandedNodeId.ToNodeId( + reference.TypeDefinition, Session.NamespaceUris); + NodeId child = ExpandedNodeId.ToNodeId( + reference.NodeId, Session.NamespaceUris); + if (typeDef.IsNull || child.IsNull) + { + continue; + } + if (await Session.NodeCache.IsTypeOfAsync( + typeDef, typeDefinition, cancellationToken).ConfigureAwait(false)) + { + matches.Add(child); + } + } + return matches.ToArrayOf(); + } + + public async ValueTask> BrowseChildNodeIdsAsync( + NodeId parent, CancellationToken cancellationToken) + { + ArrayOf references = await BrowseHierarchicalObjectsAsync( + parent, cancellationToken).ConfigureAwait(false); + var children = new List(references.Count); + for (int ii = 0; ii < references.Count; ii++) + { + NodeId nodeId = ExpandedNodeId.ToNodeId( + references[ii].NodeId, Session.NamespaceUris); + if (!nodeId.IsNull) + { + children.Add(nodeId); + } + } + return children.ToArrayOf(); + } + + public async ValueTask> ResolveChildrenAsync( + NodeId parent, + ArrayOf browseNames, + ushort namespaceIndex, + CancellationToken cancellationToken) + { + var paths = new List(browseNames.Count); + for (int ii = 0; ii < browseNames.Count; ii++) + { + paths.Add(CreateBrowsePath(parent, browseNames[ii], namespaceIndex)); + } + TranslateBrowsePathsToNodeIdsResponse response = await Session + .TranslateBrowsePathsToNodeIdsAsync( + null, paths.ToArrayOf(), cancellationToken).ConfigureAwait(false); + var results = new List(browseNames.Count); + for (int ii = 0; ii < response.Results.Count; ii++) + { + BrowsePathResult result = response.Results[ii]; + results.Add(StatusCode.IsGood(result.StatusCode) && result.Targets.Count > 0 + ? ExpandedNodeId.ToNodeId(result.Targets[0].TargetId, Session.NamespaceUris) + : NodeId.Null); + } + while (results.Count < browseNames.Count) + { + results.Add(NodeId.Null); + } + return results.ToArrayOf(); + } + + public async ValueTask ResolveChildAsync( + NodeId parent, + string browseName, + ushort namespaceIndex, + CancellationToken cancellationToken) + { + ArrayOf nodes = await ResolveChildrenAsync( + parent, [browseName], namespaceIndex, cancellationToken).ConfigureAwait(false); + return nodes.Count > 0 ? nodes[0] : NodeId.Null; + } + + public async ValueTask ResolveChildAsync( + NodeId parent, + string browseName, + CancellationToken cancellationToken) + { + if (!TryGetVisionNamespaceIndex(out ushort ns)) + { + return NodeId.Null; + } + return await ResolveChildAsync(parent, browseName, ns, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask> ResolveChildrenAsync( + NodeId parent, + ArrayOf browseNames, + CancellationToken cancellationToken) + { + if (!TryGetVisionNamespaceIndex(out ushort ns)) + { + var nulls = new List(browseNames.Count); + for (int ii = 0; ii < browseNames.Count; ii++) + { + nulls.Add(NodeId.Null); + } + return nulls.ToArrayOf(); + } + return await ResolveChildrenAsync(parent, browseNames, ns, cancellationToken) + .ConfigureAwait(false); + } + + public async ValueTask ReadValueAsync( + NodeId nodeId, CancellationToken cancellationToken) + { + if (nodeId.IsNull) + { + return DataValue.Null; + } + return await Session.ReadValueAsync(nodeId, cancellationToken).ConfigureAwait(false); + } + + public async ValueTask> ReadValuesAsync( + ArrayOf nodeIds, CancellationToken cancellationToken) + { + var reads = new List(nodeIds.Count); + for (int ii = 0; ii < nodeIds.Count; ii++) + { + if (nodeIds[ii].IsNull) + { + continue; + } + reads.Add(new ReadValueId + { + NodeId = nodeIds[ii], + AttributeId = Attributes.Value + }); + } + if (reads.Count == 0) + { + return ArrayOf.Empty; + } + ArrayOf nodesToRead = reads.ToArrayOf(); + ReadResponse response = await Session.ReadAsync( + null, 0, TimestampsToReturn.Both, nodesToRead, cancellationToken) + .ConfigureAwait(false); + ClientBase.ValidateResponse(response.Results, nodesToRead); + return response.Results; + } + + public async ValueTask ReadStructureAsync( + NodeId nodeId, CancellationToken cancellationToken) + where T : class, IEncodeable + { + DataValue value = await ReadValueAsync(nodeId, cancellationToken).ConfigureAwait(false); + if (StatusCode.IsBad(value.StatusCode)) + { + throw new ServiceResultException(value.StatusCode); + } +#pragma warning disable CS8600 // TryGetValue uses [MaybeNullWhen(false)] on encodeable overloads. + if (!value.WrappedValue.TryGetValue( + out T structure, Session.MessageContext)) +#pragma warning restore CS8600 + { + throw new ServiceResultException( + StatusCodes.BadTypeMismatch, + $"Node '{nodeId}' does not contain a {typeof(T).Name} value."); + } + return structure; + } + + public async ValueTask TryReadStructureAsync( + NodeId nodeId, CancellationToken cancellationToken) + where T : class, IEncodeable + { + if (nodeId.IsNull) + { + return null; + } + DataValue value = await ReadValueAsync(nodeId, cancellationToken).ConfigureAwait(false); + if (StatusCode.IsBad(value.StatusCode)) + { + return null; + } +#pragma warning disable CS8600 // TryGetValue uses [MaybeNullWhen(false)] on encodeable overloads. + return value.WrappedValue.TryGetValue( + out T structure, Session.MessageContext) + ? structure + : null; +#pragma warning restore CS8600 + } + + public async ValueTask> ReadStructureArrayAsync( + NodeId nodeId, CancellationToken cancellationToken) + where T : class, IEncodeable + { + DataValue value = await ReadValueAsync(nodeId, cancellationToken).ConfigureAwait(false); + if (StatusCode.IsBad(value.StatusCode)) + { + throw new ServiceResultException(value.StatusCode); + } + if (!value.WrappedValue.TryGetValue( + out ArrayOf array, Session.MessageContext)) + { + throw new ServiceResultException( + StatusCodes.BadTypeMismatch, + $"Node '{nodeId}' does not contain an array of {typeof(T).Name} values."); + } + return array; + } + + public async ValueTask> TryReadStructureArrayAsync( + NodeId nodeId, CancellationToken cancellationToken) + where T : class, IEncodeable + { + if (nodeId.IsNull) + { + return ArrayOf.Empty; + } + DataValue value = await ReadValueAsync(nodeId, cancellationToken).ConfigureAwait(false); + if (StatusCode.IsBad(value.StatusCode)) + { + return ArrayOf.Empty; + } + return value.WrappedValue.TryGetValue( + out ArrayOf array, Session.MessageContext) + ? array + : ArrayOf.Empty; + } + + public static string? ReadString(DataValue value) + { + return value.WrappedValue.TryGetValue(out string? text) ? text : null; + } + + public static bool TryReadEnum(DataValue value, out TEnum result) + where TEnum : struct, Enum + { + if (value.WrappedValue.TryGetValue(out int intValue)) + { + result = (TEnum)Enum.ToObject(typeof(TEnum), intValue); + return true; + } + if (value.WrappedValue.TryGetValue(out uint uintValue)) + { + result = (TEnum)Enum.ToObject(typeof(TEnum), uintValue); + return true; + } + result = default; + return false; + } + + public static bool TryReadNodeId(DataValue value, out NodeId nodeId) + { + if (value.WrappedValue.TryGetValue(out NodeId candidate)) + { + nodeId = candidate; + return !nodeId.IsNull; + } + nodeId = NodeId.Null; + return false; + } + + private static BrowsePath CreateBrowsePath( + NodeId parent, string browseName, ushort namespaceIndex) + { + return new BrowsePath + { + StartingNode = parent, + RelativePath = new RelativePath + { + Elements = + [ + new RelativePathElement + { + ReferenceTypeId = Opc.Ua.ReferenceTypeIds.HierarchicalReferences, + IsInverse = false, + IncludeSubtypes = true, + TargetName = new QualifiedName(browseName, namespaceIndex) + } + ] + } + }; + } + + private static void RegisterEncodeableTypes(ISession session) + { + RegisterEncodeableTypes(session.Factory); + if (!ReferenceEquals(session.MessageContext.Factory, session.Factory)) + { + RegisterEncodeableTypes(session.MessageContext.Factory); + } + } + + private static void RegisterEncodeableTypes(IEncodeableFactory factory) + { + var probe = new VisionPose3DDataType(); + if (!factory.TryGetEncodeableType(probe.BinaryEncodingId, out _)) + { + factory.Builder.AddOpcUaVision().Commit(); + } + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionFeedbackClient.cs b/src/Opc.Ua.Vision.Client/VisionFeedbackClient.cs new file mode 100644 index 0000000000..37f572eb47 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionFeedbackClient.cs @@ -0,0 +1,284 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.Vision.Client +{ + /// + /// Focused client over a single VisionFeedbackType instance — typically + /// the Feedback object of a pipeline. Wraps SubmitDetections, + /// SubmitInspectionResult, SubmitCorrection and + /// SubmitImageReference so an off-Server model can publish results the + /// Server did not compute (§9). + /// + /// + /// §9 requires a Server that does not permit a purpose (typically + /// GroundTruthLabel) to refuse the call with Bad_NotSupported. + /// This client surfaces refusals as rather + /// than swallowing them. + /// + public sealed class VisionFeedbackClient + { + private readonly VisionFeedbackTypeClient m_proxy; + + internal VisionFeedbackClient( + VisionClientOperations operations, NodeId feedbackNodeId) + { + if (operations is null) + { + throw new ArgumentNullException(nameof(operations)); + } + if (feedbackNodeId.IsNull) + { + throw new ArgumentException( + "Feedback NodeId must not be null.", nameof(feedbackNodeId)); + } + FeedbackNodeId = feedbackNodeId; + m_proxy = new VisionFeedbackTypeClient( + operations.Session, feedbackNodeId, operations.Telemetry); + } + + /// + /// Gets the feedback object NodeId. + /// + public NodeId FeedbackNodeId { get; } + + /// + /// Submits a set of detections for the given purpose (§9.2). + /// + /// + /// The purpose the detections are submitted for — for example + /// InferredResult from an off-Server model or + /// GroundTruthLabel to feed a learning job. + /// + /// + /// The detections to publish. Non-empty unless + /// says the frame was examined and found + /// to contain nothing. + /// + /// + /// The image the detections apply to, or null when the Server does + /// not require a frame reference. + /// + /// + /// The optional inline image bytes; must fit + /// MaxInlineFeedbackImageSize. Pass an empty + /// to omit inline delivery. + /// + /// + /// Asserts that the frame was examined and contains nothing, which is what + /// makes an empty a real observation rather + /// than a lost payload (§9.5). It is the terminating condition of a + /// bin-picking task and a valid negative training label. + /// + /// + /// Cancels the operation. + /// + /// + public Task SubmitDetectionsAsync( + VisionFeedbackPurposeEnum purpose, + ArrayOf detections, + VisionImageReferenceDataType? frameReference, + ByteString inlineImage, + bool sceneIsEmpty = false, + CancellationToken cancellationToken = default) + { + if (detections.Count == 0 != sceneIsEmpty) + { + // Part 9.5 pairs the two: an empty array is a deliberate observation + // only when the flag says so, and the flag with detections attached + // asserts two contradictory things about one frame. Refusing here + // rather than at the Server names which of the two is wrong. + throw new ArgumentException( + sceneIsEmpty + ? "Detections must be empty when sceneIsEmpty is true." + : "Detections must be non-empty unless sceneIsEmpty is true.", + nameof(detections)); + } + return m_proxy.SubmitDetectionsAsync( + purpose, + detections, + frameReference ?? new VisionImageReferenceDataType(), + inlineImage, + sceneIsEmpty, + cancellationToken).AsTask(); + } + + /// + /// Submits a completed inspection result (§9.3). + /// + /// + /// The stable identifier of the inspection. + /// + /// + /// The overall evaluation. + /// + /// + /// The measured characteristics; must be non-empty. + /// + /// + /// Cancels the operation. + /// + /// + public Task SubmitInspectionResultAsync( + string resultId, + VisionResultEvaluationEnum evaluation, + ArrayOf characteristics, + CancellationToken cancellationToken = default) + { + if (string.IsNullOrEmpty(resultId)) + { + throw new ArgumentException( + "ResultId must be non-empty.", nameof(resultId)); + } + if (characteristics.Count == 0) + { + throw new ArgumentException( + "At least one characteristic must be supplied.", + nameof(characteristics)); + } + return m_proxy.SubmitInspectionResultAsync( + resultId, evaluation, characteristics, cancellationToken).AsTask(); + } + + /// + /// Submits a correction against an existing result identified by + /// (§9.4). §9.4.1 requires that exactly one of + /// or + /// is non-empty. + /// + /// + /// The stable identifier of the result being corrected. + /// + /// + /// The purpose the correction is submitted for. + /// + /// + /// The corrected detections for a DetectionResultType. + /// + /// + /// The corrected characteristics for an InspectionResultType. + /// + /// + /// A human-readable explanation. + /// + /// + /// The optional inline image bytes; must fit + /// MaxInlineFeedbackImageSize. + /// + /// + /// Asserts that the referenced result should contain nothing at all — the + /// false-positive retraction (§9.5). Both corrected arrays must be empty + /// when this is set, and it is the only way to correct a result down to + /// nothing. + /// + /// + /// Cancels the operation. + /// + /// + public Task SubmitCorrectionAsync( + string resultId, + VisionFeedbackPurposeEnum purpose, + ArrayOf correctedDetections, + ArrayOf correctedCharacteristics, + LocalizedText reason, + ByteString inlineImage, + bool retractAll = false, + CancellationToken cancellationToken = default) + { + if (string.IsNullOrEmpty(resultId)) + { + throw new ArgumentException( + "ResultId must be non-empty.", nameof(resultId)); + } + bool hasDetections = correctedDetections.Count > 0; + bool hasCharacteristics = correctedCharacteristics.Count > 0; + if (retractAll) + { + if (hasDetections || hasCharacteristics) + { + throw new ArgumentException( + "Both corrected arrays must be empty when retractAll is true.", + nameof(correctedDetections)); + } + } + else if (hasDetections == hasCharacteristics) + { + // Part 9.5 asks for AT MOST one non-empty. Both is contradictory; + // neither is only meaningful with retractAll. + throw new ArgumentException( + "At most one of correctedDetections and correctedCharacteristics " + + "may be non-empty, and one must be unless retractAll is true.", + nameof(correctedDetections)); + } + return m_proxy.SubmitCorrectionAsync( + resultId, + purpose, + correctedDetections, + correctedCharacteristics, + reason.IsNull ? LocalizedText.Null : reason, + inlineImage, + retractAll, + cancellationToken).AsTask(); + } + + /// + /// Submits an image reference for the given purpose (§9.5). + /// + /// + /// The purpose the image is submitted for. + /// + /// + /// The image descriptor. Must be non-null. + /// + /// + /// The stable identifier of the target result, or an empty string. + /// + /// + /// Cancels the operation. + /// + /// + public Task SubmitImageReferenceAsync( + VisionFeedbackPurposeEnum purpose, + VisionImageReferenceDataType image, + string resultId, + CancellationToken cancellationToken = default) + { + if (image is null) + { + throw new ArgumentNullException(nameof(image)); + } + return m_proxy.SubmitImageReferenceAsync( + purpose, image, resultId ?? string.Empty, cancellationToken).AsTask(); + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionFrameGraph.cs b/src/Opc.Ua.Vision.Client/VisionFrameGraph.cs new file mode 100644 index 0000000000..931a1d157d --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionFrameGraph.cs @@ -0,0 +1,453 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.Vision.Client +{ + /// + /// Traverses the Server's CoordinateFrameType tree and composes transforms + /// between two named frames. Uses the exact conventions of §5.12: right-handed + /// frames, orientation as a unit quaternion ordered (x, y, z, w), position in + /// metres, tolerance 1e-6 for the unit-norm check. + /// + /// + /// The graph is intentionally a facade with no state: every operation re-reads + /// the relevant frames, so subsequent calls always see the Server's current + /// values. A pose is composed by walking from the source frame up to a common + /// ancestor, then down to the target, multiplying transforms in order. + /// + public sealed class VisionFrameGraph + { + private const int MaxChainDepth = 32; + + private const double UnitQuaternionTolerance = 1e-6; + + private readonly VisionClientOperations m_operations; + + internal VisionFrameGraph(VisionClientOperations operations) + { + m_operations = operations ?? throw new ArgumentNullException(nameof(operations)); + } + + /// + /// Reads a single frame snapshot from its NodeId. + /// + /// + /// The CoordinateFrameType instance NodeId. + /// + /// + /// Cancels the operation. + /// + /// + public async Task ReadAsync( + NodeId frameNodeId, + CancellationToken cancellationToken = default) + { + if (frameNodeId.IsNull) + { + throw new ArgumentException( + "Frame NodeId must not be null.", nameof(frameNodeId)); + } + string[] members = + [ + BrowseNames.FrameId, + BrowseNames.Role, + BrowseNames.ParentFrame, + BrowseNames.Transform + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + frameNodeId, members, cancellationToken).ConfigureAwait(false); + var toRead = new List(); + for (int ii = 0; ii < nodes.Count; ii++) + { + if (!nodes[ii].IsNull) + { + toRead.Add(nodes[ii]); + } + } + ArrayOf values = await m_operations.ReadValuesAsync( + toRead, cancellationToken).ConfigureAwait(false); + int cursor = 0; + string? frameId = null; + if (!nodes[0].IsNull) + { + frameId = VisionClientOperations.ReadString(values[cursor++]); + } + VisionFrameRoleEnum role = default; + if (!nodes[1].IsNull) + { + VisionClientOperations.TryReadEnum(values[cursor++], out role); + } + NodeId parent = NodeId.Null; + if (!nodes[2].IsNull) + { + VisionClientOperations.TryReadNodeId(values[cursor++], out parent); + } + VisionPose3DDataType? transform = null; + if (!nodes[3].IsNull) + { + DataValue value = values[cursor++]; +#pragma warning disable CS8600 // TryGetValue uses [MaybeNullWhen(false)] on encodeable overloads. + if (value.WrappedValue.TryGetValue( + out VisionPose3DDataType structure, + m_operations.Session.MessageContext)) + { + transform = structure; + } +#pragma warning restore CS8600 + } + return new VisionFrameSnapshot + { + NodeId = frameNodeId, + FrameId = frameId, + Role = role, + ParentFrameId = parent, + Transform = transform + }; + } + + /// + /// Composes the pose , expressed in + /// , into , walking + /// the ParentFrame chain per §5.12. + /// + /// + /// The pose to compose. + /// + /// + /// The frame the pose is currently expressed in. + /// + /// + /// The frame the pose should be expressed in. + /// + /// + /// Cancels the operation. + /// + /// + /// Any argument is null. + /// + /// + /// The frame chain cannot be resolved, is longer than 32, contains a + /// cycle, or carries a non-unit quaternion within tolerance 1e-6. + /// + /// + public async Task ComposeAsync( + VisionPose3DDataType pose, + NodeId fromFrameId, + NodeId toFrameId, + CancellationToken cancellationToken = default) + { + if (pose is null) + { + throw new ArgumentNullException(nameof(pose)); + } + if (fromFrameId.IsNull) + { + throw new ArgumentException( + "From-frame NodeId must not be null.", nameof(fromFrameId)); + } + if (toFrameId.IsNull) + { + throw new ArgumentException( + "To-frame NodeId must not be null.", nameof(toFrameId)); + } + VisionPose3DDataType transform = await ComposeTransformAsync( + fromFrameId, toFrameId, cancellationToken).ConfigureAwait(false); + return Compose(transform, pose, targetFrameId: null); + } + + /// + /// Composes the identity of frame into + /// — equivalent to a ComposeAsync call + /// with an origin pose. + /// + /// + /// The source frame NodeId. + /// + /// + /// The target frame NodeId. + /// + /// + /// Cancels the operation. + /// + /// + public Task ComposeTransformAsync( + NodeId fromFrameId, + NodeId toFrameId, + CancellationToken cancellationToken = default) + { + if (fromFrameId.IsNull) + { + throw new ArgumentException( + "From-frame NodeId must not be null.", nameof(fromFrameId)); + } + if (toFrameId.IsNull) + { + throw new ArgumentException( + "To-frame NodeId must not be null.", nameof(toFrameId)); + } + return ComposeTransformCoreAsync(fromFrameId, toFrameId, cancellationToken); + } + + private async Task ComposeTransformCoreAsync( + NodeId fromFrameId, + NodeId toFrameId, + CancellationToken cancellationToken) + { + var fromChain = await WalkToRootAsync(fromFrameId, cancellationToken) + .ConfigureAwait(false); + var toChain = await WalkToRootAsync(toFrameId, cancellationToken) + .ConfigureAwait(false); + (int fromIndex, int toIndex) = FindCommonAncestor(fromChain, toChain); + if (fromIndex < 0 || toIndex < 0) + { + throw ServiceResultException.Create( + StatusCodes.BadNoMatch, + "Frames '{0}' and '{1}' do not share a common ancestor.", + fromFrameId, + toFrameId); + } + VisionPose3DDataType up = Identity(FrameIdOf(fromChain, 0)); + for (int ii = 0; ii < fromIndex; ii++) + { + VisionFrameSnapshot frame = fromChain[ii]; + RequireTransform(frame); + up = Compose(frame.Transform!, up, FrameIdOf(fromChain, ii + 1)); + } + VisionPose3DDataType downInverse = Identity(FrameIdOf(toChain, toIndex)); + for (int ii = 0; ii < toIndex; ii++) + { + VisionFrameSnapshot frame = toChain[ii]; + RequireTransform(frame); + downInverse = Compose( + frame.Transform!, + downInverse, + FrameIdOf(toChain, ii + 1)); + } + VisionPose3DDataType down = Invert(downInverse, FrameIdOf(toChain, 0)); + return Compose(down, up, FrameIdOf(toChain, 0)); + } + + private async Task> WalkToRootAsync( + NodeId startFrameId, + CancellationToken cancellationToken) + { + var chain = new List(4); + var visited = new HashSet(); + NodeId current = startFrameId; + for (int depth = 0; depth < MaxChainDepth; depth++) + { + if (!visited.Add(current)) + { + throw ServiceResultException.Create( + StatusCodes.BadInvalidArgument, + "Frame chain starting at '{0}' contains a cycle.", + startFrameId); + } + VisionFrameSnapshot snapshot = await ReadAsync(current, cancellationToken) + .ConfigureAwait(false); + chain.Add(snapshot); + if (snapshot.ParentFrameId.IsNull) + { + return chain; + } + current = snapshot.ParentFrameId; + } + throw ServiceResultException.Create( + StatusCodes.BadInvalidArgument, + "Frame chain starting at '{0}' exceeds the {1}-frame limit.", + startFrameId, + MaxChainDepth); + } + + private static (int FromIndex, int ToIndex) FindCommonAncestor( + List fromChain, + List toChain) + { + for (int ii = 0; ii < fromChain.Count; ii++) + { + for (int jj = 0; jj < toChain.Count; jj++) + { + if (fromChain[ii].NodeId == toChain[jj].NodeId) + { + return (ii, jj); + } + } + } + return (-1, -1); + } + + private static string? FrameIdOf(List chain, int index) + { + if (index < 0 || index >= chain.Count) + { + return null; + } + return chain[index].FrameId; + } + + private static void RequireTransform(VisionFrameSnapshot frame) + { + if (frame.Transform is null) + { + throw ServiceResultException.Create( + StatusCodes.BadNotFound, + "Frame '{0}' does not report a Transform to its parent.", + frame.NodeId); + } + } + + private static VisionPose3DDataType Identity(string? frameId) + { + return new VisionPose3DDataType + { + FrameId = frameId ?? string.Empty, + Position = [0.0, 0.0, 0.0], + Orientation = [0.0, 0.0, 0.0, 1.0], + Covariance = ArrayOf.Empty + }; + } + + private static VisionPose3DDataType Compose( + VisionPose3DDataType outer, + VisionPose3DDataType inner, + string? targetFrameId) + { + (double outerX, double outerY, double outerZ) = ReadPosition(outer); + (double outerQx, double outerQy, double outerQz, double outerQw) = + ReadOrientation(outer); + (double innerX, double innerY, double innerZ) = ReadPosition(inner); + (double innerQx, double innerQy, double innerQz, double innerQw) = + ReadOrientation(inner); + + (double rx, double ry, double rz) = RotateVector( + outerQx, outerQy, outerQz, outerQw, innerX, innerY, innerZ); + + (double qx, double qy, double qz, double qw) = MultiplyQuaternions( + outerQx, outerQy, outerQz, outerQw, + innerQx, innerQy, innerQz, innerQw); + + return new VisionPose3DDataType + { + FrameId = targetFrameId ?? inner.FrameId ?? string.Empty, + Position = [outerX + rx, outerY + ry, outerZ + rz], + Orientation = [qx, qy, qz, qw], + Covariance = ArrayOf.Empty + }; + } + + private static VisionPose3DDataType Invert( + VisionPose3DDataType pose, + string? targetFrameId) + { + (double x, double y, double z) = ReadPosition(pose); + (double qx, double qy, double qz, double qw) = ReadOrientation(pose); + (double invQx, double invQy, double invQz, double invQw) = + (-qx, -qy, -qz, qw); + (double rx, double ry, double rz) = RotateVector( + invQx, invQy, invQz, invQw, -x, -y, -z); + return new VisionPose3DDataType + { + FrameId = targetFrameId ?? string.Empty, + Position = [rx, ry, rz], + Orientation = [invQx, invQy, invQz, invQw], + Covariance = ArrayOf.Empty + }; + } + + private static (double X, double Y, double Z) ReadPosition( + VisionPose3DDataType pose) + { + ArrayOf position = pose.Position; + if (position.Count < 3) + { + throw ServiceResultException.Create( + StatusCodes.BadOutOfRange, + "Pose has {0} position components, expected 3.", + position.Count); + } + return (position[0], position[1], position[2]); + } + + private static (double X, double Y, double Z, double W) ReadOrientation( + VisionPose3DDataType pose) + { + ArrayOf q = pose.Orientation; + if (q.Count != 4) + { + throw ServiceResultException.Create( + StatusCodes.BadOutOfRange, + "Pose orientation has {0} components, expected 4 (x, y, z, w).", + q.Count); + } + double qx = q[0]; + double qy = q[1]; + double qz = q[2]; + double qw = q[3]; + double norm = Math.Sqrt((qx * qx) + (qy * qy) + (qz * qz) + (qw * qw)); + if (Math.Abs(norm - 1.0) > UnitQuaternionTolerance) + { + throw ServiceResultException.Create( + StatusCodes.BadOutOfRange, + "Pose orientation quaternion norm '{0}' is outside the 1e-6 " + + "unit-norm tolerance.", + norm); + } + return (qx, qy, qz, qw); + } + + private static (double X, double Y, double Z, double W) MultiplyQuaternions( + double ax, double ay, double az, double aw, + double bx, double by, double bz, double bw) + { + double w = (aw * bw) - (ax * bx) - (ay * by) - (az * bz); + double x = (aw * bx) + (ax * bw) + (ay * bz) - (az * by); + double y = (aw * by) - (ax * bz) + (ay * bw) + (az * bx); + double z = (aw * bz) + (ax * by) - (ay * bx) + (az * bw); + return (x, y, z, w); + } + + private static (double X, double Y, double Z) RotateVector( + double qx, double qy, double qz, double qw, + double vx, double vy, double vz) + { + double tx = 2.0 * ((qy * vz) - (qz * vy)); + double ty = 2.0 * ((qz * vx) - (qx * vz)); + double tz = 2.0 * ((qx * vy) - (qy * vx)); + double rx = vx + (qw * tx) + ((qy * tz) - (qz * ty)); + double ry = vy + (qw * ty) + ((qz * tx) - (qx * tz)); + double rz = vz + (qw * tz) + ((qx * ty) - (qy * tx)); + return (rx, ry, rz); + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionInferenceService.cs b/src/Opc.Ua.Vision.Client/VisionInferenceService.cs new file mode 100644 index 0000000000..2aa349e2e6 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionInferenceService.cs @@ -0,0 +1,736 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +#if NET8_0_OR_GREATER +using System.Text.Json.Serialization; +#endif + +namespace Opc.Ua.Vision.Client +{ + /// + /// The kind of result a Vision pipeline produced, determined from the type + /// definition of the published result node. + /// +#if NET8_0_OR_GREATER + [JsonConverter(typeof(JsonStringEnumConverter))] +#endif + public enum VisionResultKind + { + /// + /// The result node could not be resolved or its type definition could + /// not be determined. + /// + Unknown = 0, + + /// + /// A DetectionResultType (§7.3) — bounding boxes and poses. + /// + Detection = 1, + + /// + /// An InspectionResultType (§7.2) — pass/fail verdicts. + /// + Inspection = 2, + + /// + /// A SegmentationResultType (§7.4) — per-pixel labels. + /// + Segmentation = 3 + } + + /// + /// The kind of result expected from a Vision inference request. + /// +#if NET8_0_OR_GREATER + [JsonConverter(typeof(JsonStringEnumConverter))] +#endif + public enum VisionExpectedResultKind + { + /// + /// Accept any result kind. + /// + Auto = 0, + + /// + /// Require a DetectionResultType result. + /// + Detection = 1, + + /// + /// Require an InspectionResultType result. + /// + Inspection = 2, + + /// + /// Require a SegmentationResultType result. + /// + Segmentation = 3 + } + + /// + /// Controls how much detail the inference summary contains. + /// +#if NET8_0_OR_GREATER + [JsonConverter(typeof(JsonStringEnumConverter))] +#endif + public enum VisionResultDetail + { + /// + /// Return a concise typed summary including bounded items. + /// + Summary = 0, + + /// + /// Return only the handle (resultId, resultNodeId, resolved, kind) with + /// no payload read. + /// + HandleOnly = 1 + } + + /// + /// Concise detection summary returned when the result is a + /// DetectionResultType. + /// + public sealed record VisionDetectionSummary + { + /// + /// Result creation time. + /// + public DateTimeUtc CreationTime { get; init; } + + /// + /// Model version used, when reported. + /// + public string? ModelVersionUsed { get; init; } + + /// + /// Frame identifier the poses are expressed in. + /// + public string? FrameId { get; init; } + + /// + /// Total number of detections in the result. + /// + public int TotalDetections { get; init; } + + /// + /// Bounded subset of detection items. + /// + public ArrayOf Items { get; init; } + } + + /// + /// One detection in a concise summary. + /// + public sealed record VisionDetectionItem + { + /// + /// Detection identifier. + /// + public string DetectionId { get; init; } = string.Empty; + + /// + /// Class label. + /// + public string ClassLabel { get; init; } = string.Empty; + + /// + /// Class numeric identifier. + /// + public uint ClassId { get; init; } + + /// + /// Confidence score. + /// + public double Confidence { get; init; } + + /// + /// Whether the detection has a 3-D pose. + /// + public bool HasPose { get; init; } + + /// + /// Concise pose (position + orientation) when available and small. + /// + public VisionPose3DDataType? Pose { get; init; } + } + + /// + /// Concise inspection summary returned when the result is an + /// InspectionResultType. + /// + public sealed record VisionInspectionSummary + { + /// + /// Result creation time. + /// + public DateTimeUtc CreationTime { get; init; } + + /// + /// Overall evaluation. + /// + public VisionResultEvaluationEnum Evaluation { get; init; } + + /// + /// Part identifier, when reported. + /// + public string? PartId { get; init; } + + /// + /// Recipe identifier, when reported. + /// + public string? RecipeId { get; init; } + + /// + /// Total number of characteristics. + /// + public int TotalCharacteristics { get; init; } + + /// + /// Bounded subset of characteristic items. + /// + public ArrayOf Items { get; init; } + } + + /// + /// One characteristic in a concise inspection summary. + /// + public sealed record VisionCharacteristicItem + { + /// + /// Characteristic name. + /// + public string Name { get; init; } = string.Empty; + + /// + /// Tolerance status. + /// + public VisionToleranceStatusEnum Status { get; init; } + + /// + /// Deviation from nominal. + /// + public double Deviation { get; init; } + } + + /// + /// Concise segmentation summary returned when the result is a + /// SegmentationResultType. + /// + public sealed record VisionSegmentationSummary + { + /// + /// Result creation time. + /// + public DateTimeUtc CreationTime { get; init; } + + /// + /// Label class names. + /// + public ArrayOf LabelClasses { get; init; } + + /// + /// Mask image width, when reported. + /// + public uint MaskWidth { get; init; } + + /// + /// Mask image height, when reported. + /// + public uint MaskHeight { get; init; } + + /// + /// Mask image format name, when reported. + /// + public string? MaskFormat { get; init; } + } + + /// + /// The complete result of a one-shot inference execution with optional + /// concise summary. Reusable from any consumer — MCP tools, Robotics, + /// or direct application code. + /// + public sealed record VisionInferenceResult + { + /// + /// The ResultId the server assigned. + /// + public required string ResultId { get; init; } + + /// + /// The NodeId the result was published at, or a null NodeId. + /// + public NodeId ResultNodeId { get; init; } = NodeId.Null; + + /// + /// Whether the result node was resolved and is addressable. + /// + public bool Resolved { get; init; } + + /// + /// The Pipeline NodeId requested to run this inference. + /// + public required NodeId RequestedPipelineNodeId { get; init; } + + /// + /// The requested pipeline's published name (BrowseName.Name), when available. + /// + public string? RequestedPipelineName { get; init; } + + /// + /// The Pipeline NodeId published by the result, when available. + /// + public NodeId PipelineId { get; init; } = NodeId.Null; + + /// + /// The sensor NodeId that produced the frame, when available. + /// + public NodeId SensorId { get; init; } = NodeId.Null; + + /// + /// The model version used to compute the result, when reported. + /// + public string? ModelVersionUsed { get; init; } + + /// + /// The result creation time, when available. + /// + public DateTimeUtc CreationTime { get; init; } + + /// + /// The frame identifier detection poses are expressed in, when available. + /// + public string? FrameId { get; init; } + + /// + /// The detected result kind, determined from the type definition. + /// + public VisionResultKind ResultKind { get; init; } + + /// + /// Concise detection summary, populated when + /// is and detail is + /// . + /// + public VisionDetectionSummary? DetectionSummary { get; init; } + + /// + /// Concise inspection summary, populated when + /// is and detail is + /// . + /// + public VisionInspectionSummary? InspectionSummary { get; init; } + + /// + /// Concise segmentation summary, populated when + /// is and detail is + /// . + /// + public VisionSegmentationSummary? SegmentationSummary { get; init; } + } + + /// + /// Reusable service that runs one-shot inference on a pipeline, resolves the + /// published result's type definition, and optionally reads a bounded concise + /// summary. Consumable from MCP tools, Robotics pick-and-place, or direct + /// application code without coupling to any tool framework. + /// + public sealed class VisionInferenceService + { + private readonly VisionClientOperations m_operations; + + internal VisionInferenceService(VisionClientOperations operations) + { + m_operations = operations + ?? throw new ArgumentNullException(nameof(operations)); + } + + /// + /// Runs a one-shot inference, resolves the result, determines its type, + /// and optionally builds a concise summary. + /// + /// + /// The pipeline client to run inference on. + /// + /// + /// Optional pipeline display/browse name for provenance in the result. + /// + /// + /// Whether to read a summary or return handle-only. + /// + /// + /// When set to a concrete kind, the service throws if the result kind + /// cannot be determined or does not match. Use + /// to accept any kind. + /// Unresolved results always return handle-only without enforcement. + /// + /// + /// Maximum number of items (detections/characteristics) in the summary. + /// Must be between 0 and 100 inclusive. + /// + /// + /// Cancels the operation. + /// + /// + /// + public async Task RunOneShotAsync( + VisionPipelineClient pipeline, + string? pipelineName, + VisionResultDetail detail, + VisionExpectedResultKind expectedKind, + int maxItems, + CancellationToken cancellationToken = default) + { + if (pipeline is null) + { + throw new ArgumentNullException(nameof(pipeline)); + } + + ValidateRequest(detail, expectedKind, maxItems); + + string resultId = await pipeline.RunInferenceAsync( + timestamp: default, cancellationToken).ConfigureAwait(false); + NodeId resultNodeId = await pipeline.ResolveResultNodeIdAsync( + resultId, cancellationToken).ConfigureAwait(false); + bool resolved = !resultNodeId.IsNull; + + if (!resolved) + { + return new VisionInferenceResult + { + ResultId = resultId, + ResultNodeId = resultNodeId, + Resolved = false, + RequestedPipelineNodeId = pipeline.PipelineNodeId, + RequestedPipelineName = pipelineName, + ResultKind = VisionResultKind.Unknown + }; + } + + VisionResultKind kind = await DetermineResultKindAsync( + resultNodeId, cancellationToken).ConfigureAwait(false); + + if (expectedKind != VisionExpectedResultKind.Auto && + kind == VisionResultKind.Unknown) + { + throw new InvalidOperationException( + $"Cannot determine result kind for resolved result node '{resultNodeId}' " + + $"while expectedKind is '{expectedKind}'."); + } + + if (expectedKind != VisionExpectedResultKind.Auto && + kind != (VisionResultKind)expectedKind) + { + throw new InvalidOperationException( + $"Expected result kind '{expectedKind}' but the pipeline produced '{kind}'."); + } + + if (detail == VisionResultDetail.HandleOnly) + { + return new VisionInferenceResult + { + ResultId = resultId, + ResultNodeId = resultNodeId, + Resolved = true, + RequestedPipelineNodeId = pipeline.PipelineNodeId, + RequestedPipelineName = pipelineName, + ResultKind = kind + }; + } + + var reader = new VisionResultReader(m_operations, resultNodeId); + return kind switch + { + VisionResultKind.Detection => await BuildDetectionResultAsync( + reader, resultId, resultNodeId, pipeline, pipelineName, maxItems, + cancellationToken).ConfigureAwait(false), + VisionResultKind.Inspection => await BuildInspectionResultAsync( + reader, resultId, resultNodeId, pipeline, pipelineName, maxItems, + cancellationToken).ConfigureAwait(false), + VisionResultKind.Segmentation => await BuildSegmentationResultAsync( + reader, resultId, resultNodeId, pipeline, pipelineName, + cancellationToken).ConfigureAwait(false), + _ => new VisionInferenceResult + { + ResultId = resultId, + ResultNodeId = resultNodeId, + Resolved = true, + RequestedPipelineNodeId = pipeline.PipelineNodeId, + RequestedPipelineName = pipelineName, + ResultKind = kind + } + }; + } + + private static void ValidateRequest( + VisionResultDetail detail, + VisionExpectedResultKind expectedKind, + int maxItems) + { + if (!IsDefined(detail)) + { + throw new ArgumentOutOfRangeException( + nameof(detail), + detail, + "Invalid detail value."); + } + + if (!IsDefined(expectedKind)) + { + throw new ArgumentOutOfRangeException( + nameof(expectedKind), + expectedKind, + "Invalid expectedKind value."); + } + + if (maxItems < 0 || maxItems > 100) + { + throw new ArgumentOutOfRangeException( + nameof(maxItems), + maxItems, + "maxItems must be between 0 and 100 inclusive."); + } + } + + private static bool IsDefined(VisionResultDetail detail) + { +#if NET8_0_OR_GREATER + return Enum.IsDefined(detail); +#else + return Enum.IsDefined(typeof(VisionResultDetail), detail); +#endif + } + + private static bool IsDefined(VisionExpectedResultKind expectedKind) + { +#if NET8_0_OR_GREATER + return Enum.IsDefined(expectedKind); +#else + return Enum.IsDefined(typeof(VisionExpectedResultKind), expectedKind); +#endif + } + + /// + /// Determines the from the type definition + /// of a resolved result node. Handles zero, one, and multiple + /// HasTypeDefinition references deterministically. + /// + /// + public async Task DetermineResultKindAsync( + NodeId resultNodeId, + CancellationToken cancellationToken = default) + { + if (resultNodeId.IsNull) + { + return VisionResultKind.Unknown; + } + + NodeId detectionType = m_operations.VisionNamespaceType( + ObjectTypes.DetectionResultType); + NodeId inspectionType = m_operations.VisionNamespaceType( + ObjectTypes.InspectionResultType); + NodeId segmentationType = m_operations.VisionNamespaceType( + ObjectTypes.SegmentationResultType); + + ArrayOf refs = await m_operations.BrowseAsync( + resultNodeId, + Opc.Ua.ReferenceTypeIds.HasTypeDefinition, + BrowseDirection.Forward, + (uint)NodeClass.ObjectType, + cancellationToken).ConfigureAwait(false); + + if (refs.Count == 0) + { + return VisionResultKind.Unknown; + } + + if (refs.Count > 1) + { + throw new InvalidOperationException( + $"Result node '{resultNodeId}' has {refs.Count} HasTypeDefinition references; " + + $"expected exactly one. First: '{refs[0].NodeId}', second: '{refs[1].NodeId}'."); + } + + NodeId typeDef = ExpandedNodeId.ToNodeId( + refs[0].NodeId, m_operations.Session.NamespaceUris); + if (typeDef.IsNull) + { + return VisionResultKind.Unknown; + } + + if (!detectionType.IsNull && + await m_operations.Session.NodeCache + .IsTypeOfAsync(typeDef, detectionType, cancellationToken) + .ConfigureAwait(false)) + { + return VisionResultKind.Detection; + } + if (!inspectionType.IsNull && + await m_operations.Session.NodeCache + .IsTypeOfAsync(typeDef, inspectionType, cancellationToken) + .ConfigureAwait(false)) + { + return VisionResultKind.Inspection; + } + if (!segmentationType.IsNull && + await m_operations.Session.NodeCache + .IsTypeOfAsync(typeDef, segmentationType, cancellationToken) + .ConfigureAwait(false)) + { + return VisionResultKind.Segmentation; + } + + return VisionResultKind.Unknown; + } + + private static async Task BuildDetectionResultAsync( + VisionResultReader reader, string resultId, NodeId resultNodeId, + VisionPipelineClient pipeline, string? pipelineName, int maxItems, + CancellationToken ct) + { + VisionDetectionResultSnapshot snap = + await reader.ReadDetectionAsync(ct).ConfigureAwait(false); + int total = snap.Detections.Count; + int take = Math.Min(total, maxItems); + var items = new List(take); + for (int i = 0; i < take; i++) + { + VisionDetectionDataType d = snap.Detections[i]; + items.Add(new VisionDetectionItem + { + DetectionId = d.DetectionId ?? string.Empty, + ClassLabel = d.ClassLabel ?? string.Empty, + ClassId = d.ClassId, + Confidence = d.Confidence, + HasPose = d.HasPose, + Pose = d.HasPose ? d.Pose : null + }); + } + return new VisionInferenceResult + { + ResultId = resultId, + ResultNodeId = resultNodeId, + Resolved = true, + RequestedPipelineNodeId = pipeline.PipelineNodeId, + RequestedPipelineName = pipelineName, + PipelineId = snap.PipelineId, + SensorId = snap.SensorId, + ModelVersionUsed = snap.ModelVersionUsed, + CreationTime = snap.CreationTime, + FrameId = snap.FrameId, + ResultKind = VisionResultKind.Detection, + DetectionSummary = new VisionDetectionSummary + { + CreationTime = snap.CreationTime, + ModelVersionUsed = snap.ModelVersionUsed, + FrameId = snap.FrameId, + TotalDetections = total, + Items = items.ToArrayOf() + } + }; + } + + private static async Task BuildInspectionResultAsync( + VisionResultReader reader, string resultId, NodeId resultNodeId, + VisionPipelineClient pipeline, string? pipelineName, int maxItems, + CancellationToken ct) + { + VisionInspectionResultSnapshot snap = + await reader.ReadInspectionAsync(ct).ConfigureAwait(false); + int total = snap.Characteristics.Count; + int take = Math.Min(total, maxItems); + var items = new List(take); + for (int i = 0; i < take; i++) + { + VisionCharacteristicDataType c = snap.Characteristics[i]; + items.Add(new VisionCharacteristicItem + { + Name = c.Name ?? string.Empty, + Status = c.Status, + Deviation = c.Deviation + }); + } + return new VisionInferenceResult + { + ResultId = resultId, + ResultNodeId = resultNodeId, + Resolved = true, + RequestedPipelineNodeId = pipeline.PipelineNodeId, + RequestedPipelineName = pipelineName, + PipelineId = snap.PipelineId, + SensorId = snap.SensorId, + ModelVersionUsed = snap.ModelVersionUsed, + CreationTime = snap.CreationTime, + ResultKind = VisionResultKind.Inspection, + InspectionSummary = new VisionInspectionSummary + { + CreationTime = snap.CreationTime, + Evaluation = snap.Evaluation, + PartId = snap.PartId, + RecipeId = snap.RecipeId, + TotalCharacteristics = total, + Items = items.ToArrayOf() + } + }; + } + + private static async Task BuildSegmentationResultAsync( + VisionResultReader reader, string resultId, NodeId resultNodeId, + VisionPipelineClient pipeline, string? pipelineName, + CancellationToken ct) + { + VisionSegmentationResultSnapshot snap = + await reader.ReadSegmentationAsync(ct).ConfigureAwait(false); + return new VisionInferenceResult + { + ResultId = resultId, + ResultNodeId = resultNodeId, + Resolved = true, + RequestedPipelineNodeId = pipeline.PipelineNodeId, + RequestedPipelineName = pipelineName, + PipelineId = snap.PipelineId, + SensorId = snap.SensorId, + CreationTime = snap.CreationTime, + ResultKind = VisionResultKind.Segmentation, + SegmentationSummary = new VisionSegmentationSummary + { + CreationTime = snap.CreationTime, + LabelClasses = snap.LabelClasses, + MaskWidth = snap.Mask?.Width ?? 0, + MaskHeight = snap.Mask?.Height ?? 0, + MaskFormat = snap.Mask?.Format.ToString() + } + }; + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionMediaClient.cs b/src/Opc.Ua.Vision.Client/VisionMediaClient.cs new file mode 100644 index 0000000000..cd1cec00b3 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionMediaClient.cs @@ -0,0 +1,437 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.Vision.Client +{ + /// + /// Focused client over a single VisionMediaManagementType instance — + /// typically the Media object of a sensor. Wraps GetClip, + /// GetStreamEndpoint, ReleaseStreamEndpoint, + /// ConfigureStreamEndpoint, SelectEndpoint, and read access to + /// LatestClip and LatestClipMetadata, following the §6 media rules + /// (by-reference default, inline gating, §6.4 status-code classification). + /// + public sealed class VisionMediaClient + { + private readonly VisionClientOperations m_operations; + private readonly VisionMediaManagementTypeClient m_proxy; + + internal VisionMediaClient(VisionClientOperations operations, NodeId mediaNodeId) + { + m_operations = operations + ?? throw new ArgumentNullException(nameof(operations)); + if (mediaNodeId.IsNull) + { + throw new ArgumentException( + "Media NodeId must not be null.", nameof(mediaNodeId)); + } + MediaNodeId = mediaNodeId; + m_proxy = new VisionMediaManagementTypeClient( + m_operations.Session, mediaNodeId, m_operations.Telemetry); + } + + /// + /// Gets the media-management object NodeId. + /// + public NodeId MediaNodeId { get; } + + /// + /// Enumerates the clip endpoints attached to this media manager. + /// + /// + /// Cancels the operation. + /// + public async IAsyncEnumerable EnumerateClipEndpointsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + FolderTypeClient? folder = await m_proxy.GetClipEndpointsAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + if (folder is null || folder.ObjectId.IsNull) + { + yield break; + } + ArrayOf refs = await m_operations + .BrowseHierarchicalObjectsAsync(folder.ObjectId, cancellationToken) + .ConfigureAwait(false); + for (int ii = 0; ii < refs.Count; ii++) + { + NodeId nodeId = ExpandedNodeId.ToNodeId( + refs[ii].NodeId, m_operations.Session.NamespaceUris); + NodeId typeDef = ExpandedNodeId.ToNodeId( + refs[ii].TypeDefinition, m_operations.Session.NamespaceUris); + if (!nodeId.IsNull && !typeDef.IsNull) + { + yield return new VisionNodeEntry( + nodeId, refs[ii].BrowseName, refs[ii].DisplayName, typeDef); + } + } + } + + /// + /// Enumerates the stream endpoints attached to this media manager. + /// + /// + /// Cancels the operation. + /// + public async IAsyncEnumerable EnumerateStreamEndpointsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + FolderTypeClient? folder = await m_proxy.GetStreamEndpointsAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + if (folder is null || folder.ObjectId.IsNull) + { + yield break; + } + ArrayOf refs = await m_operations + .BrowseHierarchicalObjectsAsync(folder.ObjectId, cancellationToken) + .ConfigureAwait(false); + for (int ii = 0; ii < refs.Count; ii++) + { + NodeId nodeId = ExpandedNodeId.ToNodeId( + refs[ii].NodeId, m_operations.Session.NamespaceUris); + NodeId typeDef = ExpandedNodeId.ToNodeId( + refs[ii].TypeDefinition, m_operations.Session.NamespaceUris); + if (!nodeId.IsNull && !typeDef.IsNull) + { + yield return new VisionNodeEntry( + nodeId, refs[ii].BrowseName, refs[ii].DisplayName, typeDef); + } + } + } + + /// + /// Calls GetClip and returns the by-reference image descriptor. When + /// is true and the still fits the + /// Server's effective inline limit, + /// carries the encoded bytes. + /// + /// + /// The ClipEndpointType NodeId, or a null NodeId to let the Server + /// apply the §6.3 selection rule (PreferredClipEndpoint first, then + /// the first endpoint in BrowseName order). + /// + /// + /// The ResultId to correlate this clip against, or a null string. + /// + /// + /// The requested acquisition time. + /// + /// + /// The desired encoded format. + /// + /// + /// Request inline delivery on top of the by-reference descriptor. + /// + /// + /// Cancels the operation. + /// + /// + /// The Server refused the call. + /// + public async Task GetClipAsync( + NodeId endpointNodeId, + string? resultId, + DateTimeUtc timestamp, + VisionClipFormatEnum format, + bool requestInline, + CancellationToken cancellationToken = default) + { + (VisionImageReferenceDataType image, NodeId endpointOut, ByteString inlineImage) = + await m_proxy.GetClipAsync( + endpointNodeId, + resultId ?? string.Empty, + timestamp, + format, + requestInline, + cancellationToken).ConfigureAwait(false); + return new VisionClipResult + { + Image = image, + EndpointNodeId = endpointOut, + InlineImage = inlineImage + }; + } + + /// + /// Reads the LatestClip variable on the clip endpoint and its + /// concurrent LatestClipMetadata. Classifies the resulting + /// per §6.4 rules 3 and 5 so a client can branch + /// on state rather than raw codes. + /// + /// + /// The ClipEndpointType instance NodeId. + /// + /// + /// Cancels the operation. + /// + /// + public async Task ReadLatestClipAsync( + NodeId clipEndpointNodeId, + CancellationToken cancellationToken = default) + { + if (clipEndpointNodeId.IsNull) + { + throw new ArgumentException( + "Clip endpoint NodeId must not be null.", nameof(clipEndpointNodeId)); + } + string[] members = + [ + BrowseNames.LatestClip, + BrowseNames.LatestClipMetadata + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + clipEndpointNodeId, members, cancellationToken).ConfigureAwait(false); + var toRead = new List(); + for (int ii = 0; ii < nodes.Count; ii++) + { + if (!nodes[ii].IsNull) + { + toRead.Add(nodes[ii]); + } + } + ArrayOf values = await m_operations.ReadValuesAsync( + toRead, cancellationToken).ConfigureAwait(false); + int cursor = 0; + DataValue latestClip = DataValue.Null; + if (!nodes[0].IsNull) + { + latestClip = values[cursor++]; + } + VisionImageReferenceDataType? metadata = null; + if (!nodes[1].IsNull) + { + DataValue metadataValue = values[cursor++]; +#pragma warning disable CS8600 // TryGetValue uses [MaybeNullWhen(false)] on encodeable overloads. + if (StatusCode.IsGood(metadataValue.StatusCode) && + metadataValue.WrappedValue.TryGetValue( + out VisionImageReferenceDataType meta, + m_operations.Session.MessageContext)) + { + metadata = meta; + } +#pragma warning restore CS8600 + } + StatusCode statusCode = latestClip.StatusCode; + ByteString bytes = ByteString.Empty; + VisionInlineClipState state; + if (StatusCode.IsGood(statusCode)) + { + if (latestClip.WrappedValue.TryGetValue(out ByteString candidate)) + { + bytes = candidate; + } + state = VisionInlineClipState.Available; + } + else + { + state = ClassifyInlineState(statusCode); + } + return new VisionInlineClipReading(bytes, metadata, statusCode, state); + } + + /// + /// Reads the LatestClipMetadata variable on the clip endpoint. Returns + /// null when the Server has not published one yet. + /// + /// + /// The ClipEndpointType instance NodeId. + /// + /// + /// Cancels the operation. + /// + /// + public async Task ReadLatestClipMetadataAsync( + NodeId clipEndpointNodeId, + CancellationToken cancellationToken = default) + { + if (clipEndpointNodeId.IsNull) + { + throw new ArgumentException( + "Clip endpoint NodeId must not be null.", nameof(clipEndpointNodeId)); + } + NodeId node = await m_operations.ResolveChildAsync( + clipEndpointNodeId, + BrowseNames.LatestClipMetadata, + cancellationToken).ConfigureAwait(false); + return await m_operations + .TryReadStructureAsync(node, cancellationToken) + .ConfigureAwait(false); + } + + /// + /// Configures the video codec, resolution, frame rate and bitrate of a + /// stream endpoint (§6.3 ConfigureStreamEndpoint). + /// + /// + /// The StreamEndpointType instance NodeId. + /// + /// + /// The desired codec. + /// + /// + /// The desired horizontal resolution in pixels. + /// + /// + /// The desired vertical resolution in pixels. + /// + /// + /// The desired frame rate in Hz. + /// + /// + /// The desired target bitrate in bits per second. + /// + /// + /// Cancels the operation. + /// + /// + public Task ConfigureStreamEndpointAsync( + NodeId streamEndpointNodeId, + VisionVideoCodecEnum codec, + uint width, + uint height, + double frameRate, + uint bitrate, + CancellationToken cancellationToken = default) + { + if (streamEndpointNodeId.IsNull) + { + throw new ArgumentException( + "Stream endpoint NodeId must not be null.", nameof(streamEndpointNodeId)); + } + return m_proxy.ConfigureStreamEndpointAsync( + streamEndpointNodeId, codec, width, height, frameRate, bitrate, + cancellationToken).AsTask(); + } + + /// + /// Opens a stream session against the stream endpoint, returning the + /// session token, URI, protocol and expiry as reported by the Server. + /// + /// + /// The StreamEndpointType instance NodeId, or a null NodeId to apply + /// the §6.3 selection rule. + /// + /// + /// The desired profile name, or an empty string to accept the Server's + /// default. + /// + /// + /// The preferred stream protocol. + /// + /// + /// Cancels the operation. + /// + /// + public async Task GetStreamEndpointAsync( + NodeId streamEndpointNodeId, + string profileName, + VisionStreamProtocolEnum preferredProtocol, + CancellationToken cancellationToken = default) + { + if (profileName is null) + { + throw new ArgumentNullException(nameof(profileName)); + } + (VisionStreamSessionDataType session, NodeId _) = await m_proxy + .GetStreamEndpointAsync( + streamEndpointNodeId, + profileName, + preferredProtocol, + cancellationToken).ConfigureAwait(false); + return session; + } + + /// + /// Releases a previously opened stream session identified by + /// . + /// + /// + /// The session token returned by . + /// + /// + /// Cancels the operation. + /// + public Task ReleaseStreamEndpointAsync( + ByteString sessionToken, + CancellationToken cancellationToken = default) + { + return m_proxy.ReleaseStreamEndpointAsync( + sessionToken, cancellationToken).AsTask(); + } + + /// + /// Sets the Server's preferred clip and stream endpoints (§6.3 + /// SelectEndpoint). A null argument leaves the corresponding preference + /// unchanged. + /// + /// + /// The stream endpoint to prefer. + /// + /// + /// The clip endpoint to prefer. + /// + /// + /// Cancels the operation. + /// + public Task SelectEndpointAsync( + NodeId streamEndpointNodeId, + NodeId clipEndpointNodeId, + CancellationToken cancellationToken = default) + { + return m_proxy.SelectEndpointAsync( + streamEndpointNodeId, + clipEndpointNodeId, + cancellationToken).AsTask(); + } + + private static VisionInlineClipState ClassifyInlineState(StatusCode statusCode) + { + uint code = statusCode.Code; + if (code == StatusCodes.BadNoDataAvailable) + { + return VisionInlineClipState.NotYetAvailable; + } + if (code == StatusCodes.BadNotSupported) + { + return VisionInlineClipState.InlineDisabled; + } + if (code == StatusCodes.BadEncodingLimitsExceeded) + { + return VisionInlineClipState.Overflow; + } + return VisionInlineClipState.Faulted; + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionMediaSnapshots.cs b/src/Opc.Ua.Vision.Client/VisionMediaSnapshots.cs new file mode 100644 index 0000000000..07aca6ffc2 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionMediaSnapshots.cs @@ -0,0 +1,135 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.Client +{ + /// + /// The result of a GetClip call. §6.4 defines the by-reference default and + /// the optional inline delivery: always carries a URI, and + /// is populated only when the caller asked for inline + /// delivery and the encoded still fits the Server's effective inline limit. + /// + public sealed record VisionClipResult + { + /// + /// The by-reference image descriptor. Its Uri is always populated when + /// the Server returned Good; the Timestamp and Digest + /// together are the correlation key of §6.4 rule 4. + /// + public required VisionImageReferenceDataType Image { get; init; } + + /// + /// The endpoint the Server actually used, after applying the §6.3 selection + /// rule (a null Endpoint argument first falls back to + /// PreferredClipEndpoint, then to the first endpoint in BrowseName + /// order). Non-null when the Server named one. + /// + public NodeId EndpointNodeId { get; init; } = NodeId.Null; + + /// + /// The encoded image bytes when inline delivery was requested and the still + /// fit the Server's effective inline limit; an empty + /// otherwise. + /// + public ByteString InlineImage { get; init; } = ByteString.Empty; + + /// + /// Gets a value indicating whether carries bytes. + /// + public bool HasInlineImage + => !InlineImage.IsNull && InlineImage.Length > 0; + } + + /// + /// The result of a call + /// against an inline-delivery clip endpoint. §6.4 rule 5 fixes the initial and + /// disabled states, and §6.4 rule 3 fixes the overflow state, so a caller can + /// distinguish them without having to inspect a raw . + /// + /// + /// The encoded image bytes on success, or an empty + /// otherwise. + /// + /// + /// The concurrent LatestClipMetadata descriptor, or null when the + /// Server did not report one. + /// + /// + /// The the Server reported on LatestClip. + /// + /// + /// The classification of + /// . §6.4 makes the classification well-defined; a + /// client should branch on this rather than reinterpreting the raw code. + /// + public sealed record VisionInlineClipReading( + ByteString Bytes, + VisionImageReferenceDataType? Metadata, + StatusCode StatusCode, + VisionInlineClipState State); + + /// + /// The state of a LatestClip read against an inline-delivery clip endpoint, + /// derived from §6.4. + /// + public enum VisionInlineClipState + { + /// + /// The clip endpoint returned a fresh image within the inline size limit. + /// + Available = 0, + + /// + /// The Server has not published a clip yet — §6.4 rule 5 requires + /// Bad_NoDataAvailable before the first acquisition. A client should + /// wait rather than treating this as a hard error. + /// + NotYetAvailable, + + /// + /// The Server has InlineDeliveryEnabled = false — §6.4 rule 5 requires + /// Bad_NotSupported in that case. A client should fall back to the + /// out-of-band URI in the metadata. + /// + InlineDisabled, + + /// + /// The last acquisition exceeded the effective inline size limit — §6.4 rule + /// 3 requires Bad_EncodingLimitsExceeded without truncation. A client + /// should fall back to the out-of-band URI in the metadata. + /// + Overflow, + + /// + /// The clip endpoint reported a different error. The raw + /// is available in . + /// + Faulted + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionNodeEntry.cs b/src/Opc.Ua.Vision.Client/VisionNodeEntry.cs new file mode 100644 index 0000000000..a7b50bd660 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionNodeEntry.cs @@ -0,0 +1,52 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.Client +{ + /// + /// A discovered instance under the Vision root. + /// + /// + /// The instance NodeId. + /// + /// + /// The instance BrowseName as reported by the Server. + /// + /// + /// The instance DisplayName as reported by the Server. + /// + /// + /// The resolved type definition NodeId. + /// + public sealed record VisionNodeEntry( + NodeId NodeId, + QualifiedName BrowseName, + LocalizedText DisplayName, + NodeId TypeDefinitionId); +} diff --git a/src/Opc.Ua.Vision.Client/VisionPipelineClient.cs b/src/Opc.Ua.Vision.Client/VisionPipelineClient.cs new file mode 100644 index 0000000000..14e24a5103 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionPipelineClient.cs @@ -0,0 +1,319 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.Vision.Client +{ + /// + /// Focused client over a single InferencePipelineType instance. Wraps + /// RunInference, StartContinuous and Stop, and reads the + /// pipeline's identity, current state, sensor binding and deployment reference + /// (§8.3). + /// + public sealed class VisionPipelineClient + { + private readonly VisionClientOperations m_operations; + private readonly InferencePipelineTypeClient m_proxy; + + internal VisionPipelineClient( + VisionClientOperations operations, NodeId pipelineNodeId) + { + m_operations = operations + ?? throw new ArgumentNullException(nameof(operations)); + if (pipelineNodeId.IsNull) + { + throw new ArgumentException( + "Pipeline NodeId must not be null.", nameof(pipelineNodeId)); + } + PipelineNodeId = pipelineNodeId; + m_proxy = new InferencePipelineTypeClient( + m_operations.Session, pipelineNodeId, m_operations.Telemetry); + } + + /// + /// Gets the pipeline object NodeId. + /// + public NodeId PipelineNodeId { get; } + + /// + /// Reads the pipeline's identity and current state. + /// + /// + /// Cancels the operation. + /// + public async Task ReadAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.PipelineId, + BrowseNames.Sensor, + BrowseNames.Deployment, + BrowseNames.State, + BrowseNames.Continuous, + BrowseNames.LearningJob + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + PipelineNodeId, members, cancellationToken).ConfigureAwait(false); + var toRead = new List(); + for (int ii = 0; ii < nodes.Count; ii++) + { + if (!nodes[ii].IsNull) + { + toRead.Add(nodes[ii]); + } + } + ArrayOf values = await m_operations.ReadValuesAsync( + toRead, cancellationToken).ConfigureAwait(false); + int cursor = 0; + string? pipelineId = null; + if (!nodes[0].IsNull) + { + pipelineId = VisionClientOperations.ReadString(values[cursor++]); + } + NodeId sensorId = NodeId.Null; + if (!nodes[1].IsNull) + { + VisionClientOperations.TryReadNodeId(values[cursor++], out sensorId); + } + NodeId deploymentId = NodeId.Null; + if (!nodes[2].IsNull) + { + VisionClientOperations.TryReadNodeId(values[cursor++], out deploymentId); + } + VisionEndpointStateEnum state = default; + if (!nodes[3].IsNull) + { + VisionClientOperations.TryReadEnum(values[cursor++], out state); + } + bool continuous = false; + if (!nodes[4].IsNull) + { + DataValue value = values[cursor++]; + if (value.WrappedValue.TryGetValue(out bool b)) + { + continuous = b; + } + } + NodeId learningJobId = NodeId.Null; + if (!nodes[5].IsNull) + { + VisionClientOperations.TryReadNodeId(values[cursor++], out learningJobId); + } + return new VisionPipelineSnapshot + { + NodeId = PipelineNodeId, + PipelineId = pipelineId, + SensorId = sensorId, + DeploymentId = deploymentId, + State = state, + Continuous = continuous, + LearningJobId = learningJobId + }; + } + + /// + /// Reads only the current State of the pipeline. Suitable for a + /// short-polling wait loop that avoids reading the full snapshot each time. + /// + /// + /// Cancels the operation. + /// + public async Task ReadStateAsync( + CancellationToken cancellationToken = default) + { + NodeId node = await m_operations.ResolveChildAsync( + PipelineNodeId, BrowseNames.State, cancellationToken).ConfigureAwait(false); + if (node.IsNull) + { + return default; + } + DataValue value = await m_operations.ReadValueAsync( + node, cancellationToken).ConfigureAwait(false); + return VisionClientOperations.TryReadEnum( + value, out VisionEndpointStateEnum result) + ? result + : default; + } + + /// + /// Runs a single inference. Returns the ResultId of the newly + /// published result. §8.4 permits the Server to reject the call with a + /// ; the exception is surfaced. + /// + /// + /// The requested acquisition timestamp; a caller can pass default to + /// let the Server acquire "now". + /// + /// + /// Cancels the operation. + /// + public async Task RunInferenceAsync( + DateTimeUtc timestamp = default, + CancellationToken cancellationToken = default) + { + return await m_proxy.RunInferenceAsync( + timestamp, cancellationToken).ConfigureAwait(false); + } + + /// + /// Resolves the NodeId of a published result from the ResultId that + /// returned. + /// + /// + /// The Part 4 method answers with the ResultId the Server assigned, which identifies + /// the result but is not addressable: every tool that reads a result needs its NodeId. + /// A Server publishes each result under the pipeline's Results folder with the + /// ResultId as its BrowseName, so the two are one enumeration apart - without this an + /// agent has to guess the Server's NodeId convention, and a wrong guess reads some + /// other node and reports an empty result rather than failing. + /// + /// The ResultId the Server assigned. + /// + /// Cancels the operation. + /// + /// + /// The result NodeId, or a null NodeId when the Server publishes no such result. + /// + public async Task ResolveResultNodeIdAsync( + string resultId, + CancellationToken cancellationToken = default) + { + if (string.IsNullOrEmpty(resultId)) + { + return NodeId.Null; + } + await foreach (VisionNodeEntry entry in EnumerateResultsAsync(cancellationToken) + .ConfigureAwait(false)) + { + if (string.Equals(entry.BrowseName.Name, resultId, StringComparison.Ordinal)) + { + return entry.NodeId; + } + // A Server may prefix the ResultId to keep BrowseNames unique within the + // folder. Match that, but only on the ResultId itself: falling back to the + // most recently published result would answer confidently with the wrong + // one, which is worse than saying it was not found. + if (entry.BrowseName.Name is { } name && + name.EndsWith(resultId, StringComparison.Ordinal)) + { + return entry.NodeId; + } + } + return NodeId.Null; + } + + /// + /// Starts continuous inference. Throws where the Server refuses. + /// + /// + /// Cancels the operation. + /// + public Task StartContinuousAsync(CancellationToken cancellationToken = default) + { + return m_proxy.StartContinuousAsync(cancellationToken).AsTask(); + } + + /// + /// Stops continuous or in-progress inference. Throws where the Server refuses. + /// + /// + /// Cancels the operation. + /// + public Task StopAsync(CancellationToken cancellationToken = default) + { + return m_proxy.StopAsync(cancellationToken).AsTask(); + } + + /// + /// Enumerates the results the pipeline has published, browsing its + /// Results folder. + /// + /// + /// Cancels the operation. + /// + public async IAsyncEnumerable EnumerateResultsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + FolderTypeClient? folder = await m_proxy.GetResultsAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + NodeId folderId = folder is null ? NodeId.Null : folder.ObjectId; + if (folderId.IsNull) + { + yield break; + } + NodeId resultType = m_operations.VisionNamespaceType( + ObjectTypes.VisionResultType); + ArrayOf refs = await m_operations + .BrowseHierarchicalObjectsAsync(folderId, cancellationToken) + .ConfigureAwait(false); + for (int ii = 0; ii < refs.Count; ii++) + { + NodeId nodeId = ExpandedNodeId.ToNodeId( + refs[ii].NodeId, m_operations.Session.NamespaceUris); + NodeId typeDef = ExpandedNodeId.ToNodeId( + refs[ii].TypeDefinition, m_operations.Session.NamespaceUris); + if (nodeId.IsNull || typeDef.IsNull) + { + continue; + } + if (!resultType.IsNull && + !await m_operations.Session.NodeCache + .IsTypeOfAsync(typeDef, resultType, cancellationToken) + .ConfigureAwait(false)) + { + continue; + } + yield return new VisionNodeEntry( + nodeId, refs[ii].BrowseName, refs[ii].DisplayName, typeDef); + } + } + + /// + /// Opens the feedback client rooted at this pipeline's Feedback + /// object, or returns null when the pipeline does not expose one. + /// + /// + /// Cancels the operation. + /// + public async Task OpenFeedbackAsync( + CancellationToken cancellationToken = default) + { + VisionFeedbackTypeClient? feedback = await m_proxy.GetFeedbackAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + NodeId feedbackId = feedback is null ? NodeId.Null : feedback.ObjectId; + return feedbackId.IsNull ? null : new VisionFeedbackClient(m_operations, feedbackId); + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionResultReader.cs b/src/Opc.Ua.Vision.Client/VisionResultReader.cs new file mode 100644 index 0000000000..5c30cc0ce3 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionResultReader.cs @@ -0,0 +1,503 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client.Subscriptions; +using Opc.Ua.Client.Subscriptions.Streaming; + +namespace Opc.Ua.Vision.Client +{ + /// + /// Focused reader for DetectionResultType, InspectionResultType + /// and SegmentationResultType instances (§7). Reads the result-shared + /// members (ResultId, CreationTime, Sensor, + /// Pipeline, ModelVersionUsed, Frame) plus the subtype + /// members, and subscribes to result changes over an + /// . + /// + public sealed class VisionResultReader + { + private readonly VisionClientOperations m_operations; + + internal VisionResultReader(VisionClientOperations operations, NodeId resultNodeId) + { + m_operations = operations + ?? throw new ArgumentNullException(nameof(operations)); + if (resultNodeId.IsNull) + { + throw new ArgumentException( + "Result NodeId must not be null.", nameof(resultNodeId)); + } + ResultNodeId = resultNodeId; + } + + /// + /// Gets the result object NodeId. + /// + public NodeId ResultNodeId { get; } + + /// + /// Reads the result as an InspectionResultType snapshot (§7.2). + /// + /// + /// Cancels the operation. + /// + public async Task ReadInspectionAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.ResultId, + BrowseNames.CreationTime, + BrowseNames.Sensor, + BrowseNames.Pipeline, + BrowseNames.ModelVersionUsed, + BrowseNames.Frame, + BrowseNames.Evaluation, + BrowseNames.PartId, + BrowseNames.RecipeId, + BrowseNames.Characteristics + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + ResultNodeId, members, cancellationToken).ConfigureAwait(false); + var toRead = ExtractPresent(nodes); + ArrayOf values = await m_operations.ReadValuesAsync( + toRead, cancellationToken).ConfigureAwait(false); + int cursor = 0; + string? resultId = TakeString(values, nodes, 0, ref cursor); + DateTimeUtc creationTime = TakeDateTime(values, nodes, 1, ref cursor); + NodeId sensor = TakeNodeId(values, nodes, 2, ref cursor); + NodeId pipeline = TakeNodeId(values, nodes, 3, ref cursor); + string? modelVersion = TakeString(values, nodes, 4, ref cursor); + VisionImageReferenceDataType? frame = TakeImageReference( + values, nodes, 5, ref cursor); + VisionResultEvaluationEnum evaluation = TakeEnum( + values, nodes, 6, ref cursor); + string? partId = TakeString(values, nodes, 7, ref cursor); + string? recipeId = TakeString(values, nodes, 8, ref cursor); + ArrayOf characteristics = + TakeCharacteristics(values, nodes, 9, ref cursor); + return new VisionInspectionResultSnapshot + { + NodeId = ResultNodeId, + ResultId = resultId, + CreationTime = creationTime, + SensorId = sensor, + PipelineId = pipeline, + ModelVersionUsed = modelVersion, + Frame = frame, + Evaluation = evaluation, + PartId = partId, + RecipeId = recipeId, + Characteristics = characteristics + }; + } + + /// + /// Reads the result as a DetectionResultType snapshot (§7.3). + /// + /// + /// Cancels the operation. + /// + public async Task ReadDetectionAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.ResultId, + BrowseNames.CreationTime, + BrowseNames.Sensor, + BrowseNames.Pipeline, + BrowseNames.ModelVersionUsed, + BrowseNames.Frame, + BrowseNames.FrameId, + BrowseNames.Detections + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + ResultNodeId, members, cancellationToken).ConfigureAwait(false); + var toRead = ExtractPresent(nodes); + ArrayOf values = await m_operations.ReadValuesAsync( + toRead, cancellationToken).ConfigureAwait(false); + int cursor = 0; + string? resultId = TakeString(values, nodes, 0, ref cursor); + DateTimeUtc creationTime = TakeDateTime(values, nodes, 1, ref cursor); + NodeId sensor = TakeNodeId(values, nodes, 2, ref cursor); + NodeId pipeline = TakeNodeId(values, nodes, 3, ref cursor); + string? modelVersion = TakeString(values, nodes, 4, ref cursor); + VisionImageReferenceDataType? frame = TakeImageReference( + values, nodes, 5, ref cursor); + string? frameId = TakeString(values, nodes, 6, ref cursor); + ArrayOf detections = TakeDetections( + values, nodes, 7, ref cursor); + return new VisionDetectionResultSnapshot + { + NodeId = ResultNodeId, + ResultId = resultId, + CreationTime = creationTime, + SensorId = sensor, + PipelineId = pipeline, + ModelVersionUsed = modelVersion, + Frame = frame, + FrameId = frameId, + Detections = detections + }; + } + + /// + /// Reads the result as a SegmentationResultType snapshot (§7.4). + /// + /// + /// Cancels the operation. + /// + public async Task ReadSegmentationAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.ResultId, + BrowseNames.CreationTime, + BrowseNames.Sensor, + BrowseNames.Pipeline, + BrowseNames.Frame, + BrowseNames.LabelClasses, + BrowseNames.Mask + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + ResultNodeId, members, cancellationToken).ConfigureAwait(false); + var toRead = ExtractPresent(nodes); + ArrayOf values = await m_operations.ReadValuesAsync( + toRead, cancellationToken).ConfigureAwait(false); + int cursor = 0; + string? resultId = TakeString(values, nodes, 0, ref cursor); + DateTimeUtc creationTime = TakeDateTime(values, nodes, 1, ref cursor); + NodeId sensor = TakeNodeId(values, nodes, 2, ref cursor); + NodeId pipeline = TakeNodeId(values, nodes, 3, ref cursor); + VisionImageReferenceDataType? frame = TakeImageReference( + values, nodes, 4, ref cursor); + ArrayOf labels = TakeStringArray(values, nodes, 5, ref cursor); + VisionImageReferenceDataType? mask = TakeImageReference( + values, nodes, 6, ref cursor); + return new VisionSegmentationResultSnapshot + { + NodeId = ResultNodeId, + ResultId = resultId, + CreationTime = creationTime, + SensorId = sensor, + PipelineId = pipeline, + Frame = frame, + LabelClasses = labels, + Mask = mask + }; + } + + /// + /// Streams detection snapshots each time the Detections variable + /// changes on the Server. + /// + /// + /// The streaming subscription to monitor over. + /// + /// + /// Cancels the observation. + /// + /// + public IAsyncEnumerable ObserveDetectionsAsync( + IStreamingSubscription streaming, + CancellationToken cancellationToken = default) + { + if (streaming is null) + { + throw new ArgumentNullException(nameof(streaming)); + } + return ObserveDetectionsCoreAsync(streaming, cancellationToken); + } + + private async IAsyncEnumerable ObserveDetectionsCoreAsync( + IStreamingSubscription streaming, + [EnumeratorCancellation] CancellationToken cancellationToken) + { + NodeId detectionsNode = await m_operations.ResolveChildAsync( + ResultNodeId, BrowseNames.Detections, cancellationToken) + .ConfigureAwait(false); + if (detectionsNode.IsNull) + { + throw ServiceResultException.Create( + StatusCodes.BadNotFound, + "Result '{0}' does not expose a Detections variable.", + ResultNodeId); + } + var monitored = new List { detectionsNode }; + await foreach (DataValueChange _ in streaming.SubscribeDataChangesAsync( + monitored, null, cancellationToken).ConfigureAwait(false)) + { + yield return await ReadDetectionAsync(cancellationToken) + .ConfigureAwait(false); + } + } + + /// + /// Streams inspection snapshots each time the Characteristics + /// variable changes on the Server. + /// + /// + /// The streaming subscription to monitor over. + /// + /// + /// Cancels the observation. + /// + /// + public IAsyncEnumerable ObserveInspectionAsync( + IStreamingSubscription streaming, + CancellationToken cancellationToken = default) + { + if (streaming is null) + { + throw new ArgumentNullException(nameof(streaming)); + } + return ObserveInspectionCoreAsync(streaming, cancellationToken); + } + + private async IAsyncEnumerable ObserveInspectionCoreAsync( + IStreamingSubscription streaming, + [EnumeratorCancellation] CancellationToken cancellationToken) + { + NodeId characteristicsNode = await m_operations.ResolveChildAsync( + ResultNodeId, BrowseNames.Characteristics, cancellationToken) + .ConfigureAwait(false); + if (characteristicsNode.IsNull) + { + throw ServiceResultException.Create( + StatusCodes.BadNotFound, + "Result '{0}' does not expose a Characteristics variable.", + ResultNodeId); + } + var monitored = new List { characteristicsNode }; + await foreach (DataValueChange _ in streaming.SubscribeDataChangesAsync( + monitored, null, cancellationToken).ConfigureAwait(false)) + { + yield return await ReadInspectionAsync(cancellationToken) + .ConfigureAwait(false); + } + } + + /// + /// Streams segmentation snapshots each time the Mask variable changes + /// on the Server. + /// + /// + /// The streaming subscription to monitor over. + /// + /// + /// Cancels the observation. + /// + /// + public IAsyncEnumerable ObserveSegmentationAsync( + IStreamingSubscription streaming, + CancellationToken cancellationToken = default) + { + if (streaming is null) + { + throw new ArgumentNullException(nameof(streaming)); + } + return ObserveSegmentationCoreAsync(streaming, cancellationToken); + } + + private async IAsyncEnumerable ObserveSegmentationCoreAsync( + IStreamingSubscription streaming, + [EnumeratorCancellation] CancellationToken cancellationToken) + { + NodeId maskNode = await m_operations.ResolveChildAsync( + ResultNodeId, BrowseNames.Mask, cancellationToken).ConfigureAwait(false); + if (maskNode.IsNull) + { + throw ServiceResultException.Create( + StatusCodes.BadNotFound, + "Result '{0}' does not expose a Mask variable.", + ResultNodeId); + } + var monitored = new List { maskNode }; + await foreach (DataValueChange _ in streaming.SubscribeDataChangesAsync( + monitored, null, cancellationToken).ConfigureAwait(false)) + { + yield return await ReadSegmentationAsync(cancellationToken) + .ConfigureAwait(false); + } + } + + private static List ExtractPresent(ArrayOf nodes) + { + var list = new List(nodes.Count); + for (int ii = 0; ii < nodes.Count; ii++) + { + if (!nodes[ii].IsNull) + { + list.Add(nodes[ii]); + } + } + return list; + } + + private static string? TakeString( + ArrayOf values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return null; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out string text) ? text : null; + } + + private static DateTimeUtc TakeDateTime( + ArrayOf values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return default; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out DateTimeUtc dt) ? dt : default; + } + + private static NodeId TakeNodeId( + ArrayOf values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return NodeId.Null; + } + DataValue value = values[cursor++]; + return VisionClientOperations.TryReadNodeId(value, out NodeId nodeId) + ? nodeId + : NodeId.Null; + } + + private static TEnum TakeEnum( + ArrayOf values, + ArrayOf nodes, + int index, + ref int cursor) + where TEnum : struct, Enum + { + if (nodes[index].IsNull) + { + return default; + } + DataValue value = values[cursor++]; + return VisionClientOperations.TryReadEnum(value, out TEnum result) + ? result + : default; + } + + private static ArrayOf TakeStringArray( + ArrayOf values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return ArrayOf.Empty; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out ArrayOf array) + ? array + : ArrayOf.Empty; + } + + private VisionImageReferenceDataType? TakeImageReference( + ArrayOf values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return null; + } + DataValue value = values[cursor++]; +#pragma warning disable CS8600 // TryGetValue uses [MaybeNullWhen(false)] on encodeable overloads. + return value.WrappedValue.TryGetValue( + out VisionImageReferenceDataType structure, + m_operations.Session.MessageContext) + ? structure + : null; +#pragma warning restore CS8600 + } + + private ArrayOf TakeDetections( + ArrayOf values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return ArrayOf.Empty; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue( + out ArrayOf array, + m_operations.Session.MessageContext) + ? array + : ArrayOf.Empty; + } + + private ArrayOf TakeCharacteristics( + ArrayOf values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return ArrayOf.Empty; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue( + out ArrayOf array, + m_operations.Session.MessageContext) + ? array + : ArrayOf.Empty; + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionResultSnapshots.cs b/src/Opc.Ua.Vision.Client/VisionResultSnapshots.cs new file mode 100644 index 0000000000..1601faa914 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionResultSnapshots.cs @@ -0,0 +1,270 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.Client +{ + /// + /// A snapshot of a CoordinateFrameType instance (§5.8). + /// + public sealed record VisionFrameSnapshot + { + /// + /// The frame's own NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The Server-stable frame identifier string. §5.12 requires this to be + /// non-empty wherever a pose is published. + /// + public string? FrameId { get; init; } + + /// + /// The ISO 9787 frame role played by this frame — for example World, + /// Base, MechanicalInterface, Tool, Object, or the + /// non-ISO Camera. + /// + public VisionFrameRoleEnum Role { get; init; } + + /// + /// The NodeId of the parent frame, or a null NodeId when this is a root. + /// + public NodeId ParentFrameId { get; init; } = NodeId.Null; + + /// + /// The transform from this frame to , or + /// null when the Server did not report it. Position is in + /// metres, Orientation is a unit quaternion ordered (x, y, z, w). + /// + public VisionPose3DDataType? Transform { get; init; } + } + + /// + /// A snapshot of an InferencePipelineType instance (§8.3). + /// + public sealed record VisionPipelineSnapshot + { + /// + /// The pipeline NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The Server-stable pipeline identifier. + /// + public string? PipelineId { get; init; } + + /// + /// The NodeId of the bound sensor, or a null NodeId when not reported. + /// + public NodeId SensorId { get; init; } = NodeId.Null; + + /// + /// The NodeId of the deployment executing inference, or a null NodeId when the + /// pipeline is not bound to a described deployment. + /// + public NodeId DeploymentId { get; init; } = NodeId.Null; + + /// + /// The current pipeline state (§6.6). + /// + public VisionEndpointStateEnum State { get; init; } + + /// + /// Whether the pipeline is currently running continuous inference. + /// + public bool Continuous { get; init; } + + /// + /// The NodeId of the associated learning job, or a null NodeId when the + /// Server retains no ground-truth corrections. + /// + public NodeId LearningJobId { get; init; } = NodeId.Null; + } + + /// + /// A snapshot of a DetectionResultType instance (§7.3). + /// + public sealed record VisionDetectionResultSnapshot + { + /// + /// The result NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The result identifier string. + /// + public string? ResultId { get; init; } + + /// + /// The time the result was published. + /// + public DateTimeUtc CreationTime { get; init; } + + /// + /// The sensor that produced the frame the result was computed from. + /// + public NodeId SensorId { get; init; } = NodeId.Null; + + /// + /// The pipeline that computed the result. + /// + public NodeId PipelineId { get; init; } = NodeId.Null; + + /// + /// The model version used to compute the result, when reported. + /// + public string? ModelVersionUsed { get; init; } + + /// + /// A descriptor for the image the detections apply to, when reported. + /// + public VisionImageReferenceDataType? Frame { get; init; } + + /// + /// The frame that detection poses are expressed in. §7.3 requires this to be + /// non-empty whenever any detection has HasPose = true. + /// + public string? FrameId { get; init; } + + /// + /// The detected instances. + /// + public ArrayOf Detections { get; init; } + } + + /// + /// A snapshot of an InspectionResultType instance (§7.2). + /// + public sealed record VisionInspectionResultSnapshot + { + /// + /// The result NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The result identifier string. + /// + public string? ResultId { get; init; } + + /// + /// The time the result was published. + /// + public DateTimeUtc CreationTime { get; init; } + + /// + /// The sensor that produced the frame the result was computed from. + /// + public NodeId SensorId { get; init; } = NodeId.Null; + + /// + /// The pipeline that computed the result. + /// + public NodeId PipelineId { get; init; } = NodeId.Null; + + /// + /// The model version used to compute the result, when reported. + /// + public string? ModelVersionUsed { get; init; } + + /// + /// A descriptor for the image the inspection was computed from, when reported. + /// + public VisionImageReferenceDataType? Frame { get; init; } + + /// + /// The overall inspection evaluation. + /// + public VisionResultEvaluationEnum Evaluation { get; init; } + + /// + /// The identifier of the inspected part, when reported. + /// + public string? PartId { get; init; } + + /// + /// The identifier of the inspection recipe, when reported. + /// + public string? RecipeId { get; init; } + + /// + /// The measured characteristics. + /// + public ArrayOf Characteristics { get; init; } + } + + /// + /// A snapshot of a SegmentationResultType instance (§7.4). + /// + public sealed record VisionSegmentationResultSnapshot + { + /// + /// The result NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The result identifier string. + /// + public string? ResultId { get; init; } + + /// + /// The time the result was published. + /// + public DateTimeUtc CreationTime { get; init; } + + /// + /// The sensor that produced the frame the result was computed from. + /// + public NodeId SensorId { get; init; } = NodeId.Null; + + /// + /// The pipeline that computed the result. + /// + public NodeId PipelineId { get; init; } = NodeId.Null; + + /// + /// A descriptor for the image the mask applies to, when reported. + /// + public VisionImageReferenceDataType? Frame { get; init; } + + /// + /// The class labels that pixel indices of refer to. + /// + public ArrayOf LabelClasses { get; init; } + + /// + /// A reference to the mask image (§7.4). Masks follow the media rules of §6 and + /// are never inlined into the result. + /// + public VisionImageReferenceDataType? Mask { get; init; } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionSensorClient.cs b/src/Opc.Ua.Vision.Client/VisionSensorClient.cs new file mode 100644 index 0000000000..c0c0a350f5 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionSensorClient.cs @@ -0,0 +1,738 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client; + +namespace Opc.Ua.Vision.Client +{ + /// + /// Focused client over a single VisionSensorType (or subtype) instance. + /// Reads identity, imaging or depth members, optics, illumination, the sensor's + /// mounted frame, and its intrinsic and hand-eye extrinsic calibrations, so a + /// caller can act on detections in world coordinates without knowing NodeIds + /// or BrowseNames. + /// + public sealed class VisionSensorClient + { + private readonly VisionClientOperations m_operations; + private readonly VisionSensorTypeClient m_proxy; + + internal VisionSensorClient(VisionClientOperations operations, NodeId sensorNodeId) + { + m_operations = operations + ?? throw new ArgumentNullException(nameof(operations)); + if (sensorNodeId.IsNull) + { + throw new ArgumentException( + "Sensor NodeId must not be null.", nameof(sensorNodeId)); + } + SensorNodeId = sensorNodeId; + m_proxy = new VisionSensorTypeClient( + m_operations.Session, sensorNodeId, m_operations.Telemetry); + } + + /// + /// Gets the sensor object NodeId. + /// + public NodeId SensorNodeId { get; } + + /// + /// Reads the sensor's identity nameplate (§5.4). + /// + /// + /// Cancels the operation. + /// + public async Task ReadIdentityAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.SensorId, + BrowseNames.RealityKind, + BrowseNames.Modality, + BrowseNames.Manufacturer, + BrowseNames.Model, + BrowseNames.SerialNumber, + BrowseNames.DeviceUri, + BrowseNames.FrameId + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + SensorNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync( + ToList(nodes), cancellationToken).ConfigureAwait(false); + var buffer = new List(values.Count); + for (int ii = 0; ii < values.Count; ii++) + { + buffer.Add(values[ii]); + } + int cursor = 0; + string? sensorId = TakeString(buffer, nodes, 0, ref cursor); + VisionRealityKindEnum reality = TakeEnum( + buffer, nodes, 1, ref cursor); + VisionSensorModalityEnum modality = TakeEnum( + buffer, nodes, 2, ref cursor); + LocalizedText manufacturer = TakeLocalizedText(buffer, nodes, 3, ref cursor); + LocalizedText model = TakeLocalizedText(buffer, nodes, 4, ref cursor); + string? serialNumber = TakeString(buffer, nodes, 5, ref cursor); + string? deviceUri = TakeString(buffer, nodes, 6, ref cursor); + string? frameId = TakeString(buffer, nodes, 7, ref cursor); + return new VisionSensorIdentity + { + NodeId = SensorNodeId, + SensorId = sensorId, + RealityKind = reality, + Modality = modality, + Manufacturer = manufacturer, + Model = model, + SerialNumber = serialNumber, + DeviceUri = deviceUri, + FrameId = frameId + }; + } + + /// + /// Reads the imaging members from the sensor, when it is an + /// ImageSensorType (§5.5). Members that are Optional on the type are + /// returned as null when the Server did not materialise them. + /// + /// + /// Cancels the operation. + /// + public async Task ReadImageMembersAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.Width, + BrowseNames.Height, + BrowseNames.PixelFormat, + BrowseNames.ExposureTime, + BrowseNames.Gain, + BrowseNames.AcquisitionFrameRate, + BrowseNames.Intrinsics + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + SensorNodeId, members, cancellationToken).ConfigureAwait(false); + if (nodes[0].IsNull && nodes[1].IsNull && nodes[2].IsNull) + { + return null; + } + ArrayOf values = await m_operations.ReadValuesAsync( + ToList(nodes), cancellationToken).ConfigureAwait(false); + var buffer = new List(values.Count); + for (int ii = 0; ii < values.Count; ii++) + { + buffer.Add(values[ii]); + } + int cursor = 0; + uint width = TakeUInt32(buffer, nodes, 0, ref cursor); + uint height = TakeUInt32(buffer, nodes, 1, ref cursor); + string? pixelFormat = TakeString(buffer, nodes, 2, ref cursor); + double? exposureTime = TakeDoubleOrNull(buffer, nodes, 3, ref cursor); + double? gain = TakeDoubleOrNull(buffer, nodes, 4, ref cursor); + double? frameRate = TakeDoubleOrNull(buffer, nodes, 5, ref cursor); + VisionIntrinsicsDataType? intrinsics = TakeIntrinsics( + buffer, nodes, 6, ref cursor); + return new VisionImageSensorSnapshot + { + Width = width, + Height = height, + PixelFormat = pixelFormat, + ExposureTime = exposureTime, + Gain = gain, + AcquisitionFrameRate = frameRate, + Intrinsics = intrinsics + }; + } + + /// + /// Reads the depth members from the sensor, when it is a + /// Depth3DSensorType (§5.6). Returns null when the sensor does + /// not carry any depth-specific members. + /// + /// + /// Cancels the operation. + /// + public async Task ReadDepthMembersAsync( + CancellationToken cancellationToken = default) + { + string[] members = + [ + BrowseNames.MinDepth, + BrowseNames.MaxDepth, + BrowseNames.DepthScale, + BrowseNames.Baseline, + BrowseNames.PointsPerFrame + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + SensorNodeId, members, cancellationToken).ConfigureAwait(false); + bool anyPresent = false; + for (int ii = 0; ii < nodes.Count; ii++) + { + if (!nodes[ii].IsNull) + { + anyPresent = true; + break; + } + } + if (!anyPresent) + { + return null; + } + ArrayOf values = await m_operations.ReadValuesAsync( + ToList(nodes), cancellationToken).ConfigureAwait(false); + var buffer = new List(values.Count); + for (int ii = 0; ii < values.Count; ii++) + { + buffer.Add(values[ii]); + } + int cursor = 0; + double minDepth = TakeDouble(buffer, nodes, 0, ref cursor); + double maxDepth = TakeDouble(buffer, nodes, 1, ref cursor); + double depthScale = TakeDouble(buffer, nodes, 2, ref cursor); + double baseline = TakeDouble(buffer, nodes, 3, ref cursor); + uint points = TakeUInt32(buffer, nodes, 4, ref cursor); + return new VisionDepth3DSensorSnapshot + { + MinDepth = minDepth, + MaxDepth = maxDepth, + DepthScale = depthScale, + Baseline = baseline, + PointsPerFrame = points + }; + } + + /// + /// Reads the optics description of the sensor, when present. + /// + /// + /// Cancels the operation. + /// + public async Task ReadOpticsAsync( + CancellationToken cancellationToken = default) + { + OpticsTypeClient? optics = await m_proxy.GetOpticsAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + if (optics is null || optics.ObjectId.IsNull) + { + return null; + } + string[] members = + [ + BrowseNames.FocalLength, + BrowseNames.Aperture, + BrowseNames.MinimumWorkingDistance + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + optics.ObjectId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync( + ToList(nodes), cancellationToken).ConfigureAwait(false); + var buffer = new List(values.Count); + for (int ii = 0; ii < values.Count; ii++) + { + buffer.Add(values[ii]); + } + int cursor = 0; + double? focalLength = TakeDoubleOrNull(buffer, nodes, 0, ref cursor); + double? aperture = TakeDoubleOrNull(buffer, nodes, 1, ref cursor); + double? workingDistance = TakeDoubleOrNull(buffer, nodes, 2, ref cursor); + return new VisionOpticsSnapshot + { + NodeId = optics.ObjectId, + FocalLength = focalLength, + Aperture = aperture, + WorkingDistance = workingDistance + }; + } + + /// + /// Reads the illumination description of the sensor, when present. + /// + /// + /// Cancels the operation. + /// + public async Task ReadIlluminationAsync( + CancellationToken cancellationToken = default) + { + IlluminationTypeClient? illumination = await m_proxy.GetIlluminationAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + if (illumination is null || illumination.ObjectId.IsNull) + { + return null; + } + string[] members = + [ + BrowseNames.Wavelength, + BrowseNames.RelativeIntensity + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + illumination.ObjectId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync( + ToList(nodes), cancellationToken).ConfigureAwait(false); + var buffer = new List(values.Count); + for (int ii = 0; ii < values.Count; ii++) + { + buffer.Add(values[ii]); + } + int cursor = 0; + double? wavelength = TakeDoubleOrNull(buffer, nodes, 0, ref cursor); + double? intensity = TakeDoubleOrNull(buffer, nodes, 1, ref cursor); + return new VisionIlluminationSnapshot + { + NodeId = illumination.ObjectId, + Wavelength = wavelength, + RelativeIntensity = intensity + }; + } + + /// + /// Reads the frame the sensor is mounted on (§5.11 MountedOn). Returns + /// a null NodeId when the sensor does not declare a mount frame. + /// + /// + /// Cancels the operation. + /// + public async Task GetMountedFrameIdAsync( + CancellationToken cancellationToken = default) + { + ArrayOf refs = await m_operations.BrowseAsync( + SensorNodeId, + m_operations.VisionReference(ReferenceTypes.MountedOn), + BrowseDirection.Forward, + (uint)NodeClass.Object, + cancellationToken).ConfigureAwait(false); + for (int ii = 0; ii < refs.Count; ii++) + { + NodeId target = ExpandedNodeId.ToNodeId( + refs[ii].NodeId, m_operations.Session.NamespaceUris); + if (!target.IsNull) + { + return target; + } + } + return NodeId.Null; + } + + /// + /// Opens the media-management client rooted at this sensor's + /// Media object. + /// + /// + /// Cancels the operation. + /// + public async Task OpenMediaAsync( + CancellationToken cancellationToken = default) + { + VisionMediaManagementTypeClient? media = await m_proxy.GetMediaAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + if (media is null || media.ObjectId.IsNull) + { + return null; + } + return new VisionMediaClient(m_operations, media.ObjectId); + } + + /// + /// Enumerates the calibrations attached to the sensor via HasCalibration + /// or nested in the sensor's Calibrations folder. + /// + /// + /// Cancels the operation. + /// + public async IAsyncEnumerable EnumerateCalibrationsAsync( + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + NodeId calibrationType = m_operations.VisionNamespaceType( + ObjectTypes.VisionCalibrationType); + if (calibrationType.IsNull) + { + yield break; + } + var refs = new List(); + ArrayOf direct = await m_operations.BrowseAsync( + SensorNodeId, + m_operations.VisionReference(ReferenceTypes.HasCalibration), + BrowseDirection.Forward, + (uint)NodeClass.Object, + cancellationToken).ConfigureAwait(false); + for (int ii = 0; ii < direct.Count; ii++) + { + refs.Add(direct[ii]); + } + FolderTypeClient? folder = await m_proxy.GetCalibrationsAsync( + m_operations.Telemetry, cancellationToken).ConfigureAwait(false); + if (folder is not null && !folder.ObjectId.IsNull) + { + ArrayOf nested = await m_operations + .BrowseHierarchicalObjectsAsync(folder.ObjectId, cancellationToken) + .ConfigureAwait(false); + for (int ii = 0; ii < nested.Count; ii++) + { + refs.Add(nested[ii]); + } + } + var seen = new HashSet(); + for (int ii = 0; ii < refs.Count; ii++) + { + NodeId nodeId = ExpandedNodeId.ToNodeId( + refs[ii].NodeId, m_operations.Session.NamespaceUris); + NodeId typeDef = ExpandedNodeId.ToNodeId( + refs[ii].TypeDefinition, m_operations.Session.NamespaceUris); + if (nodeId.IsNull || typeDef.IsNull || !seen.Add(nodeId)) + { + continue; + } + if (!await m_operations.Session.NodeCache.IsTypeOfAsync( + typeDef, calibrationType, cancellationToken).ConfigureAwait(false)) + { + continue; + } + yield return new VisionNodeEntry( + nodeId, refs[ii].BrowseName, refs[ii].DisplayName, typeDef); + } + } + + /// + /// Reads an intrinsic-calibration snapshot from the given calibration NodeId. + /// + /// + /// The IntrinsicCalibrationType instance NodeId; typically obtained + /// from . + /// + /// + /// Cancels the operation. + /// + /// + public async Task ReadIntrinsicCalibrationAsync( + NodeId calibrationNodeId, + CancellationToken cancellationToken = default) + { + if (calibrationNodeId.IsNull) + { + throw new ArgumentException( + "Calibration NodeId must not be null.", nameof(calibrationNodeId)); + } + string[] members = + [ + BrowseNames.CalibrationId, + BrowseNames.PerformedAt, + BrowseNames.Valid, + BrowseNames.ResidualError, + BrowseNames.Method, + BrowseNames.Intrinsics + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + calibrationNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync( + ToList(nodes), cancellationToken).ConfigureAwait(false); + var buffer = new List(values.Count); + for (int ii = 0; ii < values.Count; ii++) + { + buffer.Add(values[ii]); + } + int cursor = 0; + string? calibrationId = TakeString(buffer, nodes, 0, ref cursor); + DateTimeUtc performedAt = TakeDateTime(buffer, nodes, 1, ref cursor); + bool valid = TakeBool(buffer, nodes, 2, ref cursor); + double residual = TakeDouble(buffer, nodes, 3, ref cursor); + string? method = TakeString(buffer, nodes, 4, ref cursor); + VisionIntrinsicsDataType? intrinsics = TakeIntrinsics( + buffer, nodes, 5, ref cursor); + return new VisionIntrinsicCalibrationSnapshot + { + NodeId = calibrationNodeId, + CalibrationId = calibrationId, + PerformedAt = performedAt, + Valid = valid, + ResidualError = residual, + Method = method, + Intrinsics = intrinsics + }; + } + + /// + /// Reads an extrinsic-calibration snapshot from the given calibration NodeId. + /// + /// + /// The ExtrinsicCalibrationType instance NodeId; typically obtained + /// from . + /// + /// + /// Cancels the operation. + /// + /// + public async Task ReadExtrinsicCalibrationAsync( + NodeId calibrationNodeId, + CancellationToken cancellationToken = default) + { + if (calibrationNodeId.IsNull) + { + throw new ArgumentException( + "Calibration NodeId must not be null.", nameof(calibrationNodeId)); + } + string[] members = + [ + BrowseNames.CalibrationId, + BrowseNames.PerformedAt, + BrowseNames.Valid, + BrowseNames.ResidualError, + BrowseNames.Method, + BrowseNames.Mount, + BrowseNames.SourceFrame, + BrowseNames.TargetFrame, + BrowseNames.Transform + ]; + ArrayOf nodes = await m_operations.ResolveChildrenAsync( + calibrationNodeId, members, cancellationToken).ConfigureAwait(false); + ArrayOf values = await m_operations.ReadValuesAsync( + ToList(nodes), cancellationToken).ConfigureAwait(false); + var buffer = new List(values.Count); + for (int ii = 0; ii < values.Count; ii++) + { + buffer.Add(values[ii]); + } + int cursor = 0; + string? calibrationId = TakeString(buffer, nodes, 0, ref cursor); + DateTimeUtc performedAt = TakeDateTime(buffer, nodes, 1, ref cursor); + bool valid = TakeBool(buffer, nodes, 2, ref cursor); + double residual = TakeDouble(buffer, nodes, 3, ref cursor); + string? method = TakeString(buffer, nodes, 4, ref cursor); + VisionCalibrationMountEnum mount = TakeEnum( + buffer, nodes, 5, ref cursor); + NodeId source = TakeNodeId(buffer, nodes, 6, ref cursor); + NodeId target = TakeNodeId(buffer, nodes, 7, ref cursor); + VisionPose3DDataType? transform = TakePose( + buffer, nodes, 8, ref cursor); + return new VisionExtrinsicCalibrationSnapshot + { + NodeId = calibrationNodeId, + CalibrationId = calibrationId, + PerformedAt = performedAt, + Valid = valid, + ResidualError = residual, + Method = method, + Mount = mount, + SourceFrameId = source, + TargetFrameId = target, + Transform = transform + }; + } + + private static List ToList(ArrayOf nodes) + { + var list = new List(nodes.Count); + for (int ii = 0; ii < nodes.Count; ii++) + { + if (!nodes[ii].IsNull) + { + list.Add(nodes[ii]); + } + } + return list; + } + + private static string? TakeString( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return null; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out string text) ? text : null; + } + + private static LocalizedText TakeLocalizedText( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return LocalizedText.Null; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out LocalizedText text) + ? text + : LocalizedText.Null; + } + + private static TEnum TakeEnum( + List values, + ArrayOf nodes, + int index, + ref int cursor) + where TEnum : struct, Enum + { + if (nodes[index].IsNull) + { + return default; + } + DataValue value = values[cursor++]; + return VisionClientOperations.TryReadEnum(value, out TEnum result) + ? result + : default; + } + + private static double TakeDouble( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return 0.0; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out double d) ? d : 0.0; + } + + private static double? TakeDoubleOrNull( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return null; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out double d) ? d : null; + } + + private static uint TakeUInt32( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return 0; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out uint u) ? u : 0; + } + + private static bool TakeBool( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return false; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out bool b) && b; + } + + private static DateTimeUtc TakeDateTime( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return default; + } + DataValue value = values[cursor++]; + return value.WrappedValue.TryGetValue(out DateTimeUtc dt) ? dt : default; + } + + private static NodeId TakeNodeId( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return NodeId.Null; + } + DataValue value = values[cursor++]; + return VisionClientOperations.TryReadNodeId(value, out NodeId nodeId) + ? nodeId + : NodeId.Null; + } + + private VisionIntrinsicsDataType? TakeIntrinsics( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return null; + } + DataValue value = values[cursor++]; +#pragma warning disable CS8600 // TryGetValue uses [MaybeNullWhen(false)] on encodeable overloads. + return value.WrappedValue.TryGetValue( + out VisionIntrinsicsDataType structure, + m_operations.Session.MessageContext) + ? structure + : null; +#pragma warning restore CS8600 + } + + private VisionPose3DDataType? TakePose( + List values, + ArrayOf nodes, + int index, + ref int cursor) + { + if (nodes[index].IsNull) + { + return null; + } + DataValue value = values[cursor++]; +#pragma warning disable CS8600 // TryGetValue uses [MaybeNullWhen(false)] on encodeable overloads. + return value.WrappedValue.TryGetValue( + out VisionPose3DDataType structure, + m_operations.Session.MessageContext) + ? structure + : null; +#pragma warning restore CS8600 + } + } +} diff --git a/src/Opc.Ua.Vision.Client/VisionSensorSnapshots.cs b/src/Opc.Ua.Vision.Client/VisionSensorSnapshots.cs new file mode 100644 index 0000000000..7ec7ed0365 --- /dev/null +++ b/src/Opc.Ua.Vision.Client/VisionSensorSnapshots.cs @@ -0,0 +1,306 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.Client +{ + /// + /// Nameplate identity of a Vision sensor (§5.4). + /// + public sealed record VisionSensorIdentity + { + /// + /// The sensor NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The Server-unique sensor identifier from VisionSensorType.SensorId. + /// + public string? SensorId { get; init; } + + /// + /// Whether the sensor is Physical, Simulated or Hybrid (§4.3). + /// + public VisionRealityKindEnum RealityKind { get; init; } + + /// + /// What the sensor senses — Area2D, Depth3D, Thermal, and so on. + /// + public VisionSensorModalityEnum Modality { get; init; } + + /// + /// The sensor manufacturer, when reported. + /// + public LocalizedText Manufacturer { get; init; } = LocalizedText.Null; + + /// + /// The sensor model, when reported. + /// + public LocalizedText Model { get; init; } = LocalizedText.Null; + + /// + /// The sensor serial number, when reported. + /// + public string? SerialNumber { get; init; } + + /// + /// The transport-level device URI (for example a GigE Vision device id), when reported. + /// + public string? DeviceUri { get; init; } + + /// + /// The FrameId string of this sensor's own camera frame, when reported. + /// + public string? FrameId { get; init; } + } + + /// + /// Imaging members of an ImageSensorType (§5.5). + /// + public sealed record VisionImageSensorSnapshot + { + /// + /// Image width in pixels. + /// + public uint Width { get; init; } + + /// + /// Image height in pixels. + /// + public uint Height { get; init; } + + /// + /// The GenICam PFNC pixel format (for example Mono8, BayerRG12, + /// RGB8). + /// + public string? PixelFormat { get; init; } + + /// + /// Exposure time in microseconds, or null when not reported. + /// + public double? ExposureTime { get; init; } + + /// + /// Sensor gain, or null when not reported. + /// + public double? Gain { get; init; } + + /// + /// Acquisition frame rate in Hz, or null when not reported. + /// + public double? AcquisitionFrameRate { get; init; } + + /// + /// Camera intrinsics, when reported. uses + /// the corner-datum principal point convention of §5.12; a client bridging to + /// ROS or OpenCV subtracts 0.5 from Cx and Cy. + /// + public VisionIntrinsicsDataType? Intrinsics { get; init; } + } + + /// + /// Depth members of a Depth3DSensorType (§5.6). + /// + public sealed record VisionDepth3DSensorSnapshot + { + /// + /// The minimum valid depth in metres. + /// + public double MinDepth { get; init; } + + /// + /// The maximum valid depth in metres. + /// + public double MaxDepth { get; init; } + + /// + /// Scale factor applied to raw depth samples. + /// + public double DepthScale { get; init; } + + /// + /// Stereo baseline in metres, or 0 for non-stereo sensors. + /// + public double Baseline { get; init; } + + /// + /// Approximate points per frame of a point-cloud sensor. + /// + public uint PointsPerFrame { get; init; } + } + + /// + /// Optics description (§5.7). + /// + public sealed record VisionOpticsSnapshot + { + /// + /// The optics NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The lens focal length in millimetres, when reported. + /// + public double? FocalLength { get; init; } + + /// + /// The lens aperture (f-number), when reported. + /// + public double? Aperture { get; init; } + + /// + /// The working distance in metres, when reported. + /// + public double? WorkingDistance { get; init; } + } + + /// + /// Illumination description (§5.7). + /// + public sealed record VisionIlluminationSnapshot + { + /// + /// The illumination NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The dominant wavelength in nanometres, when reported. + /// + public double? Wavelength { get; init; } + + /// + /// The relative intensity in percent (0..100), when reported. + /// + public double? RelativeIntensity { get; init; } + } + + /// + /// A snapshot of an IntrinsicCalibrationType instance (§5.8). + /// + public sealed record VisionIntrinsicCalibrationSnapshot + { + /// + /// The calibration NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The stable calibration identifier. + /// + public string? CalibrationId { get; init; } + + /// + /// The time the calibration was performed. + /// + public DateTimeUtc PerformedAt { get; init; } + + /// + /// Whether the Server considers the calibration currently valid. A client should + /// treat an invalid calibration as unusable rather than substituting a default. + /// + public bool Valid { get; init; } + + /// + /// The residual re-projection error of the calibration. + /// + public double ResidualError { get; init; } + + /// + /// A description of the calibration method. + /// + public string? Method { get; init; } + + /// + /// The intrinsic parameters. + /// + public VisionIntrinsicsDataType? Intrinsics { get; init; } + } + + /// + /// A snapshot of an ExtrinsicCalibrationType instance (§5.8). + /// + public sealed record VisionExtrinsicCalibrationSnapshot + { + /// + /// The calibration NodeId. + /// + public required NodeId NodeId { get; init; } + + /// + /// The stable calibration identifier. + /// + public string? CalibrationId { get; init; } + + /// + /// The time the calibration was performed. + /// + public DateTimeUtc PerformedAt { get; init; } + + /// + /// Whether the Server considers the calibration currently valid. + /// + public bool Valid { get; init; } + + /// + /// The residual error of the calibration. + /// + public double ResidualError { get; init; } + + /// + /// A description of the calibration method. + /// + public string? Method { get; init; } + + /// + /// The camera-to-robot arrangement — EyeInHand, EyeToHand, + /// Fixed, or Unknown. + /// + public VisionCalibrationMountEnum Mount { get; init; } + + /// + /// The source frame NodeId of the transform (typically the camera frame). + /// + public NodeId SourceFrameId { get; init; } = NodeId.Null; + + /// + /// The target frame NodeId of the transform (typically the flange or a station + /// frame). §5.12 requires Transform.FrameId to equal the FrameId string + /// of this frame. + /// + public NodeId TargetFrameId { get; init; } = NodeId.Null; + + /// + /// The transform itself; Position is in metres, Orientation is a + /// unit quaternion ordered (x, y, z, w). + /// + public VisionPose3DDataType? Transform { get; init; } + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/DependencyInjection/OpenUsdSceneCameraCaptureServiceCollectionExtensions.cs b/src/Opc.Ua.Vision.OpenUsd/DependencyInjection/OpenUsdSceneCameraCaptureServiceCollectionExtensions.cs new file mode 100644 index 0000000000..9a48ce00b9 --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/DependencyInjection/OpenUsdSceneCameraCaptureServiceCollectionExtensions.cs @@ -0,0 +1,76 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Opc.Ua; +using Opc.Ua.Vision.OpenUsd; + +namespace Microsoft.Extensions.DependencyInjection +{ + /// + /// DI extension for registering the OpenUSD-backed implementation of + /// . Follows the same + /// convention as + /// Opc.Ua.OpenUsd.Client.Fluent.OpcUaOpenUsdConnectorBuilderExtensions: + /// live in the + /// Microsoft.Extensions.DependencyInjection namespace so the + /// call surfaces on any . + /// + public static class OpenUsdSceneCameraCaptureServiceCollectionExtensions + { + /// + /// Registers as the + /// singleton . The device + /// probe runs when the provider is first resolved, not at + /// registration time, so this call is safe on hosts where no + /// graphics backend is available - the resolved provider will + /// simply report + /// on every capture. + /// + /// is null. + public static IServiceCollection AddOpenUsdSceneCameraCaptureProvider( + this IServiceCollection services, + Action? configure = null) + { + if (services is null) + { + throw new ArgumentNullException(nameof(services)); + } + var options = new OpenUsdSceneCaptureOptions(); + configure?.Invoke(options); + services.TryAddSingleton(options); + services.TryAddSingleton(sp => + new OpenUsdSceneCameraCaptureProvider( + sp.GetRequiredService(), + sp.GetService())); + return services; + } + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/Encoding/PngEncoder.cs b/src/Opc.Ua.Vision.OpenUsd/Encoding/PngEncoder.cs new file mode 100644 index 0000000000..4caee3497b --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/Encoding/PngEncoder.cs @@ -0,0 +1,218 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; + +namespace Opc.Ua.Vision.OpenUsd.Encoding +{ + /// + /// Minimal pure-managed PNG encoder for 8-bit RGBA images. Emits a + /// single IHDR, an IDAT wrapping a zlib stream of DEFLATE stored blocks + /// (uncompressed - trades size for zero dependencies), and IEND. Output + /// is byte-identical to what the probe's PngWriter produced; every PNG + /// reader in the wild accepts it. + /// + /// + /// The Vision use case is a fresh render per frame, so encode time is + /// dominated by network / OPC UA framing anyway. When smaller payloads + /// matter the caller can plug a different encoder in behind + /// ; the interface never + /// exposes the encoder. + /// + internal static class PngEncoder + { + private static readonly byte[] s_signature = + [ + 0x89, (byte)'P', (byte)'N', (byte)'G', 0x0D, 0x0A, 0x1A, 0x0A + ]; + + /// + /// Encodes an RGBA8 top-down pixel buffer + /// as a PNG and returns the bytes. + /// + /// + /// or is not positive. + /// + /// + /// is not exactly width * height * 4 bytes. + /// + public static byte[] EncodeRgba8(int width, int height, ReadOnlySpan rgba) + { + if (width <= 0) + { + throw new ArgumentOutOfRangeException(nameof(width)); + } + if (height <= 0) + { + throw new ArgumentOutOfRangeException(nameof(height)); + } + int expected = checked(width * height * 4); + if (rgba.Length != expected) + { + throw new ArgumentException( + $"rgba length {rgba.Length} != width*height*4 ({expected}).", + nameof(rgba)); + } + + using var ms = new MemoryStream(expected + 512); + ms.Write(s_signature, 0, s_signature.Length); + + Span ihdr = stackalloc byte[13]; + WriteUInt32BE(ihdr, 0, (uint)width); + WriteUInt32BE(ihdr, 4, (uint)height); + ihdr[8] = 8; + ihdr[9] = 6; + ihdr[10] = 0; + ihdr[11] = 0; + ihdr[12] = 0; + WriteChunk(ms, "IHDR", ihdr); + + int rowBytes = width * 4; + int filteredSize = height * (rowBytes + 1); + byte[] filtered = new byte[filteredSize]; + for (int y = 0; y < height; y++) + { + int srcOff = y * rowBytes; + int dstOff = y * (rowBytes + 1); + filtered[dstOff] = 0; + rgba.Slice(srcOff, rowBytes).CopyTo(filtered.AsSpan(dstOff + 1, rowBytes)); + } + byte[] zlib = ZlibWrapStored(filtered); + WriteChunk(ms, "IDAT", zlib); + + WriteChunk(ms, "IEND", ReadOnlySpan.Empty); + return ms.ToArray(); + } + + private static byte[] ZlibWrapStored(byte[] payload) + { + using var ms = new MemoryStream(payload.Length + 32); + ms.WriteByte(0x78); + ms.WriteByte(0x01); + + int offset = 0; + while (offset < payload.Length) + { + int chunk = Math.Min(65535, payload.Length - offset); + bool final = offset + chunk >= payload.Length; + ms.WriteByte((byte)(final ? 1 : 0)); + ms.WriteByte((byte)(chunk & 0xFF)); + ms.WriteByte((byte)((chunk >> 8) & 0xFF)); + int nlen = ~chunk & 0xFFFF; + ms.WriteByte((byte)(nlen & 0xFF)); + ms.WriteByte((byte)((nlen >> 8) & 0xFF)); + ms.Write(payload, offset, chunk); + offset += chunk; + } + + uint adler = Adler32(payload); + ms.WriteByte((byte)((adler >> 24) & 0xFF)); + ms.WriteByte((byte)((adler >> 16) & 0xFF)); + ms.WriteByte((byte)((adler >> 8) & 0xFF)); + ms.WriteByte((byte)(adler & 0xFF)); + return ms.ToArray(); + } + + private static uint Adler32(ReadOnlySpan data) + { + const uint mod = 65521; + uint a = 1; + uint b = 0; + for (int i = 0; i < data.Length; i++) + { + a = (a + data[i]) % mod; + b = (b + a) % mod; + } + return (b << 16) | a; + } + + private static void WriteChunk(Stream s, string type, ReadOnlySpan data) + { + Span len = stackalloc byte[4]; + WriteUInt32BE(len, 0, (uint)data.Length); + s.Write(len); + + Span typeBytes = stackalloc byte[4]; + System.Text.Encoding.ASCII.GetBytes(type, typeBytes); + s.Write(typeBytes); + if (data.Length > 0) + { + s.Write(data); + } + + uint crc = Crc32.Compute(typeBytes, data); + Span crcBytes = stackalloc byte[4]; + WriteUInt32BE(crcBytes, 0, crc); + s.Write(crcBytes); + } + + private static void WriteUInt32BE(Span buf, int offset, uint value) + { + buf[offset] = (byte)((value >> 24) & 0xFF); + buf[offset + 1] = (byte)((value >> 16) & 0xFF); + buf[offset + 2] = (byte)((value >> 8) & 0xFF); + buf[offset + 3] = (byte)(value & 0xFF); + } + + private static class Crc32 + { + private static readonly uint[] s_table = BuildTable(); + + private static uint[] BuildTable() + { + uint[] t = new uint[256]; + for (uint n = 0; n < 256; n++) + { + uint c = n; + for (int k = 0; k < 8; k++) + { + c = (c & 1) != 0 ? (0xEDB88320u ^ (c >> 1)) : (c >> 1); + } + t[n] = c; + } + return t; + } + + public static uint Compute(ReadOnlySpan a, ReadOnlySpan b) + { + uint c = 0xFFFFFFFFu; + for (int i = 0; i < a.Length; i++) + { + c = s_table[(c ^ a[i]) & 0xFF] ^ (c >> 8); + } + for (int i = 0; i < b.Length; i++) + { + c = s_table[(c ^ b[i]) & 0xFF] ^ (c >> 8); + } + return c ^ 0xFFFFFFFFu; + } + } + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/EventIds.cs b/src/Opc.Ua.Vision.OpenUsd/EventIds.cs new file mode 100644 index 0000000000..2a5434239e --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/EventIds.cs @@ -0,0 +1,42 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// Centrally managed event id offsets for the source-generated log + /// messages of the Opc.Ua.Vision.OpenUsd assembly. Each per-class + /// <ClassName>Log class allocates its event ids relative + /// to the offset constant below. + /// + internal static class VisionOpenUsdEventIds + { + public const int CaptureProvider = 0; + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/ISceneCameraCaptureProvider.cs b/src/Opc.Ua.Vision.OpenUsd/ISceneCameraCaptureProvider.cs new file mode 100644 index 0000000000..04627dfd1b --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/ISceneCameraCaptureProvider.cs @@ -0,0 +1,74 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Threading; +using System.Threading.Tasks; + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// Renders a stage camera view and returns an encoded image plus the + /// metadata a Vision ClipEndpointType needs (dimensions, format, + /// timestamp). The Vision server takes a dependency on this interface + /// so it does not have to know that any specific rendering technology + /// is involved; a host with no rendering support at all can bind a + /// no-op implementation that always returns + /// . + /// + /// + /// Implementations must be safe to call concurrently from many callers; + /// they may serialize access to a graphics device internally. + /// + public interface ISceneCameraCaptureProvider + { + /// + /// Describes the graphics backend the provider will use to fulfill + /// requests. Cheap to read: probed once and cached at construction. + /// When is + /// false every call to + /// returns + /// . + /// + SceneCameraCaptureBackend Backend { get; } + + /// + /// Captures the requested camera view and returns an encoded image, + /// or a whose + /// describes why no + /// image was produced. The implementation never throws for + /// input-driven failures (missing stage, missing prim, no backend, + /// blank frame) - those become status codes instead - and never + /// returns a blank frame as if it succeeded. Cancellation surfaces + /// as . + /// + ValueTask CaptureAsync( + SceneCameraCaptureRequest request, + CancellationToken cancellationToken); + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/NugetREADME.md b/src/Opc.Ua.Vision.OpenUsd/NugetREADME.md new file mode 100644 index 0000000000..3b81d02cff --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/NugetREADME.md @@ -0,0 +1,98 @@ +# OPCFoundation.NetStandard.Opc.Ua.Vision.OpenUsd + +Offscreen renderer for the draft OPC UA Vision companion server. + +This package renders a USD stage camera's view through the OpenUSD Silk backend +(D3D12 hardware, D3D12 WARP software, or Vulkan headless) and returns encoded +PNG frames through a small `ISceneCameraCaptureProvider` abstraction. The +Vision server can depend on the abstraction without referencing OpenUSD, and +degrades to a non-rendering sensor when this optional package is not present. + +## What it does + +The `Opc.Ua.Vision.OpenUsd.OpenUsdSceneCameraCaptureProvider` fulfills a +capture request by: + +1. Opening the USD stage from the supplied path or identifier. +2. Resolving the camera prim (view + off-centre projection built from + `UsdGeomCamera.GetState(time)` and `GetTransform(time)`) - or rendering + the automatic default framing when no prim path is supplied. +3. Creating a fresh `OpenUsdSilkSession` for the request (the SDK's + reuse-with-different-camera path is known to silently render nothing). +4. Capturing an RGBA8 frame with `SilkFrameCapture.Capture` on a shared + `ISilkGraphicsDevice` picked at construction time. +5. Encoding to PNG with an in-repo, dependency-free encoder. +6. Refusing to hand back an all-zero / no-mesh frame as if it succeeded - + returning `SceneCameraCaptureStatus.BlankFrame` with a reason instead. + +## Device selection + +`OpenUsdSceneCameraCaptureProvider` tries backends in the order that makes +sense for the host: + +- **Windows**: D3D12 hardware -> D3D12 WARP (software rasterizer, no GPU) -> Vulkan. +- **Linux / other**: Vulkan (which the OpenUSD runtime bundles the SwiftShader + software ICD for, so CI without a GPU is still functional). + +The provider reports the chosen backend and whether it is a software renderer +through `Backend`, and when no backend is available reports +`IsAvailable = false` with a reason. In that case every `CaptureAsync` returns +`SceneCameraCaptureStatus.NoRenderingBackend`. + +## Registration + +```csharp +using Opc.Ua.Vision.OpenUsd; +using Microsoft.Extensions.DependencyInjection; + +services.AddOpenUsdSceneCameraCaptureProvider(o => +{ + o.PluginPath = "/path/to/plugin/usd"; // optional; auto-probes AppContext.BaseDirectory + o.PreferSoftware = false; +}); +``` + +or without DI: + +```csharp +using var provider = new OpenUsdSceneCameraCaptureProvider( + new OpenUsdSceneCaptureOptions(), telemetry: null); +SceneCameraCaptureResult result = await provider.CaptureAsync( + new SceneCameraCaptureRequest + { + StageIdentifier = "/path/to/scene.usda", + PrimPath = "/World/Cam", + Width = 640, + Height = 360, + TimeCode = 0.0, + Format = SceneCameraImageFormat.Png, + }, + cancellationToken); +``` + +## Native payload + +The OpenUSD runtime packages ship RID-specific native assets (`win-x64`, +`linux-x64`, `osx-arm64`). Publish the *consuming application* with an +explicit `RuntimeIdentifier` (e.g. `dotnet publish -r linux-x64`) so the +`plugin/usd/` tree and the backend native libraries land alongside the +executable. The provider then auto-discovers them from `AppContext.BaseDirectory`. + +When the payload is absent — the normal case on unadorned CI legs — the +provider still starts, `Backend.IsAvailable` reports `false` and every +capture returns `SceneCameraCaptureStatus.NoRenderingBackend`. This is the +degrade path the [Vision developer guide](https://github.com/OPCFoundation/UA-.NETStandard/blob/main/docs/Vision.md#rendering-degrades-rather-than-throwing) +describes: the sensor stays visible in the address space, browses still +work, and only the pixel bytes are absent, so a client can distinguish "no +GPU" from a genuine rendering fault. + +## Related packages + +| Package | Adds | +|---|---| +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Server` | The Vision server that consumes `ISceneCameraCaptureProvider` | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Client` | The client that reads the simulated sensor's `LatestClip` / `GetClip` | + +## License + +OPC Foundation MIT License 1.00 — diff --git a/src/Opc.Ua.Vision.OpenUsd/Opc.Ua.Vision.OpenUsd.csproj b/src/Opc.Ua.Vision.OpenUsd/Opc.Ua.Vision.OpenUsd.csproj new file mode 100644 index 0000000000..48c391f9ca --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/Opc.Ua.Vision.OpenUsd.csproj @@ -0,0 +1,49 @@ + + + + net10.0;net9.0;net8.0 + $(CustomTestTarget) + $(AssemblyPrefix).Vision.OpenUsd + $(PackagePrefix).Opc.Ua.Vision.OpenUsd + Opc.Ua.Vision.OpenUsd + $(NoWarn);CS1591 + enable + Offscreen renderer for the draft OPC UA Vision companion server: renders a stage camera prim's view through the OpenUSD Silk backend (D3D12 hardware, D3D12 WARP software, or Vulkan headless), returns encoded PNG frames through a small ISceneCameraCaptureProvider abstraction, and reports a clear result when no graphics backend is available so the Vision server can degrade to a non-rendering sensor. + true + NugetREADME.md + true + + false + + true + + + $(PackageId).Debug + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/Opc.Ua.Vision.OpenUsd/OpenUsdCaptureLog.cs b/src/Opc.Ua.Vision.OpenUsd/OpenUsdCaptureLog.cs new file mode 100644 index 0000000000..9165df615d --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/OpenUsdCaptureLog.cs @@ -0,0 +1,88 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Microsoft.Extensions.Logging; + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// Source-generated log messages for + /// . + /// + internal static partial class OpenUsdCaptureLog + { + [LoggerMessage(EventId = VisionOpenUsdEventIds.CaptureProvider + 0, Level = LogLevel.Information, + Message = "OpenUSD capture backend '{BackendName}' selected (device='{DeviceName}', software={IsSoftware}).")] + public static partial void BackendSelected( + this ILogger logger, string backendName, string deviceName, bool isSoftware); + + [LoggerMessage(EventId = VisionOpenUsdEventIds.CaptureProvider + 1, Level = LogLevel.Warning, + Message = "OpenUSD capture backend '{BackendName}' unavailable ({Reason}). Trying next backend.")] + public static partial void BackendUnavailable( + this ILogger logger, string backendName, string reason); + + [LoggerMessage(EventId = VisionOpenUsdEventIds.CaptureProvider + 2, Level = LogLevel.Warning, + Message = "No OpenUSD capture backend is available on this host: {Reason}. " + + "Captures will report NoRenderingBackend so the Vision server can degrade.")] + public static partial void NoBackendAvailable(this ILogger logger, string reason); + + [LoggerMessage(EventId = VisionOpenUsdEventIds.CaptureProvider + 3, Level = LogLevel.Debug, + Message = "Captured {Width}x{Height} PNG in {ElapsedMs} ms " + + "(drawCount={DrawCount}, meshCount={MeshCount}, backend={BackendName}).")] + public static partial void CaptureSucceeded(this ILogger logger, + int width, int height, long elapsedMs, int drawCount, int meshCount, string backendName); + + [LoggerMessage(EventId = VisionOpenUsdEventIds.CaptureProvider + 4, Level = LogLevel.Warning, + Message = "OpenUSD capture rendered no geometry (drawCount={DrawCount}, meshCount={MeshCount}, uniform={IsUniform}). " + + "Surfacing BlankFrame so the caller does not serve a misleading picture.")] + public static partial void BlankFrameDetected( + this ILogger logger, int drawCount, int meshCount, bool isUniform); + + [LoggerMessage(EventId = VisionOpenUsdEventIds.CaptureProvider + 5, Level = LogLevel.Warning, + Message = "Failed to open USD stage '{StageIdentifier}'.")] + public static partial void StageOpenFailed( + this ILogger logger, string stageIdentifier, Exception exception); + + [LoggerMessage(EventId = VisionOpenUsdEventIds.CaptureProvider + 6, Level = LogLevel.Warning, + Message = "Failed to resolve camera prim '{PrimPath}' on stage '{StageIdentifier}'.")] + public static partial void CameraResolveFailed( + this ILogger logger, string primPath, string stageIdentifier, Exception exception); + + [LoggerMessage(EventId = VisionOpenUsdEventIds.CaptureProvider + 7, Level = LogLevel.Warning, + Message = "SilkFrameCapture.Capture threw on backend '{BackendName}'.")] + public static partial void RenderFailed( + this ILogger logger, string backendName, Exception exception); + + [LoggerMessage(EventId = VisionOpenUsdEventIds.CaptureProvider + 8, Level = LogLevel.Warning, + Message = "PNG encoding failed for a {Width}x{Height} RGBA8 frame.")] + public static partial void EncodingFailed( + this ILogger logger, int width, int height, Exception exception); + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/OpenUsdSceneCameraCaptureProvider.cs b/src/Opc.Ua.Vision.OpenUsd/OpenUsdSceneCameraCaptureProvider.cs new file mode 100644 index 0000000000..53847e328d --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/OpenUsdSceneCameraCaptureProvider.cs @@ -0,0 +1,433 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Diagnostics; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua.Vision.OpenUsd.Encoding; +using Opc.Ua.Vision.OpenUsd.Rendering; +using OpenUsd; +using OpenUsd.Rendering; +using OpenUsd.Rendering.Silk; + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// backed by the OpenUSD + /// Silk rendering stack. Probes a graphics device on construction and + /// then serves capture requests by opening the requested stage, resolving + /// the camera prim to a CameraState, rendering through a retained + /// SilkFrameCapturer, and encoding the RGBA8 buffer as PNG. + /// + /// + /// A capturer retains the scene between captures, which is what makes + /// repeated rendering from one session correct. Before OpenUSD + /// 0.8.0-alpha the one-shot SilkFrameCapture.Capture silently + /// returned an all-zero frame on any second capture, indistinguishable + /// from a black scene, and this provider worked around it by building a + /// session per request. That defect is fixed upstream + /// (openusd-dotnet#13), so the workaround is gone. The blank-frame guard + /// stays: a frame with no draws is still worth refusing rather than + /// serving as though it were a picture of nothing. + /// + public sealed class OpenUsdSceneCameraCaptureProvider : ISceneCameraCaptureProvider, IDisposable + { + /// + /// Initializes a provider with default options and no telemetry. + /// + public OpenUsdSceneCameraCaptureProvider() + : this(new OpenUsdSceneCaptureOptions(), telemetry: null) + { + } + + /// + /// Initializes a provider with the supplied options, threading the + /// host's for source-generated + /// logging. The device probe runs synchronously here so + /// is populated by the time the constructor + /// returns. + /// + /// is null. + public OpenUsdSceneCameraCaptureProvider( + OpenUsdSceneCaptureOptions options, ITelemetryContext? telemetry) + { + m_options = options ?? throw new ArgumentNullException(nameof(options)); + m_logger = telemetry.CreateLogger(); + PluginPath = ResolvePluginPath(options.PluginPath); + + if (DeviceSelector.TrySelectDevice( + options, m_logger, out SelectedSilkDevice selected, out string reason)) + { + m_device = selected.Device; + Backend = selected.Backend; + m_backendUnavailableReason = null; + } + else + { + m_device = null; + m_backendUnavailableReason = reason; + Backend = new SceneCameraCaptureBackend + { + Name = "None", + IsAvailable = false, + IsSoftware = false, + UnavailableReason = reason + }; + } + } + + /// + public SceneCameraCaptureBackend Backend { get; } + + /// + /// The plugin path the provider probed for on construction; useful + /// for a diagnostic /health endpoint. + /// + public string? PluginPath { get; } + + /// + public async ValueTask CaptureAsync( + SceneCameraCaptureRequest request, + CancellationToken cancellationToken) + { + if (request is null) + { + throw new ArgumentNullException(nameof(request)); + } + ThrowIfDisposed(); + cancellationToken.ThrowIfCancellationRequested(); + + long start = Stopwatch.GetTimestamp(); + DateTime timestamp = request.TimestampUtc ?? DateTime.UtcNow; + + SceneCameraCaptureResult? validation = ValidateRequest(request, timestamp, start); + if (validation is not null) + { + return validation; + } + if (m_device is null) + { + return NoBackend(request, timestamp, start); + } + + await m_captureGate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + return await Task.Run( + () => CaptureCore(request, timestamp, start, cancellationToken), + cancellationToken).ConfigureAwait(false); + } + finally + { + m_captureGate.Release(); + } + } + + /// + /// Disposes the shared graphics device and internal synchronization + /// primitive. In-flight captures are allowed to finish; further + /// calls throw . + /// + public void Dispose() + { + if (Interlocked.Exchange(ref m_disposed, 1) != 0) + { + return; + } + m_captureGate.Dispose(); + + // The capturer holds the retained scene and must go before the session it rendered + // from; the device outlives both. + m_capturer?.Dispose(); + m_session?.Dispose(); + m_capturer = null; + m_session = null; + m_sessionStageIdentifier = null; + m_device?.Dispose(); + } + + private SceneCameraCaptureResult CaptureCore( + SceneCameraCaptureRequest request, + DateTime timestamp, + long start, + CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + + UsdStage? stage = null; + try + { + try + { + stage = UsdStage.Open(request.StageIdentifier); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + m_logger.StageOpenFailed(request.StageIdentifier, ex); + return Failure(SceneCameraCaptureStatus.StageOpenFailed, + $"UsdStage.Open('{request.StageIdentifier}') failed: {ex.Message}", + request, timestamp, start); + } + + CameraState camera; + try + { + // CameraState.FromStageCamera landed in OpenUSD 0.8.0-alpha (openusd-dotnet#14), + // so the projection maths is the package's own rather than derived here from + // the prim's window and clipping values. + camera = string.IsNullOrEmpty(request.PrimPath) + ? CameraState.Default + : CameraState.FromStageCamera( + stage, request.PrimPath!, request.TimeCode, request.Width, request.Height); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + m_logger.CameraResolveFailed(request.PrimPath ?? string.Empty, + request.StageIdentifier, ex); + return Failure(SceneCameraCaptureStatus.CameraResolveFailed, + $"Resolving camera prim '{request.PrimPath}' failed: {ex.Message}", + request, timestamp, start); + } + + try + { + // A capturer retains the scene between captures, so the session and the + // capturer are kept for as long as the requests keep naming the same stage. + // Rebuilding them per request would discard that scene and pay the stage-open + // cost every perception cycle. + if (m_session is null || + !string.Equals(m_sessionStageIdentifier, request.StageIdentifier, StringComparison.Ordinal)) + { + m_capturer?.Dispose(); + m_session?.Dispose(); + m_capturer = null; + m_session = null; + m_sessionStageIdentifier = null; + + m_session = PluginPath is null + ? OpenUsdSilkRuntime.Create(string.Empty, request.StageIdentifier) + : OpenUsdSilkRuntime.Create(PluginPath, request.StageIdentifier); + m_capturer = new SilkFrameCapturer(m_device!); + m_sessionStageIdentifier = request.StageIdentifier; + } + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + m_capturer?.Dispose(); + m_session?.Dispose(); + m_capturer = null; + m_session = null; + m_sessionStageIdentifier = null; + m_logger.RenderFailed(Backend.Name, ex); + return Failure(SceneCameraCaptureStatus.RenderFailed, + $"OpenUsdSilkRuntime.Create failed: {ex.Message}", + request, timestamp, start); + } + + cancellationToken.ThrowIfCancellationRequested(); + + SilkFrameCaptureResult frame; + try + { + frame = m_capturer!.Capture( + m_session!, request.Width, request.Height, request.TimeCode, camera); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + m_logger.RenderFailed(Backend.Name, ex); + return Failure(SceneCameraCaptureStatus.RenderFailed, + $"SilkFrameCapturer.Capture failed on {Backend.Name}: {ex.Message}", + request, timestamp, start); + } + + BlankFrameCheck guard = BlankFrameGuard.Check(frame); + if (guard.IsBlank) + { + m_logger.BlankFrameDetected(guard.DrawCount, guard.MeshCount, guard.IsUniform); + return Failure(SceneCameraCaptureStatus.BlankFrame, + guard.Reason ?? "render produced no visible geometry", + request, timestamp, start, frame.Width, frame.Height); + } + + byte[] png; + try + { + png = PngEncoder.EncodeRgba8(frame.Width, frame.Height, frame.Rgba.Span); + } + catch (Exception ex) when (ex is not OperationCanceledException and not IOException) + { + m_logger.EncodingFailed(frame.Width, frame.Height, ex); + return Failure(SceneCameraCaptureStatus.EncodingFailed, + $"PNG encoding failed: {ex.Message}", + request, timestamp, start, frame.Width, frame.Height); + } + + TimeSpan elapsed = Stopwatch.GetElapsedTime(start); + m_logger.CaptureSucceeded(frame.Width, frame.Height, (long)elapsed.TotalMilliseconds, + guard.DrawCount, guard.MeshCount, Backend.Name); + return new SceneCameraCaptureResult + { + Status = SceneCameraCaptureStatus.Succeeded, + Reason = null, + Image = ByteString.From(png), + Format = request.Format, + Width = frame.Width, + Height = frame.Height, + TimestampUtc = timestamp, + Elapsed = elapsed, + Backend = Backend + }; + } + finally + { + stage?.Dispose(); + } + } + + private static bool IsLocalPath(string stageIdentifier) + { + // Only a plain filesystem path is checkable here. Anything carrying a URI scheme + // (an asset-resolver path, a remote stage) is left to the native resolver. + return !Uri.TryCreate(stageIdentifier, UriKind.Absolute, out Uri? uri) || uri.IsFile; + } + + private SceneCameraCaptureResult? ValidateRequest( + SceneCameraCaptureRequest request, DateTime timestamp, long start) + { + if (string.IsNullOrWhiteSpace(request.StageIdentifier)) + { + return Failure(SceneCameraCaptureStatus.InvalidRequest, + "StageIdentifier must not be empty.", request, timestamp, start); + } + + // A stage identifier that names no readable file has been observed to tear the + // process down inside the native UsdStage.Open rather than returning an error, and + // an AccessViolationException cannot be caught here. Reject it in managed code so a + // bad request costs a failed capture rather than the whole server. + if (IsLocalPath(request.StageIdentifier) && !File.Exists(request.StageIdentifier)) + { + return Failure(SceneCameraCaptureStatus.InvalidRequest, + $"StageIdentifier '{request.StageIdentifier}' does not name a readable file.", + request, timestamp, start); + } + if (request.Width <= 0 || request.Height <= 0) + { + return Failure(SceneCameraCaptureStatus.InvalidRequest, + $"Width and height must be positive (got {request.Width}x{request.Height}).", + request, timestamp, start); + } + if (request.Width > m_options.MaxFrameWidth || request.Height > m_options.MaxFrameHeight) + { + return Failure(SceneCameraCaptureStatus.InvalidRequest, + $"Requested frame size {request.Width}x{request.Height} exceeds the configured maximum " + + $"{m_options.MaxFrameWidth}x{m_options.MaxFrameHeight}.", + request, timestamp, start); + } + if (request.Format != SceneCameraImageFormat.Png) + { + return Failure(SceneCameraCaptureStatus.InvalidRequest, + $"Image format {request.Format} is not supported by this provider.", + request, timestamp, start); + } + return null; + } + + private SceneCameraCaptureResult NoBackend( + SceneCameraCaptureRequest request, DateTime timestamp, long start) + { + return new SceneCameraCaptureResult + { + Status = SceneCameraCaptureStatus.NoRenderingBackend, + Reason = m_backendUnavailableReason + ?? "No graphics backend is available on this host.", + Image = default, + Format = request.Format, + Width = 0, + Height = 0, + TimestampUtc = timestamp, + Elapsed = Stopwatch.GetElapsedTime(start), + Backend = Backend + }; + } + + private SceneCameraCaptureResult Failure( + SceneCameraCaptureStatus status, + string reason, + SceneCameraCaptureRequest request, + DateTime timestamp, + long start, + int width = 0, + int height = 0) + { + return new SceneCameraCaptureResult + { + Status = status, + Reason = reason, + Image = default, + Format = request.Format, + Width = width == 0 ? request.Width : width, + Height = height == 0 ? request.Height : height, + TimestampUtc = timestamp, + Elapsed = Stopwatch.GetElapsedTime(start), + Backend = Backend + }; + } + + private static string? ResolvePluginPath(string? configured) + { + if (!string.IsNullOrWhiteSpace(configured)) + { + return Directory.Exists(configured) ? configured : null; + } + string candidate = Path.Combine(AppContext.BaseDirectory, "plugin", "usd"); + return Directory.Exists(candidate) ? candidate : null; + } + + private void ThrowIfDisposed() + { + if (Volatile.Read(ref m_disposed) != 0) + { + throw new ObjectDisposedException(nameof(OpenUsdSceneCameraCaptureProvider)); + } + } + + private readonly OpenUsdSceneCaptureOptions m_options; + private readonly ILogger m_logger; + private readonly ISilkGraphicsDevice? m_device; + private readonly string? m_backendUnavailableReason; + private readonly SemaphoreSlim m_captureGate = new(1, 1); + private OpenUsdSilkSession? m_session; + private SilkFrameCapturer? m_capturer; + private string? m_sessionStageIdentifier; + private int m_disposed; + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/OpenUsdSceneCaptureOptions.cs b/src/Opc.Ua.Vision.OpenUsd/OpenUsdSceneCaptureOptions.cs new file mode 100644 index 0000000000..ff6f6700c4 --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/OpenUsdSceneCaptureOptions.cs @@ -0,0 +1,84 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// Configuration for . + /// The defaults are chosen so a Vision server host does nothing beyond + /// registering the provider - the OpenUSD plugin tree is auto-discovered + /// from AppContext.BaseDirectory and the best available graphics + /// backend is picked at construction time. + /// + public sealed record class OpenUsdSceneCaptureOptions + { + /// + /// Absolute path to the OpenUSD plugin tree (plugin/usd). + /// When null the provider probes {AppContext.BaseDirectory}/plugin/usd + /// (the layout the OpenUSD runtime packages stage for the RID the + /// host was published with) and finally falls back to no plugin path + /// at all, in which case UsdStage.Open only understands the + /// built-in file formats. + /// + public string? PluginPath { get; init; } + + /// + /// When true the provider prefers the D3D12 WARP software + /// rasterizer over D3D12 hardware. Useful for CI / diagnostics + /// where the test needs a deterministic backend independent of the + /// host GPU. On non-Windows hosts this flag has no effect (only the + /// Vulkan backend is tried, which itself falls back to a software + /// ICD when no hardware Vulkan loader is present). + /// + public bool PreferSoftware { get; init; } + + /// + /// When true the provider falls back to a software backend + /// (D3D12 WARP on Windows, or a Vulkan software ICD) if no hardware + /// backend can be created. Defaults to true; set to false + /// only when a caller explicitly requires hardware acceleration and + /// would rather fail fast than serve WARP frames. + /// + public bool AllowSoftwareFallback { get; init; } = true; + + /// + /// Maximum frame width, in pixels, that + /// will honour. Requests larger than this are rejected with + /// to keep a + /// single capture from monopolising a shared software rasterizer. + /// Defaults to 8192. + /// + public int MaxFrameWidth { get; init; } = 8192; + + /// + /// Maximum frame height, in pixels. See . + /// + public int MaxFrameHeight { get; init; } = 8192; + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/Properties/AssemblyInfo.cs b/src/Opc.Ua.Vision.OpenUsd/Properties/AssemblyInfo.cs new file mode 100644 index 0000000000..2b9848014c --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/Properties/AssemblyInfo.cs @@ -0,0 +1,32 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +[assembly: CLSCompliant(false)] diff --git a/src/Opc.Ua.Vision.OpenUsd/Rendering/BlankFrameGuard.cs b/src/Opc.Ua.Vision.OpenUsd/Rendering/BlankFrameGuard.cs new file mode 100644 index 0000000000..35677c95ca --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/Rendering/BlankFrameGuard.cs @@ -0,0 +1,126 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using OpenUsd.Rendering.Silk; + +namespace Opc.Ua.Vision.OpenUsd.Rendering +{ + /// + /// Result of a check. + /// + internal readonly record struct BlankFrameCheck( + bool IsBlank, + int DrawCount, + int MeshCount, + bool IsUniform, + string? Reason); + + /// + /// Detects the "silently rendered nothing" failure mode of the OpenUSD + /// Silk backend. The probe report calls this out as the worst possible + /// failure mode for a vision system, so the provider refuses to surface + /// a blank frame as if it succeeded. + /// + /// + /// The check is defensive in depth: even though every capture uses a + /// fresh session (which avoids the known session-reuse landmine), the + /// guard runs anyway so a future backend regression or an unrelated + /// crash-into-black cannot go unreported. + /// + internal static class BlankFrameGuard + { + /// + /// Inspects and its RGBA8 pixel buffer. + /// Reports true when + /// either the render pipeline drew nothing, or every pixel in the + /// buffer has the exact same value. + /// + /// + public static BlankFrameCheck Check(SilkFrameCaptureResult result) + { + if (result is null) + { + throw new ArgumentNullException(nameof(result)); + } + + int drawCount = result.RenderResult.DrawCount; + int meshCount = result.RenderResult.Statistics.MeshCount; + + if (drawCount == 0 || meshCount == 0) + { + return new BlankFrameCheck( + IsBlank: true, + DrawCount: drawCount, + MeshCount: meshCount, + IsUniform: false, + Reason: "render pipeline reported no drawn geometry " + + $"(drawCount={drawCount}, meshCount={meshCount})"); + } + + bool uniform = IsUniformRgba8(result.Rgba.Span); + if (uniform) + { + return new BlankFrameCheck( + IsBlank: true, + DrawCount: drawCount, + MeshCount: meshCount, + IsUniform: true, + Reason: "every pixel in the returned RGBA8 buffer has the same value"); + } + + return new BlankFrameCheck(false, drawCount, meshCount, false, null); + } + + private static bool IsUniformRgba8(ReadOnlySpan rgba) + { + if (rgba.Length < 4) + { + return true; + } + uint first = ReadRgba(rgba, 0); + for (int i = 4; i < rgba.Length; i += 4) + { + if (ReadRgba(rgba, i) != first) + { + return false; + } + } + return true; + } + + private static uint ReadRgba(ReadOnlySpan rgba, int offset) + { + return ((uint)rgba[offset] << 24) | + ((uint)rgba[offset + 1] << 16) | + ((uint)rgba[offset + 2] << 8) | + rgba[offset + 3]; + } + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/Rendering/DeviceSelector.cs b/src/Opc.Ua.Vision.OpenUsd/Rendering/DeviceSelector.cs new file mode 100644 index 0000000000..a3de77d009 --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/Rendering/DeviceSelector.cs @@ -0,0 +1,198 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Runtime.Versioning; +using Microsoft.Extensions.Logging; +using OpenUsd.Rendering.Silk; +using OpenUsd.Rendering.Silk.D3D12; +using OpenUsd.Rendering.Silk.Vulkan; + +namespace Opc.Ua.Vision.OpenUsd.Rendering +{ + /// + /// Result of a single successful device probe. + /// + internal readonly record struct SelectedSilkDevice( + ISilkGraphicsDevice Device, + SceneCameraCaptureBackend Backend); + + /// + /// Picks the best available for the + /// host. Order is fixed by : + /// + /// Windows: D3D12 hardware -> D3D12 WARP -> Vulkan. + /// When is set, + /// WARP is tried first. + /// Non-Windows: Vulkan only. The OpenUSD runtime + /// packages bundle SwiftShader on linux-x64, so this succeeds even on + /// a CI host without a GPU. + /// + /// The probe order is deliberately tolerant: every backend that throws + /// during Create is caught and turned into a + /// warning, and the + /// next backend is tried. + /// + internal static class DeviceSelector + { + public static bool TrySelectDevice( + OpenUsdSceneCaptureOptions options, + ILogger logger, + out SelectedSilkDevice selected, + out string aggregateReason) + { + if (options is null) + { + throw new ArgumentNullException(nameof(options)); + } + if (logger is null) + { + throw new ArgumentNullException(nameof(logger)); + } + + List probes = BuildProbeOrder(options); + List reasons = []; + foreach (BackendProbe probe in probes) + { + ISilkGraphicsDevice? device = null; + try + { + device = probe.Factory(); + } + catch (Exception ex) when (ex is not OutOfMemoryException) + { + string reason = FormatException(ex); + reasons.Add($"{probe.Name}: {reason}"); + logger.BackendUnavailable(probe.Name, reason); + continue; + } + + SilkGraphicsCapabilities caps; + try + { + caps = device.Capabilities; + } + catch (Exception ex) when (ex is not OutOfMemoryException) + { + string reason = FormatException(ex); + reasons.Add($"{probe.Name}: {reason}"); + logger.BackendUnavailable(probe.Name, reason); + device.Dispose(); + continue; + } + + if (!options.AllowSoftwareFallback && caps.IsSoftware && !probe.PreferredAsSoftware) + { + const string reason = "created a software device but AllowSoftwareFallback is false"; + reasons.Add($"{probe.Name}: {reason}"); + logger.BackendUnavailable(probe.Name, reason); + device.Dispose(); + continue; + } + + var backend = new SceneCameraCaptureBackend + { + Name = probe.Name, + DeviceName = caps.DeviceName ?? string.Empty, + ApiVersion = caps.ApiVersion ?? string.Empty, + IsSoftware = caps.IsSoftware, + IsAvailable = true, + UnavailableReason = null + }; + logger.BackendSelected(backend.Name, backend.DeviceName, backend.IsSoftware); + selected = new SelectedSilkDevice(device, backend); + aggregateReason = string.Empty; + return true; + } + + aggregateReason = reasons.Count == 0 + ? "no graphics backend is registered for this host" + : string.Join(" | ", reasons); + logger.NoBackendAvailable(aggregateReason); + selected = default; + return false; + } + + private static List BuildProbeOrder(OpenUsdSceneCaptureOptions options) + { + List probes = []; + if (OperatingSystem.IsWindows()) + { + AddWindowsD3D12Probes(probes, options); + } + probes.Add(new BackendProbe( + Name: "Vulkan", + Factory: () => VulkanSilkGraphicsDevice.Create(), + PreferredAsSoftware: false)); + return probes; + } + + [SupportedOSPlatform("windows")] + private static void AddWindowsD3D12Probes(List probes, OpenUsdSceneCaptureOptions options) + { + if (options.PreferSoftware) + { + probes.Add(new BackendProbe( + Name: "D3D12 (WARP)", + Factory: () => D3D12SilkGraphicsDevice.Create(useWarp: true), + PreferredAsSoftware: true)); + probes.Add(new BackendProbe( + Name: "D3D12", + Factory: () => D3D12SilkGraphicsDevice.Create(useWarp: false), + PreferredAsSoftware: false)); + return; + } + probes.Add(new BackendProbe( + Name: "D3D12", + Factory: () => D3D12SilkGraphicsDevice.Create(useWarp: false), + PreferredAsSoftware: false)); + if (options.AllowSoftwareFallback) + { + probes.Add(new BackendProbe( + Name: "D3D12 (WARP)", + Factory: () => D3D12SilkGraphicsDevice.Create(useWarp: true), + PreferredAsSoftware: true)); + } + } + + private static string FormatException(Exception ex) + { + string inner = ex.InnerException is null + ? string.Empty + : $" -> {ex.InnerException.GetType().Name}: {ex.InnerException.Message}"; + return $"{ex.GetType().Name}: {ex.Message}{inner}"; + } + + private readonly record struct BackendProbe( + string Name, + Func Factory, + bool PreferredAsSoftware); + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureBackend.cs b/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureBackend.cs new file mode 100644 index 0000000000..698831fa88 --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureBackend.cs @@ -0,0 +1,99 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// Describes the graphics backend an + /// resolved at construction + /// time. Callers use it to advertise the sensor's capabilities and to + /// decide whether to bother requesting frames at all: when + /// is false every subsequent capture + /// returns . + /// + public sealed record class SceneCameraCaptureBackend + { + /// + /// A backend descriptor for hosts where no rendering path was + /// available; used both as a sentinel on + /// and as the value the + /// provider exposes when device probing failed on every backend. + /// + public static SceneCameraCaptureBackend None { get; } = new() + { + Name = "None", + IsAvailable = false, + IsSoftware = false, + UnavailableReason = "No graphics backend has been probed yet." + }; + + /// + /// Short human-readable backend name for logging and diagnostics + /// (for example "D3D12", "D3D12 (WARP)", + /// "Vulkan", or "None"). + /// + public string Name { get; init; } = string.Empty; + + /// + /// Backend adapter description (GPU name, driver / API version), + /// as reported by the Silk device. Empty when + /// is false. + /// + public string DeviceName { get; init; } = string.Empty; + + /// + /// Backend API version string as reported by the Silk device. + /// Empty when is false. + /// + public string ApiVersion { get; init; } = string.Empty; + + /// + /// true when the backend is a software rasterizer (D3D12 + /// WARP, or a Vulkan software ICD such as SwiftShader). Software + /// backends render the same output as hardware ones, but much more + /// slowly, and the Vision server may choose to lower cadence when + /// this is true. + /// + public bool IsSoftware { get; init; } + + /// + /// true when the provider will attempt capture; false + /// when every backend probe failed and the provider will short-circuit + /// every capture to . + /// + public bool IsAvailable { get; init; } + + /// + /// Set to a human-readable reason (aggregated from every backend + /// that was tried) when is false; + /// null otherwise. + /// + public string? UnavailableReason { get; init; } + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureRequest.cs b/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureRequest.cs new file mode 100644 index 0000000000..af2ab86b9c --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureRequest.cs @@ -0,0 +1,84 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// One capture request handed to an + /// . Mirrors the fields that + /// a Vision IVisionSimulatedType exposes plus the frame size + /// and encoding the caller wants back. + /// + public sealed record class SceneCameraCaptureRequest + { + /// + /// Path or URI to the USD stage that hosts the camera prim. Passed + /// verbatim to UsdStage.Open. Required. + /// + public string StageIdentifier { get; init; } = string.Empty; + + /// + /// Absolute prim path to a UsdGeomCamera on the stage (for + /// example "/World/Cam"). When empty or null the + /// provider renders with the stage's automatic default framing + /// (CameraState.Default). + /// + public string? PrimPath { get; init; } + + /// + /// Requested output width in pixels. Must be positive. + /// + public int Width { get; init; } + + /// + /// Requested output height in pixels. Must be positive. + /// + public int Height { get; init; } + + /// + /// Stage time code the frame is captured at. Defaults to zero, + /// which matches most single-frame USD stages. + /// + public double TimeCode { get; init; } + + /// + /// Encoded image format requested from the provider. + /// + public SceneCameraImageFormat Format { get; init; } = SceneCameraImageFormat.Png; + + /// + /// Optional caller-supplied timestamp attached to the resulting + /// frame; when null the provider stamps the frame with + /// when the capture completes. + /// + public DateTime? TimestampUtc { get; init; } + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureResult.cs b/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureResult.cs new file mode 100644 index 0000000000..69294affe0 --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureResult.cs @@ -0,0 +1,105 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// Outcome of one + /// call. Discriminated by ; the encoded image is + /// only meaningful when is + /// . Every non-success + /// value populates with a human-readable diagnostic + /// so the Vision server can propagate it into its own status codes + /// without inventing text. + /// + public sealed record class SceneCameraCaptureResult + { + /// + /// The outcome discriminator. Callers should switch on this to + /// decide whether is usable. + /// + public SceneCameraCaptureStatus Status { get; init; } + + /// + /// Human-readable diagnostic; null when + /// is . + /// Never a secret and safe to log or forward to a client. The + /// underlying exception, when there is one, is redacted; look at + /// the provider's log for the full stack trace. + /// + public string? Reason { get; init; } + + /// + /// Encoded image bytes. when the + /// capture did not succeed. + /// + public ByteString Image { get; init; } + + /// + /// Encoded image format. Meaningless when + /// is not + /// . + /// + public SceneCameraImageFormat Format { get; init; } + + /// + /// Actual pixel width of the rendered frame - typically the value + /// from the request, but the provider may clamp very small requests + /// upward to a minimum the graphics backend supports. + /// + public int Width { get; init; } + + /// + /// Actual pixel height of the rendered frame. + /// + public int Height { get; init; } + + /// + /// Wall-clock UTC timestamp attached to the frame. When the request + /// supplies the + /// provider echoes it back; otherwise this is when the capture + /// completed. + /// + public DateTime TimestampUtc { get; init; } + + /// + /// Wall-clock time it took to render, guard, and encode this frame. + /// A useful health signal for the caller regardless of outcome. + /// + public TimeSpan Elapsed { get; init; } + + /// + /// The graphics backend that produced this frame (or would have, + /// if it had succeeded), for observability. + /// + public SceneCameraCaptureBackend Backend { get; init; } = SceneCameraCaptureBackend.None; + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureStatus.cs b/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureStatus.cs new file mode 100644 index 0000000000..4f51dc3eef --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/SceneCameraCaptureStatus.cs @@ -0,0 +1,96 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// Outcome of a + /// call. Only yields a valid encoded image on + /// the result; every other value is a definite failure and the caller + /// must consult for a + /// human-readable diagnostic. + /// + public enum SceneCameraCaptureStatus + { + /// + /// The frame rendered, the encoder produced bytes, and the guard + /// detected drawn geometry - the image on the result is usable. + /// + Succeeded = 0, + + /// + /// No graphics backend is available on this host (for example, CI + /// on Linux without a Vulkan loader). The provider surfaces this + /// distinctly so the Vision server can degrade to a non-rendering + /// sensor rather than blaming the request. + /// + NoRenderingBackend = 1, + + /// + /// The request is missing a required field (stage identifier, + /// positive dimensions, and so on) or requests an unsupported + /// image format. + /// + InvalidRequest = 2, + + /// + /// The USD stage failed to open (file missing, wrong plugin path, + /// or a syntactically malformed stage). + /// + StageOpenFailed = 3, + + /// + /// The stage opened, but the requested camera prim path either + /// does not exist or is not a UsdGeomCamera. + /// + CameraResolveFailed = 4, + + /// + /// SilkFrameCapture.Capture threw. Typically means the + /// graphics device was lost or refused the frame. + /// + RenderFailed = 5, + + /// + /// The render pipeline reported no drawn geometry, or the returned + /// pixels are uniform. Serving this frame would be misleading, so + /// the provider surfaces it as a distinct failure. The most common + /// cause is the known session-reuse landmine in the Silk backend, + /// which the provider defends against by using a fresh session per + /// request. + /// + BlankFrame = 6, + + /// + /// Encoding the raw RGBA8 pixels to the requested output format + /// failed. Rare; a defensive escape hatch for encoder bugs. + /// + EncodingFailed = 7 + } +} diff --git a/src/Opc.Ua.Vision.OpenUsd/SceneCameraImageFormat.cs b/src/Opc.Ua.Vision.OpenUsd/SceneCameraImageFormat.cs new file mode 100644 index 0000000000..ef821ff751 --- /dev/null +++ b/src/Opc.Ua.Vision.OpenUsd/SceneCameraImageFormat.cs @@ -0,0 +1,46 @@ +/* ======================================================================== + * Copyright (c) 2005-2025 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.OpenUsd +{ + /// + /// Encoded image format for a . + /// PNG is the only format that ships with the current in-repo encoder; + /// additional formats require additional encoders and are called out at + /// the enum level so callers can pattern-match without hitting the wire. + /// + public enum SceneCameraImageFormat + { + /// + /// RFC 2083 PNG, 8-bit RGBA. Encoded by the in-repo dependency-free + /// encoder; always available. + /// + Png = 0 + } +} diff --git a/src/Opc.Ua.Vision.Server/Builders/IVisionNodeBuilder.cs b/src/Opc.Ua.Vision.Server/Builders/IVisionNodeBuilder.cs new file mode 100644 index 0000000000..e078d604d5 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Builders/IVisionNodeBuilder.cs @@ -0,0 +1,530 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Opc.Ua.Vision; + +namespace Opc.Ua.Vision.Server.Builders +{ + /// + /// Top-level fluent Vision node builder. + /// + public interface IVisionNodeBuilder + { + /// + /// Adds an under + /// Vision/Sensors. + /// + IVisionNodeBuilder AddImageSensor(string browseName, Action configure); + + /// + /// Adds a under + /// Vision/Sensors. + /// + IVisionNodeBuilder AddDepth3DSensor(string browseName, Action configure); + + /// + /// Adds a generic under + /// Vision/Sensors. Used for modalities the specification + /// does not model with a dedicated subtype (for example, a + /// thermal or event camera). + /// + IVisionNodeBuilder AddSensor(string browseName, Action configure); + + /// + /// Adds a under + /// Vision/Frames. + /// + IVisionNodeBuilder AddFrame(string browseName, Action configure); + + /// + /// Adds an under + /// Vision/Pipelines. + /// + IVisionNodeBuilder AddPipeline(string browseName, Action configure); + } + + /// + /// Configures a coordinate-frame instance. + /// + public interface IVisionFrameBuilder + { + /// + /// Sets the required non-empty frame identifier (§5.12). + /// + IVisionFrameBuilder WithFrameId(string frameId); + + /// + /// Sets the frame role. + /// + IVisionFrameBuilder WithRole(VisionFrameRoleEnum role); + + /// + /// Sets the parent frame by its Vision frame identifier. When the + /// parent has not yet been registered, the reference is deferred + /// until build completion. + /// + /// + /// Sets the parent frame's stable business identifier. The + /// referenced frame must have been added earlier via + /// ; unresolved names + /// resolve to at finalise time. + /// + IVisionFrameBuilder WithParent(string parentFrameId); + + /// + /// Sets the parent frame's directly. Use + /// this overload when the parent already exists in the address + /// space and the caller has resolved its NodeId. + /// + IVisionFrameBuilder WithParent(NodeId parentNodeId); + + /// + /// Sets the transform (frame in its parent). The transform's + /// FrameId is set to the parent frame identifier per the + /// §5.12 frame-precedence rule. + /// + IVisionFrameBuilder WithTransform(VisionPose3DDataType transform); + } + + /// + /// Configures a generic vision sensor instance. + /// + public interface IVisionSensorBuilder : IVisionSensorBuilder; + + /// + /// Configures an instance. + /// + public interface IVisionImageSensorBuilder : IVisionSensorBuilder + { + /// + /// Sets the sensor resolution. + /// + IVisionImageSensorBuilder WithResolution(uint width, uint height); + + /// + /// Sets the pixel format string per §5.5.2. + /// + IVisionImageSensorBuilder WithPixelFormat(string pixelFormat); + + /// + /// Sets the sensor intrinsics. + /// + IVisionImageSensorBuilder WithIntrinsics(VisionIntrinsicsDataType intrinsics); + } + + /// + /// Configures a instance. + /// + public interface IVisionDepth3DSensorBuilder : IVisionSensorBuilder + { + /// + /// Sets the minimum and maximum depth in metres. + /// + IVisionDepth3DSensorBuilder WithDepthRange(double minMetres, double maxMetres); + + /// + /// Sets the depth-value scale factor (metres per unit). + /// + IVisionDepth3DSensorBuilder WithDepthScale(double metresPerUnit); + + /// + /// Sets the stereo baseline in metres. + /// + IVisionDepth3DSensorBuilder WithBaseline(double metres); + } + + /// + /// Shared surface for the strongly-typed sensor builders. + /// + /// + /// The concrete builder type, so With-style methods + /// return the concrete type instead of the shared interface. + /// + public interface IVisionSensorBuilder + where TSelf : IVisionSensorBuilder + { + /// + /// Sets the deployment-scoped sensor identifier. + /// + TSelf WithSensorId(string sensorId); + + /// + /// Sets the sensor's reality kind (real, simulated, hybrid). + /// + TSelf WithRealityKind(VisionRealityKindEnum realityKind); + + /// + /// Sets the sensor modality. + /// + TSelf WithModality(VisionSensorModalityEnum modality); + + /// + /// Sets the manufacturer string. + /// + TSelf WithManufacturer(string manufacturer); + + /// + /// Sets the model string. + /// + TSelf WithModel(string model); + + /// + /// Sets the serial number. + /// + TSelf WithSerialNumber(string serialNumber); + + /// + /// Sets the device URI. + /// + TSelf WithDeviceUri(string deviceUri); + + /// + /// Sets the sensor's FrameId. If a frame with that id has + /// been registered under Vision/Frames, the Server also + /// adds a MountedOn reference to it. + /// + TSelf WithFrameId(string frameId); + + /// + /// Adds a HasScenePrim reference to another node. + /// + TSelf HasScenePrim(NodeId scenePrimNodeId); + + /// + /// Adds a MountedOn reference to another node — used when + /// the mount is not a coordinate frame with a matching + /// FrameId. + /// + TSelf MountedOn(NodeId mountNodeId); + + /// + /// Configures the sensor's optional Optics child. + /// + TSelf WithOptics(Action configure); + + /// + /// Configures the sensor's optional Illumination child. + /// + TSelf WithIllumination(Action configure); + + /// + /// Adds an intrinsic calibration under Sensor/Calibrations. + /// + TSelf AddIntrinsicCalibration(string browseName, Action configure); + + /// + /// Adds an extrinsic calibration under Sensor/Calibrations. + /// + TSelf AddExtrinsicCalibration(string browseName, Action configure); + + /// + /// Adds a stream endpoint under Sensor/Media/StreamEndpoints. + /// + TSelf AddStreamEndpoint(string browseName, Action configure); + + /// + /// Adds a clip endpoint under Sensor/Media/ClipEndpoints. + /// + TSelf AddClipEndpoint(string browseName, Action configure); + + /// + /// Binds an to this sensor's + /// media manager. All media methods delegate to the provider. + /// + TSelf UseMediaProvider(IVisionMediaProvider provider); + } + + /// + /// Configures the optics attached to a sensor. + /// + public interface IVisionOpticsBuilder + { + /// + /// Sets the focal length in metres. + /// + IVisionOpticsBuilder WithFocalLength(double metres); + + /// + /// Sets the aperture (f-number). + /// + IVisionOpticsBuilder WithAperture(double fNumber); + + /// + /// Sets the working distance in metres. + /// + IVisionOpticsBuilder WithWorkingDistance(double metres); + + /// + /// Sets the magnification. + /// + IVisionOpticsBuilder WithMagnification(double magnification); + + /// + /// Sets the lens mount type. + /// + IVisionOpticsBuilder WithMountType(string mountType); + + /// + /// Sets the lens type. + /// + IVisionOpticsBuilder WithLensType(string lensType); + } + + /// + /// Configures the illumination attached to a sensor. + /// + public interface IVisionIlluminationBuilder + { + /// + /// Sets the lamp type. + /// + IVisionIlluminationBuilder WithLampType(VisionLampTypeEnum lampType); + + /// + /// Sets the peak wavelength in nanometres. + /// + IVisionIlluminationBuilder WithWavelength(double nanometres); + + /// + /// Sets the relative intensity (0…1). + /// + IVisionIlluminationBuilder WithRelativeIntensity(double relativeIntensity); + + /// + /// Sets the lighting mode label. + /// + IVisionIlluminationBuilder WithLightingMode(VisionLightingModeEnum lightingMode); + } + + /// + /// Configures an intrinsic calibration. + /// + public interface IVisionIntrinsicCalibrationBuilder + { + /// + /// Sets the calibration identifier. + /// + IVisionIntrinsicCalibrationBuilder WithCalibrationId(string calibrationId); + + /// + /// Sets the value. + /// + IVisionIntrinsicCalibrationBuilder WithIntrinsics(VisionIntrinsicsDataType intrinsics); + + /// + /// Sets the residual reprojection error. + /// + IVisionIntrinsicCalibrationBuilder WithResidualError(double residualError); + + /// + /// Sets the calibration method label. + /// + IVisionIntrinsicCalibrationBuilder WithMethod(string method); + } + + /// + /// Configures an extrinsic calibration. + /// + public interface IVisionExtrinsicCalibrationBuilder + { + /// + /// Sets the calibration identifier. + /// + IVisionExtrinsicCalibrationBuilder WithCalibrationId(string calibrationId); + + /// + /// Sets the mount kind. + /// + IVisionExtrinsicCalibrationBuilder WithMount(VisionCalibrationMountEnum mount); + + /// + /// Sets the source and target frame identifiers. + /// + IVisionExtrinsicCalibrationBuilder WithFrames(string sourceFrame, string targetFrame); + + /// + /// Sets the extrinsic transform. The Server sets the transform's + /// FrameId equal to .FrameId or + /// the target frame if the pose's frame is empty per §5.12. + /// + IVisionExtrinsicCalibrationBuilder WithTransform(VisionPose3DDataType transform); + + /// + /// Sets the residual error. + /// + IVisionExtrinsicCalibrationBuilder WithResidualError(double residualError); + } + + /// + /// Configures a stream endpoint. + /// + public interface IVisionStreamEndpointBuilder + { + /// + /// Sets the deployment-scoped endpoint identifier. + /// + IVisionStreamEndpointBuilder WithEndpointId(string endpointId); + + /// + /// Sets the endpoint URI. + /// + IVisionStreamEndpointBuilder WithEndpointUri(string endpointUri); + + /// + /// Sets the stream protocol. + /// + IVisionStreamEndpointBuilder WithProtocol(VisionStreamProtocolEnum protocol); + + /// + /// Sets the media codec. + /// + IVisionStreamEndpointBuilder WithCodec(VisionVideoCodecEnum codec); + + /// + /// Sets the resolution. + /// + IVisionStreamEndpointBuilder WithResolution(uint width, uint height); + + /// + /// Sets the frame rate. + /// + IVisionStreamEndpointBuilder WithFrameRate(double frameRate); + + /// + /// Sets the bitrate. + /// + IVisionStreamEndpointBuilder WithBitrate(uint bitrate); + + /// + /// Sets the profile name. + /// + IVisionStreamEndpointBuilder WithDefaultProfileName(string defaultProfileName); + } + + /// + /// Configures a clip endpoint. + /// + public interface IVisionClipEndpointBuilder + { + /// + /// Sets the deployment-scoped endpoint identifier. + /// + IVisionClipEndpointBuilder WithEndpointId(string endpointId); + + /// + /// Sets the endpoint URI. + /// + IVisionClipEndpointBuilder WithEndpointUri(string endpointUri); + + /// + /// Sets the clip format. + /// + IVisionClipEndpointBuilder WithClipFormat(VisionClipFormatEnum format); + + /// + /// Sets the clip quality (encoder-specific units). + /// + IVisionClipEndpointBuilder WithQuality(uint quality); + + /// + /// Sets the clip resolution. + /// + IVisionClipEndpointBuilder WithResolution(uint width, uint height); + + /// + /// Enables or disables inline delivery of clip bytes. + /// + /// + /// When set to , the Server's + /// LatestClip variable is served with + /// and inline + /// GetClip requests are refused, as required by §6.4. + /// + IVisionClipEndpointBuilder WithInlineDelivery(bool enabled, uint maxInlineClipSize); + + /// + /// Sets the profile name. + /// + IVisionClipEndpointBuilder WithDefaultProfileName(string defaultProfileName); + } + + /// + /// Configures an inference-pipeline instance. + /// + public interface IVisionPipelineBuilder + { + /// + /// Sets the pipeline identifier. + /// + IVisionPipelineBuilder WithPipelineId(string pipelineId); + + /// + /// Sets the target sensor node id. + /// + IVisionPipelineBuilder WithSensor(NodeId sensorNodeId); + + /// + /// Sets the deployment node id. The specification deliberately + /// keeps Deployment as a plain ; the + /// Server does not require any AI Model Management dependency. + /// + IVisionPipelineBuilder WithDeployment(NodeId deploymentNodeId); + + /// + /// Sets the learning job node id. The specification deliberately + /// keeps LearningJob as a plain ; the + /// Server does not require any AI Model Management dependency. Section + /// 9.5.1 requires this to be non-null when ground-truth corrections are + /// retained so a client can tell whether its label reached a learning + /// loop. + /// + IVisionPipelineBuilder WithLearningJob(NodeId learningJobNodeId); + + /// + /// Adds a ProducedBy reference from the pipeline to the + /// referenced node (typically a controller or process instance). + /// + IVisionPipelineBuilder ProducedBy(NodeId producerNodeId); + + /// + /// Binds the inference provider. All pipeline methods delegate + /// to it. When is + /// the Server advertises the VIS-Inference-OnServer + /// facet — otherwise VIS-Inference-OffServer. + /// + IVisionPipelineBuilder UseInferenceProvider(IVisionInferenceProvider provider, bool onServer = true); + + /// + /// Binds the feedback sink for this pipeline's Feedback + /// object. + /// + IVisionPipelineBuilder UseFeedbackSink(IVisionFeedbackSink sink); + } +} diff --git a/src/Opc.Ua.Vision.Server/Builders/VisionNodeBuilder.cs b/src/Opc.Ua.Vision.Server/Builders/VisionNodeBuilder.cs new file mode 100644 index 0000000000..2096c9908f --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Builders/VisionNodeBuilder.cs @@ -0,0 +1,1388 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using Opc.Ua.Vision; + +namespace Opc.Ua.Vision.Server.Builders +{ + internal sealed class VisionNodeBuilder : IVisionNodeBuilder + { + public VisionNodeBuilder( + VisionBuildContext context, + VisionRegistry registry, + VisionMethodDispatcher dispatcher) + { + m_context = context ?? throw new ArgumentNullException(nameof(context)); + m_registry = registry ?? throw new ArgumentNullException(nameof(registry)); + m_dispatcher = dispatcher ?? throw new ArgumentNullException(nameof(dispatcher)); + } + + public IVisionNodeBuilder AddImageSensor( + string browseName, + Action configure) + { + if (string.IsNullOrEmpty(browseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(browseName)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + FolderState sensorsFolder = EnsureSensorsFolder(); + var qualifiedName = new QualifiedName(browseName, m_context.InstanceNamespaceIndex); + ImageSensorState sensor = m_context.Context.CreateInstanceOfImageSensorType( + sensorsFolder, + qualifiedName); + sensor.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + var builder = new VisionImageSensorBuilder(m_context, m_registry, m_dispatcher, sensor, browseName); + configure(builder); + builder.Finalize(sensorsFolder); + m_context.EnqueueForRegistration(sensor); + return this; + } + + public IVisionNodeBuilder AddDepth3DSensor( + string browseName, + Action configure) + { + if (string.IsNullOrEmpty(browseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(browseName)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + FolderState sensorsFolder = EnsureSensorsFolder(); + var qualifiedName = new QualifiedName(browseName, m_context.InstanceNamespaceIndex); + Depth3DSensorState sensor = m_context.Context.CreateInstanceOfDepth3DSensorType( + sensorsFolder, + qualifiedName); + sensor.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + var builder = new VisionDepth3DSensorBuilder(m_context, m_registry, m_dispatcher, sensor, browseName); + configure(builder); + builder.Finalize(sensorsFolder); + m_context.EnqueueForRegistration(sensor); + return this; + } + + public IVisionNodeBuilder AddSensor( + string browseName, + Action configure) + { + if (string.IsNullOrEmpty(browseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(browseName)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + FolderState sensorsFolder = EnsureSensorsFolder(); + var qualifiedName = new QualifiedName(browseName, m_context.InstanceNamespaceIndex); + VisionSensorState sensor = m_context.Context.CreateInstanceOfVisionSensorType( + sensorsFolder, + qualifiedName); + sensor.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + var builder = new VisionGenericSensorBuilder(m_context, m_registry, m_dispatcher, sensor, browseName); + configure(builder); + builder.Finalize(sensorsFolder); + m_context.EnqueueForRegistration(sensor); + return this; + } + + public IVisionNodeBuilder AddFrame( + string browseName, + Action configure) + { + if (string.IsNullOrEmpty(browseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(browseName)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + FolderState framesFolder = EnsureFramesFolder(); + var qualifiedName = new QualifiedName(browseName, m_context.InstanceNamespaceIndex); + CoordinateFrameState frame = m_context.Context.CreateInstanceOfCoordinateFrameType( + framesFolder, + qualifiedName); + frame.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + var builder = new VisionFrameBuilder(m_context, m_registry, frame, browseName); + configure(builder); + builder.Finalize(framesFolder); + m_context.EnqueueForRegistration(frame); + return this; + } + + public IVisionNodeBuilder AddPipeline( + string browseName, + Action configure) + { + if (string.IsNullOrEmpty(browseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(browseName)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + FolderState pipelinesFolder = EnsurePipelinesFolder(); + var qualifiedName = new QualifiedName(browseName, m_context.InstanceNamespaceIndex); + InferencePipelineState pipeline = m_context.Context.CreateInstanceOfInferencePipelineType( + pipelinesFolder, + qualifiedName); + pipeline.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + var builder = new VisionPipelineBuilder(m_context, m_registry, m_dispatcher, pipeline, browseName); + configure(builder); + builder.Finalize(pipelinesFolder); + m_context.EnqueueForRegistration(pipeline); + return this; + } + + private FolderState EnsureSensorsFolder() + { + m_context.Root.CreateOrReplaceSensors(m_context.Context, null); + return m_context.Root.Sensors!; + } + + private FolderState EnsurePipelinesFolder() + { + m_context.Root.CreateOrReplacePipelines(m_context.Context, null); + FolderState pipelines = m_context.Root.Pipelines!; + m_context.EnqueueForRegistration(pipelines); + return pipelines; + } + + private FolderState EnsureFramesFolder() + { + m_context.Root.CreateOrReplaceFrames(m_context.Context, null); + FolderState frames = m_context.Root.Frames!; + m_context.EnqueueForRegistration(frames); + return frames; + } + + private readonly VisionBuildContext m_context; + private readonly VisionRegistry m_registry; + private readonly VisionMethodDispatcher m_dispatcher; + } + + internal sealed class VisionFrameBuilder : IVisionFrameBuilder + { + public VisionFrameBuilder( + VisionBuildContext context, + VisionRegistry registry, + CoordinateFrameState frame, + string browseName) + { + m_context = context; + m_registry = registry; + m_frame = frame; + m_browseName = browseName; + } + + public IVisionFrameBuilder WithFrameId(string frameId) + { + m_frameId = frameId ?? string.Empty; + m_frame.CreateOrReplaceFrameId(m_context.Context, null); + m_frame.FrameId!.Value = m_frameId; + return this; + } + + public IVisionFrameBuilder WithRole(VisionFrameRoleEnum role) + { + m_role = role; + m_frame.CreateOrReplaceRole(m_context.Context, null); + m_frame.Role!.Value = role; + return this; + } + + public IVisionFrameBuilder WithParent(string parentFrameId) + { + m_parentFrameId = parentFrameId ?? string.Empty; + m_parentNodeId = NodeId.Null; + return this; + } + + public IVisionFrameBuilder WithParent(NodeId parentNodeId) + { + m_parentNodeId = parentNodeId; + m_parentFrameId = string.Empty; + if (!parentNodeId.IsNull) + { + m_frame.CreateOrReplaceParentFrame(m_context.Context, null); + m_frame.ParentFrame!.Value = parentNodeId; + } + return this; + } + + public IVisionFrameBuilder WithTransform(VisionPose3DDataType transform) + { + if (transform == null) + { + throw new ArgumentNullException(nameof(transform)); + } + m_frame.CreateOrReplaceTransform(m_context.Context, null); + var pose = new VisionPose3DDataType + { + FrameId = string.IsNullOrEmpty(transform.FrameId) ? m_parentFrameId : transform.FrameId, + Position = transform.Position, + Orientation = transform.Orientation, + Covariance = transform.Covariance + }; + m_frame.Transform!.Value = pose; + m_transform = pose; + return this; + } + + internal void Finalize(FolderState parent) + { + if (string.IsNullOrEmpty(m_frameId)) + { + throw ServiceResultException.Create( + StatusCodes.BadConfigurationError, + "Coordinate frame '{0}' must declare a non-empty FrameId (§5.12).", + m_browseName); + } + NodeId parentNodeId = m_parentNodeId; + if (parentNodeId.IsNull && !string.IsNullOrEmpty(m_parentFrameId)) + { + FrameRegistration? existing = m_registry.TryFindFrameByFrameId(m_parentFrameId); + if (existing != null) + { + parentNodeId = existing.NodeId; + } + } + if (!parentNodeId.IsNull) + { + m_frame.CreateOrReplaceParentFrame(m_context.Context, null); + m_frame.ParentFrame!.Value = parentNodeId; + } + parent.AddChild(m_frame); + var registration = new FrameRegistration( + m_browseName, + m_frame.NodeId, + m_frameId, + m_role, + string.IsNullOrEmpty(m_parentFrameId) ? null : m_parentFrameId, + m_transform ?? VisionCoordinateFrameMath.Identity(m_parentFrameId ?? string.Empty), + m_frame); + m_registry.AddFrame(registration); + } + + private readonly VisionBuildContext m_context; + private readonly VisionRegistry m_registry; + private readonly CoordinateFrameState m_frame; + private readonly string m_browseName; + private string m_frameId = string.Empty; + private VisionFrameRoleEnum m_role = VisionFrameRoleEnum.Other; + private string m_parentFrameId = string.Empty; + private NodeId m_parentNodeId = NodeId.Null; + private VisionPose3DDataType? m_transform; + } + + internal abstract class VisionSensorBuilderBase : IVisionSensorBuilder + where TSelf : IVisionSensorBuilder + where TSensor : VisionSensorState + { + protected VisionSensorBuilderBase( + VisionBuildContext context, + VisionRegistry registry, + VisionMethodDispatcher dispatcher, + TSensor sensor, + string browseName) + { + BuildContext = context; + Registry = registry; + m_dispatcher = dispatcher; + Sensor = sensor; + m_browseName = browseName; + } + + protected TSensor Sensor { get; } + + protected VisionBuildContext BuildContext { get; } + + protected VisionRegistry Registry { get; } + + protected abstract TSelf Self { get; } + + public TSelf WithSensorId(string sensorId) + { + Sensor.CreateOrReplaceSensorId(BuildContext.Context, null); + Sensor.SensorId!.Value = sensorId ?? string.Empty; + return Self; + } + + public TSelf WithRealityKind(VisionRealityKindEnum realityKind) + { + m_realityKind = realityKind; + Sensor.CreateOrReplaceRealityKind(BuildContext.Context, null); + Sensor.RealityKind!.Value = realityKind; + return Self; + } + + public TSelf WithModality(VisionSensorModalityEnum modality) + { + m_modality = modality; + Sensor.CreateOrReplaceModality(BuildContext.Context, null); + Sensor.Modality!.Value = modality; + return Self; + } + + public TSelf WithManufacturer(string manufacturer) + { + Sensor.CreateOrReplaceManufacturer(BuildContext.Context, null); + Sensor.Manufacturer!.Value = new LocalizedText(manufacturer ?? string.Empty); + m_hasSensorParams = true; + return Self; + } + + public TSelf WithModel(string model) + { + Sensor.CreateOrReplaceModel(BuildContext.Context, null); + Sensor.Model!.Value = new LocalizedText(model ?? string.Empty); + m_hasSensorParams = true; + return Self; + } + + public TSelf WithSerialNumber(string serialNumber) + { + Sensor.CreateOrReplaceSerialNumber(BuildContext.Context, null); + Sensor.SerialNumber!.Value = serialNumber ?? string.Empty; + m_hasSensorParams = true; + return Self; + } + + public TSelf WithDeviceUri(string deviceUri) + { + Sensor.CreateOrReplaceDeviceUri(BuildContext.Context, null); + Sensor.DeviceUri!.Value = deviceUri ?? string.Empty; + return Self; + } + + public TSelf WithFrameId(string frameId) + { + m_frameId = frameId ?? string.Empty; + Sensor.CreateOrReplaceFrameId(BuildContext.Context, null); + Sensor.FrameId!.Value = m_frameId; + return Self; + } + + public TSelf HasScenePrim(NodeId scenePrimNodeId) + { + if (!scenePrimNodeId.IsNull) + { + Sensor.AddReference(VisionReferenceTypeIds(BuildContext, "HasScenePrim"), false, scenePrimNodeId); + m_hasScenePrim = true; + } + return Self; + } + + public TSelf MountedOn(NodeId mountNodeId) + { + if (!mountNodeId.IsNull) + { + Sensor.AddReference(VisionReferenceTypeIds(BuildContext, "MountedOn"), false, mountNodeId); + } + return Self; + } + + public TSelf WithOptics(Action configure) + { + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + Sensor.CreateOrReplaceOptics(BuildContext.Context, null); + var opticsBuilder = new VisionOpticsBuilder(BuildContext, Sensor.Optics!); + configure(opticsBuilder); + m_hasOptics = true; + return Self; + } + + public TSelf WithIllumination(Action configure) + { + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + Sensor.CreateOrReplaceIllumination(BuildContext.Context, null); + var illuminationBuilder = new VisionIlluminationBuilder(BuildContext, Sensor.Illumination!); + configure(illuminationBuilder); + m_hasIllumination = true; + return Self; + } + + public TSelf AddIntrinsicCalibration( + string browseName, + Action configure) + { + if (string.IsNullOrEmpty(browseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(browseName)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + FolderState calibrations = EnsureCalibrationsFolder(); + var qualifiedName = new QualifiedName(browseName, BuildContext.InstanceNamespaceIndex); + IntrinsicCalibrationState calibration = BuildContext.Context + .CreateInstanceOfIntrinsicCalibrationType(calibrations, qualifiedName); + calibration.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + var builder = new VisionIntrinsicCalibrationBuilder(BuildContext, calibration); + configure(builder); + calibrations.AddChild(calibration); + Sensor.AddReference(VisionReferenceTypeIds(BuildContext, "HasCalibration"), false, calibration.NodeId); + m_hasCalibration = true; + return Self; + } + + public TSelf AddExtrinsicCalibration( + string browseName, + Action configure) + { + if (string.IsNullOrEmpty(browseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(browseName)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + FolderState calibrations = EnsureCalibrationsFolder(); + var qualifiedName = new QualifiedName(browseName, BuildContext.InstanceNamespaceIndex); + ExtrinsicCalibrationState calibration = BuildContext.Context + .CreateInstanceOfExtrinsicCalibrationType(calibrations, qualifiedName); + calibration.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + var builder = new VisionExtrinsicCalibrationBuilder(BuildContext, calibration); + configure(builder); + calibrations.AddChild(calibration); + Sensor.AddReference(VisionReferenceTypeIds(BuildContext, "HasCalibration"), false, calibration.NodeId); + m_hasCalibration = true; + m_hasExtrinsicCalibration = true; + return Self; + } + + public TSelf AddStreamEndpoint( + string browseName, + Action configure) + { + if (string.IsNullOrEmpty(browseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(browseName)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + VisionMediaManagementState media = EnsureMedia(); + media.CreateOrReplaceStreamEndpoints(BuildContext.Context, null); + FolderState endpoints = media.StreamEndpoints!; + var qualifiedName = new QualifiedName(browseName, BuildContext.InstanceNamespaceIndex); + StreamEndpointState endpoint = BuildContext.Context.CreateInstanceOfStreamEndpointType( + endpoints, + qualifiedName); + endpoint.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + var builder = new VisionStreamEndpointBuilder(BuildContext, endpoint); + configure(builder); + endpoints.AddChild(endpoint); + m_streamEndpoints.Add(endpoint); + return Self; + } + + public TSelf AddClipEndpoint( + string browseName, + Action configure) + { + if (string.IsNullOrEmpty(browseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(browseName)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + VisionMediaManagementState media = EnsureMedia(); + media.CreateOrReplaceClipEndpoints(BuildContext.Context, null); + FolderState endpoints = media.ClipEndpoints!; + var qualifiedName = new QualifiedName(browseName, BuildContext.InstanceNamespaceIndex); + ClipEndpointState endpoint = BuildContext.Context.CreateInstanceOfClipEndpointType( + endpoints, + qualifiedName); + endpoint.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.Organizes; + var builder = new VisionClipEndpointBuilder(BuildContext, endpoint); + configure(builder); + endpoints.AddChild(endpoint); + m_clipEndpoints.Add(endpoint); + return Self; + } + + public TSelf UseMediaProvider(IVisionMediaProvider provider) + { + if (provider == null) + { + throw new ArgumentNullException(nameof(provider)); + } + m_mediaProvider = provider; + EnsureMedia(); + return Self; + } + + internal void Finalize(FolderState parent) + { + parent.AddChild(Sensor); + HashSet facets = ComputeFacets(); + var registration = new SensorRegistration( + m_browseName, + Sensor.NodeId, + Sensor, + m_modality, + m_realityKind, + facets, + m_mediaProvider) + { + HasIntrinsicCalibration = m_hasCalibration, + HasExtrinsicCalibration = m_hasExtrinsicCalibration, + HasOptics = m_hasOptics, + HasIllumination = m_hasIllumination + }; + for (int ii = 0; ii < m_streamEndpoints.Count; ii++) + { + registration.StreamEndpoints.Add(m_streamEndpoints[ii]); + } + for (int ii = 0; ii < m_clipEndpoints.Count; ii++) + { + registration.ClipEndpoints.Add(m_clipEndpoints[ii]); + } + OnFinalize(registration); + Registry.AddSensor(registration); + if (!string.IsNullOrEmpty(m_frameId) && + Registry.TryGetFrameByFrameId(m_frameId, out FrameRegistration? mountFrame) && + mountFrame != null) + { + Sensor.AddReference(VisionReferenceTypeIds(BuildContext, "MountedOn"), false, mountFrame.NodeId); + } + if (Sensor.Media is VisionMediaManagementState media) + { + EnsureMediaMethods(media); + m_dispatcher.AttachMediaMethods(Sensor.NodeId, media); + } + } + + private void EnsureMediaMethods(VisionMediaManagementState media) + { + if (m_mediaProvider == null) + { + return; + } + ISystemContext context = BuildContext.Context; + DeclareMedia( + media.CreateOrReplaceGetStreamEndpoint(context, null), + MethodIds.VisionMediaManagementType_GetStreamEndpoint, + VisionMethodArguments.Declare); + DeclareMedia( + media.CreateOrReplaceReleaseStreamEndpoint(context, null), + MethodIds.VisionMediaManagementType_ReleaseStreamEndpoint, + VisionMethodArguments.Declare); + DeclareMedia( + media.CreateOrReplaceConfigureStreamEndpoint(context, null), + MethodIds.VisionMediaManagementType_ConfigureStreamEndpoint, + VisionMethodArguments.Declare); + DeclareMedia( + media.CreateOrReplaceSelectEndpoint(context, null), + MethodIds.VisionMediaManagementType_SelectEndpoint, + VisionMethodArguments.Declare); + DeclareMedia( + media.CreateOrReplaceGetClip(context, null), + MethodIds.VisionMediaManagementType_GetClip, + VisionMethodArguments.Declare); + } + + private void DeclareMedia( + TMethod method, + ExpandedNodeId declarationId, + Action declare) + where TMethod : MethodState + { + declare(BuildContext.Context, method); + method.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent; + method.MethodDeclarationId = ExpandedNodeId.ToNodeId( + declarationId, BuildContext.Context.NamespaceUris); + } + + protected virtual void OnFinalize(SensorRegistration registration) + { + } + + protected VisionMediaManagementState EnsureMedia() + { + Sensor.CreateOrReplaceMedia(BuildContext.Context, null); + Sensor.Media!.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent; + return Sensor.Media!; + } + + private FolderState EnsureCalibrationsFolder() + { + Sensor.CreateOrReplaceCalibrations(BuildContext.Context, null); + return Sensor.Calibrations!; + } + + private HashSet ComputeFacets() + { + var facets = new HashSet(StringComparer.Ordinal) + { + VisionConformanceUris.FacetNames.Base + }; + if (m_hasSensorParams) + { + facets.Add(VisionConformanceUris.FacetNames.SensorParams); + } + if (m_hasOptics) + { + facets.Add(VisionConformanceUris.FacetNames.Optics); + } + if (m_hasCalibration || m_hasExtrinsicCalibration) + { + facets.Add(VisionConformanceUris.FacetNames.Calibration); + } + for (int ii = 0; ii < m_streamEndpoints.Count; ii++) + { + if (m_streamEndpoints[ii].StreamProtocol?.Value == VisionStreamProtocolEnum.Rtsp) + { + facets.Add(VisionConformanceUris.FacetNames.MediaRtsp); + } + facets.Add(VisionConformanceUris.FacetNames.EndpointConfig); + } + for (int ii = 0; ii < m_clipEndpoints.Count; ii++) + { + if (m_clipEndpoints[ii].ClipFormat?.Value == VisionClipFormatEnum.Jpeg) + { + facets.Add(VisionConformanceUris.FacetNames.MediaJpeg); + } + if (m_clipEndpoints[ii].InlineDeliveryEnabled?.Value == true) + { + facets.Add(VisionConformanceUris.FacetNames.MediaInline); + } + } + if (m_realityKind == VisionRealityKindEnum.Simulated || + m_realityKind == VisionRealityKindEnum.Hybrid) + { + facets.Add(VisionConformanceUris.FacetNames.Simulation); + } + if (m_hasScenePrim) + { + facets.Add(VisionConformanceUris.FacetNames.InteropScene); + } + return facets; + } + + private NodeId VisionReferenceTypeIds(VisionBuildContext context, string referenceName) + { + ExpandedNodeId expanded = referenceName switch + { + "HasCalibration" => Opc.Ua.Vision.ReferenceTypeIds.HasCalibration, + "MountedOn" => Opc.Ua.Vision.ReferenceTypeIds.MountedOn, + "HasScenePrim" => Opc.Ua.Vision.ReferenceTypeIds.HasScenePrim, + "ProducedBy" => Opc.Ua.Vision.ReferenceTypeIds.ProducedBy, + _ => throw new ArgumentOutOfRangeException(nameof(referenceName)) + }; + return ExpandedNodeId.ToNodeId(expanded, context.Context.NamespaceUris); + } + + private readonly VisionMethodDispatcher m_dispatcher; + private readonly string m_browseName; + private readonly List m_streamEndpoints = []; + private readonly List m_clipEndpoints = []; + private VisionSensorModalityEnum m_modality; + private VisionRealityKindEnum m_realityKind = VisionRealityKindEnum.Physical; + private string m_frameId = string.Empty; + private IVisionMediaProvider? m_mediaProvider; + private bool m_hasSensorParams; + private bool m_hasOptics; + private bool m_hasIllumination; + private bool m_hasCalibration; + private bool m_hasExtrinsicCalibration; + private bool m_hasScenePrim; + } + + internal sealed class VisionGenericSensorBuilder : + VisionSensorBuilderBase, + IVisionSensorBuilder + { + public VisionGenericSensorBuilder( + VisionBuildContext context, + VisionRegistry registry, + VisionMethodDispatcher dispatcher, + VisionSensorState sensor, + string browseName) + : base(context, registry, dispatcher, sensor, browseName) + { + } + + protected override IVisionSensorBuilder Self => this; + } + + internal sealed class VisionImageSensorBuilder : + VisionSensorBuilderBase, + IVisionImageSensorBuilder + { + public VisionImageSensorBuilder( + VisionBuildContext context, + VisionRegistry registry, + VisionMethodDispatcher dispatcher, + ImageSensorState sensor, + string browseName) + : base(context, registry, dispatcher, sensor, browseName) + { + } + + protected override IVisionImageSensorBuilder Self => this; + + public IVisionImageSensorBuilder WithResolution(uint width, uint height) + { + Sensor.CreateOrReplaceWidth(BuildContext.Context, null); + Sensor.CreateOrReplaceHeight(BuildContext.Context, null); + Sensor.Width!.Value = width; + Sensor.Height!.Value = height; + return this; + } + + public IVisionImageSensorBuilder WithPixelFormat(string pixelFormat) + { + Sensor.CreateOrReplacePixelFormat(BuildContext.Context, null); + Sensor.PixelFormat!.Value = pixelFormat ?? string.Empty; + return this; + } + + public IVisionImageSensorBuilder WithIntrinsics(VisionIntrinsicsDataType intrinsics) + { + if (intrinsics == null) + { + throw new ArgumentNullException(nameof(intrinsics)); + } + Sensor.CreateOrReplaceIntrinsics(BuildContext.Context, null); + Sensor.Intrinsics!.Value = intrinsics; + m_hasIntrinsics = true; + return this; + } + + protected override void OnFinalize(SensorRegistration registration) + { + registration.HasIntrinsicCalibration = registration.HasIntrinsicCalibration || m_hasIntrinsics; + } + + private new ImageSensorState Sensor => (ImageSensorState)base.Sensor!; + + private bool m_hasIntrinsics; + } + + internal sealed class VisionDepth3DSensorBuilder : + VisionSensorBuilderBase, + IVisionDepth3DSensorBuilder + { + public VisionDepth3DSensorBuilder( + VisionBuildContext context, + VisionRegistry registry, + VisionMethodDispatcher dispatcher, + Depth3DSensorState sensor, + string browseName) + : base(context, registry, dispatcher, sensor, browseName) + { + } + + protected override IVisionDepth3DSensorBuilder Self => this; + + public IVisionDepth3DSensorBuilder WithDepthRange(double minMetres, double maxMetres) + { + Sensor.CreateOrReplaceMinDepth(BuildContext.Context, null); + Sensor.CreateOrReplaceMaxDepth(BuildContext.Context, null); + Sensor.MinDepth!.Value = minMetres; + Sensor.MaxDepth!.Value = maxMetres; + return this; + } + + public IVisionDepth3DSensorBuilder WithDepthScale(double metresPerUnit) + { + Sensor.CreateOrReplaceDepthScale(BuildContext.Context, null); + Sensor.DepthScale!.Value = metresPerUnit; + return this; + } + + public IVisionDepth3DSensorBuilder WithBaseline(double metres) + { + Sensor.CreateOrReplaceBaseline(BuildContext.Context, null); + Sensor.Baseline!.Value = metres; + return this; + } + + private new Depth3DSensorState Sensor => (Depth3DSensorState)base.Sensor!; + } + + internal sealed class VisionOpticsBuilder : IVisionOpticsBuilder + { + public VisionOpticsBuilder(VisionBuildContext context, OpticsState optics) + { + m_context = context; + m_optics = optics; + } + + public IVisionOpticsBuilder WithFocalLength(double metres) + { + m_optics.CreateOrReplaceFocalLength(m_context.Context, null); + m_optics.FocalLength!.Value = metres; + return this; + } + + public IVisionOpticsBuilder WithAperture(double fNumber) + { + m_optics.CreateOrReplaceAperture(m_context.Context, null); + m_optics.Aperture!.Value = fNumber; + return this; + } + + public IVisionOpticsBuilder WithWorkingDistance(double metres) + { + m_optics.CreateOrReplaceWorkingDistance(m_context.Context, null); + m_optics.WorkingDistance!.Value = metres; + return this; + } + + public IVisionOpticsBuilder WithMagnification(double magnification) + { + m_optics.CreateOrReplaceMagnification(m_context.Context, null); + m_optics.Magnification!.Value = magnification; + return this; + } + + public IVisionOpticsBuilder WithMountType(string mountType) + { + m_optics.CreateOrReplaceMountType(m_context.Context, null); + m_optics.MountType!.Value = mountType ?? string.Empty; + return this; + } + + public IVisionOpticsBuilder WithLensType(string lensType) + { + m_optics.CreateOrReplaceLensType(m_context.Context, null); + m_optics.LensType!.Value = lensType ?? string.Empty; + return this; + } + + private readonly VisionBuildContext m_context; + private readonly OpticsState m_optics; + } + + internal sealed class VisionIlluminationBuilder : IVisionIlluminationBuilder + { + public VisionIlluminationBuilder(VisionBuildContext context, IlluminationState illumination) + { + m_context = context; + m_illumination = illumination; + } + + public IVisionIlluminationBuilder WithLampType(VisionLampTypeEnum lampType) + { + m_illumination.CreateOrReplaceLampType(m_context.Context, null); + m_illumination.LampType!.Value = lampType; + return this; + } + + public IVisionIlluminationBuilder WithWavelength(double nanometres) + { + m_illumination.CreateOrReplaceWavelength(m_context.Context, null); + m_illumination.Wavelength!.Value = nanometres; + return this; + } + + public IVisionIlluminationBuilder WithRelativeIntensity(double relativeIntensity) + { + m_illumination.CreateOrReplaceRelativeIntensity(m_context.Context, null); + m_illumination.RelativeIntensity!.Value = relativeIntensity; + return this; + } + + public IVisionIlluminationBuilder WithLightingMode(VisionLightingModeEnum lightingMode) + { + m_illumination.CreateOrReplaceLightingMode(m_context.Context, null); + m_illumination.LightingMode!.Value = lightingMode; + return this; + } + + private readonly VisionBuildContext m_context; + private readonly IlluminationState m_illumination; + } + + internal sealed class VisionIntrinsicCalibrationBuilder : IVisionIntrinsicCalibrationBuilder + { + public VisionIntrinsicCalibrationBuilder( + VisionBuildContext context, + IntrinsicCalibrationState calibration) + { + m_context = context; + m_calibration = calibration; + } + + public IVisionIntrinsicCalibrationBuilder WithCalibrationId(string calibrationId) + { + m_calibration.CreateOrReplaceCalibrationId(m_context.Context, null); + m_calibration.CalibrationId!.Value = calibrationId ?? string.Empty; + return this; + } + + public IVisionIntrinsicCalibrationBuilder WithIntrinsics(VisionIntrinsicsDataType intrinsics) + { + if (intrinsics == null) + { + throw new ArgumentNullException(nameof(intrinsics)); + } + m_calibration.CreateOrReplaceIntrinsics(m_context.Context, null); + m_calibration.Intrinsics!.Value = intrinsics; + return this; + } + + public IVisionIntrinsicCalibrationBuilder WithResidualError(double residualError) + { + m_calibration.CreateOrReplaceResidualError(m_context.Context, null); + m_calibration.ResidualError!.Value = residualError; + return this; + } + + public IVisionIntrinsicCalibrationBuilder WithMethod(string method) + { + m_calibration.CreateOrReplaceMethod(m_context.Context, null); + m_calibration.Method!.Value = method ?? string.Empty; + return this; + } + + private readonly VisionBuildContext m_context; + private readonly IntrinsicCalibrationState m_calibration; + } + + internal sealed class VisionExtrinsicCalibrationBuilder : IVisionExtrinsicCalibrationBuilder + { + public VisionExtrinsicCalibrationBuilder( + VisionBuildContext context, + ExtrinsicCalibrationState calibration) + { + m_context = context; + m_calibration = calibration; + } + + public IVisionExtrinsicCalibrationBuilder WithCalibrationId(string calibrationId) + { + m_calibration.CreateOrReplaceCalibrationId(m_context.Context, null); + m_calibration.CalibrationId!.Value = calibrationId ?? string.Empty; + return this; + } + + public IVisionExtrinsicCalibrationBuilder WithMount(VisionCalibrationMountEnum mount) + { + m_calibration.CreateOrReplaceMount(m_context.Context, null); + m_calibration.Mount!.Value = mount; + return this; + } + + public IVisionExtrinsicCalibrationBuilder WithFrames(string sourceFrame, string targetFrame) + { + m_calibration.CreateOrReplaceSourceFrame(m_context.Context, null); + m_calibration.CreateOrReplaceTargetFrame(m_context.Context, null); + NodeId sourceNodeId = m_context.Registry.TryFindFrameByFrameId(sourceFrame)?.NodeId ?? NodeId.Null; + NodeId targetNodeId = m_context.Registry.TryFindFrameByFrameId(targetFrame)?.NodeId ?? NodeId.Null; + m_calibration.SourceFrame!.Value = sourceNodeId; + m_calibration.TargetFrame!.Value = targetNodeId; + m_sourceFrameId = sourceFrame ?? string.Empty; + m_targetFrame = targetFrame ?? string.Empty; + m_context.Registry.AddDeferredExtrinsicResolution(m_calibration, m_sourceFrameId, m_targetFrame); + return this; + } + + public IVisionExtrinsicCalibrationBuilder WithTransform(VisionPose3DDataType transform) + { + if (transform == null) + { + throw new ArgumentNullException(nameof(transform)); + } + m_calibration.CreateOrReplaceTransform(m_context.Context, null); + m_calibration.Transform!.Value = new VisionPose3DDataType + { + FrameId = string.IsNullOrEmpty(transform.FrameId) ? m_targetFrame : transform.FrameId, + Position = transform.Position, + Orientation = transform.Orientation, + Covariance = transform.Covariance + }; + return this; + } + + public IVisionExtrinsicCalibrationBuilder WithResidualError(double residualError) + { + m_calibration.CreateOrReplaceResidualError(m_context.Context, null); + m_calibration.ResidualError!.Value = residualError; + return this; + } + + private readonly VisionBuildContext m_context; + private readonly ExtrinsicCalibrationState m_calibration; + private string m_sourceFrameId = string.Empty; + private string m_targetFrame = string.Empty; + } + + internal sealed class VisionStreamEndpointBuilder : IVisionStreamEndpointBuilder + { + public VisionStreamEndpointBuilder(VisionBuildContext context, StreamEndpointState endpoint) + { + m_context = context; + m_endpoint = endpoint; + } + + public IVisionStreamEndpointBuilder WithEndpointId(string endpointId) + { + m_endpoint.CreateOrReplaceEndpointId(m_context.Context, null); + m_endpoint.EndpointId!.Value = endpointId ?? string.Empty; + return this; + } + + public IVisionStreamEndpointBuilder WithEndpointUri(string endpointUri) + { + m_endpoint.CreateOrReplaceEndpointUri(m_context.Context, null); + m_endpoint.EndpointUri!.Value = endpointUri ?? string.Empty; + return this; + } + + public IVisionStreamEndpointBuilder WithProtocol(VisionStreamProtocolEnum protocol) + { + m_endpoint.CreateOrReplaceStreamProtocol(m_context.Context, null); + m_endpoint.StreamProtocol!.Value = protocol; + return this; + } + + public IVisionStreamEndpointBuilder WithCodec(VisionVideoCodecEnum codec) + { + m_endpoint.CreateOrReplaceCodec(m_context.Context, null); + m_endpoint.Codec!.Value = codec; + return this; + } + + public IVisionStreamEndpointBuilder WithResolution(uint width, uint height) + { + m_endpoint.CreateOrReplaceWidth(m_context.Context, null); + m_endpoint.CreateOrReplaceHeight(m_context.Context, null); + m_endpoint.Width!.Value = width; + m_endpoint.Height!.Value = height; + return this; + } + + public IVisionStreamEndpointBuilder WithFrameRate(double frameRate) + { + m_endpoint.CreateOrReplaceFrameRate(m_context.Context, null); + m_endpoint.FrameRate!.Value = frameRate; + return this; + } + + public IVisionStreamEndpointBuilder WithBitrate(uint bitrate) + { + m_endpoint.CreateOrReplaceBitrate(m_context.Context, null); + m_endpoint.Bitrate!.Value = bitrate; + return this; + } + + public IVisionStreamEndpointBuilder WithDefaultProfileName(string defaultProfileName) + { + m_endpoint.CreateOrReplaceDefaultProfileName(m_context.Context, null); + m_endpoint.DefaultProfileName!.Value = defaultProfileName ?? string.Empty; + return this; + } + + private readonly VisionBuildContext m_context; + private readonly StreamEndpointState m_endpoint; + } + + internal sealed class VisionClipEndpointBuilder : IVisionClipEndpointBuilder + { + public VisionClipEndpointBuilder(VisionBuildContext context, ClipEndpointState endpoint) + { + m_context = context; + m_endpoint = endpoint; + } + + public IVisionClipEndpointBuilder WithEndpointId(string endpointId) + { + m_endpoint.CreateOrReplaceEndpointId(m_context.Context, null); + m_endpoint.EndpointId!.Value = endpointId ?? string.Empty; + return this; + } + + public IVisionClipEndpointBuilder WithEndpointUri(string endpointUri) + { + m_endpoint.CreateOrReplaceEndpointUri(m_context.Context, null); + m_endpoint.EndpointUri!.Value = endpointUri ?? string.Empty; + return this; + } + + public IVisionClipEndpointBuilder WithClipFormat(VisionClipFormatEnum format) + { + m_endpoint.CreateOrReplaceClipFormat(m_context.Context, null); + m_endpoint.ClipFormat!.Value = format; + return this; + } + + public IVisionClipEndpointBuilder WithQuality(uint quality) + { + m_endpoint.CreateOrReplaceQuality(m_context.Context, null); + m_endpoint.Quality!.Value = quality; + return this; + } + + public IVisionClipEndpointBuilder WithResolution(uint width, uint height) + { + m_endpoint.CreateOrReplaceWidth(m_context.Context, null); + m_endpoint.CreateOrReplaceHeight(m_context.Context, null); + m_endpoint.Width!.Value = width; + m_endpoint.Height!.Value = height; + return this; + } + + public IVisionClipEndpointBuilder WithInlineDelivery(bool enabled, uint maxInlineClipSize) + { + m_endpoint.CreateOrReplaceInlineDeliveryEnabled(m_context.Context, null); + m_endpoint.CreateOrReplaceMaxInlineClipSize(m_context.Context, null); + m_endpoint.CreateOrReplaceLatestClip(m_context.Context, null); + m_endpoint.CreateOrReplaceLatestClipMetadata(m_context.Context, null); + m_endpoint.InlineDeliveryEnabled!.Value = enabled; + m_endpoint.MaxInlineClipSize!.Value = maxInlineClipSize; + if (!enabled) + { + m_endpoint.LatestClip!.StatusCode = StatusCodes.BadNotSupported; + } + else + { + m_endpoint.LatestClip!.StatusCode = StatusCodes.BadNoDataAvailable; + } + m_endpoint.LatestClipMetadata!.StatusCode = StatusCodes.BadNoDataAvailable; + return this; + } + + public IVisionClipEndpointBuilder WithDefaultProfileName(string defaultProfileName) + { + m_endpoint.CreateOrReplaceDefaultProfileName(m_context.Context, null); + m_endpoint.DefaultProfileName!.Value = defaultProfileName ?? string.Empty; + return this; + } + + private readonly VisionBuildContext m_context; + private readonly ClipEndpointState m_endpoint; + } + + internal sealed class VisionPipelineBuilder : IVisionPipelineBuilder + { + public VisionPipelineBuilder( + VisionBuildContext context, + VisionRegistry registry, + VisionMethodDispatcher dispatcher, + InferencePipelineState pipeline, + string browseName) + { + m_context = context; + m_registry = registry; + m_dispatcher = dispatcher; + m_pipeline = pipeline; + m_browseName = browseName; + } + + public IVisionPipelineBuilder WithPipelineId(string pipelineId) + { + m_pipeline.CreateOrReplacePipelineId(m_context.Context, null); + m_pipeline.PipelineId!.Value = pipelineId ?? string.Empty; + return this; + } + + public IVisionPipelineBuilder WithSensor(NodeId sensorNodeId) + { + m_pipeline.CreateOrReplaceSensor(m_context.Context, null); + m_pipeline.Sensor!.Value = sensorNodeId; + return this; + } + + public IVisionPipelineBuilder WithDeployment(NodeId deploymentNodeId) + { + m_pipeline.CreateOrReplaceDeployment(m_context.Context, null); + m_pipeline.Deployment!.Value = deploymentNodeId; + return this; + } + + public IVisionPipelineBuilder WithLearningJob(NodeId learningJobNodeId) + { + m_pipeline.CreateOrReplaceLearningJob(m_context.Context, null); + m_pipeline.LearningJob!.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasProperty; + m_pipeline.LearningJob.TypeDefinitionId = global::Opc.Ua.VariableTypeIds.PropertyType; + m_pipeline.LearningJob.Value = learningJobNodeId; + return this; + } + + public IVisionPipelineBuilder ProducedBy(NodeId producerNodeId) + { + if (!producerNodeId.IsNull) + { + m_pipeline.AddReference( + ExpandedNodeId.ToNodeId( + Opc.Ua.Vision.ReferenceTypeIds.ProducedBy, + m_context.Context.NamespaceUris), + false, + producerNodeId); + } + return this; + } + + public IVisionPipelineBuilder UseInferenceProvider(IVisionInferenceProvider provider, bool onServer = true) + { + if (provider == null) + { + throw new ArgumentNullException(nameof(provider)); + } + m_inferenceProvider = provider; + m_inferenceOnServer = onServer; + return this; + } + + public IVisionPipelineBuilder UseFeedbackSink(IVisionFeedbackSink sink) + { + if (sink == null) + { + throw new ArgumentNullException(nameof(sink)); + } + m_feedbackSink = sink; + return this; + } + + internal void Finalize(FolderState parent) + { + parent.AddChild(m_pipeline); + EnsureMethods(); + var facets = new HashSet(StringComparer.Ordinal); + if (m_inferenceProvider != null) + { + facets.Add(m_inferenceOnServer + ? VisionConformanceUris.FacetNames.InferenceOnServer + : VisionConformanceUris.FacetNames.InferenceOffServer); + } + if (m_feedbackSink != null) + { + facets.Add(VisionConformanceUris.FacetNames.Feedback); + } + var registration = new PipelineRegistration( + m_browseName, + m_pipeline.NodeId, + m_pipeline, + facets) + { + InferenceProvider = m_inferenceProvider, + FeedbackSink = m_feedbackSink + }; + m_registry.AddPipeline(registration); + m_dispatcher.AttachPipelineMethods(m_pipeline.NodeId, m_pipeline); + if (m_pipeline.Feedback is VisionFeedbackState feedback) + { + m_dispatcher.AttachFeedbackMethods(m_pipeline.NodeId, feedback); + } + } + + private void EnsureMethods() + { + ISystemContext context = m_context.Context; + if (m_inferenceProvider != null) + { + FolderState results = m_pipeline.CreateOrReplaceResults(context, null); + results.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent; + results.TypeDefinitionId = global::Opc.Ua.ObjectTypeIds.FolderType; + Declare( + m_pipeline.CreateOrReplaceRunInference(context, null), + MethodIds.InferencePipelineType_RunInference, + VisionMethodArguments.Declare); + Declare( + m_pipeline.CreateOrReplaceStartContinuous(context, null), + MethodIds.InferencePipelineType_StartContinuous, + VisionMethodArguments.DeclareStartContinuous); + Declare( + m_pipeline.CreateOrReplaceStop(context, null), + MethodIds.InferencePipelineType_Stop, + VisionMethodArguments.DeclareStop); + } + if (m_feedbackSink != null) + { + m_pipeline.CreateOrReplaceFeedback(context, null); + VisionFeedbackState feedback = m_pipeline.Feedback!; + feedback.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent; + feedback.TypeDefinitionId = ExpandedNodeId.ToNodeId( + ObjectTypeIds.VisionFeedbackType, context.NamespaceUris); + Declare( + feedback.CreateOrReplaceSubmitDetections(context, null), + MethodIds.VisionFeedbackType_SubmitDetections, + VisionMethodArguments.Declare); + Declare( + feedback.CreateOrReplaceSubmitInspectionResult(context, null), + MethodIds.VisionFeedbackType_SubmitInspectionResult, + VisionMethodArguments.Declare); + Declare( + feedback.CreateOrReplaceSubmitCorrection(context, null), + MethodIds.VisionFeedbackType_SubmitCorrection, + VisionMethodArguments.Declare); + Declare( + feedback.CreateOrReplaceSubmitImageReference(context, null), + MethodIds.VisionFeedbackType_SubmitImageReference, + VisionMethodArguments.Declare); + } + } + + private void Declare( + TMethod method, + ExpandedNodeId declarationId, + Action declare) + where TMethod : MethodState + { + declare(m_context.Context, method); + method.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent; + method.MethodDeclarationId = ExpandedNodeId.ToNodeId( + declarationId, m_context.Context.NamespaceUris); + } + + private readonly VisionBuildContext m_context; + private readonly VisionRegistry m_registry; + private readonly VisionMethodDispatcher m_dispatcher; + private readonly InferencePipelineState m_pipeline; + private readonly string m_browseName; + private IVisionInferenceProvider? m_inferenceProvider; + private IVisionFeedbackSink? m_feedbackSink; + private bool m_inferenceOnServer = true; + } +} diff --git a/src/Opc.Ua.Vision.Server/Hosting/OpcUaServerVisionBuilderExtensions.cs b/src/Opc.Ua.Vision.Server/Hosting/OpcUaServerVisionBuilderExtensions.cs new file mode 100644 index 0000000000..3c84ff350d --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Hosting/OpcUaServerVisionBuilderExtensions.cs @@ -0,0 +1,359 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Diagnostics.CodeAnalysis; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Options; +using Opc.Ua.Server; +using Opc.Ua.Server.Hosting; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Hosting; + +namespace Microsoft.Extensions.DependencyInjection +{ + /// + /// Hosting extensions for OPC UA Vision servers. + /// + public static class OpcUaServerVisionBuilderExtensions + { + /// + /// Registers the standalone Vision node manager and its default + /// hosting glue. + /// + /// is null. + public static IOpcUaServerBuilder AddVision( + this IOpcUaServerBuilder builder, + Action? configure = null) + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + builder.Services.AddOptions(); + if (configure != null) + { + builder.Services.Configure(configure); + } + builder.Services.TryAddEnumerable( + ServiceDescriptor.Singleton()); + builder.Services.TryAddSingleton(); + builder.Services.AddSingleton(static services => + { + VisionServerOptions options = services + .GetRequiredService>() + .Value; + IVisionModelProvider[] providers = [.. services.GetServices()]; + return new VisionNodeManagerFactory( + providers, + options, + services.GetService()); + }); + builder.Services.AddSingleton(static services => + { + VisionServerOptions options = services + .GetRequiredService>() + .Value; + IVisionModelProvider[] providers = [.. services.GetServices()]; + return new VisionHostedNodeManagerFactory( + providers, + options, + services.GetService(), + services); + }); + builder.Services.AddSingleton(static services => + new OpcUaServerNodeManagerRegistration( + services.GetRequiredService())); + return builder; + } + + /// + /// Registers a media provider for one sensor browse name. + /// + /// + /// The media provider type. Resolved via the DI container. + /// + /// is null. + /// + public static IOpcUaServerBuilder AddVisionMediaProvider< + [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] TProvider>( + this IOpcUaServerBuilder builder, + string sensorBrowseName) + where TProvider : class, IVisionMediaProvider + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + if (string.IsNullOrEmpty(sensorBrowseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(sensorBrowseName)); + } + builder.Services.AddSingleton(); + builder.Services.AddSingleton(services => + new VisionMediaProviderRegistration( + sensorBrowseName, + services.GetRequiredService())); + return builder; + } + + /// + /// Registers a media provider instance for one sensor browse name. + /// + /// is null. + /// + public static IOpcUaServerBuilder AddVisionMediaProvider( + this IOpcUaServerBuilder builder, + string sensorBrowseName, + IVisionMediaProvider provider) + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + if (string.IsNullOrEmpty(sensorBrowseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(sensorBrowseName)); + } + if (provider == null) + { + throw new ArgumentNullException(nameof(provider)); + } + builder.Services.AddSingleton(new VisionMediaProviderRegistration(sensorBrowseName, provider)); + return builder; + } + + /// + /// Registers an inference provider for one pipeline browse name. + /// + /// + /// The inference provider type. Resolved via the DI container. + /// + /// + /// The flag controls whether the + /// server advertises the on-server or off-server inference facet + /// (§8.2). + /// + /// is null. + /// + public static IOpcUaServerBuilder AddVisionInferenceProvider< + [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] TProvider>( + this IOpcUaServerBuilder builder, + string pipelineBrowseName, + bool onServer = true) + where TProvider : class, IVisionInferenceProvider + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + if (string.IsNullOrEmpty(pipelineBrowseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(pipelineBrowseName)); + } + builder.Services.AddSingleton(); + builder.Services.AddSingleton(services => + new VisionInferenceProviderRegistration( + pipelineBrowseName, + services.GetRequiredService(), + onServer)); + return builder; + } + + /// + /// Registers an inference provider instance for one pipeline browse name. + /// + /// is null. + /// + public static IOpcUaServerBuilder AddVisionInferenceProvider( + this IOpcUaServerBuilder builder, + string pipelineBrowseName, + IVisionInferenceProvider provider, + bool onServer = true) + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + if (string.IsNullOrEmpty(pipelineBrowseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(pipelineBrowseName)); + } + if (provider == null) + { + throw new ArgumentNullException(nameof(provider)); + } + builder.Services.AddSingleton( + new VisionInferenceProviderRegistration(pipelineBrowseName, provider, onServer)); + return builder; + } + + /// + /// Registers a feedback sink for one pipeline browse name. + /// + /// + /// The feedback sink type. Resolved via the DI container. + /// + /// is null. + /// + public static IOpcUaServerBuilder AddVisionFeedbackSink< + [DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] TSink>( + this IOpcUaServerBuilder builder, + string pipelineBrowseName) + where TSink : class, IVisionFeedbackSink + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + if (string.IsNullOrEmpty(pipelineBrowseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(pipelineBrowseName)); + } + builder.Services.AddSingleton(); + builder.Services.AddSingleton(services => + new VisionFeedbackSinkRegistration( + pipelineBrowseName, + services.GetRequiredService())); + return builder; + } + + /// + /// Registers a feedback sink instance for one pipeline browse name. + /// + /// is null. + /// + public static IOpcUaServerBuilder AddVisionFeedbackSink( + this IOpcUaServerBuilder builder, + string pipelineBrowseName, + IVisionFeedbackSink sink) + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + if (string.IsNullOrEmpty(pipelineBrowseName)) + { + throw new ArgumentException("A non-empty value is required.", nameof(pipelineBrowseName)); + } + if (sink == null) + { + throw new ArgumentNullException(nameof(sink)); + } + builder.Services.AddSingleton(new VisionFeedbackSinkRegistration(pipelineBrowseName, sink)); + return builder; + } + + /// + /// Registers a Vision configurator for the standalone manager. + /// + public static IOpcUaServerBuilder ConfigureVision( + this IOpcUaServerBuilder builder, + Func configure) + { + return builder.ConfigureVisionFor(configure); + } + + /// + /// Registers a Vision configurator for the standalone manager. + /// + /// is null. + public static IOpcUaServerBuilder ConfigureVision( + this IOpcUaServerBuilder builder, + Action configure) + { + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + return builder.ConfigureVision((context, _) => + { + configure(context); + return default; + }); + } + + /// + /// Registers a Vision configurator for a specific Vision node + /// manager type. + /// + /// + /// The target Vision node manager type. Only + /// is currently supported. + /// + /// is null. + /// + public static IOpcUaServerBuilder ConfigureVisionFor( + this IOpcUaServerBuilder builder, + Func configure) + where TNodeManager : AsyncCustomNodeManager + { + if (builder == null) + { + throw new ArgumentNullException(nameof(builder)); + } + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + if (typeof(TNodeManager) != typeof(VisionNodeManager)) + { + throw new NotSupportedException( + "ConfigureVisionFor is supported only for VisionNodeManager. " + + "Use ConfigureVision for the standalone manager."); + } + builder.Services.TryAddSingleton(); + builder.Services.AddSingleton( + new DelegateVisionConfigurator(typeof(TNodeManager), configure)); + return builder; + } + + private sealed class DelegateVisionConfigurator : IVisionPostSetupConfigurator + { + public DelegateVisionConfigurator( + Type targetManagerType, + Func configure) + { + TargetManagerType = targetManagerType; + m_configure = configure; + } + + public Type TargetManagerType { get; } + + public ValueTask RunAsync(IVisionBuildContext context) + { + return m_configure(context, context.CancellationToken); + } + + private readonly Func m_configure; + } + } +} diff --git a/src/Opc.Ua.Vision.Server/Hosting/VisionHostedNodeManagerFactory.cs b/src/Opc.Ua.Vision.Server/Hosting/VisionHostedNodeManagerFactory.cs new file mode 100644 index 0000000000..0877428802 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Hosting/VisionHostedNodeManagerFactory.cs @@ -0,0 +1,80 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Diagnostics.CodeAnalysis; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Server; + +namespace Opc.Ua.Vision.Server.Hosting +{ + internal sealed class VisionHostedNodeManagerFactory : IAsyncNodeManagerFactory + { + public VisionHostedNodeManagerFactory( + ArrayOf providers, + VisionServerOptions options, + IVisionPostSetupRunner? runner, + IServiceProvider services) + { + m_providers = VisionNodeManager.NormalizeProviders(providers); + m_options = options ?? throw new ArgumentNullException(nameof(options)); + m_options.Validate(); + m_runner = runner; + m_services = services ?? throw new ArgumentNullException(nameof(services)); + } + + public ArrayOf NamespacesUris => + VisionNodeManager.GetNamespaceUris(m_providers, m_options).ToArrayOf(); + + [SuppressMessage( + "Reliability", + "CA2000:Dispose objects before losing scope", + Justification = "Ownership is transferred to the server.")] + public ValueTask CreateAsync( + IServerInternal server, + ApplicationConfiguration configuration, + CancellationToken cancellationToken = default) + { + IAsyncNodeManager manager = new VisionNodeManager( + server, + configuration, + m_providers, + m_options, + m_runner, + m_services); + return new ValueTask(manager); + } + + private readonly ArrayOf m_providers; + private readonly VisionServerOptions m_options; + private readonly IVisionPostSetupRunner? m_runner; + private readonly IServiceProvider m_services; + } +} diff --git a/src/Opc.Ua.Vision.Server/Hosting/VisionPostSetup.cs b/src/Opc.Ua.Vision.Server/Hosting/VisionPostSetup.cs new file mode 100644 index 0000000000..3d431875e8 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Hosting/VisionPostSetup.cs @@ -0,0 +1,160 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Server; + +namespace Opc.Ua.Vision.Server.Hosting +{ + /// + /// Runs Vision startup configurators. + /// + public interface IVisionPostSetupRunner + { + /// + /// Runs configurators for a node manager. + /// + ValueTask RunAsync( + AsyncCustomNodeManager manager, + VisionRootState root, + VisionServerOptions options, + CancellationToken cancellationToken); + } + + internal interface IVisionPostSetupConfigurator + { + Type TargetManagerType { get; } + + ValueTask RunAsync(IVisionBuildContext context); + } + + internal sealed class VisionPostSetupRunner : IVisionPostSetupRunner + { + public VisionPostSetupRunner( + IServiceProvider services, + IEnumerable configurators, + IEnumerable mediaRegistrations, + IEnumerable inferenceRegistrations, + IEnumerable feedbackRegistrations) + { + m_services = services; + m_configurators = configurators.ToArray().ToArrayOf(); + MediaRegistrations = mediaRegistrations.ToArray().ToArrayOf(); + InferenceRegistrations = inferenceRegistrations.ToArray().ToArrayOf(); + FeedbackRegistrations = feedbackRegistrations.ToArray().ToArrayOf(); + } + + public async ValueTask RunAsync( + AsyncCustomNodeManager manager, + VisionRootState root, + VisionServerOptions options, + CancellationToken cancellationToken) + { + if (manager == null) + { + throw new ArgumentNullException(nameof(manager)); + } + if (manager is not VisionNodeManager visionManager) + { + return; + } + VisionBuildContext context = visionManager.CreateBuildContextCore(cancellationToken); + for (int ii = 0; ii < m_configurators.Count; ii++) + { + IVisionPostSetupConfigurator configurator = m_configurators[ii]; + if (configurator.TargetManagerType.IsAssignableFrom(manager.GetType())) + { + await configurator.RunAsync(context).ConfigureAwait(false); + await context.FlushPendingRegistrationsAsync(cancellationToken).ConfigureAwait(false); + } + } + await context.FlushPendingRegistrationsAsync(cancellationToken).ConfigureAwait(false); + } + + internal ArrayOf MediaRegistrations { get; } + + internal ArrayOf InferenceRegistrations { get; } + + internal ArrayOf FeedbackRegistrations { get; } + + private readonly IServiceProvider m_services; + private readonly ArrayOf m_configurators; + } + + internal sealed class VisionMediaProviderRegistration + { + public VisionMediaProviderRegistration(string sensorBrowseName, IVisionMediaProvider provider) + { + SensorBrowseName = sensorBrowseName ?? throw new ArgumentNullException(nameof(sensorBrowseName)); + Provider = provider ?? throw new ArgumentNullException(nameof(provider)); + } + + public string SensorBrowseName { get; } + + public IVisionMediaProvider Provider { get; } + } + + internal sealed class VisionInferenceProviderRegistration + { + public VisionInferenceProviderRegistration( + string pipelineBrowseName, + IVisionInferenceProvider provider, + bool onServer) + { + PipelineBrowseName = pipelineBrowseName ?? + throw new ArgumentNullException(nameof(pipelineBrowseName)); + Provider = provider ?? throw new ArgumentNullException(nameof(provider)); + OnServer = onServer; + } + + public string PipelineBrowseName { get; } + + public IVisionInferenceProvider Provider { get; } + + public bool OnServer { get; } + } + + internal sealed class VisionFeedbackSinkRegistration + { + public VisionFeedbackSinkRegistration(string pipelineBrowseName, IVisionFeedbackSink sink) + { + PipelineBrowseName = pipelineBrowseName ?? + throw new ArgumentNullException(nameof(pipelineBrowseName)); + Sink = sink ?? throw new ArgumentNullException(nameof(sink)); + } + + public string PipelineBrowseName { get; } + + public IVisionFeedbackSink Sink { get; } + } +} diff --git a/src/Opc.Ua.Vision.Server/IVisionBuildContext.cs b/src/Opc.Ua.Vision.Server/IVisionBuildContext.cs new file mode 100644 index 0000000000..184e4189b8 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/IVisionBuildContext.cs @@ -0,0 +1,92 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using Opc.Ua.Server; +using Opc.Ua.Vision.Server.Builders; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Build-time surface exposed to Vision configurators. Sensors, + /// pipelines and coordinate frames are created through the fluent + /// entrypoints on ; low-level access to the + /// active node manager and system context is available for cases + /// where the fluent API does not model a particular customisation. + /// + public interface IVisionBuildContext + { + /// + /// Gets the active node manager. + /// + AsyncCustomNodeManager Manager { get; } + + /// + /// Gets the active system context. + /// + ISystemContext Context { get; } + + /// + /// Gets the application-owned instance namespace index. + /// + ushort InstanceNamespaceIndex { get; } + + /// + /// Gets the Vision namespace index. + /// + ushort VisionNamespaceIndex { get; } + + /// + /// Gets the well-known Server/Vision root object created by + /// the node manager. + /// + VisionRootState Root { get; } + + /// + /// Gets the fluent Vision node builder rooted at + /// . + /// + IVisionNodeBuilder Nodes { get; } + + /// + /// Gets the startup cancellation token. + /// + CancellationToken CancellationToken { get; } + + /// + /// Resolves a required application service. Throws when the + /// context was created without an . + /// + /// + /// The service type. + /// + T GetRequiredService() where T : notnull; + } +} diff --git a/src/Opc.Ua.Vision.Server/IVisionModelProvider.cs b/src/Opc.Ua.Vision.Server/IVisionModelProvider.cs new file mode 100644 index 0000000000..9885010e53 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/IVisionModelProvider.cs @@ -0,0 +1,93 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Contributes compiled predefined nodes to a Vision node manager. + /// + /// + /// The built-in loads the Vision + /// companion NodeSet. Applications can add extra providers to inject + /// vendor extensions or pre-materialised instances that are known at + /// startup. + /// + public interface IVisionModelProvider + { + /// + /// Gets the deterministic provider execution order. Providers run in + /// ascending order; the built-in provider uses int.MinValue so + /// application providers run afterwards by default. + /// + int Order { get; } + + /// + /// Gets the namespace URIs contributed by this provider. Providers + /// that replace the built-in provider must advertise the Vision + /// namespace URI. + /// + ArrayOf NamespaceUris { get; } + + /// + /// Adds predefined nodes into . + /// + void AddPredefinedNodes(NodeStateCollection nodes, ISystemContext context); + } + + /// + /// Built-in Vision model provider. + /// + public sealed class VisionModelProvider : IVisionModelProvider + { + /// + public int Order => int.MinValue; + + /// + public ArrayOf NamespaceUris => new string[] + { + global::Opc.Ua.Vision.Namespaces.Vision + }; + + /// + public void AddPredefinedNodes(NodeStateCollection nodes, ISystemContext context) + { + if (nodes == null) + { + throw new ArgumentNullException(nameof(nodes)); + } + if (context == null) + { + throw new ArgumentNullException(nameof(context)); + } + nodes.AddOpcUaVision(context); + } + } +} diff --git a/src/Opc.Ua.Vision.Server/NugetREADME.md b/src/Opc.Ua.Vision.Server/NugetREADME.md new file mode 100644 index 0000000000..64fee91118 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/NugetREADME.md @@ -0,0 +1,124 @@ +# Opc.Ua.Vision.Server + +Server hosting for the **draft** *OPC UA — Vision* companion model. + +The package materialises the well-known `Vision` object under the Server +object (i=2253), exposes fluent APIs to describe sensors, coordinate frames, +calibrations, media endpoints and inference pipelines, and wires host-supplied +providers behind the model's methods so a Server never hard-codes a +particular camera, scene or model. + +## What it gives you + +- `AddVision()` — registers the stock `VisionNodeManager` and factory in the + Generic Host pipeline. Requires no DI, Machinery or Robotics dependency: + the Vision NodeSet only requires the base UA namespace. +- `ConfigureVision(...)` / `ConfigureVisionFor(...)` — fluent + configurator that runs on server start. +- `VisionNodeManager` / `VisionNodeManagerFactory` — standalone node + manager for direct construction and for hosting extensions. +- `IVisionModelProvider` — deterministic composition of additional compiled + Vision-namespace providers. +- `IVisionMediaProvider`, `IVisionInferenceProvider`, + `IVisionFeedbackSink` — the provider abstractions a host implements to + supply media (streams, clips), run inference, and receive off-server + feedback. +- `IVisionBuildContext` and its `IVisionNodeBuilder` — the fluent surface + for adding coordinate frames, calibrations, sensors, media endpoints, + inference pipelines and feedback objects to the address space. +- Facet derivation from the materialised address space, published in + `Server.ServerCapabilities.ServerProfileArray` (VIS-Base, VIS-Media-*, + VIS-Calibration, VIS-Result-*, VIS-Feedback, VIS-Inference-OnServer / + VIS-Inference-OffServer, VIS-Simulation, VIS-Learning). + +## Example + +```csharp +using Microsoft.Extensions.Hosting; +using Opc.Ua; +using Opc.Ua.Server; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Builders; + +builder.Services + .AddOpcUa() + .AddServer(options => options.EndpointUrls.Add("opc.tcp://localhost:62855/VisionServer")) + .AddVision(options => options.InstanceNamespaceUri = "urn:example:vision:instances") + .AddVisionMediaProvider(sensorBrowseName: "Camera01") + .AddVisionInferenceProvider( + pipelineBrowseName: "Detector", + onServer: true) + .ConfigureVision((context, ct) => + { + IVisionNodeBuilder nodes = context.Nodes; + + nodes.AddFrame("World", f => f + .WithFrameId("world") + .WithRole(VisionFrameRoleEnum.World)); + + nodes.AddFrame("Flange", f => f + .WithFrameId("flange") + .WithRole(VisionFrameRoleEnum.MechanicalInterface) + .WithParent("world")); + + nodes.AddImageSensor("Camera01", s => s + .WithSensorId("cam-01") + .WithModality(VisionSensorModalityEnum.Area2D) + .WithFrameId("flange") + .WithResolution(1920u, 1080u) + .WithPixelFormat("Mono8") + .AddClipEndpoint("Clips", ep => ep + .WithEndpointId("clip-01") + .WithClipFormat(VisionClipFormatEnum.Png) + .WithResolution(1920u, 1080u) + .WithInlineDelivery(enabled: true, maxInlineClipSize: 8_388_608u))); + + nodes.AddPipeline("Detector", pipe => pipe + .WithPipelineId("pipe-01") + .WithSensor(NodeId.Null)); + + return ValueTask.CompletedTask; + }); +``` + +## Provider abstractions + +- Implement `IVisionMediaProvider` on the host to serve media without + putting pixels on OPC UA. `GetStreamAsync` returns a leased URI; + `GetClipAsync` returns a `VisionImageReferenceDataType` and, when the + caller asks for inline delivery and the encoded bytes fit the effective + limit, an inline `ByteString`. The §6.4 `Bad_NotSupported` / + `Bad_NoDataAvailable` / `Bad_EncodingLimitsExceeded` states are all + observable by a client through `LatestClip` while `LatestClipMetadata` + keeps returning the URI. +- Implement `IVisionInferenceProvider` to bind a pipeline to whatever + actually computes results — on the Server, on an edge GPU, in the cloud, + or in a simulator. The Server publishes the result nodes and applies the + spec's method conventions regardless of where inference runs. +- Implement `IVisionFeedbackSink` to receive `SubmitDetections`, + `SubmitInspectionResult`, `SubmitCorrection` and + `SubmitImageReference`. Off-server VLM agents publish through this path, + and the Server records what it did not compute. + +## Related packages + +| Package | Adds | +|---|---| +| `OPCFoundation.NetStandard.Opc.Ua.Vision` | Source-generated Vision model (required) | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Client` | High-level client for the same model | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.OpenUsd` | A reference `ISceneCameraCaptureProvider` that renders a `UsdGeomCamera` offscreen and reports `NoRenderingBackend` gracefully on CI | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Vision` | MCP tools that let a language-model agent drive a Vision server | + +See the [Vision developer guide](https://github.com/OPCFoundation/UA-.NETStandard/blob/main/docs/Vision.md) +for the hosting-API table, the full topology-builder surface, the two +perception paths (`OnServer` vs `EdgeOffServer`), facet derivation and the +bin-picking sample. + +> The namespace `http://opcfoundation.org/UA/Vision/` and every NodeId in it +> are **provisional**. The model is a working-group draft and is neither +> official nor endorsed by the OPC Foundation. + +## License + +OPC Foundation MIT License 1.00 — diff --git a/src/Opc.Ua.Vision.Server/Opc.Ua.Vision.Server.csproj b/src/Opc.Ua.Vision.Server/Opc.Ua.Vision.Server.csproj new file mode 100644 index 0000000000..4351e072ac --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Opc.Ua.Vision.Server.csproj @@ -0,0 +1,28 @@ + + + $(AssemblyPrefix).Vision.Server + $(LibTargetFrameworks) + $(PackagePrefix).Opc.Ua.Vision.Server + Opc.Ua.Vision.Server + $(NoWarn);CS1591 + enable + Server hosting for the draft OPC UA Vision companion model. Publishes the well-known Vision root under the Server object, wires host-supplied providers for media (streams, clips), inference and feedback, exposes a fluent build API, and derives conformance facets from the materialised address space. + true + NugetREADME.md + true + true + + + $(PackageId).Debug + + + + + + + + + + + + diff --git a/src/Opc.Ua.Vision.Server/Properties/AssemblyInfo.cs b/src/Opc.Ua.Vision.Server/Properties/AssemblyInfo.cs new file mode 100644 index 0000000000..7798c9bd57 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Properties/AssemblyInfo.cs @@ -0,0 +1,32 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +[assembly: CLSCompliant(false)] diff --git a/src/Opc.Ua.Vision.Server/Providers/IVisionFeedbackSink.cs b/src/Opc.Ua.Vision.Server/Providers/IVisionFeedbackSink.cs new file mode 100644 index 0000000000..16dc0de25a --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Providers/IVisionFeedbackSink.cs @@ -0,0 +1,127 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Vision; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Receives §9 feedback submissions from off-Server callers. + /// + /// + /// Vision feedback is a first-class publication path — it is how a + /// remote VLM or a supervising cell publishes results the Server + /// itself did not compute. A single sink is bound to one pipeline's + /// Feedback object. Implementations must be thread-safe. + /// + public interface IVisionFeedbackSink + { + /// + /// Publishes detections against the pipeline. The sink is + /// responsible for materialising a DetectionResultType + /// under Pipeline.Results when appropriate for the + /// 's purpose. + /// + ValueTask SubmitDetectionsAsync( + VisionSubmitDetectionsRequest request, + CancellationToken cancellationToken); + + /// + /// Publishes an inspection result, either creating a new result or + /// annotating an existing one referenced by + /// . + /// + ValueTask SubmitInspectionResultAsync( + VisionSubmitInspectionResultRequest request, + CancellationToken cancellationToken); + + /// + /// Submits a correction to a previously published result. + /// + ValueTask SubmitCorrectionAsync( + VisionSubmitCorrectionRequest request, + CancellationToken cancellationToken); + + /// + /// Registers an out-of-band image reference associated with the + /// pipeline. + /// + ValueTask SubmitImageReferenceAsync( + VisionSubmitImageReferenceRequest request, + CancellationToken cancellationToken); + } + + /// + /// Input to + /// . + /// + public sealed record VisionSubmitDetectionsRequest( + NodeId Pipeline, + VisionFeedbackPurposeEnum Purpose, + ArrayOf Detections, + VisionImageReferenceDataType FrameReference, + ByteString InlineImage, + bool SceneIsEmpty = false); + + /// + /// Input to + /// . + /// + public sealed record VisionSubmitInspectionResultRequest( + NodeId Pipeline, + string ResultId, + VisionResultEvaluationEnum Evaluation, + ArrayOf Characteristics); + + /// + /// Input to + /// . + /// + public sealed record VisionSubmitCorrectionRequest( + NodeId Pipeline, + string ResultId, + VisionFeedbackPurposeEnum Purpose, + ArrayOf CorrectedDetections, + ArrayOf CorrectedCharacteristics, + LocalizedText Reason, + ByteString InlineImage, + bool RetractAll = false); + + /// + /// Input to + /// . + /// + public sealed record VisionSubmitImageReferenceRequest( + NodeId Pipeline, + VisionFeedbackPurposeEnum Purpose, + VisionImageReferenceDataType Image, + string ResultId); +} diff --git a/src/Opc.Ua.Vision.Server/Providers/IVisionInferenceProvider.cs b/src/Opc.Ua.Vision.Server/Providers/IVisionInferenceProvider.cs new file mode 100644 index 0000000000..4f0fa798b7 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Providers/IVisionInferenceProvider.cs @@ -0,0 +1,93 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Vision; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Executes the inference for one InferencePipelineType node. + /// A pipeline binds a sensor to whatever actually computes results — + /// on the Server, on an edge GPU, in the cloud, or in a simulator. + /// + /// + /// The Server surfaces RunInference, StartContinuous + /// and Stop as OPC UA methods; providers only implement the + /// underlying operations. + /// + public interface IVisionInferenceProvider + { + /// + /// Runs a single-shot inference. The returned + /// is the id the + /// Server publishes on the pipeline's Results folder for + /// clients to inspect and reference in + /// submissions. + /// + ValueTask RunInferenceAsync( + VisionInferenceRunRequest request, + CancellationToken cancellationToken); + + /// + /// Starts continuous inference on the pipeline. Implementations + /// must be idempotent: calling this on an already-running + /// pipeline must succeed. + /// + ValueTask StartContinuousAsync( + NodeId pipeline, + CancellationToken cancellationToken); + + /// + /// Stops continuous inference on the pipeline. Implementations + /// must be idempotent. + /// + ValueTask StopAsync( + NodeId pipeline, + CancellationToken cancellationToken); + } + + /// + /// Input to . + /// + public readonly record struct VisionInferenceRunRequest( + NodeId Pipeline, + NodeId Sensor, + NodeId Deployment, + DateTimeUtc Timestamp); + + /// + /// Result of + /// . + /// + public sealed record VisionInferenceRunResult( + ServiceResult ServiceResult, + string ResultId); +} diff --git a/src/Opc.Ua.Vision.Server/Providers/IVisionMediaProvider.cs b/src/Opc.Ua.Vision.Server/Providers/IVisionMediaProvider.cs new file mode 100644 index 0000000000..ed5431a04b --- /dev/null +++ b/src/Opc.Ua.Vision.Server/Providers/IVisionMediaProvider.cs @@ -0,0 +1,155 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Vision; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Supplies the pixel and clip source that the Vision media manager + /// serves. The Server never sees pixels — the provider supplies a + /// leased URI for a live stream and optional inline bytes for a clip. + /// + /// + /// One provider instance is bound to one sensor. Providers must be + /// thread-safe: OPC UA method calls arrive concurrently. + /// + public interface IVisionMediaProvider + { + /// + /// Leases a live-stream session against the requested endpoint and + /// profile. + /// + /// + /// A session descriptor with a URI, expiry time and a per-session + /// token the caller uses in + /// . + /// + ValueTask GetStreamAsync( + VisionStreamRequest request, + CancellationToken cancellationToken); + + /// + /// Releases a session previously granted by + /// . Returns + /// even when the session is unknown so idempotent callers are safe. + /// + ValueTask ReleaseStreamAsync( + ByteString sessionToken, + CancellationToken cancellationToken); + + /// + /// Applies the requested media configuration to a stream endpoint. + /// Servers that only support single-shot configuration must return + /// . + /// + ValueTask ConfigureStreamAsync( + VisionStreamConfigurationRequest request, + CancellationToken cancellationToken); + + /// + /// Selects the preferred stream and clip endpoints on the media + /// manager. This is a pure address-space update; providers may + /// simply return . + /// + ValueTask SelectEndpointAsync( + NodeId streamEndpoint, + NodeId clipEndpoint, + CancellationToken cancellationToken); + + /// + /// Fetches a clip and, if requested and permitted, encodes it as + /// inline bytes. + /// + /// + /// The Server enforces §6.4 rules over the returned response: + /// inline delivery is refused when the clip endpoint has + /// InlineDeliveryEnabled set to false, or when the + /// encoded byte count exceeds MaxInlineClipSize. + /// + ValueTask GetClipAsync( + VisionClipRequest request, + CancellationToken cancellationToken); + } + + /// + /// Input to . + /// + public readonly record struct VisionStreamRequest( + NodeId Endpoint, + string ProfileName, + VisionStreamProtocolEnum PreferredProtocol); + + /// + /// Result of . + /// + public sealed record VisionStreamLease( + ServiceResult ServiceResult, + VisionStreamSessionDataType Session, + NodeId EndpointOut); + + /// + /// Input to . + /// + public readonly record struct VisionStreamConfigurationRequest( + NodeId Endpoint, + VisionVideoCodecEnum Codec, + uint Width, + uint Height, + double FrameRate, + uint Bitrate); + + /// + /// Input to . + /// + public readonly record struct VisionClipRequest( + NodeId Endpoint, + string ResultId, + DateTimeUtc Timestamp, + VisionClipFormatEnum Format, + bool RequestInline); + + /// + /// Result of . + /// + /// + /// should be the default (null) + /// when the caller did not request inline bytes or when the provider + /// cannot supply them; the Server surfaces + /// if the payload + /// exceeds the clip endpoint's declared limit. + /// + public sealed record VisionClipResult( + ServiceResult ServiceResult, + VisionImageReferenceDataType Image, + NodeId EndpointOut, + ByteString InlineImage); +} diff --git a/src/Opc.Ua.Vision.Server/VisionBuildContext.cs b/src/Opc.Ua.Vision.Server/VisionBuildContext.cs new file mode 100644 index 0000000000..6eeb1f3fef --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionBuildContext.cs @@ -0,0 +1,171 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Opc.Ua.Server; +using Opc.Ua.Vision.Server.Builders; + +namespace Opc.Ua.Vision.Server +{ + internal sealed class VisionBuildContext : IVisionBuildContext + { + public VisionBuildContext( + AsyncCustomNodeManager manager, + VisionRootState root, + VisionServerOptions options, + VisionRegistry registry, + VisionMethodDispatcher dispatcher, + CancellationToken cancellationToken, + IServiceProvider? services = null) + { + if (manager == null) + { + throw new ArgumentNullException(nameof(manager)); + } + if (root == null) + { + throw new ArgumentNullException(nameof(root)); + } + if (options == null) + { + throw new ArgumentNullException(nameof(options)); + } + if (registry == null) + { + throw new ArgumentNullException(nameof(registry)); + } + if (dispatcher == null) + { + throw new ArgumentNullException(nameof(dispatcher)); + } + options.Validate(); + Manager = manager; + Root = root; + Context = manager.SystemContext; + CancellationToken = cancellationToken; + m_services = services; + int instanceIndex = Context.NamespaceUris.GetIndex(options.InstanceNamespaceUri); + if (instanceIndex < 0) + { + throw ServiceResultException.Create( + StatusCodes.BadConfigurationError, + "The Vision instance namespace '{0}' is not registered.", + options.InstanceNamespaceUri); + } + InstanceNamespaceIndex = (ushort)instanceIndex; + int visionIndex = Context.NamespaceUris.GetIndex(global::Opc.Ua.Vision.Namespaces.Vision); + if (visionIndex < 0) + { + throw ServiceResultException.Create( + StatusCodes.BadConfigurationError, + "The Vision namespace '{0}' is not registered.", + global::Opc.Ua.Vision.Namespaces.Vision); + } + VisionNamespaceIndex = (ushort)visionIndex; + Registry = registry; + Nodes = new VisionNodeBuilder(this, registry, dispatcher); + } + + public AsyncCustomNodeManager Manager { get; } + + public ISystemContext Context { get; } + + public ushort InstanceNamespaceIndex { get; } + + public ushort VisionNamespaceIndex { get; } + + public VisionRootState Root { get; } + + public IVisionNodeBuilder Nodes { get; } + + internal VisionRegistry Registry { get; } + + public CancellationToken CancellationToken { get; } + + public T GetRequiredService() where T : notnull + { + if (m_services == null) + { + throw new InvalidOperationException( + "Application services are unavailable for a directly created Vision build context."); + } + return m_services.GetRequiredService(); + } + + internal void EnqueueForRegistration(NodeState node) + { + if (node == null || node.NodeId.IsNull) + { + return; + } + if (m_pendingRegistrationSet.Add(node)) + { + m_pendingRegistrations.Add(node); + } + } + + internal async ValueTask FlushPendingRegistrationsAsync(CancellationToken cancellationToken) + { + for (int ii = 0; ii < m_pendingRegistrations.Count; ii++) + { + NodeState node = m_pendingRegistrations[ii]; + NormalizeInstanceMetadata(node); + if (Manager.FindPredefinedNode(node.NodeId) == null) + { + await Manager.AddPredefinedNodeAsync(node, cancellationToken).ConfigureAwait(false); + } + } + m_pendingRegistrations.Clear(); + m_pendingRegistrationSet.Clear(); + } + + /// + /// Gives every child a reference type and a type definition. + /// + /// + /// Delegates to , which applies the same + /// normalization to every node it registers — including results + /// published at runtime, which never pass through this builder. Kept + /// here so a context whose manager is not a VisionNodeManager + /// still produces valid instance nodes. + /// + private void NormalizeInstanceMetadata(NodeState node) + { + VisionNodeManager.NormalizeInstanceMetadata(Context, node); + } + + private readonly IServiceProvider? m_services; + private readonly List m_pendingRegistrations = []; + private readonly HashSet m_pendingRegistrationSet = []; + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionConformanceUris.cs b/src/Opc.Ua.Vision.Server/VisionConformanceUris.cs new file mode 100644 index 0000000000..3a04b1001a --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionConformanceUris.cs @@ -0,0 +1,324 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +namespace Opc.Ua.Vision.Server +{ + /// + /// URIs and names of the conformance facets defined by OPC UA — Vision + /// clause 11. + /// + public static class VisionConformanceUris + { + /// + /// Base URI for Vision Server profiles. + /// + public const string ProfileBase = "http://opcfoundation.org/UA-Profile/Vision/Server/"; + + /// + /// Base URI for Vision facets. + /// + public const string FacetBase = "http://opcfoundation.org/UA-Profile/Vision/Facet/"; + + /// + /// Vision facet names from clause 11.2. + /// + public static class FacetNames + { + /// + /// Mandatory facet requiring the well-known Vision root, a + /// sensor with core members, and both mandatory media facets. + /// + public const string Base = "VIS-Base"; + + /// + /// Sensor parameters facet (§5.5). + /// + public const string SensorParams = "VIS-Sensor-Params"; + + /// + /// Optics and illumination facet (§5.7). + /// + public const string Optics = "VIS-Optics"; + + /// + /// RTSP stream endpoint facet (§6.2). + /// + public const string MediaRtsp = "VIS-Media-Rtsp"; + + /// + /// JPEG clip endpoint facet (§6.2). + /// + public const string MediaJpeg = "VIS-Media-Jpeg"; + + /// + /// Inline clip delivery facet (§6.4). + /// + public const string MediaInline = "VIS-Media-Inline"; + + /// + /// Data-channel media facet (§6.7). + /// + public const string MediaDataChannel = "VIS-Media-DataChannel"; + + /// + /// Stream configuration and selection facet (§6.5). + /// + public const string EndpointConfig = "VIS-Endpoint-Config"; + + /// + /// Coordinate frames and calibration facet (§5.8). + /// + public const string Calibration = "VIS-Calibration"; + + /// + /// Inspection result facet (§7.2). + /// + public const string ResultInspection = "VIS-Result-Inspection"; + + /// + /// Detection result facet (§7.3). + /// + public const string ResultDetection = "VIS-Result-Detection"; + + /// + /// Feedback and return-path facet (§9). + /// + public const string Feedback = "VIS-Feedback"; + + /// + /// On-server inference facet (§8.2). + /// + public const string InferenceOnServer = "VIS-Inference-OnServer"; + + /// + /// Off-server inference facet (§8.2). + /// + public const string InferenceOffServer = "VIS-Inference-OffServer"; + + /// + /// Simulated-sensor facet (§4.3, §10). + /// + public const string Simulation = "VIS-Simulation"; + + /// + /// Feedback-driven learning facet (§9.5.1). + /// + public const string Learning = "VIS-Learning"; + + /// + /// OpenUSD scene interop facet (Annex C). + /// + public const string InteropScene = "VIS-Interop-Scene"; + + /// + /// OPC 40100 interop facet (Annex D). + /// + public const string Interop40100 = "VIS-Interop-40100"; + + /// + /// Robot Intent interop facet (Annex I). + /// + public const string InteropRobotIntent = "VIS-Interop-RobotIntent"; + } + + /// + /// Vision Server profile URIs. + /// + public static class Profiles + { + /// + /// Baseline Vision Server profile — includes the mandatory + /// Base and both mandatory media facets. + /// + public const string Basic = ProfileBase + "Basic"; + + /// + /// Inspection profile — Basic plus inspection results and feedback. + /// + public const string Inspection = ProfileBase + "Inspection"; + + /// + /// Detection profile — Basic plus detection results and feedback. + /// + public const string Detection = ProfileBase + "Detection"; + + /// + /// Inference profile — Basic plus on-server inference and results. + /// + public const string Inference = ProfileBase + "Inference"; + } + + /// + /// Vision facet URIs, one per name in . + /// + public static class Facets + { + /// + /// Base facet. + /// + public const string Base = FacetBase + "Base"; + + /// + /// Sensor parameters facet. + /// + public const string SensorParams = FacetBase + "Sensor-Params"; + + /// + /// Optics facet. + /// + public const string Optics = FacetBase + "Optics"; + + /// + /// RTSP stream facet. + /// + public const string MediaRtsp = FacetBase + "Media-Rtsp"; + + /// + /// JPEG clip facet. + /// + public const string MediaJpeg = FacetBase + "Media-Jpeg"; + + /// + /// Inline media facet. + /// + public const string MediaInline = FacetBase + "Media-Inline"; + + /// + /// Data-channel media facet. + /// + public const string MediaDataChannel = FacetBase + "Media-DataChannel"; + + /// + /// Endpoint configuration facet. + /// + public const string EndpointConfig = FacetBase + "Endpoint-Config"; + + /// + /// Calibration facet. + /// + public const string Calibration = FacetBase + "Calibration"; + + /// + /// Inspection result facet. + /// + public const string ResultInspection = FacetBase + "Result-Inspection"; + + /// + /// Detection result facet. + /// + public const string ResultDetection = FacetBase + "Result-Detection"; + + /// + /// Feedback facet. + /// + public const string Feedback = FacetBase + "Feedback"; + + /// + /// On-server inference facet. + /// + public const string InferenceOnServer = FacetBase + "Inference-OnServer"; + + /// + /// Off-server inference facet. + /// + public const string InferenceOffServer = FacetBase + "Inference-OffServer"; + + /// + /// Simulation facet. + /// + public const string Simulation = FacetBase + "Simulation"; + + /// + /// Learning facet. + /// + public const string Learning = FacetBase + "Learning"; + + /// + /// OpenUSD scene interop facet. + /// + public const string InteropScene = FacetBase + "Interop-Scene"; + + /// + /// OPC 40100 interop facet. + /// + public const string Interop40100 = FacetBase + "Interop-40100"; + + /// + /// Robot Intent interop facet. + /// + public const string InteropRobotIntent = FacetBase + "Interop-RobotIntent"; + } + + /// + /// Returns the ordered list of every facet name defined by the + /// specification. + /// + public static ArrayOf AllFacets { get; } = new string[] + { + FacetNames.Base, + FacetNames.SensorParams, + FacetNames.Optics, + FacetNames.MediaRtsp, + FacetNames.MediaJpeg, + FacetNames.MediaInline, + FacetNames.MediaDataChannel, + FacetNames.EndpointConfig, + FacetNames.Calibration, + FacetNames.ResultInspection, + FacetNames.ResultDetection, + FacetNames.Feedback, + FacetNames.InferenceOnServer, + FacetNames.InferenceOffServer, + FacetNames.Simulation, + FacetNames.Learning, + FacetNames.InteropScene, + FacetNames.Interop40100, + FacetNames.InteropRobotIntent + }; + + internal static bool TryGetFacetUri(string facetName, out string facetUri) + { + const string facetNamePrefix = "VIS-"; + if (!string.IsNullOrEmpty(facetName) && + facetName.StartsWith(facetNamePrefix, StringComparison.Ordinal)) + { +#if NETSTANDARD || NETFRAMEWORK + facetUri = FacetBase + facetName.Substring(facetNamePrefix.Length); +#else + facetUri = string.Concat(FacetBase, facetName.AsSpan(facetNamePrefix.Length)); +#endif + return true; + } + facetUri = string.Empty; + return false; + } + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionCoordinateFrameMath.cs b/src/Opc.Ua.Vision.Server/VisionCoordinateFrameMath.cs new file mode 100644 index 0000000000..0f44d15250 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionCoordinateFrameMath.cs @@ -0,0 +1,365 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using Opc.Ua.Vision; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Composes rigid transforms across a coordinate-frame tree following + /// the conventions in specification §5.12. + /// + /// + /// + /// Poses are represented as values: + /// translation in metres, orientation as a unit quaternion in the + /// (x, y, z, w) ordering the specification mandates. Composition + /// applies the child transform then the parent transform, so + /// Compose(parent, child) = parent ∘ child — first the child's + /// pose in its parent, then the parent in its parent, and so on up to + /// the root. + /// + /// + /// Covariance is not composed; the spec's sentinel rules mean the + /// composed covariance is only accurate when every intermediate pose + /// carries a full 6×6 matrix in the same order as its position and + /// orientation. Callers can inspect + /// on + /// to decide whether to + /// suppress the composed covariance. + /// + /// + public static class VisionCoordinateFrameMath + { + /// + /// Position component length as defined by §5.12. + /// + public const int PositionLength = 3; + + /// + /// Orientation component length: (x, y, z, w). + /// + public const int OrientationLength = 4; + + /// + /// Snapshot of one coordinate frame, containing the frame id, the + /// optional parent frame id and the transform from this frame to + /// its parent. + /// + /// + /// 's FrameId is the parent frame + /// per §5.12 — the specification's frame-precedence rule states + /// that the transform's FrameId equals the target frame's + /// identifier. + /// + public sealed record CoordinateFrameSnapshot( + string FrameId, + VisionFrameRoleEnum Role, + string ParentFrameId, + VisionPose3DDataType Transform); + + /// + /// Returns the identity pose in . + /// + public static VisionPose3DDataType Identity(string frameId) + { + return new VisionPose3DDataType + { + FrameId = frameId ?? string.Empty, + Position = new double[] { 0.0, 0.0, 0.0 }, + Orientation = new double[] { 0.0, 0.0, 0.0, 1.0 }, + Covariance = ArrayOf.Empty + }; + } + + /// + /// Composes ∘ + /// where the child's pose is expressed in the parent's frame and + /// the result is the child's pose in 's + /// parent. + /// + /// + /// One of the poses has malformed position or orientation. + /// + public static VisionPose3DDataType Compose( + in VisionPose3DDataType parent, + in VisionPose3DDataType child) + { + ReadOnlySpan pp = ExtractPosition(parent, nameof(parent)); + ReadOnlySpan pq = ExtractOrientation(parent, nameof(parent)); + ReadOnlySpan cp = ExtractPosition(child, nameof(child)); + ReadOnlySpan cq = ExtractOrientation(child, nameof(child)); + + Span rotated = stackalloc double[3]; + RotateVector(pq, cp, rotated); + double[] position = new double[] + { + pp[0] + rotated[0], + pp[1] + rotated[1], + pp[2] + rotated[2] + }; + + Span composed = stackalloc double[4]; + MultiplyQuaternions(pq, cq, composed); + NormalizeQuaternion(composed); + double[] orientation = new double[] + { + composed[0], + composed[1], + composed[2], + composed[3] + }; + return new VisionPose3DDataType + { + FrameId = parent.FrameId ?? string.Empty, + Position = position, + Orientation = orientation, + Covariance = ArrayOf.Empty + }; + } + + /// + /// Returns the inverse of . Assumes the + /// orientation is a unit quaternion in (x, y, z, w) order. + /// + public static VisionPose3DDataType Invert(in VisionPose3DDataType pose) + { + ReadOnlySpan p = ExtractPosition(pose, nameof(pose)); + ReadOnlySpan q = ExtractOrientation(pose, nameof(pose)); + Span qInv = stackalloc double[4] { -q[0], -q[1], -q[2], q[3] }; + Span negatedP = stackalloc double[3] { -p[0], -p[1], -p[2] }; + Span rotated = stackalloc double[3]; + RotateVector(qInv, negatedP, rotated); + return new VisionPose3DDataType + { + FrameId = pose.FrameId ?? string.Empty, + Position = new double[] { rotated[0], rotated[1], rotated[2] }, + Orientation = new double[] { qInv[0], qInv[1], qInv[2], qInv[3] }, + Covariance = ArrayOf.Empty + }; + } + + /// + /// Walks starting from + /// up to + /// and composes the transforms it visits into the pose of + /// expressed in + /// . + /// + /// + /// One of the required arguments is null. + /// + /// + /// The source and target frames are not connected in the tree, or + /// a cycle is detected, or a frame identifier violates §5.12's + /// non-empty rule. + /// + public static VisionPose3DDataType TransformFromTo( + IReadOnlyDictionary frames, + string sourceFrameId, + string targetFrameId) + { + if (frames == null) + { + throw new ArgumentNullException(nameof(frames)); + } + if (string.IsNullOrEmpty(sourceFrameId)) + { + throw ServiceResultException.Create( + StatusCodes.BadInvalidArgument, + "Coordinate frame identifiers must be non-empty (§5.12)."); + } + if (string.IsNullOrEmpty(targetFrameId)) + { + throw ServiceResultException.Create( + StatusCodes.BadInvalidArgument, + "Coordinate frame identifiers must be non-empty (§5.12)."); + } + if (string.Equals(sourceFrameId, targetFrameId, StringComparison.Ordinal)) + { + return Identity(targetFrameId); + } + + List sourceToRoot = WalkToRoot(frames, sourceFrameId); + List targetToRoot = WalkToRoot(frames, targetFrameId); + + int commonSourceIndex = sourceToRoot.Count - 1; + int commonTargetIndex = targetToRoot.Count - 1; + while (commonSourceIndex > 0 && + commonTargetIndex > 0 && + string.Equals( + sourceToRoot[commonSourceIndex].FrameId, + targetToRoot[commonTargetIndex].FrameId, + StringComparison.Ordinal)) + { + commonSourceIndex--; + commonTargetIndex--; + } + + VisionPose3DDataType pose = Identity(sourceToRoot[0].FrameId); + for (int ii = 0; ii <= commonSourceIndex; ii++) + { + pose = Compose(sourceToRoot[ii].Transform, pose); + } + for (int ii = commonTargetIndex; ii >= 0; ii--) + { + pose = Compose(Invert(targetToRoot[ii].Transform), pose); + } + return new VisionPose3DDataType + { + FrameId = targetFrameId, + Position = pose.Position, + Orientation = pose.Orientation, + Covariance = ArrayOf.Empty + }; + } + + private static List WalkToRoot( + IReadOnlyDictionary frames, + string frameId) + { + var chain = new List(); + var visited = new HashSet(StringComparer.Ordinal); + string current = frameId; + while (!string.IsNullOrEmpty(current)) + { + if (!frames.TryGetValue(current, out CoordinateFrameSnapshot? snapshot)) + { + throw ServiceResultException.Create( + StatusCodes.BadNodeIdUnknown, + "Coordinate frame '{0}' is not registered in the tree.", + current); + } + if (!visited.Add(current)) + { + throw ServiceResultException.Create( + StatusCodes.BadInvalidArgument, + "Coordinate frame '{0}' participates in a cycle.", + current); + } + chain.Add(snapshot); + current = snapshot.ParentFrameId; + } + return chain; + } + + private static ReadOnlySpan ExtractPosition( + in VisionPose3DDataType pose, + string parameterName) + { + ReadOnlySpan span = pose.Position.Span; + if (span.Length != PositionLength) + { + throw new ArgumentException( + "Position must be a length-3 vector (§5.12).", + parameterName); + } + return span; + } + + private static ReadOnlySpan ExtractOrientation( + in VisionPose3DDataType pose, + string parameterName) + { + ReadOnlySpan span = pose.Orientation.Span; + if (span.Length != OrientationLength) + { + throw new ArgumentException( + "Orientation must be a length-4 quaternion (x, y, z, w) per §5.12.", + parameterName); + } + return span; + } + + private static void RotateVector( + ReadOnlySpan quaternion, + ReadOnlySpan vector, + Span result) + { + double qx = quaternion[0]; + double qy = quaternion[1]; + double qz = quaternion[2]; + double qw = quaternion[3]; + double vx = vector[0]; + double vy = vector[1]; + double vz = vector[2]; + double tx = 2.0 * ((qy * vz) - (qz * vy)); + double ty = 2.0 * ((qz * vx) - (qx * vz)); + double tz = 2.0 * ((qx * vy) - (qy * vx)); + result[0] = vx + (qw * tx) + ((qy * tz) - (qz * ty)); + result[1] = vy + (qw * ty) + ((qz * tx) - (qx * tz)); + result[2] = vz + (qw * tz) + ((qx * ty) - (qy * tx)); + } + + private static void MultiplyQuaternions( + ReadOnlySpan left, + ReadOnlySpan right, + Span result) + { + double lx = left[0]; + double ly = left[1]; + double lz = left[2]; + double lw = left[3]; + double rx = right[0]; + double ry = right[1]; + double rz = right[2]; + double rw = right[3]; + result[0] = (lw * rx) + (lx * rw) + (ly * rz) - (lz * ry); + result[1] = (lw * ry) - (lx * rz) + (ly * rw) + (lz * rx); + result[2] = (lw * rz) + (lx * ry) - (ly * rx) + (lz * rw); + result[3] = (lw * rw) - (lx * rx) - (ly * ry) - (lz * rz); + } + + private static void NormalizeQuaternion(Span quaternion) + { + double norm = Math.Sqrt( + (quaternion[0] * quaternion[0]) + + (quaternion[1] * quaternion[1]) + + (quaternion[2] * quaternion[2]) + + (quaternion[3] * quaternion[3])); + + // A zero-norm quaternion carries no orientation. Substituting the identity would + // compose a pose that looks plausible and points the wrong way, which for a grasp + // is worse than a refusal, so it is reported like every other malformed input here. + if (norm <= 0.0) + { + throw ServiceResultException.Create( + StatusCodes.BadInvalidArgument, + "An orientation quaternion has zero norm and does not describe a rotation."); + } + + quaternion[0] /= norm; + quaternion[1] /= norm; + quaternion[2] /= norm; + quaternion[3] /= norm; + } + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionFacetCalculator.cs b/src/Opc.Ua.Vision.Server/VisionFacetCalculator.cs new file mode 100644 index 0000000000..8e44db945d --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionFacetCalculator.cs @@ -0,0 +1,109 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Computes the §11 facet URIs a Vision server can claim by inspecting + /// the state recorded in the . + /// + /// + /// A facet is added to the result only when every one of its + /// requirements is present in the address space or in a bound + /// provider — hosts that need to publish a facet the calculator + /// cannot verify structurally (for example, a behavioural interop + /// facet) can add it through + /// . + /// + internal static class VisionFacetCalculator + { + public static ArrayOf Compute(VisionRegistry registry) + { + if (registry == null) + { + throw new ArgumentNullException(nameof(registry)); + } + var facets = new HashSet(StringComparer.Ordinal); + foreach (SensorRegistration sensor in registry.SensorsByNodeId.Values) + { + foreach (string facet in sensor.Facets) + { + facets.Add(facet); + } + } + foreach (PipelineRegistration pipeline in registry.PipelinesByNodeId.Values) + { + foreach (string facet in pipeline.Facets) + { + facets.Add(facet); + } + } + var result = new List(facets); + result.Sort(StringComparer.Ordinal); + return result.ToArrayOf(); + } + + public static ArrayOf ComputeProfiles(ArrayOf facets) + { + var lookup = new HashSet(StringComparer.Ordinal); + for (int ii = 0; ii < facets.Count; ii++) + { + if (!string.IsNullOrEmpty(facets[ii])) + { + lookup.Add(facets[ii]); + } + } + var profiles = new List(); + if (lookup.Contains(VisionConformanceUris.FacetNames.Base) && + lookup.Contains(VisionConformanceUris.FacetNames.MediaJpeg) && + lookup.Contains(VisionConformanceUris.FacetNames.MediaRtsp)) + { + profiles.Add(VisionConformanceUris.Profiles.Basic); + } + if (lookup.Contains(VisionConformanceUris.FacetNames.ResultInspection) && + lookup.Contains(VisionConformanceUris.FacetNames.Feedback)) + { + profiles.Add(VisionConformanceUris.Profiles.Inspection); + } + if (lookup.Contains(VisionConformanceUris.FacetNames.ResultDetection) && + lookup.Contains(VisionConformanceUris.FacetNames.Feedback)) + { + profiles.Add(VisionConformanceUris.Profiles.Detection); + } + if (lookup.Contains(VisionConformanceUris.FacetNames.InferenceOnServer)) + { + profiles.Add(VisionConformanceUris.Profiles.Inference); + } + return profiles.ToArrayOf(); + } + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionMethodArguments.cs b/src/Opc.Ua.Vision.Server/VisionMethodArguments.cs new file mode 100644 index 0000000000..8fbb837676 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionMethodArguments.cs @@ -0,0 +1,234 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using Opc.Ua.Vision.Server.Builders; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Declares the InputArguments and OutputArguments + /// Properties of the Vision Methods. + /// + /// + /// The NodeSet now browse-names its Argument Properties correctly, so the + /// generated CreateInstanceOf…MethodType factories carry the + /// signatures. The builder does not use those factories, though: the + /// Methods here are Optional children materialised by + /// CreateOrReplace…, which constructs the state object directly and + /// leaves InputArguments unset. A Method reached that way is + /// uncallable — MethodState.Call compares the supplied arguments + /// against an InputArguments Property it cannot find and concludes + /// that none were expected, so every call carrying arguments is refused + /// with Bad_TooManyArguments — and a client cannot discover the + /// signature either. Declaring them here closes that gap. Removing this + /// class makes five of the eight VisionMethodSurfaceTests fail, which + /// is the check to repeat if the generator ever starts populating them. + /// The declarations match the specification's Method definitions and the + /// generated Method state classes argument for argument. + /// + internal static class VisionMethodArguments + { + internal static void Declare(ISystemContext context, RunInferenceMethodState method) + { + SetInput(context, method, Argument("Timestamp", global::Opc.Ua.DataTypeIds.DateTime)); + SetOutput(context, method, Argument("ResultId", global::Opc.Ua.DataTypeIds.String)); + } + + internal static void DeclareStartContinuous(ISystemContext context, MethodState method) + { + SetInput(context, method); + SetOutput(context, method); + } + + internal static void DeclareStop(ISystemContext context, MethodState method) + { + SetInput(context, method); + SetOutput(context, method); + } + + internal static void Declare(ISystemContext context, SubmitDetectionsMethodState method) + { + SetInput( + context, + method, + Argument("Purpose", VisionDataType(context, DataTypeIds.VisionFeedbackPurposeEnum)), + Argument( + "Detections", + VisionDataType(context, DataTypeIds.VisionDetectionDataType), + ValueRanks.OneDimension), + Argument("FrameReference", VisionDataType(context, DataTypeIds.VisionImageReferenceDataType)), + Argument("InlineImage", global::Opc.Ua.DataTypeIds.ByteString), + Argument("SceneIsEmpty", global::Opc.Ua.DataTypeIds.Boolean)); + SetOutput(context, method); + } + + internal static void Declare(ISystemContext context, SubmitInspectionResultMethodState method) + { + SetInput( + context, + method, + Argument("ResultId", global::Opc.Ua.DataTypeIds.String), + Argument("Evaluation", VisionDataType(context, DataTypeIds.VisionResultEvaluationEnum)), + Argument( + "Characteristics", + VisionDataType(context, DataTypeIds.VisionCharacteristicDataType), + ValueRanks.OneDimension)); + SetOutput(context, method); + } + + internal static void Declare(ISystemContext context, SubmitCorrectionMethodState method) + { + SetInput( + context, + method, + Argument("ResultId", global::Opc.Ua.DataTypeIds.String), + Argument("Purpose", VisionDataType(context, DataTypeIds.VisionFeedbackPurposeEnum)), + Argument( + "CorrectedDetections", + VisionDataType(context, DataTypeIds.VisionDetectionDataType), + ValueRanks.OneDimension), + Argument( + "CorrectedCharacteristics", + VisionDataType(context, DataTypeIds.VisionCharacteristicDataType), + ValueRanks.OneDimension), + Argument("Reason", global::Opc.Ua.DataTypeIds.LocalizedText), + Argument("InlineImage", global::Opc.Ua.DataTypeIds.ByteString), + Argument("RetractAll", global::Opc.Ua.DataTypeIds.Boolean)); + SetOutput(context, method); + } + + internal static void Declare(ISystemContext context, SubmitImageReferenceMethodState method) + { + SetInput( + context, + method, + Argument("Purpose", VisionDataType(context, DataTypeIds.VisionFeedbackPurposeEnum)), + Argument("Image", VisionDataType(context, DataTypeIds.VisionImageReferenceDataType)), + Argument("ResultId", global::Opc.Ua.DataTypeIds.String)); + SetOutput(context, method); + } + + internal static void Declare(ISystemContext context, GetStreamEndpointMethodState method) + { + SetInput( + context, + method, + Argument("Endpoint", global::Opc.Ua.DataTypeIds.NodeId), + Argument("ProfileName", global::Opc.Ua.DataTypeIds.String), + Argument("PreferredProtocol", VisionDataType(context, DataTypeIds.VisionStreamProtocolEnum))); + SetOutput( + context, + method, + Argument("Session", VisionDataType(context, DataTypeIds.VisionStreamSessionDataType)), + Argument("Endpoint", global::Opc.Ua.DataTypeIds.NodeId)); + } + + internal static void Declare(ISystemContext context, ReleaseStreamEndpointMethodState method) + { + SetInput(context, method, Argument("SessionToken", global::Opc.Ua.DataTypeIds.ByteString)); + SetOutput(context, method); + } + + internal static void Declare(ISystemContext context, ConfigureStreamEndpointMethodState method) + { + SetInput( + context, + method, + Argument("Endpoint", global::Opc.Ua.DataTypeIds.NodeId), + Argument("Codec", VisionDataType(context, DataTypeIds.VisionVideoCodecEnum)), + Argument("Width", global::Opc.Ua.DataTypeIds.UInt32), + Argument("Height", global::Opc.Ua.DataTypeIds.UInt32), + Argument("FrameRate", global::Opc.Ua.DataTypeIds.Double), + Argument("Bitrate", global::Opc.Ua.DataTypeIds.UInt32)); + SetOutput(context, method); + } + + internal static void Declare(ISystemContext context, SelectEndpointMethodState method) + { + SetInput( + context, + method, + Argument("StreamEndpoint", global::Opc.Ua.DataTypeIds.NodeId), + Argument("ClipEndpoint", global::Opc.Ua.DataTypeIds.NodeId)); + SetOutput(context, method); + } + + internal static void Declare(ISystemContext context, GetClipMethodState method) + { + SetInput( + context, + method, + Argument("Endpoint", global::Opc.Ua.DataTypeIds.NodeId), + Argument("ResultId", global::Opc.Ua.DataTypeIds.String), + Argument("Timestamp", global::Opc.Ua.DataTypeIds.DateTime), + Argument("Format", VisionDataType(context, DataTypeIds.VisionClipFormatEnum)), + Argument("RequestInline", global::Opc.Ua.DataTypeIds.Boolean)); + SetOutput( + context, + method, + Argument("Image", VisionDataType(context, DataTypeIds.VisionImageReferenceDataType)), + Argument("Endpoint", global::Opc.Ua.DataTypeIds.NodeId), + Argument("InlineImage", global::Opc.Ua.DataTypeIds.ByteString)); + } + + private static void SetInput( + ISystemContext context, + MethodState method, + params Argument[] arguments) + { + method.CreateOrReplaceInputArguments(context, null).Value = arguments.ToArrayOf(); + } + + private static void SetOutput( + ISystemContext context, + MethodState method, + params Argument[] arguments) + { + method.CreateOrReplaceOutputArguments(context, null).Value = arguments.ToArrayOf(); + } + + private static Argument Argument( + string name, + NodeId dataType, + int valueRank = ValueRanks.Scalar) + { + return new Argument + { + Name = name, + DataType = dataType, + ValueRank = valueRank + }; + } + + private static NodeId VisionDataType(ISystemContext context, ExpandedNodeId dataType) + { + return ExpandedNodeId.ToNodeId(dataType, context.NamespaceUris); + } + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionMethodDispatcher.cs b/src/Opc.Ua.Vision.Server/VisionMethodDispatcher.cs new file mode 100644 index 0000000000..5081a8b7d6 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionMethodDispatcher.cs @@ -0,0 +1,805 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua.Vision; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Wires the generated Vision method delegates to the injected + /// providers, applies specification-mandated status codes, and + /// records failures via the source-generated logger. + /// + internal sealed class VisionMethodDispatcher + { + public VisionMethodDispatcher(VisionRegistry registry, ILogger logger) + { + m_registry = registry; + m_logger = logger; + } + + public void AttachMediaMethods(NodeId sensorNodeId, VisionMediaManagementState media) + { + if (media.GetStreamEndpoint != null) + { + media.GetStreamEndpoint.OnCallAsync = CreateGetStreamEndpointHandler(sensorNodeId); + } + if (media.ReleaseStreamEndpoint != null) + { + media.ReleaseStreamEndpoint.OnCallAsync = CreateReleaseStreamEndpointHandler(sensorNodeId); + } + if (media.ConfigureStreamEndpoint != null) + { + media.ConfigureStreamEndpoint.OnCallAsync = CreateConfigureStreamEndpointHandler(sensorNodeId); + } + if (media.SelectEndpoint != null) + { + media.SelectEndpoint.OnCallAsync = CreateSelectEndpointHandler(sensorNodeId); + } + if (media.GetClip != null) + { + media.GetClip.OnCallAsync = CreateGetClipHandler(sensorNodeId); + } + } + + public void AttachPipelineMethods(NodeId pipelineNodeId, InferencePipelineState pipeline) + { + if (pipeline.RunInference != null) + { + pipeline.RunInference.OnCallAsync = CreateRunInferenceHandler(pipelineNodeId); + } + if (pipeline.StartContinuous != null) + { + pipeline.StartContinuous.OnCallMethod2Async = CreateStartContinuousHandler(pipelineNodeId); + } + if (pipeline.Stop != null) + { + pipeline.Stop.OnCallMethod2Async = CreateStopHandler(pipelineNodeId); + } + } + + public void AttachFeedbackMethods(NodeId pipelineNodeId, VisionFeedbackState feedback) + { + if (feedback.SubmitDetections != null) + { + feedback.SubmitDetections.OnCallAsync = CreateSubmitDetectionsHandler(pipelineNodeId); + } + if (feedback.SubmitInspectionResult != null) + { + feedback.SubmitInspectionResult.OnCallAsync = CreateSubmitInspectionHandler(pipelineNodeId); + } + if (feedback.SubmitCorrection != null) + { + feedback.SubmitCorrection.OnCallAsync = CreateSubmitCorrectionHandler(pipelineNodeId); + } + if (feedback.SubmitImageReference != null) + { + feedback.SubmitImageReference.OnCallAsync = CreateSubmitImageReferenceHandler(pipelineNodeId); + } + } + + private GetStreamEndpointMethodStateMethodAsyncCallHandler CreateGetStreamEndpointHandler(NodeId sensorNodeId) + { + return (context, method, objectId, endpoint, profileName, protocol, ct) => + DispatchGetStreamEndpointAsync(sensorNodeId, endpoint, profileName, protocol, ct); + } + + private async ValueTask DispatchGetStreamEndpointAsync( + NodeId sensorNodeId, + NodeId endpoint, + string profileName, + VisionStreamProtocolEnum protocol, + CancellationToken cancellationToken) + { + IVisionMediaProvider? provider = ResolveMediaProvider(sensorNodeId); + if (provider == null) + { + m_logger.MediaProviderMissing(sensorNodeId); + return new GetStreamEndpointMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + try + { + VisionStreamLease lease = await provider.GetStreamAsync( + new VisionStreamRequest(endpoint, profileName, protocol), + cancellationToken).ConfigureAwait(false); + return new GetStreamEndpointMethodStateResult + { + ServiceResult = lease.ServiceResult, + Session = lease.Session, + EndpointOut = lease.EndpointOut + }; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("GetStreamEndpoint", ex); + return new GetStreamEndpointMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + } + + private ReleaseStreamEndpointMethodStateMethodAsyncCallHandler CreateReleaseStreamEndpointHandler(NodeId sensorNodeId) + { + return async (context, method, objectId, sessionToken, ct) => + { + IVisionMediaProvider? provider = ResolveMediaProvider(sensorNodeId); + if (provider == null) + { + m_logger.MediaProviderMissing(sensorNodeId); + return new ReleaseStreamEndpointMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + try + { + ServiceResult result = await provider.ReleaseStreamAsync(sessionToken, ct).ConfigureAwait(false); + return new ReleaseStreamEndpointMethodStateResult { ServiceResult = result }; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("ReleaseStreamEndpoint", ex); + return new ReleaseStreamEndpointMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + }; + } + + private ConfigureStreamEndpointMethodStateMethodAsyncCallHandler CreateConfigureStreamEndpointHandler(NodeId sensorNodeId) + { + return async (context, method, objectId, endpoint, codec, width, height, frameRate, bitrate, ct) => + { + IVisionMediaProvider? provider = ResolveMediaProvider(sensorNodeId); + if (provider == null) + { + m_logger.MediaProviderMissing(sensorNodeId); + return new ConfigureStreamEndpointMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + try + { + ServiceResult result = await provider.ConfigureStreamAsync( + new VisionStreamConfigurationRequest(endpoint, codec, width, height, frameRate, bitrate), + ct).ConfigureAwait(false); + return new ConfigureStreamEndpointMethodStateResult { ServiceResult = result }; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("ConfigureStreamEndpoint", ex); + return new ConfigureStreamEndpointMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + }; + } + + private SelectEndpointMethodStateMethodAsyncCallHandler CreateSelectEndpointHandler(NodeId sensorNodeId) + { + return async (context, method, objectId, streamEndpoint, clipEndpoint, ct) => + { + IVisionMediaProvider? provider = ResolveMediaProvider(sensorNodeId); + if (provider == null) + { + m_logger.MediaProviderMissing(sensorNodeId); + return new SelectEndpointMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + try + { + ServiceResult result = await provider.SelectEndpointAsync(streamEndpoint, clipEndpoint, ct) + .ConfigureAwait(false); + if (ServiceResult.IsGood(result) && + m_registry.TryGetSensor(sensorNodeId, out SensorRegistration? sensor) && + sensor?.Sensor.Media is VisionMediaManagementState media) + { + if (media.PreferredStreamEndpoint != null && !streamEndpoint.IsNull) + { + media.PreferredStreamEndpoint.Value = streamEndpoint; + media.PreferredStreamEndpoint.ClearChangeMasks(context, false); + } + if (media.PreferredClipEndpoint != null && !clipEndpoint.IsNull) + { + media.PreferredClipEndpoint.Value = clipEndpoint; + media.PreferredClipEndpoint.ClearChangeMasks(context, false); + } + } + return new SelectEndpointMethodStateResult { ServiceResult = result }; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("SelectEndpoint", ex); + return new SelectEndpointMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + }; + } + + private GetClipMethodStateMethodAsyncCallHandler CreateGetClipHandler(NodeId sensorNodeId) + { + return async (context, method, objectId, endpoint, resultId, timestamp, format, requestInline, ct) => + { + IVisionMediaProvider? provider = ResolveMediaProvider(sensorNodeId); + if (provider == null) + { + m_logger.MediaProviderMissing(sensorNodeId); + return new GetClipMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + ClipEndpointState? clipEndpoint = FindClipEndpoint(sensorNodeId, endpoint); + if (clipEndpoint != null && requestInline && !IsInlineDeliveryEnabled(clipEndpoint)) + { + return new GetClipMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + try + { + VisionClipResult clipResult = await provider.GetClipAsync( + new VisionClipRequest(endpoint, resultId, timestamp, format, requestInline), + ct).ConfigureAwait(false); + GetClipMethodStateResult result = EnforceInlineLimit(clipEndpoint, clipResult); + PublishLatestClip(context, clipEndpoint, result, timestamp); + return result; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("GetClip", ex); + return new GetClipMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + }; + } + + private RunInferenceMethodStateMethodAsyncCallHandler CreateRunInferenceHandler(NodeId pipelineNodeId) + { + return async (context, method, objectId, timestamp, ct) => + { + if (!m_registry.TryGetPipeline(pipelineNodeId, out PipelineRegistration? pipeline) || + pipeline == null) + { + return new RunInferenceMethodStateResult + { + ServiceResult = StatusCodes.BadNodeIdUnknown + }; + } + IVisionInferenceProvider? provider = pipeline.InferenceProvider; + if (provider == null) + { + m_logger.InferenceProviderMissing(pipelineNodeId); + return new RunInferenceMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + NodeId sensorNodeId = ReadPipelineSensor(pipeline.Pipeline); + NodeId deploymentNodeId = ReadPipelineDeployment(pipeline.Pipeline); + try + { + VisionInferenceRunResult result = await provider.RunInferenceAsync( + new VisionInferenceRunRequest(pipelineNodeId, sensorNodeId, deploymentNodeId, timestamp), + ct).ConfigureAwait(false); + return new RunInferenceMethodStateResult + { + ServiceResult = result.ServiceResult, + ResultId = result.ResultId ?? string.Empty + }; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("RunInference", ex); + return new RunInferenceMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + }; + } + + private GenericMethodCalledEventHandler2Async CreateStartContinuousHandler(NodeId pipelineNodeId) + { + return async (context, method, objectId, inputArguments, outputArguments, ct) => + { + if (!m_registry.TryGetPipeline(pipelineNodeId, out PipelineRegistration? pipeline) || + pipeline == null) + { + return StatusCodes.BadNodeIdUnknown; + } + IVisionInferenceProvider? provider = pipeline.InferenceProvider; + if (provider == null) + { + m_logger.InferenceProviderMissing(pipelineNodeId); + return StatusCodes.BadNotSupported; + } + try + { + return await provider.StartContinuousAsync(pipelineNodeId, ct).ConfigureAwait(false); + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("StartContinuous", ex); + return StatusCodes.BadInternalError; + } + }; + } + + private GenericMethodCalledEventHandler2Async CreateStopHandler(NodeId pipelineNodeId) + { + return async (context, method, objectId, inputArguments, outputArguments, ct) => + { + if (!m_registry.TryGetPipeline(pipelineNodeId, out PipelineRegistration? pipeline) || + pipeline == null) + { + return StatusCodes.BadNodeIdUnknown; + } + IVisionInferenceProvider? provider = pipeline.InferenceProvider; + if (provider == null) + { + m_logger.InferenceProviderMissing(pipelineNodeId); + return StatusCodes.BadNotSupported; + } + try + { + return await provider.StopAsync(pipelineNodeId, ct).ConfigureAwait(false); + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("Stop", ex); + return StatusCodes.BadInternalError; + } + }; + } + + private SubmitDetectionsMethodStateMethodAsyncCallHandler CreateSubmitDetectionsHandler(NodeId pipelineNodeId) + { + return async (context, method, objectId, purpose, detections, frameRef, inlineImage, sceneIsEmpty, ct) => + { + IVisionFeedbackSink? sink = ResolveFeedbackSink(pipelineNodeId); + if (sink == null) + { + m_logger.FeedbackSinkMissing(pipelineNodeId); + return new SubmitDetectionsMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + if (detections.Count == 0 != sceneIsEmpty) + { + // Part 9.5. An empty Detections is accepted only when SceneIsEmpty + // says the frame was examined and found to contain nothing, and is + // refused otherwise - the flag is what tells a deliberate empty + // observation from a lost payload. The converse is equally + // inconsistent: SceneIsEmpty with detections attached asserts two + // contradictory things about the same frame. + return new SubmitDetectionsMethodStateResult + { + ServiceResult = StatusCodes.BadInvalidArgument + }; + } + try + { + ServiceResult result = await sink.SubmitDetectionsAsync( + new VisionSubmitDetectionsRequest( + pipelineNodeId, + purpose, + detections, + frameRef, + inlineImage, + sceneIsEmpty), + ct).ConfigureAwait(false); + if (ServiceResult.IsGood(result)) + { + m_logger.FeedbackAccepted("SubmitDetections", pipelineNodeId); + } + return new SubmitDetectionsMethodStateResult { ServiceResult = result }; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("SubmitDetections", ex); + return new SubmitDetectionsMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + }; + } + + private SubmitInspectionResultMethodStateMethodAsyncCallHandler CreateSubmitInspectionHandler(NodeId pipelineNodeId) + { + return async (context, method, objectId, resultId, evaluation, characteristics, ct) => + { + IVisionFeedbackSink? sink = ResolveFeedbackSink(pipelineNodeId); + if (sink == null) + { + m_logger.FeedbackSinkMissing(pipelineNodeId); + return new SubmitInspectionResultMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + try + { + ServiceResult result = await sink.SubmitInspectionResultAsync( + new VisionSubmitInspectionResultRequest( + pipelineNodeId, + resultId ?? string.Empty, + evaluation, + characteristics), + ct).ConfigureAwait(false); + if (ServiceResult.IsGood(result)) + { + m_logger.FeedbackAccepted("SubmitInspectionResult", pipelineNodeId); + } + return new SubmitInspectionResultMethodStateResult { ServiceResult = result }; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("SubmitInspectionResult", ex); + return new SubmitInspectionResultMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + }; + } + + private SubmitCorrectionMethodStateMethodAsyncCallHandler CreateSubmitCorrectionHandler(NodeId pipelineNodeId) + { + return async (context, method, objectId, resultId, purpose, detections, characteristics, reason, inlineImage, retractAll, ct) => + { + IVisionFeedbackSink? sink = ResolveFeedbackSink(pipelineNodeId); + if (sink == null) + { + m_logger.FeedbackSinkMissing(pipelineNodeId); + return new SubmitCorrectionMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + if (string.IsNullOrEmpty(resultId)) + { + // ResultId identifies the result being corrected. Forwarding an empty one + // would ask the sink to correct an unnamed result, so refuse instead of + // quietly substituting a value the caller never supplied. + return new SubmitCorrectionMethodStateResult + { + ServiceResult = StatusCodes.BadInvalidArgument + }; + } + bool hasDetections = detections.Count > 0; + bool hasCharacteristics = characteristics.Count > 0; + if (retractAll) + { + // Part 9.5. RetractAll asserts the referenced result should contain + // nothing at all, so carrying a replacement contradicts it. + if (hasDetections || hasCharacteristics) + { + return new SubmitCorrectionMethodStateResult + { + ServiceResult = StatusCodes.BadInvalidArgument + }; + } + } + else if (hasDetections == hasCharacteristics) + { + // Part 9.5 relaxed this to AT MOST one non-empty: both populated is + // still contradictory, and neither is only meaningful with RetractAll, + // which is the branch above. + return new SubmitCorrectionMethodStateResult + { + ServiceResult = StatusCodes.BadInvalidArgument + }; + } + try + { + ServiceResult result = await sink.SubmitCorrectionAsync( + new VisionSubmitCorrectionRequest( + pipelineNodeId, + resultId, + purpose, + detections, + characteristics, + reason, + inlineImage, + retractAll), + ct).ConfigureAwait(false); + if (ServiceResult.IsGood(result)) + { + m_logger.FeedbackAccepted("SubmitCorrection", pipelineNodeId); + } + return new SubmitCorrectionMethodStateResult { ServiceResult = result }; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("SubmitCorrection", ex); + return new SubmitCorrectionMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + }; + } + + private SubmitImageReferenceMethodStateMethodAsyncCallHandler CreateSubmitImageReferenceHandler(NodeId pipelineNodeId) + { + return async (context, method, objectId, purpose, image, resultId, ct) => + { + IVisionFeedbackSink? sink = ResolveFeedbackSink(pipelineNodeId); + if (sink == null) + { + m_logger.FeedbackSinkMissing(pipelineNodeId); + return new SubmitImageReferenceMethodStateResult + { + ServiceResult = StatusCodes.BadNotSupported + }; + } + try + { + ServiceResult result = await sink.SubmitImageReferenceAsync( + new VisionSubmitImageReferenceRequest( + pipelineNodeId, + purpose, + image, + resultId ?? string.Empty), + ct).ConfigureAwait(false); + if (ServiceResult.IsGood(result)) + { + m_logger.FeedbackAccepted("SubmitImageReference", pipelineNodeId); + } + return new SubmitImageReferenceMethodStateResult { ServiceResult = result }; + } + catch (System.OperationCanceledException) + { + throw; + } +#pragma warning disable CA1031 // Do not catch general exception types. + catch (System.Exception ex) +#pragma warning restore CA1031 + { + m_logger.MethodFailed("SubmitImageReference", ex); + return new SubmitImageReferenceMethodStateResult + { + ServiceResult = StatusCodes.BadInternalError + }; + } + }; + } + + private IVisionMediaProvider? ResolveMediaProvider(NodeId sensorNodeId) + { + return m_registry.TryGetSensor(sensorNodeId, out SensorRegistration? sensor) + ? sensor?.MediaProvider + : null; + } + + private IVisionFeedbackSink? ResolveFeedbackSink(NodeId pipelineNodeId) + { + return m_registry.TryGetPipeline(pipelineNodeId, out PipelineRegistration? pipeline) + ? pipeline?.FeedbackSink + : null; + } + + private ClipEndpointState? FindClipEndpoint(NodeId sensorNodeId, NodeId endpointNodeId) + { + if (endpointNodeId.IsNull || + !m_registry.TryGetSensor(sensorNodeId, out SensorRegistration? sensor) || + sensor == null) + { + return null; + } + for (int ii = 0; ii < sensor.ClipEndpoints.Count; ii++) + { + ClipEndpointState clip = sensor.ClipEndpoints[ii]; + if (clip.NodeId == endpointNodeId) + { + return clip; + } + } + return null; + } + + private static bool IsInlineDeliveryEnabled(ClipEndpointState clip) + { + return clip.InlineDeliveryEnabled?.Value == true; + } + + /// + /// Publishes a clip that GetClip just produced onto the endpoint's + /// LatestClip and LatestClipMetadata variables. + /// + /// + /// Without this the two variables are created by the builder and then never + /// written, so LatestClip reports Bad_NoDataAvailable for the life of + /// the Server and a consumer that follows the model - read the published frame + /// first, call the method only if there is none - never gets a frame at all. A + /// clip the Server has just encoded is by definition the latest one, so the + /// dispatcher publishes it here rather than leaving every provider to remember to. + /// + private static void PublishLatestClip( + ISystemContext context, + ClipEndpointState? clip, + GetClipMethodStateResult result, + DateTimeUtc timestamp) + { + if (clip == null || !IsInlineDeliveryEnabled(clip) || ServiceResult.IsBad(result.ServiceResult)) + { + return; + } + ByteString inline = result.InlineImage; + if (inline.IsNull || inline.IsEmpty) + { + return; + } + DateTimeUtc sourceTimestamp = timestamp; + if (clip.LatestClip != null) + { + clip.LatestClip.Value = inline; + clip.LatestClip.StatusCode = StatusCodes.Good; + clip.LatestClip.Timestamp = sourceTimestamp; + clip.LatestClip.ClearChangeMasks(context, false); + } + if (clip.LatestClipMetadata != null) + { + clip.LatestClipMetadata.Value = result.Image; + clip.LatestClipMetadata.StatusCode = StatusCodes.Good; + clip.LatestClipMetadata.Timestamp = sourceTimestamp; + clip.LatestClipMetadata.ClearChangeMasks(context, false); + } + } + + private static GetClipMethodStateResult EnforceInlineLimit( + ClipEndpointState? clip, + VisionClipResult providerResult) + { + ByteString inline = providerResult.InlineImage; + if (!inline.IsNull && + !inline.IsEmpty && + clip?.MaxInlineClipSize is PropertyState limit && + limit.Value > 0u && + inline.Length > (int)limit.Value) + { + return new GetClipMethodStateResult + { + ServiceResult = StatusCodes.BadEncodingLimitsExceeded, + Image = providerResult.Image, + EndpointOut = providerResult.EndpointOut, + InlineImage = default + }; + } + return new GetClipMethodStateResult + { + ServiceResult = providerResult.ServiceResult, + Image = providerResult.Image, + EndpointOut = providerResult.EndpointOut, + InlineImage = inline + }; + } + + private static NodeId ReadPipelineSensor(InferencePipelineState pipeline) + { + NodeId value = pipeline.Sensor?.Value ?? NodeId.Null; + return value.IsNull ? NodeId.Null : value; + } + + private static NodeId ReadPipelineDeployment(InferencePipelineState pipeline) + { + NodeId value = pipeline.Deployment?.Value ?? NodeId.Null; + return value.IsNull ? NodeId.Null : value; + } + + private readonly VisionRegistry m_registry; + private readonly ILogger m_logger; + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionMethodDispatcherLog.cs b/src/Opc.Ua.Vision.Server/VisionMethodDispatcherLog.cs new file mode 100644 index 0000000000..5a52044902 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionMethodDispatcherLog.cs @@ -0,0 +1,73 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Microsoft.Extensions.Logging; + +namespace Opc.Ua.Vision.Server +{ + internal static partial class VisionMethodDispatcherLog + { + [LoggerMessage( + EventId = VisionServerEventIds.NodeManagerReady, + Level = LogLevel.Information, + Message = "Vision node manager loaded the Vision model.")] + public static partial void NodeManagerReady(this ILogger logger); + + [LoggerMessage( + EventId = VisionServerEventIds.MediaProviderMissing, + Level = LogLevel.Warning, + Message = "Vision media provider is not wired for sensor {SensorNodeId}.")] + public static partial void MediaProviderMissing(this ILogger logger, NodeId sensorNodeId); + + [LoggerMessage( + EventId = VisionServerEventIds.InferenceProviderMissing, + Level = LogLevel.Warning, + Message = "Vision inference provider is not wired for pipeline {PipelineNodeId}.")] + public static partial void InferenceProviderMissing(this ILogger logger, NodeId pipelineNodeId); + + [LoggerMessage( + EventId = VisionServerEventIds.FeedbackSinkMissing, + Level = LogLevel.Warning, + Message = "Vision feedback sink is not wired for pipeline {PipelineNodeId}.")] + public static partial void FeedbackSinkMissing(this ILogger logger, NodeId pipelineNodeId); + + [LoggerMessage( + EventId = VisionServerEventIds.MethodFailed, + Level = LogLevel.Warning, + Message = "Vision method {Method} failed.")] + public static partial void MethodFailed(this ILogger logger, string method, Exception exception); + + [LoggerMessage( + EventId = VisionServerEventIds.FeedbackAccepted, + Level = LogLevel.Information, + Message = "Vision feedback method {Method} accepted for pipeline {PipelineNodeId}.")] + public static partial void FeedbackAccepted(this ILogger logger, string method, NodeId pipelineNodeId); + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionNodeManager.cs b/src/Opc.Ua.Vision.Server/VisionNodeManager.cs new file mode 100644 index 0000000000..5d953e03ed --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionNodeManager.cs @@ -0,0 +1,475 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging; +using Opc.Ua.Server; +using Opc.Ua.Server.Fluent; +using Opc.Ua.Vision.Server.Hosting; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Standalone node manager for the OPC UA Vision companion model. + /// + /// + /// The node manager loads the Vision NodeSet through + /// instances, materialises the + /// well-known Server/Vision root and lets configurators + /// populate sensors, coordinate frames and inference pipelines + /// through the fluent surface. + /// + public sealed class VisionNodeManager : + FluentNodeManagerBase, + INodeIdFactory, + IAsyncDisposable, + IConformanceContributor + { + /// + /// Creates a standalone Vision node manager loading only the + /// built-in model provider. + /// + public VisionNodeManager(IServerInternal server, ApplicationConfiguration configuration) + : this( + server, + configuration, + new IVisionModelProvider[] { new VisionModelProvider() }, + new VisionServerOptions()) + { + } + + /// + /// Creates a Vision node manager with explicit services. + /// + public VisionNodeManager( + IServerInternal server, + ApplicationConfiguration configuration, + ArrayOf providers, + VisionServerOptions options, + IVisionPostSetupRunner? runner = null, + IServiceProvider? services = null) + : base( + server, + configuration, + server.Telemetry.CreateLogger(), + GetNamespaceUris(providers, options)) + { + m_providers = NormalizeProviders(providers); + m_options = options ?? throw new ArgumentNullException(nameof(options)); + m_options.Validate(); + m_runner = runner; + m_services = services; + m_registry = new VisionRegistry(); + m_dispatcherLogger = server.Telemetry.CreateLogger(); + m_dispatcher = new VisionMethodDispatcher(m_registry, m_dispatcherLogger); + SystemContext.NodeIdFactory = this; + RegisterEncodeables(SystemContext); + } + + /// + /// Gets the Vision root object. + /// + /// + public VisionRootState Root => m_root ?? + throw ServiceResultException.Create( + StatusCodes.BadConfigurationError, + "The Vision address space is not available yet."); + + /// + public ArrayOf ConformanceUnits => ArrayOf.Empty; + + /// + public ArrayOf ServerProfiles => ComputeServerProfileArrayEntries(); + + /// + public override NodeId New(ISystemContext context, NodeState node) + { + if (context == null) + { + throw new ArgumentNullException(nameof(context)); + } + if (node == null) + { + throw new ArgumentNullException(nameof(node)); + } + if (node is BaseInstanceState instance && instance.Parent != null) + { + return new NodeId( + $"{instance.Parent.NodeId.IdentifierAsString}_{instance.SymbolicName}", + GetInstanceNamespaceIndex(context)); + } + if (node.NodeId.IsNull) + { + return new NodeId(Guid.NewGuid(), GetInstanceNamespaceIndex(context)); + } + return node.NodeId; + } + + /// + /// Creates a direct build context for non-DI configuration. + /// + /// + /// Nodes the builder grafts onto an already created address space + /// are only browsable by their own once they + /// have been registered with the node manager. A context created + /// here never registers anything on its own, so prefer + /// , which runs the same fluent + /// surface and then registers everything it built. + /// + public IVisionBuildContext CreateVisionBuildContext(CancellationToken cancellationToken = default) + { + return CreateBuildContextCore(cancellationToken); + } + + /// + /// Configures the Vision address space through the fluent builder + /// and registers every node the builder created, so each one can be + /// browsed and read by its own . + /// + /// + /// Populates sensors, coordinate frames and inference pipelines. + /// + /// + /// Cancels the configuration. + /// + /// is null. + public async ValueTask ConfigureVisionAsync( + Action configure, + CancellationToken cancellationToken = default) + { + if (configure == null) + { + throw new ArgumentNullException(nameof(configure)); + } + VisionBuildContext context = CreateBuildContextCore(cancellationToken); + configure(context); + await context.FlushPendingRegistrationsAsync(cancellationToken).ConfigureAwait(false); + } + + /// + /// Registers a node, giving it and its children the reference type and + /// type definition an instance node is not valid without. + /// + /// + /// + /// The generated CreateOrReplace helpers materialise an optional + /// child by constructing the state object directly, which leaves both + /// unset. A child with no ReferenceTypeId is referenced by + /// nothing, so it cannot be browsed from its parent; one with no + /// TypeDefinitionId is a malformed Object that any client + /// filtering by type silently skips. + /// + /// + /// Doing it here rather than in the builder covers the case the builder + /// cannot see: a result published at runtime, long after the address + /// space was created, by an inference provider that assembled the node + /// itself. That path produced results a client could list but not read, + /// which is how this was found. + /// + /// + /// + /// The system context to resolve default type definitions against. + /// + /// The node to register. + /// Cancels the registration. + protected override ValueTask AddPredefinedNodeAsync( + ISystemContext context, + NodeState node, + CancellationToken cancellationToken = default) + { + if (node != null) + { + NormalizeInstanceMetadata(context, node); + } + return base.AddPredefinedNodeAsync(context, node!, cancellationToken); + } + + internal static void NormalizeInstanceMetadata(ISystemContext context, NodeState node) + { + var children = new List(); + node.GetChildren(context, children); + for (int ii = 0; ii < children.Count; ii++) + { + BaseInstanceState child = children[ii]; + if (child.ReferenceTypeId.IsNull) + { + child.ReferenceTypeId = child is PropertyState + ? global::Opc.Ua.ReferenceTypeIds.HasProperty + : global::Opc.Ua.ReferenceTypeIds.HasComponent; + } + if (child.TypeDefinitionId.IsNull) + { + child.TypeDefinitionId = child.GetDefaultTypeDefinitionId(context); + } + NormalizeInstanceMetadata(context, child); + } + } + + internal VisionBuildContext CreateBuildContextCore(CancellationToken cancellationToken) + { + return new VisionBuildContext( + this, + Root, + m_options, + m_registry, + m_dispatcher, + cancellationToken, + m_services); + } + + /// + /// Asynchronously disposes the node manager. + /// + public ValueTask DisposeAsync() + { + Dispose(disposing: true); + GC.SuppressFinalize(this); + return default; + } + + /// + public override async ValueTask CreateAddressSpaceAsync( + IDictionary> externalReferences, + CancellationToken cancellationToken = default) + { + await base.CreateAddressSpaceAsync(externalReferences, cancellationToken).ConfigureAwait(false); + RegisterEncodeables(SystemContext); + m_root = await GetOrCreateRootAsync(externalReferences, cancellationToken).ConfigureAwait(false); + if (m_runner != null) + { + await m_runner.RunAsync(this, m_root, m_options, cancellationToken).ConfigureAwait(false); + } + PublishServerProfiles(); + m_dispatcherLogger.NodeManagerReady(); + } + + /// + protected override ValueTask LoadPredefinedNodesAsync( + ISystemContext context, + CancellationToken cancellationToken = default) + { + if (context == null) + { + throw new ArgumentNullException(nameof(context)); + } + var nodes = new NodeStateCollection(); + for (int ii = 0; ii < m_providers.Count; ii++) + { + m_providers[ii].AddPredefinedNodes(nodes, context); + } + return new ValueTask(nodes); + } + + internal static ArrayOf NormalizeProviders( + ArrayOf providers) + { + if (providers.IsNull || providers.Count == 0) + { + return new IVisionModelProvider[] { new VisionModelProvider() }; + } + var sorted = new List(); + for (int ii = 0; ii < providers.Count; ii++) + { + sorted.Add(providers[ii]); + } + sorted.Sort(static (left, right) => left.Order.CompareTo(right.Order)); + return sorted.ToArray().ToArrayOf(); + } + + internal static string[] GetNamespaceUris( + ArrayOf providers, + VisionServerOptions options) + { + options ??= new VisionServerOptions(); + options.Validate(); + var uris = new List(); + ArrayOf normalized = NormalizeProviders(providers); + for (int ii = 0; ii < normalized.Count; ii++) + { + ArrayOf providerUris = normalized[ii].NamespaceUris; + for (int jj = 0; jj < providerUris.Count; jj++) + { + if (!uris.Contains(providerUris[jj])) + { + uris.Add(providerUris[jj]); + } + } + } + if (!uris.Contains(options.InstanceNamespaceUri)) + { + uris.Add(options.InstanceNamespaceUri); + } + return [.. uris]; + } + + internal void PublishServerProfiles() + { + ServerObjectState? serverObject = Server?.ServerObject; + BaseVariableState? profileArray = serverObject?.ServerCapabilities?.ServerProfileArray; + if (profileArray == null) + { + return; + } + ArrayOf profiles = ServerProfiles; + var merged = new List(); + if (profileArray.Value.TryGetValue(out ArrayOf existing)) + { + for (int ii = 0; ii < existing.Count; ii++) + { + if (!string.IsNullOrEmpty(existing[ii]) && !merged.Contains(existing[ii])) + { + merged.Add(existing[ii]); + } + } + } + for (int ii = 0; ii < profiles.Count; ii++) + { + if (!string.IsNullOrEmpty(profiles[ii]) && !merged.Contains(profiles[ii])) + { + merged.Add(profiles[ii]); + } + } + profileArray.Value = Variant.From(merged.ToArrayOf()); + profileArray.ClearChangeMasks(SystemContext, false); + } + + private static void RegisterEncodeables(ServerSystemContext context) + { + var probe = new global::Opc.Ua.Vision.VisionPose3DDataType(); + if (!context.EncodeableFactory.TryGetEncodeableType(probe.BinaryEncodingId, out _)) + { + context.EncodeableFactory.Builder.AddOpcUaVision().Commit(); + } + } + + private async ValueTask GetOrCreateRootAsync( + IDictionary> externalReferences, + CancellationToken cancellationToken) + { + NodeId rootId = ExpandedNodeId.ToNodeId( + global::Opc.Ua.Vision.ObjectIds.Vision, + SystemContext.NamespaceUris); + if (FindPredefinedNode(rootId) is VisionRootState existing) + { + await EnsureRootChildrenAsync(existing, cancellationToken).ConfigureAwait(false); + return existing; + } + VisionRootState root = CreateRoot(); + await AddPredefinedNodeAsync(root, cancellationToken).ConfigureAwait(false); + if (!externalReferences.TryGetValue(global::Opc.Ua.ObjectIds.Server, out IList? references)) + { + externalReferences[global::Opc.Ua.ObjectIds.Server] = references = []; + } + references.Add(new NodeStateReference( + global::Opc.Ua.ReferenceTypeIds.HasComponent, + false, + root.NodeId)); + return root; + } + + private async ValueTask EnsureRootChildrenAsync(VisionRootState root, CancellationToken cancellationToken) + { + root.CreateOrReplaceSensors(SystemContext, null); + if (root.Sensors is FolderState sensors && FindPredefinedNode(sensors.NodeId) == null) + { + await AddPredefinedNodeAsync(sensors, cancellationToken).ConfigureAwait(false); + } + } + + private VisionRootState CreateRoot() + { + var browseName = new QualifiedName("Vision", GetVisionNamespaceIndex(SystemContext)); + VisionRootState root = SystemContext.CreateInstanceOfVisionRootType( + Server.ServerObject, + browseName); + root.ReferenceTypeId = global::Opc.Ua.ReferenceTypeIds.HasComponent; + root.CreateOrReplaceSensors(SystemContext, null); + root.AddReference(global::Opc.Ua.ReferenceTypeIds.HasComponent, true, global::Opc.Ua.ObjectIds.Server); + return root; + } + + private ushort GetInstanceNamespaceIndex(ISystemContext context) + { + return (ushort)context.NamespaceUris.GetIndex(m_options.InstanceNamespaceUri); + } + + private ushort GetVisionNamespaceIndex(ServerSystemContext context) + { + return (ushort)context.NamespaceUris.GetIndex(global::Opc.Ua.Vision.Namespaces.Vision); + } + + private ArrayOf ComputeServerProfileArrayEntries() + { + ArrayOf facets = VisionFacetCalculator.Compute(m_registry); + var entries = new List(); + var facetNames = new HashSet(StringComparer.Ordinal); + for (int ii = 0; ii < facets.Count; ii++) + { + facetNames.Add(facets[ii]); + } + ArrayOf additional = m_options.AdditionalFacets; + for (int ii = 0; ii < additional.Count; ii++) + { + if (!string.IsNullOrEmpty(additional[ii])) + { + facetNames.Add(additional[ii]); + } + } + ArrayOf profiles = VisionFacetCalculator.ComputeProfiles(facetNames.ToArrayOf()); + for (int ii = 0; ii < profiles.Count; ii++) + { + if (!entries.Contains(profiles[ii])) + { + entries.Add(profiles[ii]); + } + } + foreach (string facetName in facetNames) + { + if (VisionConformanceUris.TryGetFacetUri(facetName, out string facetUri) && + !entries.Contains(facetUri)) + { + entries.Add(facetUri); + } + } + return entries.ToArrayOf(); + } + + private readonly ArrayOf m_providers; + private readonly VisionServerOptions m_options; + private readonly IVisionPostSetupRunner? m_runner; + private readonly IServiceProvider? m_services; + private readonly VisionRegistry m_registry; + private readonly VisionMethodDispatcher m_dispatcher; + private readonly ILogger m_dispatcherLogger; + private VisionRootState? m_root; + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionNodeManagerFactory.cs b/src/Opc.Ua.Vision.Server/VisionNodeManagerFactory.cs new file mode 100644 index 0000000000..3e964c129e --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionNodeManagerFactory.cs @@ -0,0 +1,94 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Diagnostics.CodeAnalysis; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Server; +using Opc.Ua.Vision.Server.Hosting; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Creates standalone Vision node managers. + /// + public sealed class VisionNodeManagerFactory : IAsyncNodeManagerFactory + { + /// + /// Creates a factory with default options and the built-in provider. + /// + public VisionNodeManagerFactory() + : this( + new IVisionModelProvider[] { new VisionModelProvider() }, + new VisionServerOptions()) + { + } + + /// + /// Creates a factory with explicit providers and options. + /// + public VisionNodeManagerFactory( + ArrayOf providers, + VisionServerOptions options, + IVisionPostSetupRunner? runner = null) + { + m_providers = VisionNodeManager.NormalizeProviders(providers); + m_options = options; + m_options.Validate(); + m_runner = runner; + } + + /// + public ArrayOf NamespacesUris => + VisionNodeManager.GetNamespaceUris(m_providers, m_options).ToArrayOf(); + + /// + [SuppressMessage( + "Reliability", + "CA2000:Dispose objects before losing scope", + Justification = "Ownership is transferred to the server.")] + public ValueTask CreateAsync( + IServerInternal server, + ApplicationConfiguration configuration, + CancellationToken cancellationToken = default) + { + IAsyncNodeManager manager = new VisionNodeManager( + server, + configuration, + m_providers, + m_options, + m_runner); + return new ValueTask(manager); + } + + private readonly ArrayOf m_providers; + private readonly VisionServerOptions m_options; + private readonly IVisionPostSetupRunner? m_runner; + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionRegistry.cs b/src/Opc.Ua.Vision.Server/VisionRegistry.cs new file mode 100644 index 0000000000..96c2b1e304 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionRegistry.cs @@ -0,0 +1,347 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using Opc.Ua.Vision; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Tracks the sensors, pipelines, coordinate frames, feedback objects + /// and provider bindings materialised inside a single Vision node + /// manager. + /// + /// + /// The registry is the single source of truth for facet computation + /// (), coordinate-frame math and + /// method-dispatch to the injected providers. + /// + internal sealed class VisionRegistry + { + public IReadOnlyDictionary Sensors => m_sensorsByBrowseName; + + public IReadOnlyDictionary SensorsByNodeId => m_sensorsByNodeId; + + public IReadOnlyDictionary Pipelines => m_pipelinesByBrowseName; + + public IReadOnlyDictionary PipelinesByNodeId => m_pipelinesByNodeId; + + public IReadOnlyDictionary Frames => m_framesByBrowseName; + + public IReadOnlyDictionary FramesByFrameId => m_framesByFrameId; + + public bool AnySensorHasFacet(string facetName) + { + foreach (KeyValuePair pair in m_sensorsByBrowseName) + { + if (pair.Value.Facets.Contains(facetName)) + { + return true; + } + } + return false; + } + + public bool AnyPipelineHasFacet(string facetName) + { + foreach (KeyValuePair pair in m_pipelinesByBrowseName) + { + if (pair.Value.Facets.Contains(facetName)) + { + return true; + } + } + return false; + } + + public void AddSensor(SensorRegistration registration) + { + if (registration == null) + { + throw new ArgumentNullException(nameof(registration)); + } + m_sensorsByBrowseName[registration.BrowseName] = registration; + m_sensorsByNodeId[registration.NodeId] = registration; + } + + public bool TryGetSensor(string browseName, out SensorRegistration? registration) + { + return m_sensorsByBrowseName.TryGetValue(browseName ?? string.Empty, out registration); + } + + public bool TryGetSensor(NodeId nodeId, out SensorRegistration? registration) + { + if (nodeId.IsNull) + { + registration = null; + return false; + } + return m_sensorsByNodeId.TryGetValue(nodeId, out registration); + } + + public void AddPipeline(PipelineRegistration registration) + { + if (registration == null) + { + throw new ArgumentNullException(nameof(registration)); + } + m_pipelinesByBrowseName[registration.BrowseName] = registration; + m_pipelinesByNodeId[registration.NodeId] = registration; + } + + public bool TryGetPipeline(string browseName, out PipelineRegistration? registration) + { + return m_pipelinesByBrowseName.TryGetValue(browseName ?? string.Empty, out registration); + } + + public bool TryGetPipeline(NodeId nodeId, out PipelineRegistration? registration) + { + if (nodeId.IsNull) + { + registration = null; + return false; + } + return m_pipelinesByNodeId.TryGetValue(nodeId, out registration); + } + + public void AddFrame(FrameRegistration registration) + { + if (registration == null) + { + throw new ArgumentNullException(nameof(registration)); + } + m_framesByBrowseName[registration.BrowseName] = registration; + m_framesByFrameId[registration.FrameId] = registration; + } + + public bool TryGetFrame(string browseName, out FrameRegistration? registration) + { + return m_framesByBrowseName.TryGetValue(browseName ?? string.Empty, out registration); + } + + public bool TryGetFrameByFrameId(string frameId, out FrameRegistration? registration) + { + return m_framesByFrameId.TryGetValue(frameId ?? string.Empty, out registration); + } + + public FrameRegistration? TryFindFrameByFrameId(string frameId) + { + return m_framesByFrameId.TryGetValue(frameId ?? string.Empty, out FrameRegistration? registration) + ? registration + : null; + } + + public void AddDeferredExtrinsicResolution( + ExtrinsicCalibrationState calibration, + string sourceFrameId, + string targetFrameId) + { + if (calibration == null) + { + return; + } + m_deferredExtrinsicResolutions.Add(new DeferredExtrinsic(calibration, sourceFrameId ?? string.Empty, targetFrameId ?? string.Empty)); + } + + public void ResolveDeferredExtrinsics() + { + foreach (DeferredExtrinsic deferred in m_deferredExtrinsicResolutions) + { + if (deferred.Calibration.SourceFrame != null && + m_framesByFrameId.TryGetValue(deferred.SourceFrameId, out FrameRegistration? source)) + { + deferred.Calibration.SourceFrame.Value = source!.NodeId; + } + if (deferred.Calibration.TargetFrame != null && + m_framesByFrameId.TryGetValue(deferred.TargetFrameId, out FrameRegistration? target)) + { + deferred.Calibration.TargetFrame.Value = target!.NodeId; + } + } + } + + public IReadOnlyDictionary ToFrameSnapshots() + { + var snapshots = new Dictionary( + StringComparer.Ordinal); + foreach (KeyValuePair pair in m_framesByFrameId) + { + snapshots.Add(pair.Key, new VisionCoordinateFrameMath.CoordinateFrameSnapshot( + pair.Value.FrameId, + pair.Value.Role, + pair.Value.ParentFrameId ?? string.Empty, + pair.Value.Transform)); + } + return snapshots; + } + + private readonly Dictionary m_sensorsByBrowseName = new(StringComparer.Ordinal); + private readonly Dictionary m_sensorsByNodeId = new(); + private readonly Dictionary m_pipelinesByBrowseName = new(StringComparer.Ordinal); + private readonly Dictionary m_pipelinesByNodeId = new(); + private readonly Dictionary m_framesByBrowseName = new(StringComparer.Ordinal); + private readonly Dictionary m_framesByFrameId = new(StringComparer.Ordinal); + private readonly List m_deferredExtrinsicResolutions = []; + + private readonly struct DeferredExtrinsic + { + public DeferredExtrinsic(ExtrinsicCalibrationState calibration, string sourceFrameId, string targetFrameId) + { + Calibration = calibration; + SourceFrameId = sourceFrameId; + TargetFrameId = targetFrameId; + } + + public ExtrinsicCalibrationState Calibration { get; } + + public string SourceFrameId { get; } + + public string TargetFrameId { get; } + } + } + + /// + /// Metadata recorded per sensor. + /// + internal sealed class SensorRegistration + { + public SensorRegistration( + string browseName, + NodeId nodeId, + VisionSensorState sensor, + VisionSensorModalityEnum modality, + VisionRealityKindEnum realityKind, + HashSet facets, + IVisionMediaProvider? mediaProvider) + { + BrowseName = browseName; + NodeId = nodeId; + Sensor = sensor; + Modality = modality; + RealityKind = realityKind; + Facets = facets; + MediaProvider = mediaProvider; + } + + public string BrowseName { get; } + + public NodeId NodeId { get; } + + public VisionSensorState Sensor { get; } + + public VisionSensorModalityEnum Modality { get; } + + public VisionRealityKindEnum RealityKind { get; set; } + + public HashSet Facets { get; } + + public IVisionMediaProvider? MediaProvider { get; set; } + + public List StreamEndpoints { get; } = []; + + public List ClipEndpoints { get; } = []; + + public bool HasIntrinsicCalibration { get; set; } + + public bool HasExtrinsicCalibration { get; set; } + + public bool HasOptics { get; set; } + + public bool HasIllumination { get; set; } + } + + /// + /// Metadata recorded per pipeline. + /// + internal sealed class PipelineRegistration + { + public PipelineRegistration( + string browseName, + NodeId nodeId, + InferencePipelineState pipeline, + HashSet facets) + { + BrowseName = browseName; + NodeId = nodeId; + Pipeline = pipeline; + Facets = facets; + } + + public string BrowseName { get; } + + public NodeId NodeId { get; } + + public InferencePipelineState Pipeline { get; } + + public HashSet Facets { get; } + + public IVisionInferenceProvider? InferenceProvider { get; set; } + + public IVisionFeedbackSink? FeedbackSink { get; set; } + } + + /// + /// Metadata recorded per coordinate frame. + /// + internal sealed class FrameRegistration + { + public FrameRegistration( + string browseName, + NodeId nodeId, + string frameId, + VisionFrameRoleEnum role, + string? parentFrameId, + VisionPose3DDataType transform, + CoordinateFrameState frame) + { + BrowseName = browseName; + NodeId = nodeId; + FrameId = frameId; + Role = role; + ParentFrameId = parentFrameId; + Transform = transform; + Frame = frame; + } + + public string BrowseName { get; } + + public NodeId NodeId { get; } + + public string FrameId { get; } + + public VisionFrameRoleEnum Role { get; } + + public string? ParentFrameId { get; } + + public VisionPose3DDataType Transform { get; } + + public CoordinateFrameState Frame { get; } + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionServerEventIds.cs b/src/Opc.Ua.Vision.Server/VisionServerEventIds.cs new file mode 100644 index 0000000000..d374e2d8de --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionServerEventIds.cs @@ -0,0 +1,67 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Vision.Server +{ + /// + /// Logger event identifiers used by the Vision server surface. + /// + internal static class VisionServerEventIds + { + /// + /// The Vision node manager finished creating its address space. + /// + public const int NodeManagerReady = 16000; + + /// + /// A Vision media provider is missing for a sensor. + /// + public const int MediaProviderMissing = 16001; + + /// + /// A Vision inference provider is missing for a pipeline. + /// + public const int InferenceProviderMissing = 16002; + + /// + /// A Vision feedback sink is missing for a pipeline. + /// + public const int FeedbackSinkMissing = 16003; + + /// + /// A Vision method failed and produced a Bad_ status. + /// + public const int MethodFailed = 16004; + + /// + /// A published result was accepted from an off-server caller. + /// + public const int FeedbackAccepted = 16005; + } +} diff --git a/src/Opc.Ua.Vision.Server/VisionServerOptions.cs b/src/Opc.Ua.Vision.Server/VisionServerOptions.cs new file mode 100644 index 0000000000..9f73bea5d4 --- /dev/null +++ b/src/Opc.Ua.Vision.Server/VisionServerOptions.cs @@ -0,0 +1,115 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +namespace Opc.Ua.Vision.Server +{ + /// + /// Options for the standalone Vision node manager. + /// + public sealed class VisionServerOptions + { + /// + /// Default application-owned namespace for Vision instances. + /// + public const string DefaultInstanceNamespaceUri = + "urn:opcua-netstandard:vision:instances"; + + /// + /// Default version reported on Server/Vision. + /// + public const string DefaultSpecificationVersion = "0.1.0"; + + /// + /// Gets or sets the application-owned namespace URI used for the + /// Vision instances created underneath the well-known Vision root. + /// + /// + /// This namespace is application-specific and must be distinct from + /// the OPC UA base namespace and the Vision companion namespace. + /// + public string InstanceNamespaceUri { get; set; } = DefaultInstanceNamespaceUri; + + /// + /// Gets or sets the specification version this Server reports. + /// + public string SpecificationVersion { get; set; } = DefaultSpecificationVersion; + + /// + /// Gets or sets the additional facets the Server declares beyond those + /// the facet calculator derives from the address space. This is the + /// escape hatch for facets whose requirements cannot be inspected + /// structurally (for example, an interop facet requiring behavioural + /// contract that is met by the host). + /// + public ArrayOf AdditionalFacets { get; set; } = ArrayOf.Empty; + + /// + /// Validates the option values. + /// + /// + /// One of the required options is empty or invalid. + /// + /// + /// The instance namespace is one of the standard model namespaces. + /// + public void Validate() + { + if (string.IsNullOrWhiteSpace(InstanceNamespaceUri)) + { + throw new ArgumentException( + "VisionServerOptions.InstanceNamespaceUri must not be empty.", + nameof(InstanceNamespaceUri)); + } + if (!Uri.TryCreate(InstanceNamespaceUri, UriKind.Absolute, out Uri? uri) || + string.IsNullOrEmpty(uri.Scheme)) + { + throw new ArgumentException( + "VisionServerOptions.InstanceNamespaceUri must be an absolute URI or URN.", + nameof(InstanceNamespaceUri)); + } + if (string.IsNullOrWhiteSpace(SpecificationVersion)) + { + throw new ArgumentException( + "VisionServerOptions.SpecificationVersion must not be empty.", + nameof(SpecificationVersion)); + } + if (InstanceNamespaceUri == global::Opc.Ua.Namespaces.OpcUa || + InstanceNamespaceUri == global::Opc.Ua.Vision.Namespaces.Vision) + { + throw ServiceResultException.Create( + StatusCodes.BadConfigurationError, + "VisionServerOptions.InstanceNamespaceUri '{0}' is a model namespace. " + + "Configure a distinct application-owned namespace for Vision instances.", + InstanceNamespaceUri); + } + } + } +} diff --git a/src/Opc.Ua.Vision/Model/Opc.Ua.Vision.NodeSet2.xml b/src/Opc.Ua.Vision/Model/Opc.Ua.Vision.NodeSet2.xml new file mode 100644 index 0000000000..ccfc52d7a0 --- /dev/null +++ b/src/Opc.Ua.Vision/Model/Opc.Ua.Vision.NodeSet2.xml @@ -0,0 +1,2211 @@ + + + + + http://opcfoundation.org/UA/Vision/ + + + + + + + + i=1 + i=6 + i=7 + i=9 + i=11 + i=12 + i=14 + i=15 + i=17 + i=20 + i=21 + i=294 + i=290 + i=296 + i=887 + i=24 + i=47 + i=46 + i=45 + i=35 + i=40 + i=37 + i=17603 + i=38 + i=78 + i=80 + i=11508 + i=11510 + + + VisionRealityKindEnum + Whether the sensor observes the physical world, a simulation, or both. This is the sim/real switch: every other member of the model means the same thing regardless of its value. + Vision DataTypes + + i=29 + ns=1;i=3901 + + A physical device observing the real world.A synthetic sensor rendered by a simulator (e.g. NVIDIA Isaac Sim); ground truth may be available.A physical device whose output is augmented or replayed through a simulation. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3001 + + PhysicalSimulatedHybrid + + + VisionStreamProtocolEnum + Wire protocol of a continuous media stream. Rtsp is the mandatory default: a conformant Server exposes at least one StreamEndpoint using it. + Vision DataTypes + + i=29 + ns=1;i=3902 + + RTSP (RFC 7826/2326). MANDATORY default streaming protocol.RTSP over TLS.WebRTC.Secure Reliable Transport.HTTP Live Streaming.Motion JPEG over HTTP.GenICam GenDC container stream.A protocol identified by the endpoint URI scheme.The stream is carried on an OPC UA data channel multiplexed onto the SecureChannel the client already has, per the OPC UA - Data Channels errata proposal. OPTIONAL and never a default: see clause 6.7. That proposal is a DRAFT in this repository, not a released OPC UA specification, so a Server is fully conformant without it. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3002 + + RtspRtspsWebRtcSrtHlsMjpegGenDcOtherDataChannel + + + VisionClipFormatEnum + Encoding of a still clip. Jpeg is the mandatory default: a conformant Server exposes at least one ClipEndpoint using it. + Vision DataTypes + + i=29 + ns=1;i=3903 + + JPEG (ISO/IEC 10918). MANDATORY default clip format.PNG.TIFF.BMP.WebP.GenICam GenDC container.A format identified by the accompanying media type. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3003 + + JpegPngTiffBmpWebPGenDcOther + + + VisionVideoCodecEnum + Codec carried by a stream endpoint. + Vision DataTypes + + i=29 + ns=1;i=3904 + + Uncompressed frames. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3004 + + H264H265MjpegAv1RawOther + + + VisionEndpointStateEnum + Runtime lifecycle state of a media endpoint or deployment. + Vision DataTypes + + i=29 + ns=1;i=3905 + + Declared but not serving.Able to serve; no active session.Serving at least one session.Serving below configured quality.Unable to serve. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3005 + + InactiveReadyActiveDegradedFaulted + + + VisionEndpointAuthenticationEnum + Authentication a client must present to the media endpoint. This is the media-plane credential, independent of the OPC UA session. + Vision DataTypes + + i=29 + ns=1;i=3906 + + No authentication. Appropriate only on an isolated network.HTTP/RTSP Basic.HTTP/RTSP Digest.Bearer token, typically the time-limited token returned by GetStreamEndpoint or GetClip.Client certificate. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3006 + + NoneBasicDigestTokenMutualTls + + + VisionResultEvaluationEnum + Overall verdict of a result. Value semantics are aligned with the ResultEvaluationEnum of OPC 40001-101 so that a client already consuming Machinery results needs no new interpretation rules. + Vision DataTypes + + i=29 + ns=1;i=3909 + + No verdict available.Within tolerance / accepted.Out of tolerance / rejected.A verdict was not possible, typically because the measurement uncertainty spans a tolerance limit. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3009 + + UndefinedOkNotOkNotDecidable + + + VisionToleranceStatusEnum + Per-characteristic tolerance outcome. + Vision DataTypes + + i=29 + ns=1;i=3910 + + Uncertainty spans a tolerance limit. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3010 + + InToleranceOutOfToleranceIndeterminate + + + VisionFeedbackPurposeEnum + Why a client is pushing information back into the vision system. + Vision DataTypes + + i=29 + ns=1;i=3911 + + Render the geometry onto the outgoing stream.Record a downstream verdict against the result.Treat the payload as a corrected label for training.Use the payload as an acquisition or processing trigger. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3011 + + OverlayReconciliationGroundTruthLabelTrigger + + + VisionCalibrationMountEnum + Physical relationship between a camera and the kinematic chain it is calibrated against. + Vision DataTypes + + i=29 + ns=1;i=3912 + + Camera mounted on the moving flange or tool.Camera fixed in the workspace observing the tool.Camera fixed with no associated kinematic chain. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3012 + + EyeInHandEyeToHandFixedUnknown + + + VisionFrameRoleEnum + Role of a coordinate frame, following the ISO 9787 frame vocabulary. The mechanical interface and the tool are DISTINCT roles: a camera on a robot flange is calibrated to the mechanical interface, while a pick pose has to reach the tool centre point, and a model that cannot tell them apart cannot express the offset between them. + Vision DataTypes + + i=29 + ns=1;i=3913 + + The flange at the end of the last link, to which an end effector is fitted. This is what an eye-in-hand extrinsic calibration resolves to.A tool frame, whose origin is a tool centre point.A camera frame. Numbered after the ISO 9787 roles because it is not one of them. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3013 + + WorldBaseMechanicalInterfaceToolObjectOtherCamera + + + VisionDistortionModelEnum + Lens distortion model the coefficients belong to. + Vision DataTypes + + i=29 + ns=1;i=3914 + + Radial and tangential (k1, k2, p1, p2, k3...).Fisheye / equidistant. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3014 + + NoneBrownConradyKannalaBrandtRationalPolynomialOther + + + VisionSensorModalityEnum + What the sensor measures. + Vision DataTypes + + i=29 + ns=1;i=3915 + + Two-dimensional area-scan imaging.Line-scan imaging.Depth or point-cloud sensing.Event / neuromorphic camera. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3015 + + Area2DLine2DDepth3DThermalMultispectralEventOther + + + VisionLampTypeEnum + Emitter technology of a light source. The named values are those OPC 40100-2 gives as examples for ILampType.LampType, which is a free String there. + Vision DataTypes + + i=29 + ns=1;i=3916 + + Light-emitting diode. The default because it is what the overwhelming majority of machine-vision illuminators use.Includes the line and pattern projectors of laser-triangulation and structured-light sensors.An emitter technology none of the above names. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3016 + + LedFluorescentLaserXenonHalogenOther + + + VisionLightingModeEnum + How a light source is being driven. The named values are those OPC 40100-2 gives as examples for ILightingControllerType.LightingMode, which is an unconstrained UInt32 there. + Vision DataTypes + + i=29 + ns=1;i=3917 + + Constant output, not synchronised to acquisition.Pulsed in synchronisation with acquisition, which is what makes a short exposure viable on a moving part.Driven at a carrier frequency, so the contribution of the illuminator can be separated from ambient light.A drive mode none of the above names. + + + EnumStrings + + i=78 + i=68 + ns=1;i=3017 + + ContinuousStrobeModulatedOther + + + VisionPose3DDataType + A rigid-body pose expressed in a named coordinate frame. Position is metres; Orientation is a unit quaternion ordered (x, y, z, w). Covariance is an optional row-major 6x6 matrix over (x, y, z, rx, ry, rz); an empty array means the uncertainty is not reported. + Vision DataTypes + + i=22 + ns=1;i=5001 + + FrameId of the CoordinateFrame this pose is expressed in.Translation (x, y, z) in metres.Unit quaternion (x, y, z, w).Row-major 6x6 covariance, or empty. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=1;i=3050 + + + + VisionBoundingBox2DDataType + A box in image space. The origin is the top-left pixel; Rotation is degrees clockwise about the box centre, so 0 denotes an axis-aligned box. + Vision DataTypes + + i=22 + ns=1;i=5002 + + Box centre x, in pixels.Box centre y, in pixels.Box width, in pixels.Box height, in pixels.Rotation about the centre, in degrees. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=1;i=3051 + + + + VisionBoundingBox3DDataType + An oriented box in three-dimensional space, given by a centre pose and an extent. + Vision DataTypes + + i=22 + ns=1;i=5003 + + Centre pose of the box.Extent (x, y, z) in metres. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=1;i=3052 + + + + VisionImageReferenceDataType + A reference to image bytes that are NOT carried in the OPC UA payload. This is the default way results point at imagery: the Uri is resolved out-of-band, typically through a ClipEndpoint, and verified against Digest. + Vision DataTypes + + i=22 + ns=1;i=5004 + + Location of the encoded image.Cryptographic digest of the referenced bytes.Digest algorithm; default SHA-256.Container or encoding of the image.Pixel format using GenICam PFNC naming, for example Mono8, BayerRG12 or RGB8.Image width in pixels.Image height in pixels.Encoded size, so a client can decide whether to fetch.Acquisition time of the frame. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=1;i=3053 + + + + VisionIntrinsicsDataType + Pinhole intrinsics plus a distortion model, in pixel units, valid for the stated image size. For a simulated sensor these are derived from the USD camera aperture and focal-length attributes. + Vision DataTypes + + i=22 + ns=1;i=5005 + + Focal length in pixels, x.Focal length in pixels, y.Principal point x, in pixels.Principal point y, in pixels.Axis skew; 0 for square pixels.Model the coefficients belong to.Coefficients, model-ordered.Image width these intrinsics are valid for.Image height these intrinsics are valid for. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=1;i=3054 + + + + VisionDetectionDataType + One detected instance. This is the robotics-vision payload: a class, a score, and enough geometry to act on - a 2-D box for image-space work, a 3-D box and a 6-DoF pose for picking and visual servoing. Field naming follows the ROS 2 vision_msgs conventions so that a bridge is mechanical. + Vision DataTypes + + i=22 + ns=1;i=5006 + + Identifier unique within the result.Human-readable class name.Numeric class identifier within the model label set.Score in the range 0.0 to 1.0.True when BoundingBox2D is meaningful.Image-space box.True when BoundingBox3D is meaningful.Object-space box.True when Pose is meaningful.6-DoF pose, for example a grasp pose.Stable identity across frames, when tracking is done. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=1;i=3055 + + + + VisionCharacteristicDataType + One measured characteristic of an inspected part. This is the machine-vision payload, and the field set deliberately mirrors QIF (ISO 23952) Results, including measurement uncertainty, so that a QIF document can be produced from it without inventing information. + Vision DataTypes + + i=22 + ns=1;i=5007 + + Identifier of the characteristic measured.Human-readable characteristic name.Design or target value.Measured value.Actual minus Nominal.Lower tolerance limit, relative to Nominal.Upper tolerance limit, relative to Nominal.Expanded measurement uncertainty (ISO 14253), in the same unit as Actual. 0 means not reported.Engineering unit of Nominal, Actual and Deviation.Per-characteristic outcome. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=1;i=3056 + + + + VisionStreamSessionDataType + A leased media session. The Uri may embed a single-use or time-limited credential, which is why it is returned by a Method rather than published as a browsable Variable. + Vision DataTypes + + i=22 + ns=1;i=5008 + + Opaque token identifying the lease.Media URI to open.Protocol of the returned URI.Expiry after which the Uri is no longer valid. + + + Default Binary + Default Binary encoding of the structure. + + i=76 + ns=1;i=3057 + + + + HasCalibration + Links a sensor to a calibration currently valid for it. + Vision ReferenceTypes + IsCalibrationOf + + i=32 + + + + MountedOn + Links a sensor to the CoordinateFrame it is rigidly mounted on, for example a robot flange frame for an eye-in-hand camera. + Vision ReferenceTypes + HasMounted + + i=32 + + + + HasScenePrim + Links a sensor to the materialized USD prim representing it, when the Server also implements OPC UA - OpenUSD Scene Materialization. The target is expected to be a UsdGeomCameraType instance. Optional: PrimPath remains the portable descriptor. + Vision ReferenceTypes + IsScenePrimOf + + i=32 + + + + ProducedBy + Links a result to the inference pipeline that produced it. + Vision ReferenceTypes + Produces + + i=32 + + + + OpticsType + The lens in front of a sensor. Member names are deliberately aligned with the ILensType of OPC 40100-2 so that a Server implementing both models reports one set of values under two vocabularies. + Vision + + i=58 + ns=1;i=6001 + ns=1;i=6002 + ns=1;i=6003 + ns=1;i=6004 + ns=1;i=6005 + ns=1;i=6006 + ns=1;i=6007 + ns=1;i=6008 + + + + FocalLength + Focal length in millimetres. + + i=80 + i=68 + ns=1;i=1005 + + + + Aperture + Current aperture as an f-number. + + i=80 + i=68 + ns=1;i=1005 + + + + WorkingDistance + Current object-to-lens distance in metres. + + i=80 + i=68 + ns=1;i=1005 + + + + MinimumWorkingDistance + Smallest usable object-to-lens distance in metres. + + i=80 + i=68 + ns=1;i=1005 + + + + Magnification + Image size divided by object size. + + i=80 + i=68 + ns=1;i=1005 + + + + OpticalFormat + Largest sensor diagonal the lens covers, for example 2/3 inch. + + i=80 + i=68 + ns=1;i=1005 + + + + MountType + Lens mount, for example C, CS, F or M12. + + i=80 + i=68 + ns=1;i=1005 + + + + LensType + Lens class, for example Entocentric, Telecentric or Fisheye. + + i=80 + i=68 + ns=1;i=1005 + + + + IlluminationType + A controlled light source associated with a sensor. Member names align with the ILampType and ILightingControllerType of OPC 40100-2. + Vision + + i=58 + ns=1;i=6009 + ns=1;i=6010 + ns=1;i=6011 + ns=1;i=6012 + ns=1;i=6013 + + + + LampType + Emitter technology of this light source. + + i=80 + i=68 + ns=1;i=1006 + + + + Wavelength + Dominant emission wavelength in nanometres. + + i=80 + i=68 + ns=1;i=1006 + + + + RelativeIntensity + Current output as a percentage of full capability. + + i=80 + i=68 + ns=1;i=1006 + + + + LightingMode + How this light source is currently being driven. + + i=80 + i=68 + ns=1;i=1006 + + + + Quality + Remaining emitter quality as a percentage; 100 is new. + + i=80 + i=68 + ns=1;i=1006 + + + + MediaEndpointType + Abstract base for a media access point. The endpoint DESCRIBES where media can be obtained; on the default path the media itself never traverses OPC UA. Subtypes add the protocol- or format-specific members. + Vision + + i=58 + ns=1;i=6014 + ns=1;i=6015 + ns=1;i=6016 + ns=1;i=6017 + ns=1;i=6018 + ns=1;i=6019 + ns=1;i=6196 + ns=1;i=6197 + + + + EndpointId + Identifier of this endpoint, unique within the sensor. + + i=78 + i=68 + ns=1;i=1007 + + + + EndpointUri + Base URI at which the media is served. May be a template that GetStreamEndpoint or GetClip resolves into a session-specific URI. + + i=78 + i=68 + ns=1;i=1007 + + + + State + Runtime state of this endpoint. + + i=78 + i=68 + ns=1;i=1007 + + + + Authentication + Credential the media plane requires. Independent of the OPC UA session. + + i=78 + i=68 + ns=1;i=1007 + + + + SecureTransport + True when the media transport itself provides confidentiality, for example RTSPS, SRT with encryption, or HTTPS. Mandatory because clause 12.2 evaluates credential issuance on it: a Server SHALL NOT return a URI embedding a credential unless this is true and the OPC UA SecureChannel is SignAndEncrypt. A client SHALL treat false as meaning the media transport offers no confidentiality, whatever Authentication states. + + i=78 + i=68 + ns=1;i=1007 + + + + DefaultProfileName + Name of the profile this endpoint uses when GetStreamEndpoint is called with an empty ProfileName. A profile is a Server-local named configuration and has no node of its own, so this is the one profile name a client can rely on without prior knowledge of the Server. Empty means the endpoint has a single configuration and takes no profile name at all. See clause 6.3. + + i=80 + i=68 + ns=1;i=1007 + + + + StreamEndpointType + A continuous media stream. A conformant Server SHALL expose at least one instance whose StreamProtocol is Rtsp; every other protocol is optional. This is the default way to obtain live imagery. + Vision + + ns=1;i=1007 + ns=1;i=6020 + ns=1;i=6021 + ns=1;i=6022 + ns=1;i=6023 + ns=1;i=6024 + ns=1;i=6025 + ns=1;i=6026 + ns=1;i=6027 + ns=1;i=6028 + + + + StreamProtocol + Protocol of this stream. Rtsp is the mandatory default. + + i=78 + i=68 + ns=1;i=1008 + + + + ProtocolVersion + Version of StreamProtocol served, e.g. '1.0' for RTSP/1.0 (RFC 2326) or '2.0' for RTSP/2.0 (RFC 7826). RTSP 2.0 is not backward compatible with 1.0, so a client that must interoperate without probing reads this. + + i=78 + i=68 + ns=1;i=1008 + + + + Codec + Codec carried by the stream. + + i=80 + i=68 + ns=1;i=1008 + + + + Width + Streamed frame width in pixels. + + i=80 + i=68 + ns=1;i=1008 + + + + Height + Streamed frame height in pixels. + + i=80 + i=68 + ns=1;i=1008 + + + + FrameRate + Streamed frames per second. + + i=80 + i=68 + ns=1;i=1008 + + + + Bitrate + Target bitrate in bits per second. + + i=80 + i=68 + ns=1;i=1008 + + + + MaxSessions + Maximum concurrent sessions this endpoint will serve. + + i=80 + i=68 + ns=1;i=1008 + + + + ActiveSessions + Sessions currently leased. + + i=80 + i=68 + ns=1;i=1008 + + + + ClipEndpointType + A still-image access point. A conformant Server SHALL expose at least one instance whose ClipFormat is Jpeg; every other format is optional. In addition to the default URI path, this type MAY publish the encoded image inline as a ByteString so that clients can Read or Subscribe to it - but only within MaxInlineClipSize, which SHALL NOT exceed the Server's ServerCapabilities.MaxByteStringLength. Inline delivery serves single stills; it is not a substitute for a StreamEndpoint. + Vision + + ns=1;i=1007 + ns=1;i=6029 + ns=1;i=6030 + ns=1;i=6031 + ns=1;i=6032 + ns=1;i=6033 + ns=1;i=6034 + ns=1;i=6035 + ns=1;i=6036 + ns=1;i=6037 + + + + ClipFormat + Encoding of clips from this endpoint. Jpeg is the mandatory default. + + i=78 + i=68 + ns=1;i=1009 + + + + Quality + Encoder quality 0 to 100 where the format defines one, for example JPEG. + + i=80 + i=68 + ns=1;i=1009 + + + + Width + Clip width in pixels. + + i=80 + i=68 + ns=1;i=1009 + + + + Height + Clip height in pixels. + + i=80 + i=68 + ns=1;i=1009 + + + + Retention + How long a generated clip remains retrievable at its Uri. + + i=80 + i=68 + ns=1;i=1009 + + + + InlineDeliveryEnabled + True when LatestClip is published. False, the default, means clients use the URI path exclusively. + + i=80 + i=68 + ns=1;i=1009 + + + + MaxInlineClipSize + Largest inline payload this endpoint will publish, in bytes. SHALL NOT exceed Server.ServerCapabilities.MaxByteStringLength, and a Read or Publish response carrying the value is additionally bounded by the Session's MaxResponseMessageSize. See clause 6.4. + + i=80 + i=68 + ns=1;i=1009 + + + + LatestClip + The most recently produced clip, encoded per ClipFormat. Subscribable: the value changes once per acquisition, which suits one-image-per-part inspection. When the encoded image exceeds MaxInlineClipSize the Server SHALL set the StatusCode to Bad_EncodingLimitsExceeded, and the client SHALL fall back to LatestClipMetadata.Uri, which remains valid. + + i=80 + i=63 + ns=1;i=1009 + + + + LatestClipMetadata + Descriptor of LatestClip, including the out-of-band Uri and Digest. Populated whenever a clip exists, whether or not the bytes are published inline. + + i=80 + i=63 + ns=1;i=1009 + + + + VisionMediaManagementType + Container and control surface for a sensor's media endpoints. Holds the endpoint folders and the Methods that select, configure and lease them. + Vision + + i=58 + ns=1;i=6038 + ns=1;i=6040 + ns=1;i=6042 + ns=1;i=6043 + ns=1;i=6044 + ns=1;i=6047 + ns=1;i=6049 + ns=1;i=6051 + ns=1;i=6053 + + + + StreamEndpoints + StreamEndpointType instances offered by this sensor. At least one uses Rtsp. + + i=78 + i=61 + ns=1;i=1010 + ns=1;i=6039 + + + + <StreamEndpoint> + A stream endpoint offered by this sensor. + + i=11510 + ns=1;i=1008 + ns=1;i=6038 + + + + ClipEndpoints + ClipEndpointType instances offered by this sensor. At least one uses Jpeg. + + i=78 + i=61 + ns=1;i=1010 + ns=1;i=6041 + + + + <ClipEndpoint> + A clip endpoint offered by this sensor. + + i=11510 + ns=1;i=1009 + ns=1;i=6040 + + + + PreferredStreamEndpoint + The StreamEndpoint a client should use unless it has a reason not to. + + i=80 + i=68 + ns=1;i=1010 + + + + PreferredClipEndpoint + The ClipEndpoint a client should use unless it has a reason not to. + + i=80 + i=68 + ns=1;i=1010 + + + + GetStreamEndpoint + Lease a stream. Returns a session descriptor whose Uri may embed a time-limited credential; the client opens that Uri with the media protocol. The preferred protocol is advisory - the Server returns what it can serve, which is at minimum RTSP. + + i=78 + ns=1;i=1010 + ns=1;i=6045 + ns=1;i=6046 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6044 + + i=297Endpointi=17-1StreamEndpoint to lease. Null selects PreferredStreamEndpoint, or, when that is also null, the first endpoint in StreamEndpoints in BrowseName order that satisfies the request.i=297ProfileNamei=12-1Requested profile, or empty for the default.i=297PreferredProtocolns=1;i=3002-1Advisory protocol preference. + + + OutputArguments + + i=78 + i=68 + ns=1;i=6044 + + i=297Sessionns=1;i=3057-1The leased session.i=297Endpointi=17-1The StreamEndpoint that was leased. + + + ReleaseStreamEndpoint + Release a previously leased stream session. A Server SHALL also expire leases automatically at ExpiresAt. + + i=78 + ns=1;i=1010 + ns=1;i=6048 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6047 + + i=297SessionTokeni=15-1Token from GetStreamEndpoint. + + + ConfigureStreamEndpoint + Change the encoding parameters of a stream endpoint. A Server MAY reject or clamp values it cannot serve; the resulting effective values are readable on the endpoint. + + i=80 + ns=1;i=1010 + ns=1;i=6050 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6049 + + i=297Endpointi=17-1StreamEndpoint to configure.i=297Codecns=1;i=3004-1Requested codec.i=297Widthi=7-1Requested width in pixels.i=297Heighti=7-1Requested height in pixels.i=297FrameRatei=11-1Requested frames per second.i=297Bitratei=7-1Requested bitrate in bits per second. + + + SelectEndpoint + Designate the preferred stream and clip endpoints, updating PreferredStreamEndpoint and PreferredClipEndpoint. + + i=80 + ns=1;i=1010 + ns=1;i=6052 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6051 + + i=297StreamEndpointi=17-1Preferred stream endpoint, or null to leave unchanged.i=297ClipEndpointi=17-1Preferred clip endpoint, or null to leave unchanged. + + + GetClip + Obtain a still image: either the frame associated with a given ResultId, or the frame nearest a timestamp. The returned descriptor always carries a Uri. The bytes are returned inline only when RequestInline is true AND the encoded image fits MaxInlineClipSize; otherwise InlineImage is empty and the client uses the Uri. + + i=78 + ns=1;i=1010 + ns=1;i=6054 + ns=1;i=6055 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6053 + + i=297Endpointi=17-1ClipEndpoint to use. Null selects PreferredClipEndpoint, or, when that is also null, the first endpoint in ClipEndpoints in BrowseName order that supports Format.i=297ResultIdi=12-1Result whose frame is wanted, or empty.i=297Timestampi=294-1Frame nearest this time, used when ResultId is empty.i=297Formatns=1;i=3003-1Requested encoding; Jpeg is always supported.i=297RequestInlinei=1-1Ask for the bytes inline in addition to the Uri. + + + OutputArguments + + i=78 + i=68 + ns=1;i=6053 + + i=297Imagens=1;i=3053-1Descriptor of the clip.i=297Endpointi=17-1The ClipEndpoint that served the clip.i=297InlineImagei=15-1Encoded bytes, or empty when not requested or too large. + + + CoordinateFrameType + A named coordinate frame. Frames form a tree through ParentFrame, so a client can compose a chain from a camera frame to a world frame. Roles follow ISO 9787, which standardises WHICH frames exist - note that no standard defines how to CALIBRATE between them, which is why ExtrinsicCalibrationType carries the result explicitly. + Vision + + i=58 + ns=1;i=6056 + ns=1;i=6057 + ns=1;i=6058 + ns=1;i=6059 + + + + FrameId + Identifier referenced by VisionPose3DDataType.FrameId. + + i=78 + i=68 + ns=1;i=1011 + + + + Role + Role of this frame. + + i=78 + i=68 + ns=1;i=1011 + + + + ParentFrame + The frame this one is expressed in; null for a root frame. + + i=80 + i=68 + ns=1;i=1011 + + + + Transform + Pose of this frame within ParentFrame. + + i=80 + i=63 + ns=1;i=1011 + + + + VisionCalibrationType + Abstract base for a calibration result, carrying the provenance a client needs in order to decide whether to trust it. + Vision + + i=58 + ns=1;i=6060 + ns=1;i=6061 + ns=1;i=6062 + ns=1;i=6063 + ns=1;i=6064 + + + + CalibrationId + Identifier of this calibration. + + i=78 + i=68 + ns=1;i=1012 + + + + PerformedAt + When the calibration was computed. + + i=78 + i=68 + ns=1;i=1012 + + + + Valid + False when the Server knows the calibration is stale, for example after a mount change. + + i=78 + i=68 + ns=1;i=1012 + + + + ResidualError + Reprojection or fit residual, in the natural unit of the method. + + i=80 + i=68 + ns=1;i=1012 + + + + Method + Free-text method identifier, for example Zhang, Tsai-Lenz or Daniilidis. + + i=80 + i=68 + ns=1;i=1012 + + + + IntrinsicCalibrationType + Camera intrinsics and lens distortion for a specific image size. + Vision + + ns=1;i=1012 + ns=1;i=6065 + + + + Intrinsics + The intrinsic parameters. + + i=78 + i=63 + ns=1;i=1013 + + + + ExtrinsicCalibrationType + The rigid transform between two frames. For a robot cell this is the hand-eye calibration. No ISO, IEC or VDI standard defines the calibration PROCEDURE, so this type carries the resulting transform, the mounting arrangement it applies to, and its residual, which is what a consumer actually needs. + Vision + + ns=1;i=1012 + ns=1;i=6066 + ns=1;i=6067 + ns=1;i=6068 + ns=1;i=6069 + + + + Mount + Mounting arrangement the transform applies to. + + i=78 + i=68 + ns=1;i=1014 + + + + SourceFrame + Frame the transform maps from. + + i=78 + i=68 + ns=1;i=1014 + + + + TargetFrame + Frame the transform maps to. + + i=78 + i=68 + ns=1;i=1014 + + + + Transform + Pose of SourceFrame expressed in TargetFrame. + + i=78 + i=63 + ns=1;i=1014 + + + + VisionSensorType + Abstract base for anything that produces imagery or range data. The RealityKind property is what makes the model sim/real symmetric: a physical camera and a simulated one expose the same members, so a client written against this type works against either. + Vision + + i=58 + ns=1;i=6070 + ns=1;i=6071 + ns=1;i=6072 + ns=1;i=6073 + ns=1;i=6074 + ns=1;i=6075 + ns=1;i=6076 + ns=1;i=6077 + ns=1;i=6078 + ns=1;i=6079 + ns=1;i=6080 + ns=1;i=6081 + + + + SensorId + Identifier of the sensor, unique within the Server. + + i=78 + i=68 + ns=1;i=1002 + + + + RealityKind + Whether this sensor is physical, simulated or hybrid. + + i=78 + i=68 + ns=1;i=1002 + + + + Modality + What the sensor measures. + + i=78 + i=68 + ns=1;i=1002 + + + + Manufacturer + Device manufacturer. + + i=80 + i=68 + ns=1;i=1002 + + + + Model + Device model designation. + + i=80 + i=68 + ns=1;i=1002 + + + + SerialNumber + Device serial number. + + i=80 + i=68 + ns=1;i=1002 + + + + DeviceUri + Transport-level device identifier, for example a GigE Vision or USB3 Vision device id. Lets a client correlate this sensor with the GenICam layer that actually moves the pixels. + + i=80 + i=68 + ns=1;i=1002 + + + + FrameId + FrameId of this sensor's own camera frame. + + i=80 + i=68 + ns=1;i=1002 + + + + Media + Media endpoints and their control surface. + + i=78 + ns=1;i=1010 + ns=1;i=1002 + + + + Optics + Lens in front of this sensor. + + i=80 + ns=1;i=1005 + ns=1;i=1002 + + + + Illumination + Light source associated with this sensor. + + i=80 + ns=1;i=1006 + ns=1;i=1002 + + + + Calibrations + VisionCalibrationType instances for this sensor. + + i=80 + i=61 + ns=1;i=1002 + + + + ImageSensorType + A two-dimensional imaging sensor. The acquisition parameters use GenICam SFNC 2.8 names and semantics so that a Server bridging a GenICam device can map them one to one, and a client that knows SFNC needs no translation table. This is the layer OPC 40100-2 leaves empty: its VisionImageSensorType adds no members at all. + Vision + + ns=1;i=1002 + ns=1;i=6082 + ns=1;i=6083 + ns=1;i=6084 + ns=1;i=6085 + ns=1;i=6086 + ns=1;i=6087 + ns=1;i=6088 + ns=1;i=6089 + ns=1;i=6090 + ns=1;i=6091 + ns=1;i=6092 + ns=1;i=6093 + ns=1;i=6094 + ns=1;i=6095 + ns=1;i=6096 + + + + Width + Image width in pixels (SFNC Width). + + i=78 + i=68 + ns=1;i=1003 + + + + Height + Image height in pixels (SFNC Height). + + i=78 + i=68 + ns=1;i=1003 + + + + PixelFormat + Pixel format using GenICam PFNC naming, for example Mono8, BayerRG12 or RGB8 (SFNC PixelFormat). + + i=78 + i=68 + ns=1;i=1003 + + + + ExposureTime + Exposure time in microseconds (SFNC ExposureTime). + + i=80 + i=68 + ns=1;i=1003 + + + + Gain + Analog gain (SFNC Gain). + + i=80 + i=68 + ns=1;i=1003 + + + + AcquisitionFrameRate + Frames per second (SFNC AcquisitionFrameRate). + + i=80 + i=68 + ns=1;i=1003 + + + + TriggerMode + Trigger mode, On or Off (SFNC TriggerMode). + + i=80 + i=68 + ns=1;i=1003 + + + + TriggerSource + Active trigger source (SFNC TriggerSource). + + i=80 + i=68 + ns=1;i=1003 + + + + OffsetX + Region-of-interest x offset (SFNC OffsetX). + + i=80 + i=68 + ns=1;i=1003 + + + + OffsetY + Region-of-interest y offset (SFNC OffsetY). + + i=80 + i=68 + ns=1;i=1003 + + + + BinningHorizontal + Horizontal binning factor (SFNC BinningHorizontal). + + i=80 + i=68 + ns=1;i=1003 + + + + BinningVertical + Vertical binning factor (SFNC BinningVertical). + + i=80 + i=68 + ns=1;i=1003 + + + + ReverseX + Horizontal flip (SFNC ReverseX). + + i=80 + i=68 + ns=1;i=1003 + + + + ReverseY + Vertical flip (SFNC ReverseY). + + i=80 + i=68 + ns=1;i=1003 + + + + Intrinsics + Currently applicable intrinsics, mirroring the active IntrinsicCalibration. + + i=80 + i=63 + ns=1;i=1003 + + + + Depth3DSensorType + A sensor producing depth or point-cloud data. Point clouds are large and are obtained through a media endpoint, not read as an OPC UA array. + Vision + + ns=1;i=1002 + ns=1;i=6097 + ns=1;i=6098 + ns=1;i=6099 + ns=1;i=6100 + ns=1;i=6101 + ns=1;i=6199 + ns=1;i=6200 + + + + MinDepth + Smallest reportable range in metres. + + i=80 + i=68 + ns=1;i=1004 + + + + MaxDepth + Largest reportable range in metres. + + i=80 + i=68 + ns=1;i=1004 + + + + DepthScale + Metres represented by one unit of the raw depth map. + + i=80 + i=68 + ns=1;i=1004 + + + + Baseline + Stereo baseline in metres, where applicable. + + i=80 + i=68 + ns=1;i=1004 + + + + PointsPerFrame + Nominal point count per frame. + + i=80 + i=68 + ns=1;i=1004 + + + + IVisionSimulatedType + Applied to a sensor whose RealityKind is Simulated or Hybrid. It names the simulator and the scene prim being rendered, which is what makes a synthetic sensor addressable in the same terms as a physical one. When the Server also implements OPC UA - OpenUSD Scene Materialization, PrimPath resolves to a UsdGeomCameraType instance and HasScenePrim points at it directly. + Vision + + i=17602 + ns=1;i=6102 + ns=1;i=6103 + ns=1;i=6104 + ns=1;i=6105 + ns=1;i=6106 + + + + SimulatorUri + Identifier of the simulator or renderer, for example an Isaac Sim instance. + + i=78 + i=68 + ns=1;i=1030 + + + + StageIdentifier + Root layer identifier of the stage being rendered. Uses the same identity contract as OPC UA - OpenUSD Bindings and Scene Materialization. + + i=78 + i=68 + ns=1;i=1030 + + + + PrimPath + Absolute SdfPath of the camera prim this sensor renders from. + + i=78 + i=68 + ns=1;i=1030 + + + + GroundTruthAvailable + True when the simulator can emit annotator ground truth alongside imagery. + + i=80 + i=68 + ns=1;i=1030 + + + + RandomizationSeed + Seed of the active domain-randomization run, so a dataset can be reproduced. + + i=80 + i=68 + ns=1;i=1030 + + + + VisionResultType + Abstract base for a vision result. Unlike OPC 40100-1, whose ResultContent is BaseDataType[] and explicitly not defined, the subtypes of this type define their content. The trust members exist so that a high-risk deployment can log which model version produced a decision and where its explanation lives. + Vision + + i=58 + ns=1;i=6137 + ns=1;i=6138 + ns=1;i=6139 + ns=1;i=6140 + ns=1;i=6141 + ns=1;i=6142 + ns=1;i=6143 + ns=1;i=6144 + + + + ResultId + Identifier of the result, unique within the Server. + + i=78 + i=68 + ns=1;i=1020 + + + + CreationTime + When the result was produced. + + i=78 + i=68 + ns=1;i=1020 + + + + Sensor + Sensor the frame came from. + + i=80 + i=68 + ns=1;i=1020 + + + + Pipeline + Pipeline that produced it. + + i=80 + i=68 + ns=1;i=1020 + + + + ModelVersionUsed + Version of the model that produced the result. Required in practice for auditability when the model can be updated in the field. + + i=80 + i=68 + ns=1;i=1020 + + + + Confidence + Overall confidence in the range 0.0 to 1.0, where the model reports one. + + i=80 + i=68 + ns=1;i=1020 + + + + ExplanationUri + Location of an explanation artefact, for example a saliency map. Treated as untrusted input. + + i=80 + i=68 + ns=1;i=1020 + + + + Frame + Reference to the frame this result was computed from. + + i=80 + i=63 + ns=1;i=1020 + + + + InspectionResultType + A machine-vision inspection outcome: a verdict plus the characteristics that produced it. Carrying nominal, actual, tolerance AND uncertainty is what makes the verdict reproducible by a third party, and is why VisionResultEvaluationEnum has a NotDecidable value. + Vision + + ns=1;i=1020 + ns=1;i=6145 + ns=1;i=6146 + ns=1;i=6147 + ns=1;i=6148 + + + + Evaluation + Overall verdict. + + i=78 + i=68 + ns=1;i=1021 + + + + PartId + Identifier of the inspected part. + + i=80 + i=68 + ns=1;i=1021 + + + + RecipeId + Identifier of the inspection recipe or program applied. + + i=80 + i=68 + ns=1;i=1021 + + + + Characteristics + Measured characteristics. + + i=78 + i=63 + ns=1;i=1021 + + + + DetectionResultType + A robotics-vision perception outcome: zero or more detected instances, each optionally carrying a 6-DoF pose suitable for picking or servoing. + Vision + + ns=1;i=1020 + ns=1;i=6149 + ns=1;i=6150 + + + + Detections + Detected instances. + + i=78 + i=63 + ns=1;i=1022 + + + + FrameId + FrameId that detection poses are expressed in. + + i=80 + i=68 + ns=1;i=1022 + + + + SegmentationResultType + A per-pixel labelling outcome. The mask itself is referenced, not inlined. + Vision + + ns=1;i=1020 + ns=1;i=6151 + ns=1;i=6152 + + + + LabelClasses + Class labels present in the mask. + + i=80 + i=68 + ns=1;i=1023 + + + + Mask + Reference to the encoded mask image. + + i=78 + i=63 + ns=1;i=1023 + + + + VisionFeedbackType + The return path into the vision system. It serves three purposes at once: drawing geometry onto the outgoing stream, recording a downstream verdict against a result, and - most importantly - accepting corrected labels that become training data. That last purpose is what turns a deployed inspection system into a learning one. Every Method here is a WRITE and requires explicit authorization. + Vision + + i=58 + ns=1;i=6153 + ns=1;i=6154 + ns=1;i=6155 + ns=1;i=6156 + ns=1;i=6157 + ns=1;i=6159 + ns=1;i=6161 + ns=1;i=6163 + + + + OverlayEnabled + True when submitted geometry is rendered onto the outgoing stream. + + i=80 + i=68 + ns=1;i=1024 + + + + OverlayStyle + Vendor-defined overlay style identifier. + + i=80 + i=68 + ns=1;i=1024 + + + + OverlayTtl + How long submitted overlay geometry remains rendered. + + i=80 + i=68 + ns=1;i=1024 + + + + MaxInlineFeedbackImageSize + Largest inline image this surface accepts, in bytes. SHALL NOT exceed Server.ServerCapabilities.MaxByteStringLength, and a Call request carrying the value is additionally bounded by the Session's MaxRequestMessageSize. An oversized payload is rejected with Bad_EncodingLimitsExceeded and the client uses SubmitImageReference instead. See clause 6.4. + + i=80 + i=68 + ns=1;i=1024 + + + + SubmitDetections + Push detected geometry back into the vision system. With Purpose set to Overlay the boxes are drawn on the stream; with Purpose set to GroundTruthLabel they are retained as corrected labels for the associated learning job. + + i=80 + ns=1;i=1024 + ns=1;i=6158 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6157 + + i=297Purposens=1;i=3011-1Why the geometry is being sent.i=297Detectionsns=1;i=305510The detections.i=297FrameReferencens=1;i=3053-1Frame the detections belong to.i=297InlineImagei=15-1Optional annotated image, accepted only within MaxInlineFeedbackImageSize; otherwise use SubmitImageReference.i=297SceneIsEmptyi=1-1True asserts that the frame was examined and contains nothing to report, which is a deliberate observation and not a failed one. It is the only way Detections may be empty: an empty array with this false is rejected, so a call that lost its payload is still caught. False with a non-empty Detections is the ordinary case. Last in the list because argument order is part of the wire contract. See clause 9.5. + + + SubmitInspectionResult + Record a downstream inspection verdict against a result, for reconciliation with what the vision system originally reported. + + i=80 + ns=1;i=1024 + ns=1;i=6160 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6159 + + i=297ResultIdi=12-1Result being reconciled.i=297Evaluationns=1;i=3009-1Downstream verdict.i=297Characteristicsns=1;i=305610Downstream measurements, where available. + + + SubmitCorrection + Submit a human-in-the-loop or downstream correction of a previous result. This is the primary source of labelled data for retraining. + + i=80 + ns=1;i=1024 + ns=1;i=6162 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6161 + + i=297ResultIdi=12-1Result being corrected.i=297Purposens=1;i=3011-1Normally GroundTruthLabel.i=297CorrectedDetectionsns=1;i=305510Corrected detections, where the result was a detection.i=297CorrectedCharacteristicsns=1;i=305610Corrected characteristics, where the result was an inspection.i=297Reasoni=21-1Why the correction was made.i=297InlineImagei=15-1Optional corrected or annotated image, accepted only within MaxInlineFeedbackImageSize; otherwise use SubmitImageReference.i=297RetractAlli=1-1True asserts that the corrected result should contain nothing at all - every detection or characteristic it reported was a false positive and nothing replaces it. It is the only way both corrected arrays may be empty. This is the most valuable correction shape for a learning loop, because a false positive is the error an operator is most able to label with confidence. Last in the list because argument order is part of the wire contract. See clause 9.5. + + + SubmitImageReference + The default way to hand an image back: by reference. Used whenever the image exceeds MaxInlineFeedbackImageSize, and preferred in all cases. + + i=80 + ns=1;i=1024 + ns=1;i=6164 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6163 + + i=297Purposens=1;i=3011-1Why the image is being sent.i=297Imagens=1;i=3053-1Descriptor of the image.i=297ResultIdi=12-1Associated result, or empty. + + + InferencePipelineType + Binds a sensor to a deployment and publishes the results. The same type serves on-server and off-server inference: when the deployment is remote the Server publishes results it did not compute, and the only observable difference is DeploymentType.InferenceLocation. + Vision + + i=58 + ns=1;i=6165 + ns=1;i=6166 + ns=1;i=6167 + ns=1;i=6168 + ns=1;i=6169 + ns=1;i=6170 + ns=1;i=6171 + ns=1;i=6172 + ns=1;i=6175 + ns=1;i=6176 + ns=1;i=6198 + + + + PipelineId + Identifier of the pipeline. + + i=78 + i=68 + ns=1;i=1018 + + + + Sensor + Sensor supplying frames. + + i=78 + i=68 + ns=1;i=1018 + + + + Deployment + The deployment executing inference. This is a NodeId, not a reference, and the node it names is NOT defined by this specification - see clause 8.2. Where the Server also implements OPC UA - AI Model Management and Inference it names a DeploymentType instance there, which is what clause 8's provenance argument assumes; a Server that describes its deployment some other way names that node instead. Nothing in this NodeSet references the other model's identifiers, so adopting or ignoring it changes nothing about loading this one. + + i=78 + i=68 + ns=1;i=1018 + + + + State + Runtime state of the pipeline. + + i=78 + i=68 + ns=1;i=1018 + + + + Continuous + True while the pipeline runs on every frame. + + i=80 + i=68 + ns=1;i=1018 + + + + Results + Recent VisionResultType instances produced by this pipeline. + + i=80 + i=61 + ns=1;i=1018 + + + + Feedback + Feedback surface for pushing results back into the vision system. + + i=80 + ns=1;i=1024 + ns=1;i=1018 + + + + RunInference + Run inference once, on the current or a specified frame, and return the identifier of the result that was produced. + + i=80 + ns=1;i=1018 + ns=1;i=6173 + ns=1;i=6174 + + + + InputArguments + + i=78 + i=68 + ns=1;i=6172 + + i=297Timestampi=294-1Frame nearest this time, or null for the newest. + + + OutputArguments + + i=78 + i=68 + ns=1;i=6172 + + i=297ResultIdi=12-1Identifier of the produced result. + + + StartContinuous + Begin running inference on every acquired frame. + + i=80 + ns=1;i=1018 + + + + Stop + Stop continuous inference. + + i=80 + ns=1;i=1018 + + + + VisionRootType + The single well-known entry point for everything in this model. A client starts here, enumerates Sensors, and follows references outward. Mirrors the discovery pattern of OPC UA - OpenUSD Bindings. + Vision + + i=58 + ns=1;i=6191 + ns=1;i=6192 + ns=1;i=6194 + + + + Sensors + VisionSensorType instances known to this Server. + + i=78 + i=61 + ns=1;i=1001 + + + + Pipelines + InferencePipelineType instances. + + i=80 + i=61 + ns=1;i=1001 + + + + Frames + CoordinateFrameType instances. + + i=80 + i=61 + ns=1;i=1001 + + + + Vision + The well-known Vision entry point, a component of the Server object. A conformant Server exposes exactly one. + + ns=1;i=1001 + i=2253 + + + + DataChannelSource + NodeId of the Object through which this endpoint's bytes can also be obtained on an OPC UA data channel, per the OPC UA - Data Channels errata proposal. Non-null means the data channel path is offered IN ADDITION to the endpoint's out-of-band path; null or absent means out-of-band only. The target is created by the Server - typically a DataChannelSourceType instance, or any Object implementing IDataChannelSourceType - and is NOT defined by this specification. That proposal is a DRAFT: a conformant Server may leave this null forever. See clause 6.7. + + i=80 + i=68 + ns=1;i=1007 + + + + DataChannelContentType + IANA media type the data channel carries, for example video/H264 or image/jpeg. Mirrors IDataChannelSourceType.ContentType so a client can learn the payload type from this model alone, without the Data Channels model being present. Meaningful only where DataChannelSource is non-null. + + i=80 + i=68 + ns=1;i=1007 + + + + LearningJob + LearningJobType instance that consumes GroundTruthLabel corrections submitted through this pipeline's Feedback object, or null where the Server retains none. A NodeId and not a reference, for the same reason Deployment is: this model takes no dependency on the model that defines the job. Section 9.5.1 requires this to be non-null wherever such a correction is retained - without it a client cannot establish whether its label reached a learning loop at all. + + i=80 + i=68 + ns=1;i=1018 + + + + DepthWidth + Width in pixels of the sensor's native depth image. Present on a device whose depth output is an ordered image - structured-light, time-of-flight and stereo sensors - and absent on one whose output is an unordered point cloud, where there is no image to have a shape. PointsPerFrame is a nominal count and is not a substitute: it cannot be used to reproject a depth pixel, nor to size a decoder. See clause 5.6. + + i=80 + i=68 + ns=1;i=1004 + + + + DepthHeight + Height in pixels of the sensor's native depth image, under the same condition as DepthWidth. The two are present or absent together. + + i=80 + i=68 + ns=1;i=1004 + + + diff --git a/src/Opc.Ua.Vision/NugetREADME.md b/src/Opc.Ua.Vision/NugetREADME.md new file mode 100644 index 0000000000..82d467ae27 --- /dev/null +++ b/src/Opc.Ua.Vision/NugetREADME.md @@ -0,0 +1,56 @@ +# Opc.Ua.Vision + +Server/client-independent foundation for the **draft** *OPC UA — Vision* +companion specification. + +The Vision NodeSet is **source-generated** here directly over the base OPC UA +namespace (no DI, Machinery or Robotics dependency), exposing generated +ObjectTypes, ReferenceTypes, enums, typed node states and client proxies, plus +the `AddOpcUaVision` model loader. The generated `ObjectTypeIds`, +`ReferenceTypeIds` and `DataTypeIds` classes are the source of truth for the +model. + +The model covers the vision domain end-to-end: a `VisionRootType` topology +grouping vision sensors (`VisionSensorType`, `ImageSensorType`, +`Depth3DSensorType`) together with their `OpticsType` and `IlluminationType` +components; `CoordinateFrameType`, `VisionCalibrationType`, +`IntrinsicCalibrationType` and `ExtrinsicCalibrationType` for the spatial +grounding of every sensor; media surfaces (`MediaEndpointType`, +`StreamEndpointType`, `ClipEndpointType`, `VisionMediaManagementType`) for how +image and video data leaves the server; inference (`InferencePipelineType`); +results (`VisionResultType`, `InspectionResultType`, `DetectionResultType`, +`SegmentationResultType`) and their `VisionFeedbackType` correction cycle; and +the `IVisionSimulatedType` marker interface for simulated sensors. The +`HasCalibration`, `MountedOn`, `HasScenePrim` and `ProducedBy` reference types +carry the semantic relationships between these components. + +## Numerical conventions + +Every DataType in this package respects §5.12 of the specification: + +- Positions are metres. +- Orientations are unit quaternions ordered `(x, y, z, w)`. +- `VisionIntrinsicsDataType.Cx` / `Cy` are corner-datum principal-point + coordinates. +- An empty covariance array is the sentinel for "not reported" — not a + zero matrix. + +## Related packages + +| Package | Adds | +|---|---| +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Server` | Hosting a Vision server: `AddVision`, `ConfigureVision`, provider abstractions, fluent topology builders, facet derivation | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Client` | `VisionClient` discovery, sensors, media, inference, results, `VisionFrameGraph` pose composition, `VisionFeedbackClient` | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.OpenUsd` | Rendering a simulated sensor's camera view offscreen from an OpenUSD stage | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Vision` | MCP tools for perception agents, including `vision_get_frame` returning an MCP `ImageContentBlock` | + +See the [Vision developer guide](https://github.com/OPCFoundation/UA-.NETStandard/blob/main/docs/Vision.md) +for the full end-to-end story, code examples and the bin-picking sample. + +> The namespace `http://opcfoundation.org/UA/Vision/` and every NodeId in it are +> **provisional**. The model is a working-group draft and is neither official +> nor endorsed by the OPC Foundation. + +## License + +OPC Foundation MIT License 1.00 — diff --git a/src/Opc.Ua.Vision/Opc.Ua.Vision.csproj b/src/Opc.Ua.Vision/Opc.Ua.Vision.csproj new file mode 100644 index 0000000000..33210862c7 --- /dev/null +++ b/src/Opc.Ua.Vision/Opc.Ua.Vision.csproj @@ -0,0 +1,54 @@ + + + $(AssemblyPrefix).Vision + $(LibTargetFrameworks) + $(PackagePrefix).Opc.Ua.Vision + Opc.Ua.Vision + $(NoWarn);CS1591;CS0108 + enable + Server/client-independent OPC UA Vision (draft) companion contracts and source-generated Vision model, exposing model identifiers, typed states and proxies, and a model loader for vision sensors, calibrations, media endpoints, inference pipelines, and vision results over the base OPC UA namespace. + true + NugetREADME.md + true + true + + + $(PackageId).Debug + + + + + + + + + + + + Analyzer + false + + + + + + + + + http://opcfoundation.org/UA/Vision/ + Opc.Ua.Vision + + + + + v105 + true + true + + + + + diff --git a/src/Opc.Ua.Vision/Properties/AssemblyInfo.cs b/src/Opc.Ua.Vision/Properties/AssemblyInfo.cs new file mode 100644 index 0000000000..7798c9bd57 --- /dev/null +++ b/src/Opc.Ua.Vision/Properties/AssemblyInfo.cs @@ -0,0 +1,32 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +[assembly: CLSCompliant(false)] diff --git a/src/Opc.Ua.XRegistry/Opc.Ua.XRegistry.NodeSet2.xml b/src/Opc.Ua.XRegistry/Opc.Ua.XRegistry.NodeSet2.xml index cb081108d6..c401e8b41f 100644 --- a/src/Opc.Ua.XRegistry/Opc.Ua.XRegistry.NodeSet2.xml +++ b/src/Opc.Ua.XRegistry/Opc.Ua.XRegistry.NodeSet2.xml @@ -1,4 +1,4 @@ - + diff --git a/tests/Opc.Ua.AI.Tests/AddressSpaceShapeTests.cs b/tests/Opc.Ua.AI.Tests/AddressSpaceShapeTests.cs new file mode 100644 index 0000000000..d20207f9ea --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/AddressSpaceShapeTests.cs @@ -0,0 +1,214 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; +using ObjectIds = Opc.Ua.ObjectIds; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Verifies the shape of the address space itself. + /// + /// + /// These are the faults that produce a Server which builds, starts, answers + /// calls, and is wrong. None of them fails anything: a duplicated entry point + /// looks like a populated Server to whoever browses the right one, and a + /// NodeId collision looks like nothing at all until a type node has been + /// overwritten by an instance node. + /// + [TestFixture] + [Category("AIModelManagement")] + [SetCulture("en-us")] + [SetUICulture("en-us")] + public sealed class AddressSpaceShapeTests + { + [Test] + public async Task ExactlyOneEntryPointHangsOffTheServerObjectAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + int roots = nm.CountIndexed(); + + // The model already declares the entry point and parents it to the + // Server Object. Building a second one leaves two Objects with the same + // BrowseName under the Server, one populated and one empty, and which + // one a client finds depends on browse order - so half the time the + // Server appears to publish nothing at all. + Assert.That(roots, Is.EqualTo(1)); + + Assert.That( + nm.RootId, + Is.EqualTo(new NodeId( + Opc.Ua.AI.Objects.AiModelManagement, + nm.NamespaceIndex)), + "the entry point must be the one the model declares"); + } + + [Test] + public async Task DynamicNodeIdsCannotCollideWithTheModelsOwnAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + + // Enough churn to walk a numeric counter into the model's id range, + // which starts at 1001 and runs to 7001. + for (int index = 0; index < 40; index++) + { + await deployment.BeginTransfer!.OnCallAsync!( + nm.SystemContext, + deployment.BeginTransfer, + nm.PrimaryDeploymentId, + "application/json", + 16, + CancellationToken.None).ConfigureAwait(false); + } + + // Every type the model declares must still be the type it declared. A + // numeric counter starting at 1 overwrites these silently: the + // predefined-node index takes the last writer, so AiRootType quietly + // became an inference job's FinishedAt property. + foreach (uint identifier in new uint[] + { + Opc.Ua.AI.ObjectTypes.AiRootType, + Opc.Ua.AI.ObjectTypes.ModelType, + Opc.Ua.AI.ObjectTypes.DeploymentType, + Opc.Ua.AI.ObjectTypes.InferenceTransferType + }) + { + var id = new NodeId(identifier, nm.NamespaceIndex); + NodeState? node = nm.IndexedNode(id); + + Assert.That(node, Is.Not.Null, $"{id} is missing"); + Assert.That( + node, + Is.InstanceOf(), + $"{id} was overwritten by an instance node"); + } + } + + [Test] + public async Task TheJobsFolderIsResolvableOnceSomethingIsInItAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + + BeginTransferMethodStateResult begun = await deployment.BeginTransfer!.OnCallAsync!( + nm.SystemContext, + deployment.BeginTransfer, + nm.PrimaryDeploymentId, + "application/json", + 16, + CancellationToken.None).ConfigureAwait(false); + + var root = nm.FindPredefinedNode(nm.RootId); + + Assert.That(root.Jobs, Is.Not.Null); + + // Created lazily on the first transfer, the folder existed on the + // NodeState tree - so it appeared in a Browse of the root - while being + // absent from the index, so browsing IT returned BadNodeIdUnknown. The + // collection the specification defines was unreachable. + Assert.That( + nm.IndexedNode(root.Jobs!.NodeId), Is.Not.Null, + "the Jobs folder must be indexed, not only present on the tree"); + + Assert.That(nm.IndexedNode(begun.Transfer), Is.Not.Null); + } + + [Test] + public async Task JobsAreReclaimedRatherThanAccumulatingAsync() + { + using AINodeManager nm = await AIServerTestHarness + .CreateAsync( + new InferenceBackends(new FakeInferenceBackend("primary")), + new AIOptions + { + EnableFallback = false, + AsyncInferenceDelay = TimeSpan.Zero, + MaxRetainedJobs = 3 + }) + .ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + var started = new List(); + + for (int index = 0; index < 6; index++) + { + InvokeAsyncMethodStateResult result = + await deployment.InvokeAsync!.OnCallAsync!( + nm.SystemContext, + deployment.InvokeAsync, + nm.PrimaryDeploymentId, + ByteString.From(Encoding.UTF8.GetBytes("{}")), + string.Empty, + "application/json", + ArrayOf.Empty, + CancellationToken.None).ConfigureAwait(false); + + started.Add(result.Job); + } + + int live = 0; + + foreach (NodeId job in started) + { + if (nm.IndexedNode(job) is not null) + { + live++; + } + } + + // Each job retains its request and response payloads, so an uncapped set + // grows in bytes as well as nodes - and any session that can call the + // Method can grow it. Transfers already had a cap; jobs did not. + Assert.That(live, Is.LessThanOrEqualTo(3)); + Assert.That( + nm.IndexedNode(started[^1]), Is.Not.Null, + "the most recent job must survive"); + } + + private static Task CreateAsync() + { + return AIServerTestHarness.CreateAsync( + new InferenceBackends(new FakeInferenceBackend("primary")), + new AIOptions { EnableFallback = false }); + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/AiClientTests.cs b/tests/Opc.Ua.AI.Tests/AiClientTests.cs new file mode 100644 index 0000000000..34a81cfff1 --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/AiClientTests.cs @@ -0,0 +1,144 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Reflection; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.AI.Client; + +namespace Opc.Ua.AI.Tests +{ + [TestFixture] + [Category("AI")] + [Category("Client")] + public sealed class AIClientTests + { + [Test] + public void RootReportsAINamespaceAndWellKnownFolders() + { + var harness = new AISessionHarness(); + + Assert.Multiple(() => + { + Assert.That(harness.Client.IsAINamespaceAvailable, Is.True); + Assert.That(harness.Client.AIRootId, Is.EqualTo(harness.AIRootId)); + Assert.That(harness.Client.ModelsFolderId, Is.EqualTo(harness.ModelsFolderId)); + Assert.That(harness.Client.DeploymentsFolderId, Is.EqualTo(harness.DeploymentsFolderId)); + }); + } + + [Test] + public async Task DiscoverModelsReturnsTypedModelInstances() + { + var harness = new AISessionHarness(); + harness.AddModel("ModelA"); + + ArrayOf nodes = await harness.Client.DiscoverModelsAsync().ConfigureAwait(false); + + Assert.That(nodes.Count, Is.EqualTo(1)); + Assert.That(nodes[0], Is.EqualTo(harness.ModelNodeId)); + } + + [Test] + public async Task EnumerateDeploymentsYieldsBrowseMetadata() + { + var harness = new AISessionHarness(); + harness.AddDeployment("Primary"); + + var entries = new List(); + await foreach (AINodeEntry entry in harness.Client.EnumerateDeploymentsAsync()) + { + entries.Add(entry); + } + + Assert.That(entries, Has.Count.EqualTo(1)); + Assert.That(entries[0].BrowseName.Name, Is.EqualTo("Primary")); + Assert.That(entries[0].NodeId, Is.EqualTo(harness.DeploymentNodeId)); + } + + [Test] + public async Task ModelReadReturnsNamedSnapshotValues() + { + var harness = new AISessionHarness(); + harness.AddValueChild(harness.ModelNodeId, BrowseNames.ModelId, new NodeId(2100u, 3), "model-1"); + harness.AddValueChild(harness.ModelNodeId, BrowseNames.Name, new NodeId(2101u, 3), "demo"); + harness.AddValueChild(harness.ModelNodeId, BrowseNames.Version, new NodeId(2102u, 3), "1.0"); + harness.AddValueChild(harness.ModelNodeId, BrowseNames.Digest, new NodeId(2103u, 3), ByteString.From([1, 2, 3])); + + AIModelSnapshot snapshot = await harness.Client.Model(harness.ModelNodeId).ReadAsync() + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(snapshot.ModelId, Is.EqualTo("model-1")); + Assert.That(snapshot.Name, Is.EqualTo("demo")); + Assert.That(snapshot.Version, Is.EqualTo("1.0")); + Assert.That(snapshot.Digest.Length, Is.EqualTo(3)); + }); + } + + [Test] + public void AIBrowseClientIsNotPublicApi() + { + Type[] exported = typeof(AIClient).Assembly.GetExportedTypes(); + + Assert.That(exported.Any(t => t.Name == "AIBrowseClient"), Is.False); + } + + [Test] + public void PublicApiDoesNotExposeObjectOrByteArrayOnClientTypes() + { + Type[] exported = typeof(AIClient).Assembly.GetExportedTypes() + .Where(t => t.Namespace == typeof(AIClient).Namespace) + .ToArray(); + + foreach (Type type in exported) + { + foreach (MethodInfo method in type.GetMethods(BindingFlags.Public | BindingFlags.Instance)) + { + if (method.DeclaringType == typeof(object) || + method.Name is nameof(object.Equals) or nameof(object.GetHashCode)) + { + continue; + } + Assert.That(method.ReturnType, Is.Not.EqualTo(typeof(byte[])), type.FullName + "." + method.Name); + Assert.That(method.ReturnType, Is.Not.EqualTo(typeof(object)), type.FullName + "." + method.Name); + foreach (ParameterInfo parameter in method.GetParameters()) + { + Assert.That(parameter.ParameterType, Is.Not.EqualTo(typeof(byte[])), method.Name); + Assert.That(parameter.ParameterType, Is.Not.EqualTo(typeof(object)), method.Name); + } + } + } + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/AiHostingTests.cs b/tests/Opc.Ua.AI.Tests/AiHostingTests.cs new file mode 100644 index 0000000000..8fc3eef39e --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/AiHostingTests.cs @@ -0,0 +1,190 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Options; +using Moq; +using NUnit.Framework; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; +using Opc.Ua.AI.Server.Hosting; +using Opc.Ua.Server.Fluent; +using Opc.Ua.Server.Hosting; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Tests for the AI hosting extensions on . + /// + [TestFixture] + [Category("AI")] + [Category("Hosting")] + public sealed class AIHostingTests + { + [Test] + public void AddAIThrowsOnNullBuilder() + { + Assert.Throws(() => + OpcUaServerAIBuilderExtensions.AddAI(null!)); + } + + [Test] + public void AddAIRegistersNodeManagerFactoryAndOptions() + { + IServiceCollection services = new ServiceCollection(); + services.AddSingleton(new StubChatClientFactory()); + + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddAI( + ai => ai.PrimaryDeploymentId = "configured", + backend => backend.Site = InferenceSite.EdgeOffServer, + fallback => fallback.Enabled = false); + + using ServiceProvider provider = services.BuildServiceProvider(); + AINodeManagerFactory factory = provider.GetRequiredService(); + AIOptions aiOptions = provider.GetRequiredService>().Value; + InferenceBackendOptions backendOptions = + provider.GetRequiredService>().Value; + var registrations = provider.GetServices(); + + Assert.Multiple(() => + { + Assert.That(factory, Is.Not.Null); + Assert.That(aiOptions.PrimaryDeploymentId, Is.EqualTo("configured")); + Assert.That(backendOptions.Site, Is.EqualTo(InferenceSite.EdgeOffServer)); + Assert.That(registrations.Any(r => r.AsyncFactory is AINodeManagerFactory), Is.True); + }); + } + + [Test] + public void AddAIDefaultsToChatClientBackend() + { + IServiceCollection services = new ServiceCollection(); + var factory = new StubChatClientFactory(); + services.AddSingleton(factory); + + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddAI(configureFallbackBackend: options => options.Enabled = false); + + using ServiceProvider provider = services.BuildServiceProvider(); + InferenceBackends backends = provider.GetRequiredService(); + + Assert.Multiple(() => + { + Assert.That(backends.Primary, Is.TypeOf()); + Assert.That(backends.Fallback, Is.Null); + Assert.That(factory.CreatedNames, Does.Contain(string.Empty)); + }); + } + + [Test] + public void AddAIRegistersRestBackendWhenConfigured() + { + IServiceCollection services = new ServiceCollection(); + + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddAI( + configureBackend: options => + { + options.Kind = InferenceBackendKind.RestChatCompletions; + options.Authentication = BackendAuthentication.Anonymous; + }, + configureFallbackBackend: options => options.Enabled = false); + + using ServiceProvider provider = services.BuildServiceProvider(); + InferenceBackends backends = provider.GetRequiredService(); + + Assert.That(backends.Primary, Is.TypeOf()); + } + + [Test] + public void AddAICreatesFallbackFromNamedOptions() + { + IServiceCollection services = new ServiceCollection(); + var factory = new StubChatClientFactory(); + services.AddSingleton(factory); + + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddAI(configureFallbackBackend: options => + { + options.Enabled = true; + options.Site = InferenceSite.OnServer; + }); + + using ServiceProvider provider = services.BuildServiceProvider(); + InferenceBackends backends = provider.GetRequiredService(); + InferenceBackendOptions fallbackOptions = provider + .GetRequiredService>() + .Get(AINodeManagerFactory.FallbackOptionsName); + + Assert.Multiple(() => + { + Assert.That(backends.Fallback, Is.TypeOf()); + Assert.That(fallbackOptions.Site, Is.EqualTo(InferenceSite.OnServer)); + Assert.That(factory.CreatedNames, Does.Contain(AINodeManagerFactory.FallbackOptionsName)); + }); + } + + [Test] + public void AddAIRequiresChatClientFactoryForDefaultBackend() + { + IServiceCollection services = new ServiceCollection(); + + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddAI(configureFallbackBackend: options => options.Enabled = false); + + using ServiceProvider provider = services.BuildServiceProvider(); + + Assert.Throws(() => + provider.GetRequiredService()); + } + + private sealed class StubChatClientFactory : IChatClientFactory + { + public List CreatedNames { get; } = []; + + public IChatClient CreateChatClient( + string backendName, + InferenceBackendOptions options) + { + CreatedNames.Add(backendName); + return Mock.Of(); + } + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/AiLearningJobTests.cs b/tests/Opc.Ua.AI.Tests/AiLearningJobTests.cs new file mode 100644 index 0000000000..5c7ef35a85 --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/AiLearningJobTests.cs @@ -0,0 +1,166 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Verifies the learning job that accounts for submitted ground-truth samples. + /// + [TestFixture] + [Category("AIModelManagement")] + [SetCulture("en-us")] + [SetUICulture("en-us")] + public sealed class AILearningJobTests + { + [Test] + public async Task LearningJobIsIndexedByItsOwnNodeIdAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + NodeState? node = nm.IndexedNode(nm.LearningJobId); + + Assert.Multiple(() => + { + Assert.That(nm.LearningJobId, Is.Not.EqualTo(NodeId.Null)); + Assert.That(node, Is.InstanceOf()); + }); + } + + [Test] + public async Task DisabledLearningLoopOmitsTheLearningJobAsync() + { + using AINodeManager nm = await AIServerTestHarness + .CreateAsync( + new InferenceBackends(new FakeInferenceBackend("primary")), + new AIOptions + { + EnableFallback = false, + EnableLearningLoop = false + }) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(nm.LearningJobId, Is.EqualTo(NodeId.Null)); + Assert.That(nm.CountIndexed(), Is.Zero); + }); + } + + [Test] + public async Task SamplesCollectedStartsAtZeroAndIncrementsAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + LearningJobState job = nm.FindPredefinedNode(nm.LearningJobId); + + Assert.That(job.SamplesCollected!.Value, Is.Zero); + + bool added = await nm.RecordLearningSampleAsync("sample-1").ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(added, Is.True); + Assert.That(job.SamplesCollected.Value, Is.EqualTo(1)); + }); + } + + [Test] + public async Task DuplicateSampleIdIncrementsOnlyOnceAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + bool first = await nm.RecordLearningSampleAsync("sample-1").ConfigureAwait(false); + bool second = await nm.RecordLearningSampleAsync("sample-1").ConfigureAwait(false); + LearningJobState job = nm.FindPredefinedNode(nm.LearningJobId); + + Assert.Multiple(() => + { + Assert.That(first, Is.True); + Assert.That(second, Is.False); + Assert.That(job.SamplesCollected!.Value, Is.EqualTo(1)); + }); + } + + [Test] + public async Task NegativeExampleCountsExactlyLikePositiveExampleAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + bool positive = await nm + .RecordLearningSampleAsync("positive-1", AILearningSampleKind.Positive) + .ConfigureAwait(false); + bool negative = await nm + .RecordLearningSampleAsync("negative-1", AILearningSampleKind.Negative) + .ConfigureAwait(false); + LearningJobState job = nm.FindPredefinedNode(nm.LearningJobId); + + Assert.Multiple(() => + { + Assert.That(positive, Is.True); + Assert.That(negative, Is.True); + Assert.That(job.SamplesCollected!.Value, Is.EqualTo(2)); + }); + } + + [Test] + public async Task ConcurrentSampleIncrementsDoNotLoseCountsAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + Task[] tasks = Enumerable + .Range(0, 250) + .Select(index => nm.RecordLearningSampleAsync($"sample-{index}").AsTask()) + .ToArray(); + + bool[] added = await Task.WhenAll(tasks).ConfigureAwait(false); + LearningJobState job = nm.FindPredefinedNode(nm.LearningJobId); + + Assert.Multiple(() => + { + Assert.That(added, Is.All.True); + Assert.That(job.SamplesCollected!.Value, Is.EqualTo(250)); + }); + } + + private static Task CreateAsync() + { + return AIServerTestHarness.CreateAsync( + new InferenceBackends(new FakeInferenceBackend("primary")), + new AIOptions { EnableFallback = false }); + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/AiServerTestHarness.cs b/tests/Opc.Ua.AI.Tests/AiServerTestHarness.cs new file mode 100644 index 0000000000..5faafc9d56 --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/AiServerTestHarness.cs @@ -0,0 +1,144 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using System.Threading.Tasks; +using Microsoft.Extensions.Options; +using Moq; +using Opc.Ua; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; +using Opc.Ua.Server; +using Opc.Ua.Tests; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Builds the node manager the AI tests are exercised against. + /// + /// + /// A mocked rather than a running Server, because + /// every claim these tests make is about the address space and the routing, and + /// neither needs a socket. The tests that do need one say so. + /// + internal static class AIServerTestHarness + { + /// + /// Creates a node manager with its address space already built. + /// + public static async Task CreateAsync( + InferenceBackends backends, + AIOptions? options = null, + InferenceBackendOptions? backendOptions = null) + { + Mock server = CreateServer(); + +#pragma warning disable CA2000 // the caller disposes the node manager + var manager = new AINodeManager( + server.Object, + null!, + backends, + Options.Create(options ?? new AIOptions()), + Options.Create(backendOptions ?? new InferenceBackendOptions())); +#pragma warning restore CA2000 + + await manager + .CreateAddressSpaceAsync(new Dictionary>()) + .ConfigureAwait(false); + + return manager; + } + + /// + /// Creates the mocked server the node manager runs inside. + /// + public static Mock CreateServer() + { + ITelemetryContext telemetry = NUnitTelemetryContext.Create(); + var namespaceUris = new NamespaceTable(); + namespaceUris.GetIndexOrAppend(Opc.Ua.AI.Namespaces.AI); + namespaceUris.GetIndexOrAppend(Opc.Ua.AI.Namespaces.xRegistry); + + var server = new Mock(); + var masterNodeManager = new Mock(); + server.Setup(s => s.NamespaceUris).Returns(namespaceUris); + server.Setup(s => s.ServerUris).Returns(new StringTable()); + server.Setup(s => s.TypeTree).Returns(CreateTypeTable(namespaceUris)); + server.Setup(s => s.Factory).Returns(EncodeableFactory.Create()); + server.Setup(s => s.Telemetry).Returns(telemetry); + server.Setup(s => s.NodeManager).Returns(masterNodeManager.Object); + server.Setup(s => s.DefaultSystemContext) + .Returns(new ServerSystemContext(server.Object)); + return server; + } + + /// + /// Seeds the base types the AI model derives from. + /// + /// + /// A real Server loads these with the core NodeSet. The type table rejects a + /// subtype whose supertype it does not already know, so anything the model + /// reaches for has to be here - including the Part 10 program state machine, + /// which AiJobType subtypes so that a long inference has a lifecycle + /// clients already understand. + /// + public static TypeTable CreateTypeTable(NamespaceTable namespaceUris) + { + var typeTable = new TypeTable(namespaceUris); + + typeTable.AddSubtype(Opc.Ua.ObjectTypeIds.BaseObjectType, NodeId.Null); + typeTable.AddSubtype(Opc.Ua.ObjectTypeIds.FolderType, Opc.Ua.ObjectTypeIds.BaseObjectType); + typeTable.AddSubtype(Opc.Ua.ObjectTypeIds.FileType, Opc.Ua.ObjectTypeIds.BaseObjectType); + typeTable.AddSubtype( + Opc.Ua.ObjectTypeIds.StateMachineType, Opc.Ua.ObjectTypeIds.BaseObjectType); + typeTable.AddSubtype( + Opc.Ua.ObjectTypeIds.FiniteStateMachineType, Opc.Ua.ObjectTypeIds.StateMachineType); + typeTable.AddSubtype( + Opc.Ua.ObjectTypeIds.ProgramStateMachineType, Opc.Ua.ObjectTypeIds.FiniteStateMachineType); + + typeTable.AddSubtype(Opc.Ua.VariableTypeIds.BaseVariableType, NodeId.Null); + typeTable.AddSubtype( + Opc.Ua.VariableTypeIds.BaseDataVariableType, Opc.Ua.VariableTypeIds.BaseVariableType); + typeTable.AddSubtype( + Opc.Ua.VariableTypeIds.PropertyType, Opc.Ua.VariableTypeIds.BaseVariableType); + + typeTable.AddSubtype(Opc.Ua.DataTypeIds.BaseDataType, NodeId.Null); + typeTable.AddSubtype(Opc.Ua.DataTypeIds.Structure, Opc.Ua.DataTypeIds.BaseDataType); + typeTable.AddSubtype(Opc.Ua.DataTypeIds.Enumeration, Opc.Ua.DataTypeIds.BaseDataType); + + typeTable.AddSubtype(Opc.Ua.ReferenceTypeIds.References, NodeId.Null); + typeTable.AddSubtype( + Opc.Ua.ReferenceTypeIds.NonHierarchicalReferences, Opc.Ua.ReferenceTypeIds.References); + typeTable.AddSubtype( + Opc.Ua.ReferenceTypeIds.HierarchicalReferences, Opc.Ua.ReferenceTypeIds.References); + + return typeTable; + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/AiSessionHarness.cs b/tests/Opc.Ua.AI.Tests/AiSessionHarness.cs new file mode 100644 index 0000000000..0bdee3928d --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/AiSessionHarness.cs @@ -0,0 +1,258 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Moq; +using Opc.Ua.AI.Client; +using Opc.Ua.Client; + +namespace Opc.Ua.AI.Tests +{ + internal sealed class AISessionHarness + { + private readonly Dictionary<(NodeId Parent, string BrowseName), NodeId> m_children = []; + private readonly Dictionary> m_browse = []; + private readonly Dictionary m_values = []; + + public AISessionHarness() + { + Telemetry = new Mock().Object; + NamespaceUris.GetIndexOrAppend(Opc.Ua.Namespaces.OpcUa); + NamespaceUris.GetIndexOrAppend(Namespaces.AI); + MessageContext = ServiceMessageContext.Create(Telemetry); + MessageContext.NamespaceUris.GetIndexOrAppend(Opc.Ua.Namespaces.OpcUa); + MessageContext.NamespaceUris.GetIndexOrAppend(Namespaces.AI); + Session.SetupGet(s => s.NamespaceUris).Returns(NamespaceUris); + Session.SetupGet(s => s.MessageContext).Returns(MessageContext); + Session.SetupGet(s => s.Factory).Returns(MessageContext.Factory); + Session.SetupGet(s => s.OperationLimits).Returns(new OperationLimits()); + Session.SetupGet(s => s.ServerCapabilities).Returns(new ServerCapabilities()); + Session.SetupGet(s => s.ContinuationPointPolicy).Returns(ContinuationPointPolicy.Default); + Session.SetupGet(s => s.NodeCache).Returns(NodeCache.Object); + NodeCache + .Setup(c => c.IsTypeOfAsync( + It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(new ValueTask(true)); + SetupTranslate(); + SetupBrowse(); + SetupRead(); + AddChild(AIRootId, BrowseNames.Models, ModelsFolderId); + AddChild(AIRootId, BrowseNames.Deployments, DeploymentsFolderId); + Client = new AIClient(Session.Object, Telemetry); + } + + public Mock Session { get; } = new(MockBehavior.Loose); + + public Mock NodeCache { get; } = new(MockBehavior.Loose); + + public ITelemetryContext Telemetry { get; } + + public NamespaceTable NamespaceUris { get; } = new(); + + public ServiceMessageContext MessageContext { get; } + + public AIClient Client { get; } + + public ushort AINamespaceIndex => (ushort)NamespaceUris.GetIndex(Namespaces.AI); + + public NodeId AIRootId => NodeId.Create(Objects.AiModelManagement, Namespaces.AI, NamespaceUris); + + public NodeId ModelsFolderId => NodeId.Create(Objects.AiRootType_Models, Namespaces.AI, NamespaceUris); + + public NodeId DeploymentsFolderId => NodeId.Create(Objects.AiRootType_Deployments, Namespaces.AI, NamespaceUris); + + public NodeId ModelNodeId { get; } = new(2000u, 3); + + public NodeId DeploymentNodeId { get; } = new(3000u, 3); + + public void AddModel(string browseName = "Model1") + { + AddBrowse(ModelsFolderId, [Ref(ModelNodeId, browseName, ObjectTypes.ModelType)]); + } + + public void AddDeployment(string browseName = "Deployment1") + { + AddBrowse(DeploymentsFolderId, [Ref(DeploymentNodeId, browseName, ObjectTypes.DeploymentType)]); + } + + public ReferenceDescription Ref(NodeId nodeId, string browseName, uint typeId) + { + return new ReferenceDescription + { + NodeId = new ExpandedNodeId(nodeId), + BrowseName = new QualifiedName(browseName, AINamespaceIndex), + DisplayName = new LocalizedText(browseName), + NodeClass = NodeClass.Object, + TypeDefinition = new ExpandedNodeId(new NodeId(typeId, AINamespaceIndex)), + ReferenceTypeId = Opc.Ua.ReferenceTypeIds.HierarchicalReferences, + IsForward = true + }; + } + + public void AddBrowse(NodeId folder, IReadOnlyList references) + { + m_browse[folder] = [.. references]; + } + + public void AddChild(NodeId parent, string browseName, NodeId child) + { + m_children[(parent, browseName)] = child; + } + + public void AddValueChild(NodeId parent, string browseName, NodeId nodeId, Variant value) + { + AddChild(parent, browseName, nodeId); + m_values[nodeId] = value; + } + + private void SetupTranslate() + { + Session.Setup(s => s.TranslateBrowsePathsToNodeIdsAsync( + It.IsAny(), + It.IsAny>(), + It.IsAny())) + .Returns, CancellationToken>((_, paths, _) => + { + var results = new List(); + for (int ii = 0; ii < paths.Count; ii++) + { + BrowsePath path = paths[ii]; + NodeId current = path.StartingNode; + bool found = true; + for (int jj = 0; jj < path.RelativePath.Elements.Count; jj++) + { + string name = path.RelativePath.Elements[jj].TargetName.Name ?? string.Empty; + if (!m_children.TryGetValue((current, name), out NodeId next)) + { + found = false; + break; + } + current = next; + } + results.Add(found ? GoodPath(current) : BadPath()); + } + return new ValueTask( + new TranslateBrowsePathsToNodeIdsResponse + { + ResponseHeader = new ResponseHeader(), + Results = results.ToArrayOf(), + DiagnosticInfos = default + }); + }); + } + + private void SetupBrowse() + { + Session.Setup(s => s.BrowseAsync( + It.IsAny(), + It.IsAny(), + It.IsAny(), + It.IsAny>(), + It.IsAny())) + .Returns, CancellationToken>( + (_, _, _, descriptions, _) => + { + var results = new List(descriptions.Count); + for (int ii = 0; ii < descriptions.Count; ii++) + { + List refs = m_browse.TryGetValue( + descriptions[ii].NodeId, out List? value) + ? value + : []; + results.Add(new BrowseResult + { + StatusCode = StatusCodes.Good, + References = refs.ToArrayOf(), + ContinuationPoint = default + }); + } + return new ValueTask(new BrowseResponse + { + ResponseHeader = new ResponseHeader(), + Results = results.ToArrayOf(), + DiagnosticInfos = default + }); + }); + Session.Setup(s => s.BrowseNextAsync( + It.IsAny(), + It.IsAny(), + It.IsAny>(), + It.IsAny())) + .Returns(new ValueTask(new BrowseNextResponse + { + ResponseHeader = new ResponseHeader(), + Results = [new BrowseResult { StatusCode = StatusCodes.Good, References = [] }], + DiagnosticInfos = default + })); + } + + private void SetupRead() + { + Session.Setup(s => s.ReadAsync( + It.IsAny(), + It.IsAny(), + It.IsAny(), + It.IsAny>(), + It.IsAny())) + .Returns, CancellationToken>( + (_, _, _, nodes, _) => + { + var values = new List(); + for (int ii = 0; ii < nodes.Count; ii++) + { + values.Add(m_values.TryGetValue(nodes[ii].NodeId, out Variant variant) + ? new DataValue(variant, StatusCodes.Good, System.DateTime.UtcNow, System.DateTime.UtcNow) + : new DataValue(Variant.Null, StatusCodes.BadNodeIdUnknown)); + } + return new ValueTask(new ReadResponse + { + ResponseHeader = new ResponseHeader(), + Results = values.ToArrayOf(), + DiagnosticInfos = default + }); + }); + } + + private static BrowsePathResult GoodPath(NodeId nodeId) + { + return new BrowsePathResult + { + StatusCode = StatusCodes.Good, + Targets = [new BrowsePathTarget { TargetId = new ExpandedNodeId(nodeId) }] + }; + } + + private static BrowsePathResult BadPath() + { + return new BrowsePathResult { StatusCode = StatusCodes.BadNoMatch, Targets = [] }; + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/AssemblyInfo.cs b/tests/Opc.Ua.AI.Tests/AssemblyInfo.cs new file mode 100644 index 0000000000..9cde0c2cf1 --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/AssemblyInfo.cs @@ -0,0 +1,34 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +// The OPC UA surface under test is not CLS compliant, so claiming compliance +// here would be false. +[assembly: CLSCompliant(false)] diff --git a/tests/Opc.Ua.AI.Tests/ChatClientInferenceBackendTests.cs b/tests/Opc.Ua.AI.Tests/ChatClientInferenceBackendTests.cs new file mode 100644 index 0000000000..4fd1d7b081 --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/ChatClientInferenceBackendTests.cs @@ -0,0 +1,589 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.AI; +using Moq; +using NUnit.Framework; +using Opc.Ua.AI.Inference; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Covers the projection. + /// + /// + /// The point of routing through Microsoft.Extensions.AI is that one + /// abstraction covers a hosted service and an on-device runtime, which is the + /// property clause 8.1 asserts about where inference runs. These tests use a + /// stub , so they exercise the projection both ways + /// without a model, a network or a GPU - which is also what lets them run in + /// CI. What they mostly check is that nothing is invented: usage the client + /// did not report must not be estimated, and a model it did not name must not + /// be guessed at. + /// + [TestFixture] + public sealed class ChatClientInferenceBackendTests + { + [Test] + public async Task InvokeProjectsMessagesAndReportsWhatTheClientAnswered() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "pong")) + { + ModelId = "served-model", + FinishReason = ChatFinishReason.Stop, + Usage = new UsageDetails + { + InputTokenCount = 11, + OutputTokenCount = 22, + TotalTokenCount = 33 + } + }); + using var backend = new ChatClientInferenceBackend(client, InferenceSite.EdgeOffServer); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"ping"}]}""", "asked-model"), + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.True); + Assert.That(Encoding.UTF8.GetString(result.Payload.Span), Is.EqualTo("pong")); + Assert.That(result.ModelUsed, Is.EqualTo("served-model"), + "The model that actually answered is not always the one asked for, " + + "and a caller that cannot see the substitution cannot tell a degraded " + + "answer from a good one."); + Assert.That(result.InputUnits, Is.EqualTo(11UL)); + Assert.That(result.OutputUnits, Is.EqualTo(22UL)); + Assert.That(result.TotalUnits, Is.EqualTo(33UL)); + Assert.That(result.Finish, Is.EqualTo(InferenceFinish.Stop)); + Assert.That(client.LastMessages, Has.Count.EqualTo(1)); + Assert.That(client.LastMessages![0].Role, Is.EqualTo(ChatRole.User)); + Assert.That(client.LastMessages![0].Text, Is.EqualTo("ping")); + Assert.That(client.LastOptions?.ModelId, Is.EqualTo("asked-model")); + }); + } + + [Test] + public async Task InvokeReportsTheRequestedModelWhenTheClientNamesNone() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok"))); + using var backend = new ChatClientInferenceBackend(client); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"hi"}]}""", "asked-model"), + CancellationToken.None).ConfigureAwait(false); + + // The request named a model and nothing observed contradicts it, so it + // is reported rather than second-guessed. + Assert.That(result.ModelUsed, Is.EqualTo("asked-model")); + } + + [Test] + public async Task InvokeKeepsPlainStringContentAsATextMessage() + { + List? captured = null; + Mock client = CreateCapturingClient((messages, _, _) => captured = new List(messages)); + using var backend = new ChatClientInferenceBackend(client.Object); + + await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"ping"}]}"""), + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(captured, Is.Not.Null); + Assert.That(captured![0].Role, Is.EqualTo(ChatRole.User)); + Assert.That(captured[0].Text, Is.EqualTo("ping")); + }); + } + + [Test] + public async Task InvokeProjectsTextAndDataImagePartsToMultimodalContent() + { + byte[] imageBytes = [1, 2, 3, 4]; + string image = Convert.ToBase64String(imageBytes); + List? captured = null; + Mock client = CreateCapturingClient((messages, _, _) => captured = new List(messages)); + using var backend = new ChatClientInferenceBackend(client.Object); + + await backend.InvokeAsync( + Request( + """ + {"messages":[{"role":"user","content":[ + {"type":"text","text":"Measure the bore diameter."}, + {"type":"image_url","image_url":{"url":"data:image/png;base64,$IMAGE$","detail":"high"}} + ]}]} + """.Replace("$IMAGE$", image, StringComparison.Ordinal)), + CancellationToken.None).ConfigureAwait(false); + + IList contents = captured![0].Contents; + var text = (TextContent)contents[0]; + var data = (DataContent)contents[1]; + Assert.Multiple(() => + { + Assert.That(contents, Has.Count.EqualTo(2)); + Assert.That(text.Text, Is.EqualTo("Measure the bore diameter.")); + Assert.That(data.MediaType, Is.EqualTo("image/png")); + Assert.That(data.Data.ToArray(), Is.EqualTo(imageBytes).AsCollection); + Assert.That(data.AdditionalProperties?["detail"], Is.EqualTo("high")); + }); + } + + [Test] + public async Task InvokeProjectsRemoteImagePartToUriContent() + { + List? captured = null; + Mock client = CreateCapturingClient((messages, _, _) => captured = new List(messages)); + using var backend = new ChatClientInferenceBackend(client.Object); + + await backend.InvokeAsync( + Request(""" + {"messages":[{"role":"user","content":[ + {"type":"image_url","image_url":{"url":"https://example.test/frame.png"}} + ]}]} + """), + CancellationToken.None).ConfigureAwait(false); + + var uri = (UriContent)captured![0].Contents[0]; + Assert.Multiple(() => + { + Assert.That(uri.Uri, Is.EqualTo(new Uri("https://example.test/frame.png"))); + Assert.That(uri.MediaType, Is.EqualTo("image/png")); + }); + } + + [Test] + public async Task InvokeRefusesMalformedImageDataUriBeforeCallingTheClient() + { + List? captured = null; + Mock client = CreateCapturingClient((messages, _, _) => captured = new List(messages)); + using var backend = new ChatClientInferenceBackend(client.Object); + + InferenceResult result = await backend.InvokeAsync( + Request(""" + {"messages":[{"role":"user","content":[ + {"type":"image_url","image_url":{"url":"data:image/png;base64,not-base64"}} + ]}]} + """), + CancellationToken.None).ConfigureAwait(false); + + // A malformed image is refused before inference so callers do not get + // an answer about a picture the model never received. + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.False); + Assert.That(result.Message, Does.Contain("base64")); + Assert.That(captured, Is.Null); + }); + } + + [Test] + [TestCase("data:foo;base64,AAAA", TestName = "MediaTypeWithoutASubtype")] + [TestCase("data:image;base64,AAAA", TestName = "MediaTypeMissingItsSubtype")] + [TestCase("data:image//png;base64,AAAA", TestName = "MediaTypeWithTwoSeparators")] + public async Task InvokeRefusesAnInvalidImageMediaTypeAsAStructuredResult(string url) + { + List? captured = null; + Mock client = CreateCapturingClient((messages, _, _) => captured = new List(messages)); + using var backend = new ChatClientInferenceBackend(client.Object); + + InferenceResult result = await backend.InvokeAsync( + Request( + "{\"messages\":[{\"role\":\"user\",\"content\":[" + + "{\"type\":\"image_url\",\"image_url\":{\"url\":\"" + + url + + "\"}}" + + "]}]}"), + CancellationToken.None).ConfigureAwait(false); + + // DataContent validates the media type itself and throws + // ArgumentException, not FormatException. Left uncaught that escapes + // InvokeAsync entirely, so the caller gets a bare Bad status carrying an + // internal exception message rather than a refusal naming the problem - + // and on the chunked-transfer path the transfer is stranded Executing, + // holding a concurrency slot until it expires. + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.False); + Assert.That(result.Finish, Is.EqualTo(InferenceFinish.Error)); + Assert.That(result.Message, Does.Contain("media type")); + Assert.That(captured, Is.Null); + }); + } + + [Test] + public async Task InvokeRefusesANonStringRoleAsAStructuredResult() + { + List? captured = null; + Mock client = CreateCapturingClient( + (messages, _, _) => captured = new List(messages)); + using var backend = new ChatClientInferenceBackend(client.Object); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":5,"content":"hi"}]}"""), + CancellationToken.None).ConfigureAwait(false); + + // JsonElement.GetString throws on a non-string element, which would + // escape the handler that turns a bad payload into a refusal. + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.False); + Assert.That(result.Message, Does.Contain("role")); + Assert.That(captured, Is.Null); + }); + } + + [Test] + public async Task InvokeReportsZeroUnitsWhenTheClientReportsNoUsage() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok"))); + using var backend = new ChatClientInferenceBackend(client); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"hi"}]}"""), + CancellationToken.None).ConfigureAwait(false); + + // Estimating usage a backend did not report would produce a number that + // looks metered and is not, and it would be billed against. + Assert.Multiple(() => + { + Assert.That(result.InputUnits, Is.Zero); + Assert.That(result.OutputUnits, Is.Zero); + Assert.That(result.TotalUnits, Is.Zero); + }); + } + + [Test] + public async Task InvokeRefusesAPayloadThatIsNotValidJson() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "unused"))); + using var backend = new ChatClientInferenceBackend(client); + + InferenceResult result = await backend.InvokeAsync( + Request("{ not json"), + CancellationToken.None).ConfigureAwait(false); + + // Malformed input from a caller is expected rather than exceptional, + // and the reason is more use than a transport fault. + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.False); + Assert.That(result.Finish, Is.EqualTo(InferenceFinish.Error)); + Assert.That(result.Message, Does.Contain("not a valid chat completions body")); + Assert.That(client.LastMessages, Is.Null, "The client must not be reached."); + }); + } + + [Test] + public async Task InvokeRefusesAPayloadCarryingNoMessages() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "unused"))); + using var backend = new ChatClientInferenceBackend(client); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[]}"""), + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.False); + Assert.That(result.Message, Does.Contain("no messages")); + Assert.That(client.LastMessages, Is.Null); + }); + } + + [Test] + public async Task InvokeMapsTheRolesTheContractDefines() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok"))); + using var backend = new ChatClientInferenceBackend(client); + + await backend.InvokeAsync( + Request(""" + {"messages":[ + {"role":"system","content":"s"}, + {"role":"user","content":"u"}, + {"role":"assistant","content":"a"}, + {"role":"tool","content":"t"}]} + """), + CancellationToken.None).ConfigureAwait(false); + + Assert.That( + new[] + { + client.LastMessages![0].Role, + client.LastMessages![1].Role, + client.LastMessages![2].Role, + client.LastMessages![3].Role + }, + Is.EqualTo(new[] + { + ChatRole.System, ChatRole.User, ChatRole.Assistant, ChatRole.Tool + }).AsCollection); + } + + [Test] + public async Task InvokeAppliesTheCallParametersTheBackendSupports() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok"))); + using var backend = new ChatClientInferenceBackend(client); + + await backend.InvokeAsync( + Request( + """{"messages":[{"role":"user","content":"hi"}]}""", + parameters: new Dictionary + { + ["temperature"] = "0.25", + ["max_tokens"] = "64", + ["top_p"] = "0.9" + }), + CancellationToken.None).ConfigureAwait(false); + + // Parsed invariantly: on a machine whose locale writes 0,25 a + // culture-sensitive parse silently drops the decimal point. + Assert.Multiple(() => + { + Assert.That(client.LastOptions!.Temperature, Is.EqualTo(0.25f)); + Assert.That(client.LastOptions!.MaxOutputTokens, Is.EqualTo(64)); + Assert.That(client.LastOptions!.TopP, Is.EqualTo(0.9f)); + }); + } + + [Test] + public void InvokeRefusesACallParameterItCannotHonour() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok"))); + using var backend = new ChatClientInferenceBackend(client); + + // A caller whose parameter was silently dropped believes it took + // effect, and there is no later point at which it can find out. + Assert.That( + async () => await backend.InvokeAsync( + Request( + """{"messages":[{"role":"user","content":"hi"}]}""", + parameters: new Dictionary { ["seed"] = "1" }), + CancellationToken.None).ConfigureAwait(false), + Throws.ArgumentException); + } + + [Test] + public async Task ProbeReportsUnreachableRatherThanRaisingWhenTheClientFails() + { + var client = new StubChatClient(new InvalidOperationException("no route to host")); + using var backend = new ChatClientInferenceBackend(client); + + BackendProbe probe = await backend.ProbeAsync(CancellationToken.None) + .ConfigureAwait(false); + + // A probe exists so a commissioning engineer learns the endpoint is + // wrong before a deployment depends on it, so it reports rather than + // throws. + Assert.Multiple(() => + { + Assert.That(probe.Reachable, Is.False); + Assert.That(probe.Detail, Does.Contain("no route to host")); + }); + } + + [Test] + public async Task ProbeReportsReachableAndNamesTheModelThatAnswered() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "pong")) + { + ModelId = "served-model" + }); + using var backend = new ChatClientInferenceBackend(client); + + BackendProbe probe = await backend.ProbeAsync(CancellationToken.None) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(probe.Reachable, Is.True); + Assert.That(probe.Detail, Is.EqualTo("served-model")); + }); + } + + [Test] + public async Task ListModelsFiltersAndBoundsTheSuppliedCatalogue() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok"))); + using var backend = new ChatClientInferenceBackend( + client, + InferenceSite.OnServer, + new[] + { + new BackendModel { Name = "alpha-small" }, + new BackendModel { Name = "alpha-large" }, + new BackendModel { Name = "beta" } + }); + + IReadOnlyList all = await backend + .ListModelsAsync(null, 0, CancellationToken.None).ConfigureAwait(false); + IReadOnlyList filtered = await backend + .ListModelsAsync("alpha", 0, CancellationToken.None).ConfigureAwait(false); + IReadOnlyList bounded = await backend + .ListModelsAsync(null, 2, CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(all, Has.Count.EqualTo(3)); + Assert.That(filtered, Has.Count.EqualTo(2)); + Assert.That(bounded, Has.Count.EqualTo(2)); + }); + } + + [Test] + public void SiteIsWhatTheHostStatedRatherThanSomethingInferred() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok"))); + using var backend = new ChatClientInferenceBackend(client, InferenceSite.OnServer); + + // An IChatClient over a local runtime and one over a hosted service are + // the same type, so the client cannot be asked where it runs. + Assert.That(backend.Site, Is.EqualTo(InferenceSite.OnServer)); + } + + [Test] + public void ConstructorRefusesANullClient() + { + Assert.That( + () => new ChatClientInferenceBackend(null!), + Throws.ArgumentNullException); + } + + [Test] + public void InvokeRefusesANullRequest() + { + var client = new StubChatClient( + new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok"))); + using var backend = new ChatClientInferenceBackend(client); + + Assert.That( + async () => await backend.InvokeAsync(null!, CancellationToken.None) + .ConfigureAwait(false), + Throws.ArgumentNullException); + } + + private static InferenceRequest Request( + string body, + string model = "", + IReadOnlyDictionary? parameters = null) + { + return new InferenceRequest + { + Model = model, + Payload = Encoding.UTF8.GetBytes(body), + ContentType = "application/json", + Parameters = parameters ?? new Dictionary() + }; + } + + private static Mock CreateCapturingClient( + Action, ChatOptions?, CancellationToken> callback) + { + var client = new Mock(); + client + .Setup(c => c.GetResponseAsync( + It.IsAny>(), + It.IsAny(), + It.IsAny())) + .Callback(callback) + .ReturnsAsync(new ChatResponse(new ChatMessage(ChatRole.Assistant, "ok"))); + return client; + } + + private sealed class StubChatClient : IChatClient + { + private readonly ChatResponse? m_response; + private readonly Exception? m_failure; + + public StubChatClient(ChatResponse response) + { + m_response = response; + } + + public StubChatClient(Exception failure) + { + m_failure = failure; + } + + public List? LastMessages { get; private set; } + + public ChatOptions? LastOptions { get; private set; } + + public Task GetResponseAsync( + IEnumerable messages, + ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + LastMessages = new List(messages); + LastOptions = options; + if (m_failure != null) + { + throw m_failure; + } + return Task.FromResult(m_response!); + } + + public IAsyncEnumerable GetStreamingResponseAsync( + IEnumerable messages, + ChatOptions? options = null, + CancellationToken cancellationToken = default) + { + throw new NotSupportedException(); + } + + public object? GetService(Type serviceType, object? serviceKey = null) + { + return null; + } + + public void Dispose() + { + } + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/CredentialResolverTests.cs b/tests/Opc.Ua.AI.Tests/CredentialResolverTests.cs new file mode 100644 index 0000000000..17b639ad9b --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/CredentialResolverTests.cs @@ -0,0 +1,317 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.AI.Inference; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Covers the credential resolvers, which are the only things in the stack + /// that ever hold a secret value. + /// + /// + /// Clause 9.2 forbids a Server from exposing credential material through any + /// Attribute, and argues it from the fact that an address space is browsable, + /// subscribable and historisable - a secret placed there is not exposed once, + /// it is published, distributed and archived. These resolvers are where the + /// name a deployment publishes turns into the value it stands for, so their + /// refusals matter more than their successes. + /// + [TestFixture] + public sealed class CredentialResolverTests + { + [Test] + public async Task FileResolverReadsTheKeyMountedAtTheDirectory() + { + using var mount = new TempMount(); + mount.Write("api-key", "s3cret"); + + var resolver = new FileCredentialResolver(mount.Path); + string? value = await resolver.ResolveAsync("api-key", CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(value, Is.EqualTo("s3cret")); + } + + [Test] + public async Task FileResolverTrimsTheTrailingNewlineAMountLeaves() + { + using var mount = new TempMount(); + mount.Write("api-key", "s3cret\n"); + + var resolver = new FileCredentialResolver(mount.Path); + string? value = await resolver.ResolveAsync("api-key", CancellationToken.None) + .ConfigureAwait(false); + + // Presenting the newline in a header fails in a way that looks like a + // wrong key rather than a stray byte, which is a bad afternoon. + Assert.That(value, Is.EqualTo("s3cret")); + } + + [Test] + public async Task FileResolverReturnsNullForAKeyTheMountDoesNotCarry() + { + using var mount = new TempMount(); + + var resolver = new FileCredentialResolver(mount.Path); + string? value = await resolver.ResolveAsync("absent", CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(value, Is.Null); + } + + [Test] + public void FileResolverRefusesAReferenceThatCouldEscapeTheMount() + { + using var mount = new TempMount(); + var resolver = new FileCredentialResolver(mount.Path); + + // A reference names a key, not a path. One containing a separator is + // configuration this Server controls, so it is a mistake worth + // surfacing rather than input worth repairing. + Assert.Multiple(() => + { + Assert.That( + async () => await resolver.ResolveAsync("../etc/passwd", CancellationToken.None) + .ConfigureAwait(false), + Throws.ArgumentException); + Assert.That( + async () => await resolver.ResolveAsync("sub/key", CancellationToken.None) + .ConfigureAwait(false), + Throws.ArgumentException); + Assert.That( + async () => await resolver.ResolveAsync("..\\key", CancellationToken.None) + .ConfigureAwait(false), + Throws.ArgumentException); + }); + } + + [Test] + public async Task NullResolverNeverProducesAValue() + { + string? value = await NullCredentialResolver.Instance + .ResolveAsync("anything", CancellationToken.None).ConfigureAwait(false); + + Assert.That(value, Is.Null); + } + + [Test] + public async Task WorkloadIdentityReadsTheTokenTheAudiencePointsAt() + { + using var mount = new TempMount(); + string path = mount.Write("token", "projected-token\n"); + + // Workload identity has no secret to name, so the configured audience + // is the token to read. Every platform projects it as a file; reading + // it directly is what keeps this free of a cloud SDK. + var resolver = new WorkloadIdentityCredentialResolver(path); + string? value = await resolver.ResolveAsync(string.Empty, CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(value, Is.EqualTo("projected-token")); + } + + [Test] + public async Task WorkloadIdentityFallsBackToTheReferenceWhenNoAudienceIsConfigured() + { + using var mount = new TempMount(); + string path = mount.Write("token", "from-reference"); + + var resolver = new WorkloadIdentityCredentialResolver(); + string? value = await resolver.ResolveAsync(path, CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(value, Is.EqualTo("from-reference")); + } + + [Test] + public async Task WorkloadIdentityReturnsNullWhenNoTokenIsProjected() + { + using var mount = new TempMount(); + + var resolver = new WorkloadIdentityCredentialResolver( + Path.Combine(mount.Path, "never-written")); + string? value = await resolver.ResolveAsync(string.Empty, CancellationToken.None) + .ConfigureAwait(false); + + // A Server whose platform projected nothing sends no Authorization + // header, which is a clearer failure than an empty one. + Assert.That(value, Is.Null); + } + + [Test] + public async Task WorkloadIdentityUsesASuppliedAcquisitionDelegate() + { + string? seen = null; + var resolver = new WorkloadIdentityCredentialResolver( + (scope, _) => + { + seen = scope; + return new ValueTask("delegated"); + }, + "the-scope"); + + string? value = await resolver.ResolveAsync("ignored", CancellationToken.None) + .ConfigureAwait(false); + + // This is the seam a host uses to bring its own identity SDK without + // this assembly depending on one, and the seam a test uses to exercise + // the path without a platform. + Assert.Multiple(() => + { + Assert.That(value, Is.EqualTo("delegated")); + Assert.That(seen, Is.EqualTo("the-scope")); + }); + } + + [Test] + public void WorkloadIdentityRefusesANullAcquisitionDelegate() + { + Assert.That( + () => new WorkloadIdentityCredentialResolver(null!, "scope"), + Throws.ArgumentNullException); + } + + [Test] + public async Task WorkloadIdentityReadsTheTokenAGoogleExternalAccountConfigNames() + { + using var mount = new TempMount(); + string token = mount.Write("gcp-token", "google-projected-token\n"); + string config = mount.Write( + "external-account.json", + "{\"type\":\"external_account\",\"credential_source\":{\"file\":\"" + + token.Replace("\\", "\\\\", StringComparison.Ordinal) + + "\"}}"); + + // Google is the one platform that does not name the token in a variable + // of its own: it names a configuration, and the configuration names the + // token. A variable invented to look like the Azure and AWS ones would + // never match anything, so this pins the mechanism that actually exists. + using var env = new TempEnvironment("GOOGLE_APPLICATION_CREDENTIALS", config); + var resolver = new WorkloadIdentityCredentialResolver(); + string? value = await resolver.ResolveAsync(string.Empty, CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(value, Is.EqualTo("google-projected-token")); + } + + [Test] + public async Task WorkloadIdentityIgnoresAMalformedGoogleExternalAccountConfig() + { + using var mount = new TempMount(); + string config = mount.Write("external-account.json", "{ not json at all"); + + using var env = new TempEnvironment("GOOGLE_APPLICATION_CREDENTIALS", config); + var resolver = new WorkloadIdentityCredentialResolver(); + string? value = await resolver.ResolveAsync(string.Empty, CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(value, Is.Null); + } + + [Test] + public async Task WorkloadIdentityReadsTheTokenTheAwsVariableNames() + { + using var mount = new TempMount(); + string token = mount.Write("aws-token", "aws-projected-token"); + + // AWS_WEB_IDENTITY_TOKEN_FILE is the AWS SDK's own variable, which EKS + // populates for a service account bound to an IAM role. + using var env = new TempEnvironment("AWS_WEB_IDENTITY_TOKEN_FILE", token); + var resolver = new WorkloadIdentityCredentialResolver(); + string? value = await resolver.ResolveAsync(string.Empty, CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(value, Is.EqualTo("aws-projected-token")); + } + + [Test] + public void FileResolverRefusesANullDirectory() + { + Assert.That( + () => new FileCredentialResolver(null!), + Throws.ArgumentNullException); + } + + private sealed class TempEnvironment : IDisposable + { + public TempEnvironment(string name, string? value) + { + m_name = name; + m_previous = Environment.GetEnvironmentVariable(name); + Environment.SetEnvironmentVariable(name, value); + } + + public void Dispose() + { + Environment.SetEnvironmentVariable(m_name, m_previous); + } + + private readonly string m_name; + private readonly string? m_previous; + } + + private sealed class TempMount : IDisposable + { + public TempMount() + { + Path = System.IO.Path.Combine( + System.IO.Path.GetTempPath(), + "opcua-ai-cred-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(Path); + } + + public string Path { get; } + + public string Write(string name, string content) + { + string full = System.IO.Path.Combine(Path, name); + File.WriteAllText(full, content); + return full; + } + + public void Dispose() + { + try + { + Directory.Delete(Path, recursive: true); + } + catch (IOException) + { + // A leftover temp directory is not worth failing a test over. + } + } + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/CredentialSafetyTests.cs b/tests/Opc.Ua.AI.Tests/CredentialSafetyTests.cs new file mode 100644 index 0000000000..cad28b6a2c --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/CredentialSafetyTests.cs @@ -0,0 +1,213 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Verifies that no credential material reaches the address space. + /// + /// + /// + /// The specification forbids it and a sample is the thing people copy, so this + /// is checked by walking every Variable the Server publishes rather than by + /// inspecting the two or three places a secret was expected to appear. A leak + /// that only happens somewhere unexpected is the only kind worth testing for. + /// + /// + /// The credential resolver is also checked directly, because "the address space + /// is clean" and "the resolver refuses a path" are different claims and only one + /// of them is about browsing. + /// + /// + [TestFixture] + [Category("AIModelManagement")] + [SetCulture("en-us")] + [SetUICulture("en-us")] + public sealed class CredentialSafetyTests + { + private const string Secret = "sk-this-must-never-be-browsable-1234567890"; + + [Test] + public async Task NoPublishedVariableCarriesCredentialMaterialAsync() + { + var backendOptions = new InferenceBackendOptions + { + Authentication = BackendAuthentication.ApiKey, + CredentialReference = "inference-api-key", + EndpointUri = "https://example.invalid/openai/", + Site = InferenceSite.Cloud, + EgressPermitted = true + }; + + using AINodeManager nm = await AIServerTestHarness + .CreateAsync( + new InferenceBackends(new FakeInferenceBackend("primary")), + new AIOptions { EnableFallback = false }, + backendOptions) + .ConfigureAwait(false); + + var offenders = new List(); + + foreach (NodeState node in Walk(nm)) + { + if (node is not BaseVariableState variable) + { + continue; + } + + string? text = variable.Value.ToString(); + + if (!string.IsNullOrEmpty(text) && + text.Contains(Secret, StringComparison.OrdinalIgnoreCase)) + { + offenders.Add(variable.BrowseName.ToString()); + } + } + + Assert.That(offenders, Is.Empty); + } + + [Test] + public async Task TheSourcePublishesTheReferenceRatherThanTheSecretAsync() + { + var backendOptions = new InferenceBackendOptions + { + Authentication = BackendAuthentication.ApiKey, + CredentialReference = "inference-api-key" + }; + + using AINodeManager nm = await AIServerTestHarness + .CreateAsync( + new InferenceBackends(new FakeInferenceBackend("primary")), + new AIOptions { EnableFallback = false }, + backendOptions) + .ConfigureAwait(false); + + ModelSourceState? source = null; + + foreach (NodeState node in Walk(nm)) + { + if (node is ModelSourceState found) + { + source = found; + break; + } + } + + Assert.That(source, Is.Not.Null); + + Assert.Multiple(() => + { + // A client is entitled to know WHICH credential is configured, so it + // can tell whether the right one is. It is not entitled to the value. + Assert.That( + source!.CredentialReference!.Value, + Is.EqualTo("inference-api-key")); + Assert.That( + source.AuthenticationKind!.Value, + Is.EqualTo(AuthenticationKindEnum.ApiKey)); + }); + } + + [Test] + public async Task TheFileResolverRefusesAReferenceThatCouldEscapeTheMountAsync() + { + string directory = Path.Combine( + Path.GetTempPath(), + "ai-cred-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(directory); + + try + { + await File.WriteAllTextAsync( + Path.Combine(directory, "key"), Secret).ConfigureAwait(false); + + var resolver = new FileCredentialResolver(directory); + + string? resolved = await resolver + .ResolveAsync("key", CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(resolved, Is.EqualTo(Secret)); + + // A reference is configuration this Server controls, so one carrying + // a separator is a mistake worth surfacing rather than input worth + // repairing. Sanitising it quietly would hide the misconfiguration. + foreach (string escape in new[] { "../key", "sub/key", "sub\\key", ".." }) + { + Assert.ThrowsAsync( + async () => await resolver + .ResolveAsync(escape, CancellationToken.None) + .ConfigureAwait(false), + "'{0}' must be refused", + escape); + } + } + finally + { + Directory.Delete(directory, recursive: true); + } + } + + /// + /// Every node the Server publishes, children included. + /// + private static IEnumerable Walk(AINodeManager nm) + { + var root = nm.FindPredefinedNode(nm.RootId); + var pending = new Stack(); + pending.Push(root); + + while (pending.Count > 0) + { + NodeState node = pending.Pop(); + yield return node; + + var children = new List(); + node.GetChildren(nm.SystemContext, children); + + foreach (BaseInstanceState child in children) + { + pending.Push(child); + } + } + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/FakeInferenceBackend.cs b/tests/Opc.Ua.AI.Tests/FakeInferenceBackend.cs new file mode 100644 index 0000000000..0b94cda3cb --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/FakeInferenceBackend.cs @@ -0,0 +1,154 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.AI.Inference; + +namespace Opc.Ua.AI.Tests +{ + /// + /// A backend that answers without a network. + /// + /// + /// + /// This is a test double, not a third configurable provider. The distinction + /// matters: a fake that shipped as a supported option would let the sample look + /// healthy while never having reached a model, and the useful thing about the + /// sample is precisely that it does. + /// + /// + /// It records what it was asked, because several of the claims worth testing are + /// about which backend was called rather than what came back. + /// + /// + internal sealed class FakeInferenceBackend : IInferenceBackend + { + private readonly List m_requests = []; + + /// + /// Creates a fake. + /// + /// + /// Names this instance in the answers it produces, so a test can tell which + /// of two fakes actually replied. + /// + public FakeInferenceBackend(string name) + { + Name = name; + } + + /// Identifies this fake in its own answers. + public string Name { get; } + + /// + public InferenceSite Site { get; set; } = InferenceSite.OnServer; + + /// Whether calls succeed. + public bool Healthy { get; set; } = true; + + /// Whether the probe reports the backend as reachable. + public bool Reachable { get; set; } = true; + + /// + /// How a failing call fails. Defaults to an ordinary error; set to + /// to exercise a safety refusal, + /// which must not be retried anywhere else. + /// + public InferenceFinish FailureKind { get; set; } = InferenceFinish.Error; + + /// Models this fake offers. + public List Models { get; } = []; + + /// Everything this fake was asked, in order. + public IReadOnlyList Requests => m_requests; + + /// + public ValueTask> ListModelsAsync( + string? filter, + uint maxResults, + CancellationToken ct) + { + IReadOnlyList models = Models; + return ValueTask.FromResult(models); + } + + /// + public ValueTask InvokeAsync( + InferenceRequest request, + CancellationToken ct) + { + lock (m_requests) + { + m_requests.Add(request); + } + + if (!Healthy) + { + return ValueTask.FromResult(new InferenceResult + { + Ok = false, + Finish = FailureKind, + Message = Name + " is unhealthy." + }); + } + + // The answer names the fake that produced it, which is what lets a test + // distinguish "the fallback answered" from "the primary recovered". + byte[] payload = Encoding.UTF8.GetBytes( + FormattableString.Invariant($"{{\"answeredBy\":\"{Name}\"}}")); + + return ValueTask.FromResult(new InferenceResult + { + Ok = true, + Payload = payload, + ContentType = "application/json", + ModelUsed = request.Model, + UsageUnit = "tokens", + InputUnits = 1, + OutputUnits = 2, + TotalUnits = 3, + Finish = InferenceFinish.Stop + }); + } + + /// + public ValueTask ProbeAsync(CancellationToken ct) + { + return ValueTask.FromResult(new BackendProbe + { + Reachable = Reachable, + Detail = Name + }); + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/FallbackTests.cs b/tests/Opc.Ua.AI.Tests/FallbackTests.cs new file mode 100644 index 0000000000..ee423bd361 --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/FallbackTests.cs @@ -0,0 +1,252 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Verifies that a substituted model is reported as one. + /// + /// + /// This is the fixture worth having. A fallback that answers without saying so + /// is indistinguishable from a healthy primary at every layer above it: the call + /// succeeds, the payload is well formed, and the caller attributes a smaller + /// model's answer to the model it asked for. Nothing else in the sample fails + /// this quietly. + /// + [TestFixture] + [Category("AIModelManagement")] + [SetCulture("en-us")] + [SetUICulture("en-us")] + public sealed class FallbackTests + { + [Test] + public async Task AHealthyPrimaryAnswersAndReportsItsOwnModelAsync() + { + var primary = new FakeInferenceBackend("primary"); + var fallback = new FakeInferenceBackend("fallback"); + + using AINodeManager nm = await CreateAsync(primary, fallback) + .ConfigureAwait(false); + + InvokeMethodStateResult result = await InvokeAsync(nm, nm.PrimaryDeploymentId) + .ConfigureAwait(false); + + var primaryModel = nm.FindPredefinedNode(ModelOf(nm, nm.PrimaryDeploymentId)); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(AnsweredBy(result), Is.EqualTo("primary")); + Assert.That(result.ModelUsed, Is.EqualTo(primaryModel.NodeId)); + Assert.That(fallback.Requests, Is.Empty, "the fallback must not be consulted"); + }); + } + + [Test] + public async Task AFailedPrimaryFallsBackAndReportsTheSubstitutedModelAsync() + { + var primary = new FakeInferenceBackend("primary") { Healthy = false }; + var fallback = new FakeInferenceBackend("fallback"); + + using AINodeManager nm = await CreateAsync(primary, fallback) + .ConfigureAwait(false); + + InvokeMethodStateResult result = await InvokeAsync(nm, nm.PrimaryDeploymentId) + .ConfigureAwait(false); + + NodeId primaryModelId = ModelOf(nm, nm.PrimaryDeploymentId); + NodeId fallbackModelId = ModelOf(nm, nm.FallbackDeploymentId); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(AnsweredBy(result), Is.EqualTo("fallback")); + + // The claim this whole fixture exists for. + Assert.That( + result.ModelUsed, + Is.EqualTo(fallbackModelId), + "ModelUsed must name the model that actually answered"); + Assert.That( + result.ModelUsed, + Is.Not.EqualTo(primaryModelId), + "reporting the requested model would hide the substitution"); + }); + } + + [Test] + public async Task FallbackIsNotTakenWhenThePolicySaysFailAsync() + { + var primary = new FakeInferenceBackend("primary") { Healthy = false }; + var fallback = new FakeInferenceBackend("fallback"); + + // No fallback deployment means no FallsBackTo and the policy stays Fail, + // which is the default a caller gets unless someone chose otherwise. + using AINodeManager nm = await CreateAsync( + primary, + fallback, + new AIOptions { EnableFallback = false }) + .ConfigureAwait(false); + + InvokeMethodStateResult result = await InvokeAsync(nm, nm.PrimaryDeploymentId) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.False); + Assert.That(fallback.Requests, Is.Empty); + Assert.That(nm.FallbackDeploymentId, Is.EqualTo(NodeId.Null)); + }); + } + + [Test] + public async Task ASafetyRefusalIsNotRetriedOnTheFallbackAsync() + { + var primary = new FakeInferenceBackend("primary") + { + Healthy = false, + FailureKind = InferenceFinish.Filtered + }; + var fallback = new FakeInferenceBackend("fallback"); + + using AINodeManager nm = await CreateAsync(primary, fallback) + .ConfigureAwait(false); + + await InvokeAsync(nm, nm.PrimaryDeploymentId).ConfigureAwait(false); + + // A content filter declining is a result, not a fault. Sending the same + // payload to a second model until one accepts it would turn a policy + // into an obstacle, so the substitution must not happen here even + // though the deployment is configured to fall back. + Assert.That( + fallback.Requests, + Is.Empty, + "a filtered response must not be retried elsewhere"); + } + + [Test] + public async Task ConsecutiveFailuresResetOnSuccessAsync() + { + var primary = new FakeInferenceBackend("primary") { Healthy = false }; + var fallback = new FakeInferenceBackend("fallback"); + + using AINodeManager nm = await CreateAsync(primary, fallback) + .ConfigureAwait(false); + + await InvokeAsync(nm, nm.PrimaryDeploymentId).ConfigureAwait(false); + await InvokeAsync(nm, nm.PrimaryDeploymentId).ConfigureAwait(false); + + Assert.That(FailuresOf(nm, nm.PrimaryDeploymentId), Is.EqualTo(2u)); + + primary.Healthy = true; + await InvokeAsync(nm, nm.PrimaryDeploymentId).ConfigureAwait(false); + + // Resets rather than decrements, because the question the member answers + // is "is it failing now", not "how often has it ever failed". + Assert.That(FailuresOf(nm, nm.PrimaryDeploymentId), Is.Zero); + } + + private static Task CreateAsync( + IInferenceBackend primary, + IInferenceBackend fallback, + AIOptions? options = null) + { + return AIServerTestHarness.CreateAsync( + new InferenceBackends(primary, fallback), + options); + } + + private static async Task InvokeAsync( + AINodeManager nm, + NodeId deploymentId) + { + var deployment = nm.FindPredefinedNode(deploymentId); + + return await deployment.Invoke!.OnCallAsync!( + nm.SystemContext, + deployment.Invoke, + deploymentId, + ByteString.From(Encoding.UTF8.GetBytes("{}")), + string.Empty, + "application/json", + ArrayOf.Empty, + 5000, + CancellationToken.None).ConfigureAwait(false); + } + + /// + /// Follows UsesModel, the way an auditing client would. + /// + private static NodeId ModelOf(AINodeManager nm, NodeId deploymentId) + { + var deployment = nm.FindPredefinedNode(deploymentId); + var references = new System.Collections.Generic.List(); + deployment.GetReferences(nm.SystemContext, references); + + NodeId usesModel = ExpandedNodeId.ToNodeId( + Opc.Ua.AI.ReferenceTypeIds.UsesModel, + nm.SystemContext.NamespaceUris); + + foreach (IReference reference in references) + { + if (!reference.IsInverse && reference.ReferenceTypeId == usesModel) + { + return ExpandedNodeId.ToNodeId( + reference.TargetId, + nm.SystemContext.NamespaceUris); + } + } + + return NodeId.Null; + } + + private static uint FailuresOf(AINodeManager nm, NodeId deploymentId) + { + var deployment = nm.FindPredefinedNode(deploymentId); + return deployment.ConsecutiveFailures?.Value ?? 0; + } + + private static string AnsweredBy(InvokeMethodStateResult result) + { + string json = Encoding.UTF8.GetString(result.ResponsePayload.Span); + return json.Contains("\"fallback\"", System.StringComparison.Ordinal) + ? "fallback" + : "primary"; + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/JobAndSourceTests.cs b/tests/Opc.Ua.AI.Tests/JobAndSourceTests.cs new file mode 100644 index 0000000000..56f22c771a --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/JobAndSourceTests.cs @@ -0,0 +1,267 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Diagnostics; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; +using ObjectIds = Opc.Ua.ObjectIds; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Verifies the asynchronous path and the model source. + /// + [TestFixture] + [Category("AIModelManagement")] + [SetCulture("en-us")] + [SetUICulture("en-us")] + public sealed class JobAndSourceTests + { + [Test] + public async Task AnAsynchronousInferenceReturnsAJobThatLaterCarriesTheResultAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + + InvokeAsyncMethodStateResult started = + await deployment.InvokeAsync!.OnCallAsync!( + nm.SystemContext, + deployment.InvokeAsync, + nm.PrimaryDeploymentId, + ByteString.From(Encoding.UTF8.GetBytes("{}")), + string.Empty, + "application/json", + ArrayOf.Empty, + CancellationToken.None).ConfigureAwait(false); + + Assert.That(started.Job, Is.Not.EqualTo(NodeId.Null)); + + var job = nm.FindPredefinedNode(started.Job); + + // Running before Halted: the caller has a NodeId to watch, and the + // result belongs to the job rather than to the call that asked for it. + Assert.That( + job.CurrentState!.Id!.Value, + Is.EqualTo(Opc.Ua.ObjectIds.ProgramStateMachineType_Running)); + + await WaitForHaltedAsync(job).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(job.ResponsePayload!.Value.Length, Is.GreaterThan(0)); + Assert.That(job.ModelUsed!.Value, Is.Not.EqualTo(NodeId.Null)); + Assert.That(job.Progress!.Value, Is.EqualTo(100)); + Assert.That(job.FinishedAt, Is.Not.Null); + }); + } + + [Test] + public async Task AFailedJobHaltsAndRecordsWhyAsync() + { + var primary = new FakeInferenceBackend("primary") { Healthy = false }; + + using AINodeManager nm = await AIServerTestHarness + .CreateAsync( + new InferenceBackends(primary), + new AIOptions + { + EnableFallback = false, + AsyncInferenceDelay = TimeSpan.Zero + }) + .ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + + InvokeAsyncMethodStateResult started = + await deployment.InvokeAsync!.OnCallAsync!( + nm.SystemContext, + deployment.InvokeAsync, + nm.PrimaryDeploymentId, + ByteString.From(Encoding.UTF8.GetBytes("{}")), + string.Empty, + "application/json", + ArrayOf.Empty, + CancellationToken.None).ConfigureAwait(false); + + var job = nm.FindPredefinedNode(started.Job); + await WaitForHaltedAsync(job).ConfigureAwait(false); + + Assert.Multiple(() => + { + // Halted either way. Whether the inference succeeded is answered by + // whether ResponsePayload or LastError is set, not by the state the + // program ended in. + Assert.That(job.LastError!.Value.Text, Is.Not.Empty); + + // The member is published either way, so a client can browse to it + // and subscribe before the job finishes. On failure it carries a + // null ByteString - absent rather than empty, because an empty + // answer and no answer are different things and a client deciding + // whether to retry needs to tell them apart. + Assert.That(job.ResponsePayload, Is.Not.Null); + Assert.That(job.ResponsePayload!.Value.IsNull, Is.True); + }); + } + + [Test] + public async Task TheSourceReportsWhatItCanReachAsync() + { + var primary = new FakeInferenceBackend("primary"); + primary.Models.Add(new BackendModel + { + Publisher = "contoso", + Name = "weld-inspect", + Version = "2.1.0" + }); + + using AINodeManager nm = await AIServerTestHarness + .CreateAsync( + new InferenceBackends(primary), + new AIOptions { EnableFallback = false }) + .ConfigureAwait(false); + + ModelSourceState source = FindSource(nm); + + TestConnectionMethodStateResult probe = await source.TestConnection!.OnCallAsync!( + nm.SystemContext, + source.TestConnection, + source.NodeId, + CancellationToken.None).ConfigureAwait(false); + + ListModelsMethodStateResult listed = await source.ListModels!.OnCallAsync!( + nm.SystemContext, + source.ListModels, + source.NodeId, + string.Empty, + 0, + ByteString.Empty, + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(probe.Reachable, Is.True); + Assert.That(source.Reachability!.Value, Is.EqualTo(ReachabilityEnum.Reachable)); + + // Answered from the source, not from what this Server has already + // imported: the question is what COULD be deployed. + Assert.That(listed.Models.Count, Is.EqualTo(1)); + Assert.That(listed.Models[0].Name, Is.EqualTo("weld-inspect")); + }); + } + + [Test] + public async Task AnUnreachableSourceSaysSoAsync() + { + var primary = new FakeInferenceBackend("primary") { Reachable = false }; + + using AINodeManager nm = await AIServerTestHarness + .CreateAsync( + new InferenceBackends(primary), + new AIOptions { EnableFallback = false }) + .ConfigureAwait(false); + + ModelSourceState source = FindSource(nm); + + TestConnectionMethodStateResult probe = await source.TestConnection!.OnCallAsync!( + nm.SystemContext, + source.TestConnection, + source.NodeId, + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(probe.Reachable, Is.False); + Assert.That( + source.Reachability!.Value, + Is.EqualTo(ReachabilityEnum.Unreachable)); + Assert.That(source.ConsecutiveFailures!.Value, Is.EqualTo(1u)); + }); + } + + private static Task CreateAsync() + { + return AIServerTestHarness.CreateAsync( + new InferenceBackends(new FakeInferenceBackend("primary")), + new AIOptions + { + EnableFallback = false, + AsyncInferenceDelay = TimeSpan.Zero + }); + } + + private static ModelSourceState FindSource(AINodeManager nm) + { + var root = nm.FindPredefinedNode(nm.RootId); + var children = new System.Collections.Generic.List(); + root.Sources!.GetChildren(nm.SystemContext, children); + + foreach (BaseInstanceState child in children) + { + if (child is ModelSourceState source) + { + return source; + } + } + + throw new InvalidOperationException("No model source was published."); + } + + /// + /// Waits for the job to leave Running. + /// + /// + /// Polls rather than sleeping a fixed interval, so the test is neither + /// flaky on a loaded machine nor slower than it needs to be on an idle one. + /// + private static async Task WaitForHaltedAsync(InferenceJobState job) + { + var stopwatch = Stopwatch.StartNew(); + + while (stopwatch.Elapsed < TimeSpan.FromSeconds(10)) + { + if (job.CurrentState!.Id!.Value == Opc.Ua.ObjectIds.ProgramStateMachineType_Halted) + { + return; + } + + await Task.Delay(20).ConfigureAwait(false); + } + + Assert.Fail("The job did not reach Halted."); + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/Opc.Ua.AI.Tests.csproj b/tests/Opc.Ua.AI.Tests/Opc.Ua.AI.Tests.csproj new file mode 100644 index 0000000000..0c456c5beb --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/Opc.Ua.AI.Tests.csproj @@ -0,0 +1,47 @@ + + + Exe + + net10.0;net9.0;net8.0 + $(CustomTestTarget) + net10.0 + true + Opc.Ua.AI.Tests + Opc.Ua.AI.Tests + false + enable + false + + false + false + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + all + runtime; build; native; contentfiles; analyzers + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + + + + + diff --git a/tests/Opc.Ua.AI.Tests/ProvenanceTests.cs b/tests/Opc.Ua.AI.Tests/ProvenanceTests.cs new file mode 100644 index 0000000000..aecada16d4 --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/ProvenanceTests.cs @@ -0,0 +1,212 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Verifies that a result can be traced back to the artefact that produced it. + /// + /// + /// The walk is result to ModelUsed, model to Digest, and model to + /// the source it was ImportedFrom. Every link has to be present and + /// resolvable, because a chain that breaks anywhere answers nothing at all: the + /// question it exists for - "which weights produced this output" - has no + /// partial answer. + /// + [TestFixture] + [Category("AIModelManagement")] + [SetCulture("en-us")] + [SetUICulture("en-us")] + public sealed class ProvenanceTests + { + [Test] + public async Task AResultResolvesToAModelThatCarriesItsIdentityAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + + InvokeMethodStateResult result = await deployment.Invoke!.OnCallAsync!( + nm.SystemContext, + deployment.Invoke, + nm.PrimaryDeploymentId, + ByteString.From(Encoding.UTF8.GetBytes("{}")), + string.Empty, + "application/json", + ArrayOf.Empty, + 5000, + CancellationToken.None).ConfigureAwait(false); + + Assert.That(result.ModelUsed, Is.Not.EqualTo(NodeId.Null)); + + // The step that matters: the NodeId a caller was handed must resolve. + var model = nm.FindPredefinedNode(result.ModelUsed); + + Assert.Multiple(() => + { + Assert.That(model, Is.Not.Null); + Assert.That(model.ModelId, Is.Not.Null); + Assert.That(model.ModelId!.Value, Is.Not.Empty); + Assert.That(model.Digest, Is.Not.Null, "the walk terminates at a digest"); + Assert.That(model.DigestAlgorithm, Is.Not.Null); + }); + } + + [Test] + public async Task EveryPublishedModelIsReachableFromTheSourceItCameFromAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + NodeId modelId = TargetOf( + nm, deployment, Opc.Ua.AI.ReferenceTypeIds.UsesModel); + + Assert.That(modelId, Is.Not.EqualTo(NodeId.Null), "UsesModel must be present"); + + var model = nm.FindPredefinedNode(modelId); + NodeId sourceId = TargetOf( + nm, model, Opc.Ua.AI.ReferenceTypeIds.ImportedFrom); + + Assert.That( + sourceId, + Is.Not.EqualTo(NodeId.Null), + "a model this Server did not author must say where it came from"); + + var source = nm.FindPredefinedNode(sourceId); + + Assert.Multiple(() => + { + Assert.That(source, Is.Not.Null); + Assert.That(source.EndpointUri, Is.Not.Null); + Assert.That(source.SourceId!.Value, Is.Not.Empty); + }); + } + + [Test] + public async Task ADigestIsEmptyRatherThanInventedWhenTheBackendDeclaresNoneAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + var model = nm.FindPredefinedNode( + TargetOf(nm, deployment, Opc.Ua.AI.ReferenceTypeIds.UsesModel)); + + // A hosted endpoint that will not say which weights answered cannot be + // made to say so by hashing its name. A digest that looks like an + // artefact digest but is not one is worse than none, because it will be + // compared against a real one and appear to disagree. + Assert.Multiple(() => + { + Assert.That(model.Digest!.Value.Length, Is.Zero); + Assert.That(model.DigestAlgorithm!.Value, Is.Empty); + }); + } + + [Test] + public async Task ADeclaredDigestIsPublishedVerbatimAsync() + { + byte[] digest = [1, 2, 3, 4]; + + var backendOptions = new InferenceBackendOptions(); + backendOptions.Models.Add(new BackendModel + { + Publisher = "contoso", + Name = "weld-inspect", + Version = "2.1.0", + Digest = digest, + DigestAlgorithm = "SHA-256" + }); + + using AINodeManager nm = await AIServerTestHarness + .CreateAsync( + new InferenceBackends(new FakeInferenceBackend("primary")), + new AIOptions { EnableFallback = false }, + backendOptions) + .ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + var model = nm.FindPredefinedNode( + TargetOf(nm, deployment, Opc.Ua.AI.ReferenceTypeIds.UsesModel)); + + Assert.Multiple(() => + { + Assert.That(model.Digest!.Value.ToArray(), Is.EqualTo(digest)); + Assert.That(model.DigestAlgorithm!.Value, Is.EqualTo("SHA-256")); + Assert.That( + model.ModelId!.Value, + Is.EqualTo("contoso/weld-inspect:2.1.0"), + "identity is the publisher, name and version triple"); + }); + } + + private static Task CreateAsync() + { + return AIServerTestHarness.CreateAsync( + new InferenceBackends( + new FakeInferenceBackend("primary"), + new FakeInferenceBackend("fallback"))); + } + + /// + /// Follows one forward reference of the given type. + /// + private static NodeId TargetOf( + AINodeManager nm, + NodeState node, + ExpandedNodeId referenceTypeId) + { + var references = new List(); + node.GetReferences(nm.SystemContext, references); + + NodeId wanted = ExpandedNodeId.ToNodeId( + referenceTypeId, nm.SystemContext.NamespaceUris); + + foreach (IReference reference in references) + { + if (!reference.IsInverse && reference.ReferenceTypeId == wanted) + { + return ExpandedNodeId.ToNodeId( + reference.TargetId, nm.SystemContext.NamespaceUris); + } + } + + return NodeId.Null; + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/RestChatCompletionsBackendTests.cs b/tests/Opc.Ua.AI.Tests/RestChatCompletionsBackendTests.cs new file mode 100644 index 0000000000..6e4a639355 --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/RestChatCompletionsBackendTests.cs @@ -0,0 +1,686 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Net.Http.Headers; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using NUnit.Framework; +using Opc.Ua.AI.Inference; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Covers the OpenAI-compatible REST chat-completions backend. + /// + /// + /// This backend sits at the boundary a deployment usually discovers last: + /// model catalogues, credentials, throttling and timeout behavior all come from + /// a service outside the Server. The tests keep that boundary under a stub + /// , so they pin the wire decisions without a + /// network or a vendor account. + /// + [TestFixture] + public sealed class RestChatCompletionsBackendTests + { + [Test] + public async Task ListModelsUsesEndpointCatalogueAndAppliesFilterAndBound() + { + using var http = Http( + out StubHttpMessageHandler handler, + ModelList( + """ + {"data":[ + {"id":"alpha-small","owned_by":"endpoint-owner"}, + {"id":"beta-large","owned_by":"endpoint-owner"}]} + """), + ModelList( + """ + {"data":[ + {"id":"alpha-small","owned_by":"endpoint-owner"}, + {"id":"beta-large","owned_by":"endpoint-owner"}]} + """), + ModelList( + """ + {"data":[ + {"id":"alpha-small","owned_by":"endpoint-owner"}, + {"id":"beta-large","owned_by":"endpoint-owner"}]} + """)); + InferenceBackendOptions options = Options(); + options.Models.Add(new BackendModel + { + Name = "alpha-small", + Publisher = "configured-publisher", + Version = "configured-version", + Framework = "configured-framework" + }); + using var backend = Backend(options, http); + + IReadOnlyList all = await backend + .ListModelsAsync(null, 0, CancellationToken.None).ConfigureAwait(false); + IReadOnlyList filtered = await backend + .ListModelsAsync("configured", 0, CancellationToken.None).ConfigureAwait(false); + IReadOnlyList bounded = await backend + .ListModelsAsync(null, 1, CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(all, Has.Count.EqualTo(2)); + Assert.That(all[0].Name, Is.EqualTo("alpha-small")); + Assert.That(all[0].Publisher, Is.EqualTo("configured-publisher")); + Assert.That(all[0].Version, Is.EqualTo("configured-version")); + Assert.That(all[0].Framework, Is.EqualTo("configured-framework")); + Assert.That(all[1].Publisher, Is.EqualTo("endpoint-owner")); + Assert.That(filtered.Select(m => m.Name), Has.One.EqualTo("alpha-small")); + Assert.That(bounded, Has.Count.EqualTo(1)); + Assert.That(handler.Requests, Has.Count.EqualTo(3)); + Assert.That(handler.Requests.All(r => r.Method == HttpMethod.Get), Is.True); + }); + } + + [Test] + public async Task ListModelsFallsBackToConfigurationWhenEndpointListIsAbsentOrEmpty() + { + using var http = Http( + out _, + ModelList("""{"object":"list"}"""), + ModelList("""{"data":[]}""")); + InferenceBackendOptions options = Options(); + options.Models.Add(new BackendModel { Name = "configured-one", Publisher = "operator" }); + options.Models.Add(new BackendModel { Name = "configured-two", Publisher = "operator" }); + using var backend = Backend(options, http); + + IReadOnlyList absent = await backend + .ListModelsAsync(null, 0, CancellationToken.None).ConfigureAwait(false); + IReadOnlyList empty = await backend + .ListModelsAsync(null, 0, CancellationToken.None).ConfigureAwait(false); + + // An endpoint that declines to publish its catalogue should not erase + // the operator's configured catalogue from the address space. + Assert.Multiple(() => + { + Assert.That(string.Join("|", absent.Select(m => m.Name)), Is.EqualTo("configured-one|configured-two")); + Assert.That(string.Join("|", empty.Select(m => m.Name)), Is.EqualTo("configured-one|configured-two")); + }); + } + + [Test] + public async Task ListModelsFallsBackToConfigurationWhenEndpointIsUnreachableOrTimesOut() + { + using var http = Http( + out _, + (_, _) => throw new HttpRequestException("synthetic route failure"), + (_, _) => throw new TaskCanceledException("synthetic timeout")); + InferenceBackendOptions options = Options(); + options.Models.Add(new BackendModel { Name = "configured", Publisher = "operator" }); + using var backend = Backend(options, http); + + IReadOnlyList unreachable = await backend + .ListModelsAsync(null, 0, CancellationToken.None).ConfigureAwait(false); + IReadOnlyList timedOut = await backend + .ListModelsAsync(null, 0, CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(unreachable.Select(m => m.Name), Has.One.EqualTo("configured")); + Assert.That(timedOut.Select(m => m.Name), Has.One.EqualTo("configured")); + }); + } + + [Test] + public async Task InvokeSendsTheOpaquePayloadAndReportsWhatTheEndpointAnswered() + { + using var http = Http( + out StubHttpMessageHandler handler, + Json( + HttpStatusCode.OK, + """ + { + "model":"served-model", + "usage":{"prompt_tokens":11,"completion_tokens":22,"total_tokens":40}, + "choices":[{"finish_reason":"length"}] + } + """)); + using var backend = Backend(Options(), http); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"ping"}]}""", "asked-model"), + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.True); + Assert.That(result.Payload.ToArray(), Is.EqualTo(ResponseBytes(handler)).AsCollection); + Assert.That(result.ContentType, Is.EqualTo("application/json")); + Assert.That(result.ModelUsed, Is.EqualTo("served-model"), + "A service can route to a different model than the one requested; " + + "callers need the model that actually answered."); + Assert.That(result.InputUnits, Is.EqualTo(11UL)); + Assert.That(result.OutputUnits, Is.EqualTo(22UL)); + Assert.That(result.TotalUnits, Is.EqualTo(40UL)); + Assert.That(result.Finish, Is.EqualTo(InferenceFinish.Length)); + Assert.That(handler.Requests[0].Method, Is.EqualTo(HttpMethod.Post)); + Assert.That(handler.Requests[0].Path, Is.EqualTo("/v1/chat/completions")); + Assert.That( + handler.Requests[0].Body, + Is.EqualTo("""{"messages":[{"role":"user","content":"ping"}]}""")); + Assert.That(handler.Requests[0].ContentType, Is.EqualTo("application/json")); + }); + } + + [Test] + public async Task InvokeReportsZeroUsageWhenEndpointReturnsNone() + { + using var http = Http( + out _, + (_, ct) => + { + ct.ThrowIfCancellationRequested(); + return Task.FromResult(Json(HttpStatusCode.OK, """{"choices":[{"finish_reason":"stop"}]}""")); + }); + using var backend = Backend(Options(), http); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"hi"}]}""", "asked-model"), + CancellationToken.None).ConfigureAwait(false); + + // Estimating usage a backend did not report would produce a number that + // looks metered and is not, and it would be billed against. + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.True); + Assert.That(result.ModelUsed, Is.EqualTo("asked-model")); + Assert.That(result.InputUnits, Is.Zero); + Assert.That(result.OutputUnits, Is.Zero); + Assert.That(result.TotalUnits, Is.Zero); + }); + } + + [Test] + public async Task InvokeReportsHttpFailureWithResponseBody() + { + using var http = Http(out _, Json(HttpStatusCode.InternalServerError, "synthetic backend fault")); + using var backend = Backend(Options(), http); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"hi"}]}"""), + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.False); + Assert.That(result.Finish, Is.EqualTo(InferenceFinish.Error)); + Assert.That(result.RetryAfter, Is.EqualTo(TimeSpan.Zero)); + Assert.That(result.Message, Does.Contain("HTTP 500")); + Assert.That(result.Message, Does.Contain("synthetic backend fault")); + }); + } + + [Test] + public async Task InvokeReportsRetryAfterForCapacityFailures() + { + HttpResponseMessage response = Json(HttpStatusCode.TooManyRequests, "synthetic capacity limit"); + response.Headers.RetryAfter = new RetryConditionHeaderValue(TimeSpan.FromSeconds(13)); + using var http = Http(out _, response); + using var backend = Backend(Options(), http); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"hi"}]}"""), + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.False); + Assert.That(result.Finish, Is.EqualTo(InferenceFinish.Error)); + Assert.That(result.RetryAfter, Is.EqualTo(TimeSpan.FromSeconds(13))); + Assert.That(result.Message, Does.Contain("HTTP 429")); + }); + } + + [Test] + public async Task InvokeReturnsCancelledWhenTheRequestTimeoutExpires() + { + using var http = Http( + out _, + async (_, ct) => + { + await Task.Delay(TimeSpan.FromMinutes(5), ct).ConfigureAwait(false); + return Json(HttpStatusCode.OK, """{"choices":[{"finish_reason":"stop"}]}"""); + }); + using var backend = Backend(Options(), http); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"hi"}]}""", timeout: TimeSpan.FromMilliseconds(20)), + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.False); + Assert.That(result.Finish, Is.EqualTo(InferenceFinish.Cancelled)); + Assert.That(result.Message, Does.Contain("timeout")); + }); + } + + [Test] + public void InvokePropagatesCallerCancellation() + { + using var cts = new CancellationTokenSource(); + using var http = Http( + out _, + (_, ct) => + { + cts.Cancel(); + throw new OperationCanceledException(ct); + }); + using var backend = Backend(Options(), http); + + Assert.That( + async () => await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"hi"}]}"""), + cts.Token).ConfigureAwait(false), + Throws.InstanceOf()); + } + + [Test] + public async Task InvokePassesMalformedSuccessfulBodyThroughWithoutInventingMetadata() + { + using var http = Http(out _, Json(HttpStatusCode.OK, "{ not json")); + using var backend = Backend(Options(), http); + + InferenceResult result = await backend.InvokeAsync( + Request("""{"messages":[{"role":"user","content":"hi"}]}""", "asked-model"), + CancellationToken.None).ConfigureAwait(false); + + // The implementation treats a 200 response body as the caller's payload + // even when the optional metadata envelope cannot be parsed. + Assert.Multiple(() => + { + Assert.That(result.Ok, Is.True); + Assert.That(Encoding.UTF8.GetString(result.Payload.Span), Is.EqualTo("{ not json")); + Assert.That(result.ModelUsed, Is.EqualTo("asked-model")); + Assert.That(result.InputUnits, Is.Zero); + Assert.That(result.OutputUnits, Is.Zero); + Assert.That(result.TotalUnits, Is.Zero); + Assert.That(result.Finish, Is.EqualTo(InferenceFinish.Stop)); + }); + } + + [Test] + public async Task ProbeReportsReachableHttpStatus() + { + using var http = Http(out _, Json(HttpStatusCode.OK, """{"data":[]}""")); + using var backend = Backend(Options(), http); + + BackendProbe probe = await backend.ProbeAsync(CancellationToken.None) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(probe.Reachable, Is.True); + Assert.That(probe.Throttled, Is.False); + Assert.That(probe.Detail, Is.EqualTo("HTTP 200.")); + }); + } + + [Test] + public async Task ProbeReportsUnreachableRatherThanThrowing() + { + using var http = Http( + out _, + (_, _) => throw new HttpRequestException("synthetic route failure")); + using var backend = Backend(Options(), http); + + BackendProbe probe = await backend.ProbeAsync(CancellationToken.None) + .ConfigureAwait(false); + + // A probe exists so a commissioning engineer learns the endpoint is + // wrong before a deployment depends on it, so it reports rather than + // throws. + Assert.Multiple(() => + { + Assert.That(probe.Reachable, Is.False); + Assert.That(probe.Throttled, Is.False); + Assert.That(probe.Detail, Does.Contain("synthetic route failure")); + }); + } + + [Test] + public async Task ProbeReportsThrottledAndRetryAfter() + { + HttpResponseMessage response = Json(HttpStatusCode.TooManyRequests, "synthetic capacity limit"); + response.Headers.RetryAfter = new RetryConditionHeaderValue(TimeSpan.FromSeconds(17)); + using var http = Http(out _, response); + using var backend = Backend(Options(), http); + + BackendProbe probe = await backend.ProbeAsync(CancellationToken.None) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(probe.Reachable, Is.True); + Assert.That(probe.Throttled, Is.True); + Assert.That(probe.RetryAfter, Is.EqualTo(TimeSpan.FromSeconds(17))); + Assert.That(probe.Detail, Does.Contain("capacity")); + }); + } + + [Test] + public async Task ProbeReportsTimeoutAsUnreachable() + { + using var http = Http(out _, (_, _) => throw new TaskCanceledException("synthetic timeout")); + using var backend = Backend(Options(), http); + + BackendProbe probe = await backend.ProbeAsync(CancellationToken.None) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(probe.Reachable, Is.False); + Assert.That(probe.Throttled, Is.False); + Assert.That(probe.Detail, Does.Contain("did not answer")); + }); + } + + [Test] + public async Task ApiKeyAuthenticationAddsConfiguredHeaderFromResolvedCredential() + { + var credentials = new StubCredentialResolver("synthetic-api-key-value-for-test-only"); + InferenceBackendOptions options = Options(); + options.Authentication = BackendAuthentication.ApiKey; + options.CredentialReference = "synthetic-api-key-reference"; + options.ApiKeyHeader = "x-test-api-key"; + using var http = Http(out StubHttpMessageHandler handler, Json(HttpStatusCode.OK, """{"data":[]}""")); + using var backend = Backend(options, http, credentials); + + await backend.ProbeAsync(CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(credentials.LastReference, Is.EqualTo("synthetic-api-key-reference")); + Assert.That( + handler.Requests[0].Headers["x-test-api-key"], + Has.One.EqualTo("synthetic-api-key-value-for-test-only")); + Assert.That(handler.Requests[0].Authorization, Is.Null); + }); + } + + [Test] + public async Task BearerStyleAuthenticationAddsAuthorizationHeader() + { + var bearerCredentials = new StubCredentialResolver("synthetic-bearer-token-for-test-only"); + InferenceBackendOptions bearer = Options(); + bearer.Authentication = BackendAuthentication.BearerToken; + bearer.CredentialReference = "synthetic-bearer-reference"; + using var bearerHttp = Http(out StubHttpMessageHandler bearerHandler, Json(HttpStatusCode.OK, "{}")); + using var bearerBackend = Backend(bearer, bearerHttp, bearerCredentials); + + var workloadCredentials = new StubCredentialResolver("synthetic-workload-token-for-test-only"); + InferenceBackendOptions workload = Options(); + workload.Authentication = BackendAuthentication.WorkloadIdentity; + workload.CredentialReference = "synthetic-workload-reference"; + using var workloadHttp = Http(out StubHttpMessageHandler workloadHandler, Json(HttpStatusCode.OK, "{}")); + using var workloadBackend = Backend(workload, workloadHttp, workloadCredentials); + + await bearerBackend.ProbeAsync(CancellationToken.None).ConfigureAwait(false); + await workloadBackend.ProbeAsync(CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(bearerHandler.Requests[0].Authorization, Is.EqualTo( + "Bearer synthetic-bearer-token-for-test-only")); + Assert.That(workloadHandler.Requests[0].Authorization, Is.EqualTo( + "Bearer synthetic-workload-token-for-test-only")); + }); + } + + [Test] + public async Task AnonymousAuthenticationSendsNoCredentialHeaders() + { + var credentials = new StubCredentialResolver("synthetic-credential-value-for-test-only"); + InferenceBackendOptions options = Options(); + options.Authentication = BackendAuthentication.Anonymous; + options.CredentialReference = "synthetic-reference"; + using var http = Http(out StubHttpMessageHandler handler, Json(HttpStatusCode.OK, "{}")); + using var backend = Backend(options, http, credentials); + + await backend.ProbeAsync(CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(credentials.LastReference, Is.EqualTo("synthetic-reference")); + Assert.That(handler.Requests[0].Authorization, Is.Null); + Assert.That(handler.Requests[0].Headers.ContainsKey("api-key"), Is.False); + }); + } + + [Test] + public void ConstructorRefusesNullDependencies() + { + InferenceBackendOptions options = Options(); + var credentials = new StubCredentialResolver(null); + + Assert.Multiple(() => + { + Assert.That( + () => new RestChatCompletionsBackend( + null!, + credentials, + NullLogger.Instance), + Throws.ArgumentNullException); + Assert.That( + () => new RestChatCompletionsBackend( + options, + null!, + NullLogger.Instance), + Throws.ArgumentNullException); + Assert.That( + () => new RestChatCompletionsBackend(options, credentials, null!), + Throws.ArgumentNullException); + }); + } + + private static RestChatCompletionsBackend Backend( + InferenceBackendOptions options, + HttpClient http, + ICredentialResolver? credentials = null) + { + return new RestChatCompletionsBackend( + options, + credentials ?? new StubCredentialResolver(null), + NullLogger.Instance, + http); + } + + private static HttpClient Http( + out StubHttpMessageHandler handler, + params HttpResponseMessage[] responses) + { + handler = new StubHttpMessageHandler( + responses.Select< + HttpResponseMessage, + Func>>( + response => (_, _) => Task.FromResult(response)).ToArray()); + return new HttpClient(handler) { BaseAddress = new Uri("https://unit.test/") }; + } + + private static HttpClient Http( + out StubHttpMessageHandler handler, + params Func>[] responders) + { + handler = new StubHttpMessageHandler(responders); + return new HttpClient(handler) { BaseAddress = new Uri("https://unit.test/") }; + } + + private static InferenceBackendOptions Options() + { + return new InferenceBackendOptions + { + EndpointUri = "https://unit.test/", + ChatCompletionsPath = "v1/chat/completions", + ProbePath = "v1/models" + }; + } + + private static InferenceRequest Request( + string body, + string model = "", + TimeSpan timeout = default) + { + return new InferenceRequest + { + Model = model, + Payload = Encoding.UTF8.GetBytes(body), + ContentType = "application/json", + Timeout = timeout + }; + } + + private static HttpResponseMessage Json(HttpStatusCode status, string body) + { + return new HttpResponseMessage(status) + { + Content = new StringContent(body, Encoding.UTF8, "application/json") + }; + } + + private static HttpResponseMessage ModelList(string body) + { + return Json(HttpStatusCode.OK, body); + } + + private static byte[] ResponseBytes(StubHttpMessageHandler handler) + { + return handler.LastResponseBody ?? Array.Empty(); + } + + private sealed class StubHttpMessageHandler : HttpMessageHandler + { + private readonly Queue>> m_responders; + + public StubHttpMessageHandler( + IEnumerable>> responders) + { + m_responders = new Queue>>( + responders); + } + + public List Requests { get; } = []; + + public byte[]? LastResponseBody { get; private set; } + + protected override async Task SendAsync( + HttpRequestMessage request, + CancellationToken cancellationToken) + { + Requests.Add(await RequestSnapshot.CreateAsync(request, cancellationToken).ConfigureAwait(false)); + if (m_responders.Count == 0) + { + throw new InvalidOperationException("No synthetic response was configured."); + } + + HttpResponseMessage response = await m_responders + .Dequeue()(request, cancellationToken).ConfigureAwait(false); + LastResponseBody = response.Content is null + ? [] + : await response.Content.ReadAsByteArrayAsync(cancellationToken).ConfigureAwait(false); + return response; + } + } + + private sealed record RequestSnapshot + { + public HttpMethod Method { get; init; } = HttpMethod.Get; + + public string Path { get; init; } = string.Empty; + + public string? Body { get; init; } + + public string? ContentType { get; init; } + + public Dictionary Headers { get; init; } = new(StringComparer.OrdinalIgnoreCase); + + public string? Authorization { get; init; } + + public static async Task CreateAsync( + HttpRequestMessage request, + CancellationToken cancellationToken) + { + var headers = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (KeyValuePair> header in request.Headers) + { + headers.Add(header.Key, header.Value.ToArray()); + } + + if (request.Content != null) + { + foreach (KeyValuePair> header in request.Content.Headers) + { + headers.Add(header.Key, header.Value.ToArray()); + } + } + + return new RequestSnapshot + { + Method = request.Method, + Path = request.RequestUri?.AbsolutePath ?? string.Empty, + Body = request.Content is null + ? null + : await request.Content.ReadAsStringAsync(cancellationToken).ConfigureAwait(false), + ContentType = request.Content?.Headers.ContentType?.MediaType, + Headers = headers, + Authorization = request.Headers.Authorization?.ToString() + }; + } + } + + private sealed class StubCredentialResolver : ICredentialResolver + { + private readonly string? m_value; + + public StubCredentialResolver(string? value) + { + m_value = value; + } + + public string? LastReference { get; private set; } + + public ValueTask ResolveAsync(string reference, CancellationToken ct) + { + LastReference = reference; + return new ValueTask(m_value); + } + } + } +} diff --git a/tests/Opc.Ua.AI.Tests/TransferTests.cs b/tests/Opc.Ua.AI.Tests/TransferTests.cs new file mode 100644 index 0000000000..beac90016b --- /dev/null +++ b/tests/Opc.Ua.AI.Tests/TransferTests.cs @@ -0,0 +1,372 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.AI; +using Opc.Ua.AI.Inference; +using Opc.Ua.AI.Server; + +namespace Opc.Ua.AI.Tests +{ + /// + /// Verifies the chunked path for payloads too large to pass inline. + /// + /// + /// The property worth protecting is that nothing a caller is entitled to + /// changes because the bytes arrived in chunks. A large payload is a transport + /// concern; if the transfer path dropped ModelUsed the audit trail would + /// have a hole exactly where the largest requests are. + /// + [TestFixture] + [Category("AIModelManagement")] + [SetCulture("en-us")] + [SetUICulture("en-us")] + public sealed class TransferTests + { + [Test] + public async Task AnOversizePayloadIsRefusedInlineAndNamesTheTransferAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + byte[] oversize = new byte[InlineLimit + 1]; + + InvokeMethodStateResult result = await deployment.Invoke!.OnCallAsync!( + nm.SystemContext, + deployment.Invoke, + nm.PrimaryDeploymentId, + ByteString.From(oversize), + string.Empty, + "application/octet-stream", + ArrayOf.Empty, + 5000, + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.TransferRequired, Is.True); + Assert.That(result.Transfer, Is.Not.EqualTo(NodeId.Null)); + + // The refusal names the transfer that will carry it, so a caller + // that reads the answer can act on it without a second round trip + // to work out what to do next. + Assert.That( + nm.FindPredefinedNode(result.Transfer), + Is.Not.Null); + }); + } + + [Test] + public async Task ATransferCarriesThePayloadAndReportsTheModelUsedAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + + BeginTransferMethodStateResult begun = await deployment.BeginTransfer!.OnCallAsync!( + nm.SystemContext, + deployment.BeginTransfer, + nm.PrimaryDeploymentId, + "application/json", + 1024, + CancellationToken.None).ConfigureAwait(false); + + Assert.That(begun.Accepted, Is.True); + + var transfer = nm.FindPredefinedNode(begun.Transfer); + + WriteRequest(nm, transfer, "{\"prompt\":\"hello\"}"); + + ExecuteMethodStateResult executed = await transfer.Execute!.OnCallAsync!( + nm.SystemContext, + transfer.Execute, + transfer.NodeId, + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(executed.Accepted, Is.True); + Assert.That(transfer.State!.Value, Is.EqualTo(TransferStateEnum.Completed)); + + // The same output an inline call would have produced. + Assert.That(transfer.ModelUsed!.Value, Is.Not.EqualTo(NodeId.Null)); + Assert.That(transfer.ResponseContentType!.Value, Is.EqualTo("application/json")); + Assert.That(ReadResponse(nm, transfer), Does.Contain("primary")); + }); + } + + [Test] + public async Task ATransferLargerThanTheServerAcceptsIsRefusedBeforeAnyBytesArriveAsync() + { + using AINodeManager nm = await CreateAsync( + new AIOptions { MaxTransferSize = 4096 }) + .ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + + BeginTransferMethodStateResult begun = await deployment.BeginTransfer!.OnCallAsync!( + nm.SystemContext, + deployment.BeginTransfer, + nm.PrimaryDeploymentId, + "application/json", + 4097, + CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(begun.Accepted, Is.False); + Assert.That(begun.Transfer, Is.EqualTo(NodeId.Null)); + Assert.That( + (StatusCode)begun.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadRequestTooLarge)); + }); + } + + [Test] + public async Task TheResponseFileIsNotWritableAsync() + { + using AINodeManager nm = await CreateAsync().ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + + BeginTransferMethodStateResult begun = await deployment.BeginTransfer!.OnCallAsync!( + nm.SystemContext, + deployment.BeginTransfer, + nm.PrimaryDeploymentId, + "application/json", + 16, + CancellationToken.None).ConfigureAwait(false); + + var transfer = nm.FindPredefinedNode(begun.Transfer); + + uint handle = 0; + const byte writeEraseExisting = 6; + + ServiceResult opened = transfer.Response!.Open!.OnCall!( + nm.SystemContext, + transfer.Response.Open, + transfer.Response.NodeId, + writeEraseExisting, + ref handle); + + // A client that could overwrite a model's answer could forge one. + Assert.That(ServiceResult.IsGood(opened), Is.False); + } + + [Test] + public async Task ATransferAbortedMidInferenceDoesNotWriteIntoDisposedBuffersAsync() + { + var backend = new BlockingFakeBackend(); + + using AINodeManager nm = await AIServerTestHarness + .CreateAsync( + new InferenceBackends(backend), + new AIOptions { EnableFallback = false }, + new InferenceBackendOptions { MaxInlinePayloadSize = InlineLimit }) + .ConfigureAwait(false); + + var deployment = nm.FindPredefinedNode(nm.PrimaryDeploymentId); + + BeginTransferMethodStateResult begun = await deployment.BeginTransfer!.OnCallAsync!( + nm.SystemContext, + deployment.BeginTransfer, + nm.PrimaryDeploymentId, + "application/json", + 64, + CancellationToken.None).ConfigureAwait(false); + + var transfer = nm.FindPredefinedNode(begun.Transfer); + WriteRequest(nm, transfer, "{}"); + + // Execute starts and blocks inside the backend. + Task executing = transfer.Execute!.OnCallAsync!( + nm.SystemContext, + transfer.Execute, + transfer.NodeId, + CancellationToken.None).AsTask(); + + await backend.Entered.ConfigureAwait(false); + + // Abort while it is in flight. This removes the entry and disposes the + // buffers the completing call is about to write into. + transfer.Abort!.OnCallMethod2Async!( + nm.SystemContext, + transfer.Abort, + transfer.NodeId, + ArrayOf.Empty, + [], + CancellationToken.None).AsTask().Wait(TimeSpan.FromSeconds(5)); + + backend.Release(); + + ExecuteMethodStateResult result = await executing.ConfigureAwait(false); + + // The answer is dropped rather than written into a disposed buffer, + // which is what the caller that aborted was asking for. Before the + // liveness re-check this threw ObjectDisposedException out of a Method + // call, which no client can do anything sensible with. + Assert.Multiple(() => + { + Assert.That(result.Accepted, Is.False); + Assert.That( + (StatusCode)result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInvalidState)); + }); + } + + /// + /// A backend that lets a test hold an inference open. + /// + private sealed class BlockingFakeBackend : IInferenceBackend + { + private readonly TaskCompletionSource m_entered = + new(TaskCreationOptions.RunContinuationsAsynchronously); + + private readonly TaskCompletionSource m_release = + new(TaskCreationOptions.RunContinuationsAsynchronously); + + public Task Entered => m_entered.Task; + + public void Release() + { + m_release.TrySetResult(); + } + + public InferenceSite Site => InferenceSite.OnServer; + + public ValueTask> ListModelsAsync( + string? filter, uint maxResults, CancellationToken ct) + { + return ValueTask.FromResult>([]); + } + + public async ValueTask InvokeAsync( + InferenceRequest request, CancellationToken ct) + { + m_entered.TrySetResult(); + await m_release.Task.ConfigureAwait(false); + + return new InferenceResult + { + Ok = true, + Payload = Encoding.UTF8.GetBytes("{\"ok\":true}"), + ContentType = "application/json" + }; + } + + public ValueTask ProbeAsync(CancellationToken ct) + { + return ValueTask.FromResult(new BackendProbe { Reachable = true }); + } + } + + private const uint InlineLimit = 512; + + private static Task CreateAsync( + AIOptions? options = null) + { + return AIServerTestHarness.CreateAsync( + new InferenceBackends(new FakeInferenceBackend("primary")), + options ?? new AIOptions { EnableFallback = false }, + new InferenceBackendOptions { MaxInlinePayloadSize = InlineLimit }); + } + + private static void WriteRequest( + AINodeManager nm, + InferenceTransferState transfer, + string body) + { + uint handle = 0; + const byte writeEraseExisting = 6; + + ServiceResult opened = transfer.Request!.Open!.OnCall!( + nm.SystemContext, + transfer.Request.Open, + transfer.Request.NodeId, + writeEraseExisting, + ref handle); + + Assert.That(ServiceResult.IsGood(opened), Is.True); + + ServiceResult written = transfer.Request.Write!.OnCall!( + nm.SystemContext, + transfer.Request.Write, + transfer.Request.NodeId, + handle, + ByteString.From(Encoding.UTF8.GetBytes(body))); + + Assert.That(ServiceResult.IsGood(written), Is.True); + + _ = transfer.Request.Close!.OnCall!( + nm.SystemContext, + transfer.Request.Close, + transfer.Request.NodeId, + handle); + } + + private static string ReadResponse( + AINodeManager nm, + InferenceTransferState transfer) + { + uint handle = 0; + const byte read = 1; + + _ = transfer.Response!.Open!.OnCall!( + nm.SystemContext, + transfer.Response.Open, + transfer.Response.NodeId, + read, + ref handle); + + ByteString data = default; + + _ = transfer.Response.Read!.OnCall!( + nm.SystemContext, + transfer.Response.Read, + transfer.Response.NodeId, + handle, + 4096, + ref data); + + _ = transfer.Response.Close!.OnCall!( + nm.SystemContext, + transfer.Response.Close, + transfer.Response.NodeId, + handle); + + return Encoding.UTF8.GetString(data.Span); + } + } +} diff --git a/tests/Opc.Ua.Aot.Tests/McpAotTests.cs b/tests/Opc.Ua.Aot.Tests/McpAotTests.cs index ac58806565..2dd8c0efde 100644 --- a/tests/Opc.Ua.Aot.Tests/McpAotTests.cs +++ b/tests/Opc.Ua.Aot.Tests/McpAotTests.cs @@ -27,6 +27,8 @@ * http://opcfoundation.org/License/MIT/1.00/ * ======================================================================*/ +#nullable enable + using System.Text.Json; using Microsoft.Extensions.DependencyInjection; using ModelContextProtocol.Server; diff --git a/tests/Opc.Ua.Client.Tests/Subscription/Fakes/FakeSubscriptionManagerContext.cs b/tests/Opc.Ua.Client.Tests/Subscription/Fakes/FakeSubscriptionManagerContext.cs index 1327cdbfe5..732e3a05f0 100644 --- a/tests/Opc.Ua.Client.Tests/Subscription/Fakes/FakeSubscriptionManagerContext.cs +++ b/tests/Opc.Ua.Client.Tests/Subscription/Fakes/FakeSubscriptionManagerContext.cs @@ -144,11 +144,41 @@ public ValueTask DeleteSubscriptionsAsync( new DeleteSubscriptionsResponse()); } + /// + /// Identifiers the fake session claims outside the manager's registry, + /// standing in for subscriptions created through the classic API. + /// + public HashSet SessionOwnedSubscriptionIds { get; } = []; + + /// + public int SessionSubscriptionCount => SessionOwnedSubscriptionIds.Count; + + /// Recorded dispatches to session-owned subscriptions. + public int SessionDispatchCount => Volatile.Read(ref m_sessionDispatchCount); + + public bool TryDispatchToSessionSubscription( + uint subscriptionId, + NotificationMessage message, + ArrayOf availableSequenceNumbers, + ArrayOf stringTable, + bool moreNotifications) + { + if (!SessionOwnedSubscriptionIds.Contains(subscriptionId)) + { + return false; + } + Interlocked.Increment(ref m_sessionDispatchCount); + return true; + } + + private int m_sessionDispatchCount; + /// /// Appends a recorded call. Publish workers run on background /// threads while the test thread inspects the recordings, so the /// backing lists must never be mutated without synchronization. /// + /// /// /// private void Record(List recordings, T call) @@ -163,6 +193,7 @@ private void Record(List recordings, T call) /// Returns a stable copy of a recording so assertions cannot /// observe a list that is being appended to concurrently. /// + /// /// private IReadOnlyList Snapshot(List recordings) { @@ -175,6 +206,7 @@ private IReadOnlyList Snapshot(List recordings) /// /// Reads the number of recorded calls without allocating a snapshot. /// + /// /// private int Count(List recordings) { diff --git a/tests/Opc.Ua.Client.Tests/Subscription/SubscriptionManagerTests.cs b/tests/Opc.Ua.Client.Tests/Subscription/SubscriptionManagerTests.cs index 11148a472f..1fdf109941 100644 --- a/tests/Opc.Ua.Client.Tests/Subscription/SubscriptionManagerTests.cs +++ b/tests/Opc.Ua.Client.Tests/Subscription/SubscriptionManagerTests.cs @@ -988,6 +988,106 @@ await WaitUntilAsync(() => session.DeleteCallsCount > 0, testCt) } } + /// + /// A subscription created through the classic Session.AddSubscription + /// API is unknown to this manager's registry, but it is live and owned by + /// the application. Deleting it as abandoned takes it down on the server + /// while the caller still believes it is streaming, which shows up as a + /// twin that silently stops updating. + /// + [Test] + [CancelAfter(30_000)] + public async Task PublishWorkerKeepsSubscriptionOwnedBySessionAsync( + CancellationToken testCt) + { + using ILoggerFactory loggerFactory = LoggerFactory.Create(b => b.AddDebug()); + var session = new FakeSubscriptionManagerContext(); + OptionsMonitor createdOptions = + OptionsFactory.Create(); + + var created = new FakeManagedSubscription { Id = 1u, Created = true }; + + var sut = new SubscriptionManager(session, + loggerFactory, DiagnosticsMasks.None); + await using (sut.ConfigureAwait(false)) + { + session.CreateSubscriptionFactory = (handler, options, queue) => created; + sut.Add(m_mockNotificationDataHandler.Object, createdOptions); + + // The session holds this one outside the manager's registry. + session.SessionOwnedSubscriptionIds.Add(4242u); + + int publishCount = 0; + session.OnPublishAsync = (h, a, ct) => + { + Interlocked.Increment(ref publishCount); + return new ValueTask( + CreatePublishResponse(4242u, h.RequestHandle)); + }; + + sut.MinPublishWorkerCount = 1; + sut.MaxPublishWorkerCount = 1; + sut.Resume(); + + await WaitUntilAsync(() => Volatile.Read(ref publishCount) >= 5, + testCt).ConfigureAwait(false); + + Assert.That(session.DeleteCalls, Is.Empty, + "A subscription the session owns must never be deleted as abandoned."); + Assert.That(session.SessionDispatchCount, Is.GreaterThan(0), + "Notifications for a session-owned subscription must be delivered to it."); + } + } + + /// + /// A session with only a classic subscription still needs a Publish worker. + /// The manager does not own that subscription, but it owns the shared Publish + /// pipeline that dispatches responses to it. + /// + [Test] + [CancelAfter(30_000)] + public async Task ClassicSessionSubscriptionStartsPublishWorkerAsync( + CancellationToken testCt) + { + ILoggerFactory loggerFactory = m_telemetry.LoggerFactory; + var session = new FakeSubscriptionManagerContext(); + session.SessionOwnedSubscriptionIds.Add(4242u); + var publishSeen = new TaskCompletionSource( + TaskCreationOptions.RunContinuationsAsynchronously); + session.OnPublishAsync = (header, acknowledgements, ct) => + { + publishSeen.TrySetResult(true); + return new ValueTask( + CreatePublishResponse(4242u, header.RequestHandle)); + }; + + var sut = new SubscriptionManager( + session, + loggerFactory, + DiagnosticsMasks.None) + { + MinPublishWorkerCount = 1, + MaxPublishWorkerCount = 1 + }; + await using (sut.ConfigureAwait(false)) + { + sut.Resume(); + sut.Update(); + + await publishSeen.Task.WaitAsync(testCt).ConfigureAwait(false); + await WaitUntilAsync( + () => session.SessionDispatchCount > 0, + testCt).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(sut.Count, Is.Zero); + Assert.That(sut.PublishWorkerCount, Is.EqualTo(1)); + Assert.That(session.DeleteCalls, Is.Empty); + }); + } + } + /// /// While an identifier stays unresolved the server keeps answering with /// the same undeliverable response. The worker must back off instead of diff --git a/tests/Opc.Ua.OpenUsd.Tests/OpenUsdTranslationProfileTests.cs b/tests/Opc.Ua.OpenUsd.Tests/OpenUsdTranslationProfileTests.cs new file mode 100644 index 0000000000..1e2120a7f4 --- /dev/null +++ b/tests/Opc.Ua.OpenUsd.Tests/OpenUsdTranslationProfileTests.cs @@ -0,0 +1,92 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.OpenUsd.Client; + +namespace Opc.Ua.OpenUsd.Tests +{ + /// + /// Pins which source values the connector's §5.8 translation profile accepts. + /// + /// + /// This matters more than it looks. The profile fails closed - an unaccepted value + /// leaves the target unresolved with no error anywhere - so a server publishing a + /// position in a shape the profile does not take produces a viewport that silently + /// never moves the prim, while every subscription counter says the data is flowing. + /// + [TestFixture] + [Category("OpenUsd")] + [Parallelizable] + public sealed class OpenUsdTranslationProfileTests + { + [Test] + public void StructuredCartesianCoordinatesAreAccepted() + { + var binding = new OpenUsdConnector.BindingInfo + { + Kind = OpenUsdRenderTargetKind.Translation, + Scale = 1.0, + Offset = 0.0 + }; + var value = new Variant(new ExtensionObject(new ThreeDCartesianCoordinates + { + X = 1.5, + Y = -2.5, + Z = 3.5 + })); + + Variant converted = OpenUsdConnector.Convert(binding, value); + + Assert.That(converted.IsNull, Is.False, + "A structured 3D coordinate is the source shape the translation profile is " + + "defined for; leaving it unresolved would stop any prim following it."); + Assert.That(converted.ToString(), Does.Contain("1.5")); + } + + [Test] + public void PlainDoubleArrayIsNotAccepted() + { + var binding = new OpenUsdConnector.BindingInfo + { + Kind = OpenUsdRenderTargetKind.Translation, + Scale = 1.0, + Offset = 0.0 + }; + + Variant converted = OpenUsdConnector.Convert(binding, new Variant(new[] { 1.5, -2.5, 3.5 })); + + Assert.That(converted.IsNull, Is.True, + "A bare double[3] is not a structured 3D source. This is the behaviour that " + + "silently stopped the bin-picking parts moving, so it is pinned here to " + + "make the requirement visible rather than surprising."); + } + } +} diff --git a/tests/Opc.Ua.Robotics.Intent.Tests/RobotIntentMcpPalletScenarioTests.cs b/tests/Opc.Ua.Robotics.Intent.Tests/RobotIntentMcpPalletScenarioTests.cs index 67e625cba3..ef079848be 100644 --- a/tests/Opc.Ua.Robotics.Intent.Tests/RobotIntentMcpPalletScenarioTests.cs +++ b/tests/Opc.Ua.Robotics.Intent.Tests/RobotIntentMcpPalletScenarioTests.cs @@ -29,7 +29,6 @@ #if NET10_0 using System; -using System.Globalization; using System.Linq; using System.Net; using System.Net.Sockets; @@ -92,7 +91,7 @@ public async Task PalletStackingScenarioUsesMcpToolsAndPublishesPayloadMotion() .SubmitLinearMoveAsync( robotics, controllerId, - LinearMoveJson("refused-without-authority", 0.12, 0.16, 0.28), + LinearMoveInput("refused-without-authority", 0.12, 0.16, 0.28), kAgentSession, CancellationToken.None) .ConfigureAwait(false); @@ -115,7 +114,7 @@ await RoboticsControlTools .SubmitPickAsync( robotics, controllerId, - PickJson("direct-pick-slot-01", bin.NodeId, tool.NodeId), + PickInput("direct-pick-slot-01", bin.NodeId, tool.NodeId), kAgentSession, CancellationToken.None) .ConfigureAwait(false); @@ -131,8 +130,12 @@ await RoboticsControlTools .SubmitJointMoveAsync( robotics, controllerId, - "{\"intentId\":\"direct-stack-slot-01\",\"jointTargets\":[0.1,-1.0,1.5,-0.9,0.75,0.0]," + - "\"blockingMode\":\"None\"}", + new JointMoveIntentInput + { + IntentId = "direct-stack-slot-01", + JointTargets = [0.1, -1.0, 1.5, -0.9, 0.75, 0.0], + BlockingMode = BlockingModeEnum.None + }, info.AxisCount, kAgentSession, CancellationToken.None) @@ -152,7 +155,7 @@ await RoboticsControlTools .SubmitPlaceAsync( robotics, controllerId, - PlaceJson("direct-place-slot-01", fixtureLocation.NodeId, tool.NodeId), + PlaceInput("direct-place-slot-01", fixtureLocation.NodeId, tool.NodeId), kAgentSession, CancellationToken.None) .ConfigureAwait(false); @@ -169,7 +172,12 @@ await RoboticsControlTools .SubmitWaitAsync( robotics, controllerId, - "{\"intentId\":\"pause-resume-cancel\",\"duration\":1500,\"blockingMode\":\"None\"}", + new WaitIntentInput + { + IntentId = "pause-resume-cancel", + Duration = 1500, + BlockingMode = BlockingModeEnum.None + }, kAgentSession, CancellationToken.None) .ConfigureAwait(false); @@ -177,8 +185,13 @@ await RoboticsControlTools IntentCommandOutcome pause = await RoboticsControlTools .PauseAsync(robotics, controllerId, kAgentSession, CancellationToken.None) .ConfigureAwait(false); - ArrayOf operationsWhilePaused = await RoboticsMonitoringTools - .ListOperationsAsync(robotics, controllerId, kAgentSession, CancellationToken.None) + OperationListResult operationsWhilePaused = await RoboticsMonitoringTools + .ListOperationsAsync( + robotics, + controllerId, + query: null, + kAgentSession, + CancellationToken.None) .ConfigureAwait(false); IntentCommandOutcome resume = await RoboticsControlTools .ResumeAsync(robotics, controllerId, kAgentSession, CancellationToken.None) @@ -198,7 +211,12 @@ await RoboticsControlTools pausable, kAgentSession).ConfigureAwait(false); IntentSubmissionResult retryRefusal = await RoboticsControlTools - .RetryAsync(robotics, controllerId, refusedWithoutAuthority.IntentId, kAgentSession, CancellationToken.None) + .RetryAsync( + robotics, + controllerId, + refusedWithoutAuthority.IntentId, + kAgentSession, + CancellationToken.None) .ConfigureAwait(false); MissionSubmissionResult stackMission = await RoboticsMissionTools @@ -207,7 +225,7 @@ await RoboticsControlTools controllerId, "stack-slot-02", 0, - StackingMissionStepsJson(bin.NodeId, fixtureLocation.NodeId, tool.NodeId), + StackingMissionSteps(bin.NodeId, fixtureLocation.NodeId, tool.NodeId), null, "stack pallet slot 02", kAgentSession, @@ -222,15 +240,25 @@ await RoboticsControlTools controllerId, "cancelled-pallet-demo", 0, - "[{\"stepId\":\"wait\",\"released\":true,\"intent\":{\"kind\":\"wait\"," + - "\"intentId\":\"cancelled-pallet-demo-wait\",\"duration\":1000}}]", + [ + WaitStep( + "wait", + "cancelled-pallet-demo-wait", + duration: 1000, + released: true) + ], null, "cancelled pallet demonstration", kAgentSession, CancellationToken.None) .ConfigureAwait(false); - ArrayOf missionsBeforeCancel = await RoboticsMonitoringTools - .ListMissionsAsync(robotics, controllerId, kAgentSession, CancellationToken.None) + MissionListResult missionsBeforeCancel = await RoboticsMonitoringTools + .ListMissionsAsync( + robotics, + controllerId, + query: null, + kAgentSession, + CancellationToken.None) .ConfigureAwait(false); MissionUpdateOutcome update = await RoboticsMissionTools .UpdateMissionAsync( @@ -238,10 +266,18 @@ await RoboticsControlTools controllerId, "cancelled-pallet-demo", 1, - "[{\"stepId\":\"wait\",\"released\":true,\"intent\":{\"kind\":\"wait\"," + - "\"intentId\":\"cancelled-pallet-demo-wait\",\"duration\":1000}}," + - "{\"stepId\":\"replacement\",\"released\":false,\"intent\":{\"kind\":\"wait\"," + - "\"intentId\":\"cancelled-pallet-demo-replacement\",\"duration\":100}}]", + [ + WaitStep( + "wait", + "cancelled-pallet-demo-wait", + duration: 1000, + released: true), + WaitStep( + "replacement", + "cancelled-pallet-demo-replacement", + duration: 100, + released: false) + ], kAgentSession, CancellationToken.None) .ConfigureAwait(false); @@ -277,13 +313,13 @@ await RoboticsControlTools Assert.That(placed.Result.State, Is.EqualTo(ExecutionStateEnum.Succeeded)); Assert.That(firstSlotFilled, Is.True); Assert.That(pause.Accepted, Is.True); - Assert.That(operationsWhilePaused, Is.Not.Empty); + Assert.That(operationsWhilePaused.Returned, Is.GreaterThan(0)); Assert.That(resume.Accepted, Is.True); Assert.That(cancelIntent.Accepted, Is.True); Assert.That(cancelled.Result.State, Is.EqualTo(ExecutionStateEnum.Cancelled)); Assert.That(retryRefusal.Accepted, Is.False); Assert.That(missionToCancel.Accepted, Is.True); - Assert.That(missionsBeforeCancel.Count, Is.GreaterThanOrEqualTo(0)); + Assert.That(missionsBeforeCancel.Total, Is.GreaterThan(0)); Assert.That(update.Result, Is.EqualTo(MissionUpdateResultEnum.Accepted)); Assert.That(cancelMission.Accepted, Is.True); Assert.That(cancelAll, Is.GreaterThanOrEqualTo(0)); @@ -310,9 +346,15 @@ private static void AssertAgentCanPlanStacking( Assert.Multiple(() => { - Assert.That(info.Lookups.Locations.ToArray()!.Select(entry => entry.Name), Does.Contain("Bin")); - Assert.That(info.Lookups.Locations.ToArray()!.Select(entry => entry.Name), Does.Contain("Fixture")); - Assert.That(info.Lookups.Outputs.ToArray()!.Select(entry => entry.Name), Does.Contain("HeldPartPosition")); + Assert.That( + info.Lookups.Locations.ToArray()!.Select(entry => entry.Name), + Does.Contain("Bin")); + Assert.That( + info.Lookups.Locations.ToArray()!.Select(entry => entry.Name), + Does.Contain("Fixture")); + Assert.That( + info.Lookups.Outputs.ToArray()!.Select(entry => entry.Name), + Does.Contain("HeldPartPosition")); Assert.That( info.Lookups.Outputs.ToArray()!.Select(entry => entry.Name), Does.Contain("PayloadSlot01Filled")); @@ -484,80 +526,102 @@ private static RobotIntentNodeLookupEntry Find( return entry!; } - private static string LinearMoveJson(string intentId, double x, double y, double z) + private static LinearMoveIntentInput LinearMoveInput( + string intentId, + double x, + double y, + double z) { - // JSON numbers are culture-invariant. Interpolating a double directly would emit a - // comma decimal separator under a culture such as de-DE, turning a three-element - // position into a six-element one that no longer describes the requested pose. - string px = x.ToString(CultureInfo.InvariantCulture); - string py = y.ToString(CultureInfo.InvariantCulture); - string pz = z.ToString(CultureInfo.InvariantCulture); - return $$""" + return new LinearMoveIntentInput + { + IntentId = intentId, + Target = new PoseDto { - "intentId": "{{intentId}}", - "target": { - "position": [{{px}}, {{py}}, {{pz}}], - "orientation": [0, 0, 0, 1], - "frameId": "world" - }, - "constraints": { "cartesianSpeed": 0.25 }, - "blockingMode": "None" - } - """; + Position = new PosePositionDto { X = x, Y = y, Z = z }, + Orientation = new QuaternionDto { W = 1.0 }, + FrameId = "world" + }, + Constraints = new MotionConstraintsDto { CartesianSpeed = 0.25 }, + BlockingMode = BlockingModeEnum.None + }; } - private static string PickJson(string intentId, NodeId source, NodeId tool) + private static PickIntentInput PickInput(string intentId, NodeId source, NodeId tool) { - return $$""" - { - "intentId": "{{intentId}}", - "source": "{{source}}", - "tool": "{{tool}}", - "blockingMode": "None" - } - """; + return new PickIntentInput + { + IntentId = intentId, + Source = source.ToString(), + Tool = tool.ToString(), + BlockingMode = BlockingModeEnum.None + }; + } + + private static PlaceIntentInput PlaceInput( + string intentId, + NodeId destination, + NodeId tool) + { + return new PlaceIntentInput + { + IntentId = intentId, + Destination = destination.ToString(), + Tool = tool.ToString(), + BlockingMode = BlockingModeEnum.None + }; } - private static string PlaceJson(string intentId, NodeId destination, NodeId tool) + private static MissionStepInput[] StackingMissionSteps( + NodeId bin, + NodeId fixture, + NodeId tool) { - return $$""" + return + [ + new MissionStepInput + { + StepId = "pick-slot-02", + Released = true, + Intent = new MissionIntentInput + { + Kind = IntentKind.Pick, + Pick = PickInput("mission-pick-slot-02", bin, tool) + } + }, + new MissionStepInput { - "intentId": "{{intentId}}", - "destination": "{{destination}}", - "tool": "{{tool}}", - "blockingMode": "None" + StepId = "place-slot-02", + Released = true, + Intent = new MissionIntentInput + { + Kind = IntentKind.Place, + Place = PlaceInput("mission-place-slot-02", fixture, tool) + } } - """; + ]; } - private static string StackingMissionStepsJson(NodeId bin, NodeId fixture, NodeId tool) + private static MissionStepInput WaitStep( + string stepId, + string intentId, + double duration, + bool released) { - return $$""" - [ - { - "stepId": "pick-slot-02", - "released": true, - "intent": { - "kind": "pick", - "intentId": "mission-pick-slot-02", - "source": "{{bin}}", - "tool": "{{tool}}", - "blockingMode": "None" - } - }, - { - "stepId": "place-slot-02", - "released": true, - "intent": { - "kind": "place", - "intentId": "mission-place-slot-02", - "destination": "{{fixture}}", - "tool": "{{tool}}", - "blockingMode": "None" + return new MissionStepInput + { + StepId = stepId, + Released = released, + Intent = new MissionIntentInput + { + Kind = IntentKind.Wait, + Wait = new WaitIntentInput + { + IntentId = intentId, + Duration = duration, + BlockingMode = BlockingModeEnum.None } - } - ] - """; + } + }; } private static double Distance(double[] left, double[] right) diff --git a/tests/Opc.Ua.Robotics.Intent.Tests/RobotIntentMcpToolLiveChannelTests.cs b/tests/Opc.Ua.Robotics.Intent.Tests/RobotIntentMcpToolLiveChannelTests.cs index e74d7af9f6..0a21b4710f 100644 --- a/tests/Opc.Ua.Robotics.Intent.Tests/RobotIntentMcpToolLiveChannelTests.cs +++ b/tests/Opc.Ua.Robotics.Intent.Tests/RobotIntentMcpToolLiveChannelTests.cs @@ -83,7 +83,12 @@ await sessionManager.ConnectAsync( .RequestControlAsync(robotics, controllerId, kSessionName, CancellationToken.None) .ConfigureAwait(false); IntentSubmissionResult submission = await RoboticsControlTools - .SubmitLinearMoveAsync(robotics, controllerId, kLinearMoveJson, kSessionName, CancellationToken.None) + .SubmitLinearMoveAsync( + robotics, + controllerId, + kLinearMoveInput, + kSessionName, + CancellationToken.None) .ConfigureAwait(false); IntentOperationWaitResult completed = await WaitForCompletionThroughMcpAsync( robotics, @@ -149,10 +154,19 @@ private static OpcUaSessionManager CreateSessionManager() private const string kSessionName = "mcp-live-robotics"; - private const string kLinearMoveJson = - "{\"intentId\":\"mcp-live-linear\",\"target\":{\"position\":[0.1,0.2,0.3]," + - "\"orientation\":[0,0,0,1],\"frameId\":\"world\"},\"constraints\":{\"cartesianSpeed\":0.05}," + - "\"bufferMode\":\"Aborting\",\"blockingMode\":\"None\"}"; + private static readonly LinearMoveIntentInput kLinearMoveInput = new() + { + IntentId = "mcp-live-linear", + Target = new PoseDto + { + Position = new PosePositionDto { X = 0.1, Y = 0.2, Z = 0.3 }, + Orientation = new QuaternionDto { W = 1.0 }, + FrameId = "world" + }, + Constraints = new MotionConstraintsDto { CartesianSpeed = 0.05 }, + BufferMode = BufferModeEnum.Aborting, + BlockingMode = BlockingModeEnum.None + }; private sealed class LiveChannelFixture : IAsyncDisposable { diff --git a/tests/Opc.Ua.Robotics.Tests/BinPickingPalletizerKinematicsTests.cs b/tests/Opc.Ua.Robotics.Tests/BinPickingPalletizerKinematicsTests.cs new file mode 100644 index 0000000000..9e7012715d --- /dev/null +++ b/tests/Opc.Ua.Robotics.Tests/BinPickingPalletizerKinematicsTests.cs @@ -0,0 +1,391 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.RobotIntent; +using Robotics.IntentEnabledRobot.Kinematics; +using Robotics.IntentEnabledRobot.Simulation; +using Vision.BinPickingCell; + +namespace Opc.Ua.Robotics.Tests +{ + [TestFixture] + [Category("Robotics")] + [Parallelizable] + public sealed class BinPickingPalletizerKinematicsTests + { + [Test] + public void ZeroPoseKeepsToolDown() + { + var kinematics = new BinPickingPalletizerKinematics(); + + SimulatedArmForwardPose pose = kinematics.Forward([0.0, 0.0, 0.0, 0.0]); + ArrayOf axis = PoseMath.RotateVector(pose.ToolPose.Orientation.Span, [1.0, 0.0, 0.0]); + + Assert.Multiple(() => + { + Assert.That(pose.JointFramePoses.Count, Is.EqualTo(4)); + Assert.That(pose.ToolPose.Position[0], + Is.EqualTo(BinPickingPalletizerGeometry.MaximumReachMetres).Within(1e-9)); + Assert.That(pose.ToolPose.Position[2], + Is.EqualTo( + BinPickingPalletizerGeometry.ShoulderHeightMetres - + BinPickingPalletizerGeometry.FlangeToTcpMetres).Within(1e-9)); + Assert.That(axis[0], Is.Zero.Within(1e-9)); + Assert.That(axis[1], Is.Zero.Within(1e-9)); + Assert.That(axis[2], Is.EqualTo(-1.0).Within(1e-9)); + }); + } + + [Test] + public void ExecutorStartsAtPalletizerInitialConfiguration() + { + var kinematics = new BinPickingPalletizerKinematics(); + var executor = new global::Robotics.IntentEnabledRobot.Simulation.SimulatedArmExecutor( + kinematics); + + Assert.That( + executor.CurrentSnapshot.JointAngles.Span.ToArray(), + Is.EqualTo(kinematics.InitialJointAngles.Span.ToArray())); + Assert.That( + executor.CurrentSnapshot.ToolPose.Position.Span.ToArray(), + Is.EqualTo( + kinematics.Forward(kinematics.InitialJointAngles.Span) + .ToolPose.Position.Span.ToArray())); + } + + [TestCase(0.60, 0.00, -0.10, 0.0)] + [TestCase(-0.60, 0.00, -0.08, 0.35)] + [TestCase(0.55, -0.08, -0.16, -0.5)] + public void InverseRoundTripsWorkPositions( + double x, + double y, + double z, + double toolRoll) + { + var kinematics = new BinPickingPalletizerKinematics(); + var target = new Pose3DDataType + { + FrameId = BinPickingPalletizerGeometry.RobotBaseFrameId, + Position = new[] { x, y, z }.ToArrayOf(), + Orientation = BinPickingPalletizerKinematics.ToolDownOrientation( + Math.Atan2(y, x), + toolRoll) + }; + double[] reference = [Math.Atan2(y, x), 0.5, -1.0, toolRoll]; + + bool solved = kinematics.TrySelectNearestConfiguration( + target, + reference, + out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure); + + Assert.That(solved, Is.True, failure.ToString()); + SimulatedArmForwardPose actual = kinematics.Forward(solution!.JointAngles.Span); + Assert.Multiple(() => + { + Assert.That(actual.ToolPose.Position[0], Is.EqualTo(x).Within(1e-6)); + Assert.That(actual.ToolPose.Position[1], Is.EqualTo(y).Within(1e-6)); + Assert.That(actual.ToolPose.Position[2], Is.EqualTo(z).Within(1e-6)); + Assert.That( + QuaternionEquivalent( + actual.ToolPose.Orientation.Span, + target.Orientation.Span), + Is.True); + }); + } + + [Test] + public void HeldObjectEnvelopeParticipatesInCollisionChecks() + { + var kinematics = new BinPickingPalletizerKinematics(); + Pose3DDataType tool = kinematics.Forward(kinematics.InitialJointAngles.Span).ToolPose; + ReadOnlySpan position = tool.Position.Span; + kinematics.Collisions = new SimulatedCollisionModel( + ArrayOf.Create( + [ + new SimulatedObstacleBox( + "HeldPathObstacle", + position[0], + position[1], + 0.02, + 0.02, + position[2] - 0.045, + position[2] - 0.025) + ]), + ArrayOf.Create([0.0, 0.047, 0.042, 0.018])); + + bool toolOnlyClear = kinematics.ClearsWorkSurface( + kinematics.InitialJointAngles.Span); + kinematics.SetHeldObjectEnvelope(0.04, 0.04, 0.04); + bool heldObjectClear = kinematics.ClearsWorkSurface( + kinematics.InitialJointAngles.Span); + + Assert.Multiple(() => + { + Assert.That(toolOnlyClear, Is.True); + Assert.That(heldObjectClear, Is.False); + }); + } + + [Test] + public void InverseReturnsTwoDistinctElbowBranches() + { + var kinematics = new BinPickingPalletizerKinematics(); + Pose3DDataType target = Target(0.60, 0.0, -0.10); + + SimulatedArmIkResult result = kinematics.Inverse( + target, + [0.0, 0.5, -1.0, 0.0]); + + Assert.That(result.Solutions.Count, Is.EqualTo(2)); + Assert.That( + Math.Sign(result.Solutions[0].JointAngles[2]), + Is.Not.EqualTo(Math.Sign(result.Solutions[1].JointAngles[2]))); + } + + [Test] + public void SidewaysToolOrientationIsRefused() + { + var kinematics = new BinPickingPalletizerKinematics(); + var target = new Pose3DDataType + { + FrameId = BinPickingPalletizerGeometry.RobotBaseFrameId, + Position = new[] { 0.60, 0.0, 0.0 }.ToArrayOf(), + Orientation = new[] { 0.0, 0.0, 0.0, 1.0 }.ToArrayOf() + }; + + SimulatedArmIkResult result = kinematics.Inverse( + target, + [0.0, 0.0, 0.0, 0.0]); + + Assert.Multiple(() => + { + Assert.That(result.Succeeded, Is.False); + Assert.That(result.Failure, Is.EqualTo(SimulatedArmKinematicFailure.Kinematics)); + }); + } + + [Test] + public void NearbyVerticalSamplesStayOnOneBranch() + { + var kinematics = new BinPickingPalletizerKinematics(); + bool initialSolved = kinematics.TrySelectNearestConfiguration( + Target(0.60, 0.0, 0.12), + [0.0, 0.4, -1.1, 0.0], + out SimulatedArmIkSolution? initial, + out SimulatedArmKinematicFailure initialFailure); + Assert.That(initialSolved, Is.True, initialFailure.ToString()); + double[] reference = initial!.JointAngles.Span.ToArray(); + double previousElbowSign = Math.Sign(reference[2]); + + for (int step = 1; step <= 20; step++) + { + double z = 0.12 - (step * 0.012); + bool solved = kinematics.TrySelectNearestConfiguration( + Target(0.60, 0.0, z), + reference, + out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure); + + Assert.That(solved, Is.True, $"step={step}, failure={failure}"); + double elbowSign = Math.Sign(solution!.JointAngles[2]); + if (previousElbowSign != 0.0) + { + Assert.That(elbowSign, Is.EqualTo(previousElbowSign)); + } + Assert.That(MaxJointDelta(reference, solution.JointAngles.Span), + Is.LessThan(0.20)); + reference = solution.JointAngles.Span.ToArray(); + previousElbowSign = elbowSign; + } + } + + [Test] + public void FixtureDescentHasAContinuousClearBranch() + { + var kinematics = new BinPickingPalletizerKinematics + { + MinimumLinkHeight = + BinPickingCellGeometry.BenchTopMetres - + BinPickingCellGeometry.RobotBaseHeightMetres, + Collisions = BinPickingCellGeometry.CreateCollisionModel() + }; + ArrayOf orientation = + BinPickingPalletizerKinematics.ToolDownOrientation(0.0, Math.PI / 2.0); + Pose3DDataType transit = new() + { + FrameId = BinPickingPalletizerGeometry.RobotBaseFrameId, + Position = new[] { -0.60, 0.0, 0.32 }.ToArrayOf(), + Orientation = orientation + }; + bool transitSolved = kinematics.TrySelectNearestConfiguration( + transit, + kinematics.InitialJointAngles.Span, + out SimulatedArmIkSolution? transitSolution, + out SimulatedArmKinematicFailure transitFailure); + Assert.That(transitSolved, Is.True, transitFailure.ToString()); + double[] reference = transitSolution!.JointAngles.Span.ToArray(); + + for (int step = 1; step <= 32; step++) + { + double z = 0.32 + ((-0.119 - 0.32) * step / 32.0); + Pose3DDataType target = new() + { + FrameId = BinPickingPalletizerGeometry.RobotBaseFrameId, + Position = new[] { -0.60, 0.0, z }.ToArrayOf(), + Orientation = orientation + }; + bool solved = kinematics.TrySelectNearestConfiguration( + target, + reference, + out SimulatedArmIkSolution? solution, + out SimulatedArmKinematicFailure failure); + Assert.That(solved, Is.True, $"step={step}, z={z:F4}, failure={failure}"); + Assert.That(MaxJointDelta(reference, solution!.JointAngles.Span), Is.LessThan(0.20)); + reference = solution.JointAngles.Span.ToArray(); + } + } + + [Test] + public void EveryBinPickingWorkPoseHasAClearSolution() + { + var kinematics = new BinPickingPalletizerKinematics + { + MinimumLinkHeight = + BinPickingCellGeometry.BenchTopMetres - + BinPickingCellGeometry.RobotBaseHeightMetres, + Collisions = BinPickingCellGeometry.CreateCollisionModel() + }; + var targets = new System.Collections.Generic.List<(string Name, double X, double Y, double Z)> + { + ( + "Bin approach", + BinPickingPartsCatalog.BinCentreX, + 0.0, + BinPickingCellGeometry.BenchTopMetres + 0.20 - + BinPickingCellGeometry.RobotBaseHeightMetres), + ( + "Fixture approach", + BinPickingCellGeometry.FixtureCentreX, + 0.0, + BinPickingCellGeometry.FixturePlateTopMetres + 0.20 - + BinPickingCellGeometry.RobotBaseHeightMetres), + ("Bin transit", BinPickingPartsCatalog.BinCentreX, 0.0, 0.32), + ("Fixture transit", BinPickingCellGeometry.FixtureCentreX, 0.0, 0.32) + }; + foreach (BinPickingPart part in BinPickingPartsCatalog.Parts) + { + targets.Add( + ( + "Home " + part.ClassLabel, + part.InitialWorldPosition[0], + part.InitialWorldPosition[1], + part.InitialWorldPosition[2] + + global::Robotics.IntentEnabledRobot.Simulation.SimulatedArmExecutor.HeldPartTcpOffset - + BinPickingCellGeometry.RobotBaseHeightMetres)); + } + + foreach ((string name, double x, double y, double z) in targets) + { + double baseYaw = Math.Atan2(y, x); + var target = new Pose3DDataType + { + FrameId = BinPickingPalletizerGeometry.RobotBaseFrameId, + Position = new[] { x, y, z }.ToArrayOf(), + Orientation = BinPickingPalletizerKinematics.ToolDownOrientation( + baseYaw, + Math.PI / 2.0) + }; + + bool solved = kinematics.TrySelectNearestConfiguration( + target, + kinematics.InitialJointAngles.Span, + out _, + out SimulatedArmKinematicFailure failure); + + Assert.That(solved, Is.True, $"{name}: {failure}"); + } + } + + [Test] + public void OutsideWorkspaceIsRefused() + { + var kinematics = new BinPickingPalletizerKinematics(); + + SimulatedArmIkResult result = kinematics.Inverse( + Target(1.20, 0.0, 0.0), + [0.0, 0.0, 0.0, 0.0]); + + Assert.Multiple(() => + { + Assert.That(result.Succeeded, Is.False); + Assert.That(result.Failure, Is.EqualTo(SimulatedArmKinematicFailure.Unreachable)); + }); + } + + private static Pose3DDataType Target(double x, double y, double z) + { + return new Pose3DDataType + { + FrameId = BinPickingPalletizerGeometry.RobotBaseFrameId, + Position = new[] { x, y, z }.ToArrayOf(), + Orientation = BinPickingPalletizerKinematics.ToolDownOrientation( + Math.Atan2(y, x), + 0.0) + }; + } + + private static bool QuaternionEquivalent( + ReadOnlySpan left, + ReadOnlySpan right) + { + double dot = + (left[0] * right[0]) + + (left[1] * right[1]) + + (left[2] * right[2]) + + (left[3] * right[3]); + return Math.Abs(Math.Abs(dot) - 1.0) <= 1e-6; + } + + private static double MaxJointDelta( + ReadOnlySpan left, + ReadOnlySpan right) + { + double maximum = 0.0; + for (int ii = 0; ii < left.Length; ii++) + { + maximum = Math.Max(maximum, Math.Abs(left[ii] - right[ii])); + } + return maximum; + } + } +} diff --git a/tests/Opc.Ua.Robotics.Tests/BinPickingTargetProviderTests.cs b/tests/Opc.Ua.Robotics.Tests/BinPickingTargetProviderTests.cs new file mode 100644 index 0000000000..586444fe0b --- /dev/null +++ b/tests/Opc.Ua.Robotics.Tests/BinPickingTargetProviderTests.cs @@ -0,0 +1,210 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using NUnit.Framework; +using Opc.Ua; +using Opc.Ua.Vision; +using Vision.BinPickingCell; + +namespace Opc.Ua.Robotics.Tests +{ + [TestFixture] + public class BinPickingTargetProviderTests + { + [Test] + public void OnServerStaleTargetFallsBackToCurrentWorldState() + { + var worldState = new BinPickingWorldState(); + var provider = CreateProvider(worldState, BinPickingInferenceLocation.OnServer); + provider.PublishWorldState( + "old-result", + DateTimeUtc.From(DateTime.UtcNow - TimeSpan.FromMinutes(1)), + worldState.Snapshot()); + + bool resolved = provider.TryResolve("RedCube", out BinPickingTarget target); + + Assert.Multiple(() => + { + Assert.That(resolved, Is.True); + Assert.That(target.ResultId, Is.EqualTo("simulation-world-state")); + Assert.That(target.SourceFrameId, Is.EqualTo("world")); + Assert.That(target.WorldX, Is.EqualTo(0.520).Within(1e-9)); + Assert.That(target.WorldY, Is.EqualTo(-0.080).Within(1e-9)); + }); + } + + [Test] + public void OffServerStaleTargetIsNotReplacedWithSimulationTruth() + { + var worldState = new BinPickingWorldState(); + var provider = CreateProvider(worldState, BinPickingInferenceLocation.EdgeOffServer); + provider.PublishWorldState( + "old-result", + DateTimeUtc.From(DateTime.UtcNow - TimeSpan.FromMinutes(1)), + worldState.Snapshot()); + + Assert.That(provider.TryResolve("RedCube", out _), Is.False); + } + + [Test] + public void OffServerDetectionComposesCameraPoseAndPreservesProvenance() + { + var worldState = new BinPickingWorldState(); + var provider = CreateProvider(worldState, BinPickingInferenceLocation.EdgeOffServer); + BinPickingPart red = BinPickingPartsCatalog.TryGet("RedCube")!; + VisionDetectionDataType detection = Detection( + red.ClassLabel, + "camera_eih", + red.InitialWorldPosition[0], + red.InitialWorldPosition[1], + red.InitialWorldPosition[2]); + + provider.PublishDetections( + "agent-result", + DateTimeUtc.From(DateTime.UtcNow), + new[] { detection }.ToArrayOf(), + IdentityCamera(), + "camera_eih"); + + bool resolved = provider.TryResolve(red.ClassLabel, out BinPickingTarget target); + Assert.Multiple(() => + { + Assert.That(resolved, Is.True); + Assert.That(target.ResultId, Is.EqualTo("agent-result")); + Assert.That(target.SourceFrameId, Is.EqualTo("camera_eih")); + Assert.That(target.WorldX, Is.EqualTo(red.InitialWorldPosition[0]).Within(1e-9)); + Assert.That(target.WorldY, Is.EqualTo(red.InitialWorldPosition[1]).Within(1e-9)); + Assert.That(target.WorldZ, Is.EqualTo(red.InitialWorldPosition[2]).Within(1e-9)); + }); + } + + [Test] + public void OffServerDetectionRejectsAnUncalibratedSourceFrame() + { + var worldState = new BinPickingWorldState(); + var provider = CreateProvider(worldState, BinPickingInferenceLocation.EdgeOffServer); + BinPickingPart red = BinPickingPartsCatalog.TryGet("RedCube")!; + VisionDetectionDataType detection = Detection( + red.ClassLabel, + "wrong_camera", + red.InitialWorldPosition[0], + red.InitialWorldPosition[1], + red.InitialWorldPosition[2]); + + ServiceResultException? exception = Assert.Throws(() => + provider.PublishDetections( + "agent-result", + DateTimeUtc.From(DateTime.UtcNow), + new[] { detection }.ToArrayOf(), + IdentityCamera(), + "camera_eih")); + + Assert.Multiple(() => + { + Assert.That(exception!.StatusCode, Is.EqualTo((StatusCode)StatusCodes.BadInvalidArgument)); + Assert.That(provider.TryResolve(red.ClassLabel, out _), Is.False); + }); + } + + [Test] + public void OffServerDetectionBatchRejectsResidualWithoutPublishingEarlierTargets() + { + var worldState = new BinPickingWorldState(); + var provider = CreateProvider(worldState, BinPickingInferenceLocation.EdgeOffServer); + BinPickingPart red = BinPickingPartsCatalog.TryGet("RedCube")!; + BinPickingPart green = BinPickingPartsCatalog.TryGet("GreenCylinder")!; + VisionDetectionDataType[] detections = + [ + Detection( + red.ClassLabel, + "camera_eih", + red.InitialWorldPosition[0], + red.InitialWorldPosition[1], + red.InitialWorldPosition[2]), + Detection( + green.ClassLabel, + "camera_eih", + green.InitialWorldPosition[0] + 0.20, + green.InitialWorldPosition[1], + green.InitialWorldPosition[2]) + ]; + + _ = Assert.Throws(() => + provider.PublishDetections( + "agent-result", + DateTimeUtc.From(DateTime.UtcNow), + detections.ToArrayOf(), + IdentityCamera(), + "camera_eih")); + + Assert.That(provider.TryResolve(red.ClassLabel, out _), Is.False); + } + + private static BinPickingTargetProvider CreateProvider( + BinPickingWorldState worldState, + BinPickingInferenceLocation inferenceLocation) + { + return new BinPickingTargetProvider( + worldState, + new BinPickingCellOptions { InferenceLocation = inferenceLocation }); + } + + private static VisionDetectionDataType Detection( + string classLabel, + string frameId, + double x, + double y, + double z) + { + return new VisionDetectionDataType + { + ClassLabel = classLabel, + Confidence = 0.99, + HasPose = true, + Pose = new VisionPose3DDataType + { + FrameId = frameId, + Position = new[] { x, y, z }.ToArrayOf(), + Orientation = new[] { 0.0, 0.0, 0.0, 1.0 }.ToArrayOf() + } + }; + } + + private static VisionPose3DDataType IdentityCamera() + { + return new VisionPose3DDataType + { + FrameId = "world", + Position = new[] { 0.0, 0.0, 0.0 }.ToArrayOf(), + Orientation = new[] { 0.0, 0.0, 0.0, 1.0 }.ToArrayOf() + }; + } + } +} diff --git a/tests/Opc.Ua.Robotics.Tests/BinPickingWorldStateTests.cs b/tests/Opc.Ua.Robotics.Tests/BinPickingWorldStateTests.cs new file mode 100644 index 0000000000..c3186f6bf5 --- /dev/null +++ b/tests/Opc.Ua.Robotics.Tests/BinPickingWorldStateTests.cs @@ -0,0 +1,194 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using NUnit.Framework; +using Vision.BinPickingCell; + +namespace Opc.Ua.Robotics.Tests +{ + /// + /// Covers the bin-picking cell's world model, whose location label decides whether the + /// camera still reports a part. The label has to follow the part's coordinates: when it + /// followed the last operation instead, a part the robot had returned to the bin stayed + /// invisible to the detector for good, and a stack-then-return cycle could only run once. + /// + [TestFixture] + [Category("Robotics")] + [Parallelizable] + public sealed class BinPickingWorldStateTests + { + [Test] + public void EveryPartStartsInTheBin() + { + var state = new BinPickingWorldState(); + + IReadOnlyList parts = state.Snapshot(); + + Assert.That(parts, Is.Not.Empty); + Assert.Multiple(() => + { + foreach (BinPickingPartSnapshot part in parts) + { + Assert.That(part.Location, Is.EqualTo(BinPickingPartLocation.InBin), + part.Part.ClassLabel + " starts in the bin."); + Assert.That(BinPickingPartsCatalog.IsInsideBin(part.WorldX, part.WorldY), Is.True, + part.Part.ClassLabel + " starts inside the bin footprint."); + } + }); + } + + [Test] + public void PickedPartIsHeldAndNoLongerReported() + { + var state = new BinPickingWorldState(); + + bool marked = state.MarkHeld(SampleLabel, 0.30, 0.0, 1.00); + + Assert.That(marked, Is.True); + Assert.That(Find(state, SampleLabel).Location, Is.EqualTo(BinPickingPartLocation.Held), + "A part in the gripper must not be reported as sitting in the bin."); + } + + [Test] + public void PartPlacedOnTheFixtureIsNotInTheBin() + { + var state = new BinPickingWorldState(); + _ = state.MarkHeld(SampleLabel, 0.30, 0.0, 1.00); + + _ = state.MarkPlaced(SampleLabel, FixtureX, FixtureY, 0.85); + + Assert.That(Find(state, SampleLabel).Location, Is.EqualTo(BinPickingPartLocation.Placed)); + } + + [Test] + public void PartReturnedToTheBinIsInTheBinAgain() + { + var state = new BinPickingWorldState(); + _ = state.MarkHeld(SampleLabel, 0.30, 0.0, 1.00); + _ = state.MarkPlaced(SampleLabel, FixtureX, FixtureY, 0.85); + + _ = state.MarkPlaced( + SampleLabel, + BinPickingPartsCatalog.BinCentreX, + BinPickingPartsCatalog.BinCentreY, + 0.85); + + Assert.That(Find(state, SampleLabel).Location, Is.EqualTo(BinPickingPartLocation.InBin), + "A part put back in the bin has to become visible to the camera again, or a " + + "stack-then-return cycle can only ever run once."); + } + + [Test] + public void PartReturnedToItsOwnStartingSpotIsInTheBin() + { + var state = new BinPickingWorldState(); + BinPickingPart part = BinPickingPartsCatalog.Parts[0]; + _ = state.MarkHeld(part.ClassLabel, 0.30, 0.0, 1.00); + + _ = state.MarkPlaced( + part.ClassLabel, + part.InitialWorldPosition[0], + part.InitialWorldPosition[1], + part.InitialWorldPosition[2]); + + Assert.That(Find(state, part.ClassLabel).Location, Is.EqualTo(BinPickingPartLocation.InBin)); + } + + [Test] + public void BinFootprintCoversEveryAuthoredPartPositionAndExcludesTheFixture() + { + Assert.Multiple(() => + { + foreach (BinPickingPart part in BinPickingPartsCatalog.Parts) + { + Assert.That( + BinPickingPartsCatalog.IsInsideBin(part.InitialWorldPosition[0], part.InitialWorldPosition[1]), + Is.True, + part.ClassLabel + " is authored inside the bin."); + } + Assert.That(BinPickingPartsCatalog.IsInsideBin(FixtureX, FixtureY), Is.False, + "The fixture stands well clear of the bin, so a part stacked there is not in the bin."); + }); + } + + [Test] + public void ResetPutsEveryPartBackWhereItStarted() + { + var state = new BinPickingWorldState(); + _ = state.MarkHeld(SampleLabel, 0.30, 0.0, 1.00); + _ = state.MarkPlaced(SampleLabel, FixtureX, FixtureY, 0.85); + + state.Reset(); + + Assert.Multiple(() => + { + foreach (BinPickingPartSnapshot snapshot in state.Snapshot()) + { + Assert.That(snapshot.WorldX, Is.EqualTo(snapshot.Part.InitialWorldPosition[0]).Within(1e-9)); + Assert.That(snapshot.WorldY, Is.EqualTo(snapshot.Part.InitialWorldPosition[1]).Within(1e-9)); + Assert.That(snapshot.WorldZ, Is.EqualTo(snapshot.Part.InitialWorldPosition[2]).Within(1e-9)); + Assert.That(snapshot.Location, Is.EqualTo(BinPickingPartLocation.InBin)); + } + }); + } + + [Test] + public void UnknownClassLabelIsRejected() + { + var state = new BinPickingWorldState(); + + Assert.Multiple(() => + { + Assert.That(state.MarkHeld("NoSuchPart", 0.0, 0.0, 0.0), Is.False); + Assert.That(state.MarkPlaced("NoSuchPart", 0.0, 0.0, 0.0), Is.False); + }); + } + + private static BinPickingPartSnapshot Find(BinPickingWorldState state, string classLabel) + { + foreach (BinPickingPartSnapshot snapshot in state.Snapshot()) + { + if (snapshot.Part.ClassLabel == classLabel) + { + return snapshot; + } + } + Assert.Fail("The catalogue has no part called " + classLabel + "."); + return null!; + } + + /// + /// The fixture the parts get stacked on, from Assets/Cell.usda. + /// + private const double FixtureX = BinPickingPartsCatalog.FixtureCentreX; + private const double FixtureY = 0.0; + private const string SampleLabel = "RedCube"; + } +} diff --git a/tests/Opc.Ua.Robotics.Tests/IntentClientRuntimeTests.cs b/tests/Opc.Ua.Robotics.Tests/IntentClientRuntimeTests.cs index 51a1521030..4e9587469f 100644 --- a/tests/Opc.Ua.Robotics.Tests/IntentClientRuntimeTests.cs +++ b/tests/Opc.Ua.Robotics.Tests/IntentClientRuntimeTests.cs @@ -52,10 +52,7 @@ namespace Opc.Ua.Robotics.Client.Tests [Category("Robotics")] public sealed class IntentClientRuntimeTests { - [TestCase(ExecutionStateEnum.Succeeded)] - [TestCase(ExecutionStateEnum.Failed)] - [TestCase(ExecutionStateEnum.Cancelled)] - [TestCase(ExecutionStateEnum.Retriable)] + [TestCaseSource(nameof(TerminalExecutionStates))] public async Task OperationHandleCompletesOnTerminalStates(ExecutionStateEnum state) { FakeRobotIntentTransport transport = new() @@ -66,9 +63,9 @@ public async Task OperationHandleCompletesOnTerminalStates(ExecutionStateEnum st await using IntentOperationHandle handle = await controller.TrackOperationAsync( "i1", - new NodeId(10)); + new NodeId(10)).ConfigureAwait(false); - IntentResultDataType result = await AwaitWithTimeoutAsync(handle.Completion, s_handshakeTimeout); + IntentResultDataType result = await AwaitWithTimeoutAsync(handle.Completion, s_handshakeTimeout).ConfigureAwait(false); Assert.That(result.State, Is.EqualTo(state)); } @@ -84,9 +81,9 @@ public async Task OperationHandleReadsInitialStateAfterSubscribing() await using IntentOperationHandle handle = await controller.TrackOperationAsync( "i1", - new NodeId(10)); + new NodeId(10)).ConfigureAwait(false); - Assert.That(await AwaitWithTimeoutAsync(handle.Completion, s_handshakeTimeout), Is.Not.Null); + Assert.That(await AwaitWithTimeoutAsync(handle.Completion, s_handshakeTimeout).ConfigureAwait(false), Is.Not.Null); Assert.That(transport.SubscribeCount, Is.EqualTo(1)); Assert.That(transport.ReadSnapshotCount, Is.EqualTo(1)); } @@ -106,7 +103,7 @@ public async Task OperationHandleWaitsForResultAfterTerminalStateNotification() await using IntentOperationHandle handle = await controller.TrackOperationAsync( "i1", - new NodeId(10)); + new NodeId(10)).ConfigureAwait(false); transport.PublishChange("ExecutionState", Variant.From((int)ExecutionStateEnum.Succeeded)); Task early = await Task.WhenAny(handle.Completion, Task.Delay(100)).ConfigureAwait(false); @@ -116,7 +113,7 @@ public async Task OperationHandleWaitsForResultAfterTerminalStateNotification() State = ExecutionStateEnum.Succeeded }; transport.PublishChange("Result", Variant.FromStructure(expected)); - IntentResultDataType result = await AwaitWithTimeoutAsync(handle.Completion, s_handshakeTimeout); + IntentResultDataType result = await AwaitWithTimeoutAsync(handle.Completion, s_handshakeTimeout).ConfigureAwait(false); Assert.Multiple(() => { @@ -137,10 +134,10 @@ public async Task OperationHandleRereadsAfterReconnect() await using IntentOperationHandle handle = await controller.TrackOperationAsync( "i1", - new NodeId(10)); + new NodeId(10)).ConfigureAwait(false); transport.Snapshot = Snapshot(ExecutionStateEnum.Succeeded); transport.PublishReconnect(); - IntentResultDataType result = await AwaitWithTimeoutAsync(handle.Completion, s_handshakeTimeout); + IntentResultDataType result = await AwaitWithTimeoutAsync(handle.Completion, s_handshakeTimeout).ConfigureAwait(false); Assert.That(result.State, Is.EqualTo(ExecutionStateEnum.Succeeded)); Assert.That(transport.ReadSnapshotCount, Is.GreaterThanOrEqualTo(2)); @@ -158,8 +155,8 @@ public async Task CancelRefusalIsReturnedNotThrown() await using IntentOperationHandle handle = await controller.TrackOperationAsync( "i1", - new NodeId(10)); - IntentCommandOutcome outcome = await handle.CancelAsync(StopModeEnum.QuickStop); + new NodeId(10)).ConfigureAwait(false); + IntentCommandOutcome outcome = await handle.CancelAsync(StopModeEnum.QuickStop).ConfigureAwait(false); Assert.Multiple(() => { @@ -190,11 +187,11 @@ public async Task OperationHandleCommandMethodsDelegateToTransport() await using IntentOperationHandle handle = await controller.TrackOperationAsync( "i1", - new NodeId(10)); + new NodeId(10)).ConfigureAwait(false); - IntentCommandOutcome pause = await handle.PauseAsync(); - IntentCommandOutcome resume = await handle.ResumeAsync(); - IntentSubmissionResult retry = await handle.RetryAsync(); + IntentCommandOutcome pause = await handle.PauseAsync().ConfigureAwait(false); + IntentCommandOutcome resume = await handle.ResumeAsync().ConfigureAwait(false); + IntentSubmissionResult retry = await handle.RetryAsync().ConfigureAwait(false); Assert.Multiple(() => { @@ -220,12 +217,134 @@ public async Task CancellingDoesNotCompleteOperationHandle() await using IntentOperationHandle handle = await controller.TrackOperationAsync( "i1", - new NodeId(10)); - Task completed = await Task.WhenAny(handle.Completion, Task.Delay(100)); + new NodeId(10)).ConfigureAwait(false); + Task completed = await Task.WhenAny(handle.Completion, Task.Delay(100)).ConfigureAwait(false); Assert.That(completed, Is.Not.SameAs(handle.Completion)); } + [Test] + public async Task MissionHandleReadsInitialStateAfterSubscribing() + { + FakeRobotIntentTransport transport = new() + { + MissionSnapshot = MissionSnapshot(ExecutionStateEnum.Succeeded) + }; + RobotIntentControllerClient controller = new(transport); + + await using MissionHandle handle = await controller.TrackMissionAsync( + "mission-1", + new NodeId(20)).ConfigureAwait(false); + + MissionSnapshot terminal = await AwaitWithTimeoutAsync(handle.Completion, TimeSpan.FromSeconds(1)).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(terminal.ExecutionState, Is.EqualTo(ExecutionStateEnum.Succeeded)); + Assert.That(transport.SubscribeCount, Is.EqualTo(1)); + Assert.That(transport.ReadMissionSnapshotCount, Is.EqualTo(1)); + }); + } + + [Test] + public async Task MissionHandleRefreshesAfterReconnect() + { + FakeRobotIntentTransport transport = new() + { + MissionSnapshot = MissionSnapshot(ExecutionStateEnum.Executing) + }; + RobotIntentControllerClient controller = new(transport); + + await using MissionHandle handle = await controller.TrackMissionAsync( + "mission-1", + new NodeId(20)).ConfigureAwait(false); + transport.MissionSnapshot = MissionSnapshot(ExecutionStateEnum.Failed) with + { + Failure = IntentFailureEnum.Other, + FailureMessage = new LocalizedText("executor failure") + }; + transport.PublishReconnect(); + + MissionSnapshot terminal = await AwaitWithTimeoutAsync(handle.Completion, TimeSpan.FromSeconds(1)).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(terminal.ExecutionState, Is.EqualTo(ExecutionStateEnum.Failed)); + Assert.That(terminal.Failure, Is.EqualTo(IntentFailureEnum.Other)); + Assert.That(terminal.FailureMessage.Text, Is.EqualTo("executor failure")); + Assert.That(transport.ReadMissionSnapshotCount, Is.GreaterThanOrEqualTo(2)); + }); + } + + [Test] + public async Task MissionHandleTimeoutRemainsIncompleteWhenRefreshObservesTerminalState() + { + FakeRobotIntentTransport transport = new() + { + MissionSnapshot = MissionSnapshot(ExecutionStateEnum.Executing) + }; + RobotIntentControllerClient controller = new(transport); + + await using MissionHandle handle = await controller.TrackMissionAsync( + "mission-1", + new NodeId(20)).ConfigureAwait(false); + transport.MissionSnapshot = MissionSnapshot(ExecutionStateEnum.Succeeded); + + MissionWaitResult result = await handle.WaitForCompletionAsync(TimeSpan.Zero).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Completed, Is.False); + Assert.That(result.Current.ExecutionState, Is.EqualTo(ExecutionStateEnum.Succeeded)); + Assert.That(handle.Completion.IsCompleted, Is.True); + }); + } + + [Test] + public async Task MissionHandleCancellationDelegatesToTransport() + { + FakeRobotIntentTransport transport = new() + { + MissionSnapshot = MissionSnapshot(ExecutionStateEnum.Executing), + CancelMissionOutcome = new IntentCommandOutcome(false) + }; + RobotIntentControllerClient controller = new(transport); + + await using MissionHandle handle = await controller.TrackMissionAsync( + "mission-1", + new NodeId(20)).ConfigureAwait(false); + IntentCommandOutcome outcome = await handle.CancelAsync(StopModeEnum.QuickStop).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(outcome.Accepted, Is.False); + Assert.That(transport.CancelMissionCount, Is.EqualTo(1)); + Assert.That(transport.LastCancelMissionId, Is.EqualTo("mission-1")); + Assert.That(transport.LastCancelMissionStopMode, Is.EqualTo(StopModeEnum.QuickStop)); + }); + } + + [Test] + public async Task MissionHandleDisposalStopsItsSubscription() + { + FakeRobotIntentTransport transport = new() + { + MissionSnapshot = MissionSnapshot(ExecutionStateEnum.Executing) + }; + RobotIntentControllerClient controller = new(transport); + MissionHandle handle = await controller.TrackMissionAsync("mission-1", new NodeId(20)).ConfigureAwait(false); + await WaitUntilAsync( + () => transport.ActiveSubscriptionCount == 1, + TimeSpan.FromSeconds(1)).ConfigureAwait(false); + + await handle.DisposeAsync().ConfigureAwait(false); + + await WaitUntilAsync( + () => transport.ActiveSubscriptionCount == 0, + TimeSpan.FromSeconds(1)).ConfigureAwait(false); + Assert.That(transport.ActiveSubscriptionCount, Is.Zero); + } + [Test] public async Task AuthorityLeaseReleasesOnDisposeAndReportsLoss() { @@ -236,7 +355,7 @@ public async Task AuthorityLeaseReleasesOnDisposeAndReportsLoss() }; RobotIntentControllerClient controller = new(transport); - await using (CommandAuthorityLease lease = await controller.RequestAuthorityAsync()) + await using (CommandAuthorityLease lease = await controller.RequestAuthorityAsync().ConfigureAwait(false)) { transport.ControlOwner = new NodeId(2); transport.PublishOwner(new NodeId(2)); @@ -259,7 +378,7 @@ public async Task AuthorityLeaseKeepsGrantWhenInitialNotificationIsOwnOwner() }; RobotIntentControllerClient controller = new(transport); - await using CommandAuthorityLease lease = await controller.RequestAuthorityAsync(); + await using CommandAuthorityLease lease = await controller.RequestAuthorityAsync().ConfigureAwait(false); int notifications = 0; lease.OwnerChanged += _ => Interlocked.Increment(ref notifications); transport.PublishOwner(owner); @@ -282,10 +401,10 @@ public async Task AuthorityLeaseDisposeIsIdempotentAndReleasesOnce() ControlOwner = new NodeId(1) }; RobotIntentControllerClient controller = new(transport); - CommandAuthorityLease lease = await controller.RequestAuthorityAsync(); + CommandAuthorityLease lease = await controller.RequestAuthorityAsync().ConfigureAwait(false); - await lease.DisposeAsync(); - Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync()); + await lease.DisposeAsync().ConfigureAwait(false); + Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync().ConfigureAwait(false)); Assert.That(transport.ReleaseCount, Is.EqualTo(1)); } @@ -300,10 +419,10 @@ public async Task AuthorityLeaseDisposeAfterClosedSessionDoesNotThrow() ReleaseException = ServiceResultException.Create(StatusCodes.BadSessionClosed, "closed") }; RobotIntentControllerClient controller = new(transport); - CommandAuthorityLease lease = await controller.RequestAuthorityAsync(); + CommandAuthorityLease lease = await controller.RequestAuthorityAsync().ConfigureAwait(false); - Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync()); - Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync()); + Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync().ConfigureAwait(false)); + Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync().ConfigureAwait(false)); Assert.That(transport.ReleaseCount, Is.EqualTo(1)); } @@ -317,10 +436,10 @@ public async Task AuthorityLeaseDisposeAfterRefusalDoesNotRelease() ControlOwner = new NodeId(2) }; RobotIntentControllerClient controller = new(transport); - CommandAuthorityLease lease = await controller.RequestAuthorityAsync(); + CommandAuthorityLease lease = await controller.RequestAuthorityAsync().ConfigureAwait(false); - await lease.DisposeAsync(); - await lease.DisposeAsync(); + await lease.DisposeAsync().ConfigureAwait(false); + await lease.DisposeAsync().ConfigureAwait(false); Assert.That(transport.ReleaseCount, Is.Zero); } @@ -333,7 +452,7 @@ public async Task OperationHandleDisposeIsIdempotentDuringCallback() Snapshot = Snapshot(ExecutionStateEnum.Executing) }; RobotIntentControllerClient controller = new(transport); - IntentOperationHandle handle = await controller.TrackOperationAsync("i1", new NodeId(10)); + IntentOperationHandle handle = await controller.TrackOperationAsync("i1", new NodeId(10)).ConfigureAwait(false); var callbackEntered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); using var allowCallbackToExit = new ManualResetEventSlim(false); handle.Changed += _ => @@ -343,12 +462,12 @@ public async Task OperationHandleDisposeIsIdempotentDuringCallback() }; transport.PublishChange(Variant.From((int)ExecutionStateEnum.Succeeded)); - await AwaitWithTimeoutAsync(callbackEntered.Task, s_handshakeTimeout); + await AwaitWithTimeoutAsync(callbackEntered.Task, s_handshakeTimeout).ConfigureAwait(false); Task dispose = handle.DisposeAsync().AsTask(); allowCallbackToExit.Set(); - Assert.DoesNotThrowAsync(async () => await dispose); - Assert.DoesNotThrowAsync(async () => await handle.DisposeAsync()); + Assert.DoesNotThrowAsync(async () => await dispose.ConfigureAwait(false)); + Assert.DoesNotThrowAsync(async () => await handle.DisposeAsync().ConfigureAwait(false)); } [Test] @@ -361,9 +480,9 @@ public async Task MissionUpdateRejectsNonIncreasingIdLocally() .HorizonStep("a", RobotIntentBuilder.Wait(1).Build()) .Build(); - _ = await controller.SubmitMissionAsync(mission); + _ = await controller.SubmitMissionAsync(mission).ConfigureAwait(false); - MissionUpdateOutcome outcome = await controller.UpdateMissionAsync("m1", 3, []); + MissionUpdateOutcome outcome = await controller.UpdateMissionAsync("m1", 3, []).ConfigureAwait(false); Assert.That(outcome.Result, Is.EqualTo(MissionUpdateResultEnum.Outdated)); Assert.That(transport.UpdateMissionCount, Is.Zero); @@ -378,13 +497,13 @@ public async Task MissionUpdateIdsAreTrackedPerMission() _ = await controller.SubmitMissionAsync(RobotIntentBuilder.Mission("a") .WithMissionUpdateId(5) .HorizonStep("a1", RobotIntentBuilder.Wait(1).Build()) - .Build()); + .Build()).ConfigureAwait(false); _ = await controller.SubmitMissionAsync(RobotIntentBuilder.Mission("b") .WithMissionUpdateId(1) .HorizonStep("b1", RobotIntentBuilder.Wait(1).Build()) - .Build()); - MissionUpdateOutcome accepted = await controller.UpdateMissionAsync("b", 2, []); - MissionUpdateOutcome outdated = await controller.UpdateMissionAsync("a", 4, []); + .Build()).ConfigureAwait(false); + MissionUpdateOutcome accepted = await controller.UpdateMissionAsync("b", 2, []).ConfigureAwait(false); + MissionUpdateOutcome outdated = await controller.UpdateMissionAsync("a", 4, []).ConfigureAwait(false); Assert.Multiple(() => { @@ -440,8 +559,8 @@ public async Task ChannelLeaseRenewsAndSurfacesRefusalMessage() }; await using RealTimeChannelLease lease = new(transport, "rt1", TimeSpan.FromMilliseconds(30)); - await lease.OpenAsync(); - await AwaitWithTimeoutAsync(transport.WaitForOpenChannelCountAsync(2), s_handshakeTimeout); + await lease.OpenAsync().ConfigureAwait(false); + await AwaitWithTimeoutAsync(transport.WaitForOpenChannelCountAsync(2), s_handshakeTimeout).ConfigureAwait(false); Assert.That(transport.OpenChannelCount, Is.GreaterThan(1)); Assert.That(lease.EndpointUrl, Is.EqualTo("opc.tcp://rt")); @@ -451,7 +570,7 @@ public async Task ChannelLeaseRenewsAndSurfacesRefusalMessage() Granted = false, Message = new LocalizedText("busy") }; - await lease.RenewAsync(); + await lease.RenewAsync().ConfigureAwait(false); Assert.That(lease.Granted, Is.False); Assert.That(lease.Message.Text, Is.EqualTo("busy")); @@ -479,8 +598,8 @@ public async Task ChannelLeaseRenewLoopRetriesAfterServiceFailure() }); await using RealTimeChannelLease lease = new(transport, "rt1", TimeSpan.FromMilliseconds(30)); - await lease.OpenAsync(); - await WaitUntilAsync(() => transport.OpenChannelCount >= 3, s_handshakeTimeout); + await lease.OpenAsync().ConfigureAwait(false); + await WaitUntilAsync(() => transport.OpenChannelCount >= 3, s_handshakeTimeout).ConfigureAwait(false); Assert.That(transport.OpenChannelCount, Is.GreaterThanOrEqualTo(3)); Assert.That(lease.Granted, Is.True); @@ -503,10 +622,10 @@ public async Task ChannelLeaseDisposeClosesAfterRenewFailure() transport.EnqueueChannelFault(ServiceResultException.Create(StatusCodes.BadTimeout, "transient")); var lease = new RealTimeChannelLease(transport, "rt1", TimeSpan.FromMilliseconds(30)); - await lease.OpenAsync(); - await AwaitWithTimeoutAsync(transport.WaitForOpenChannelCountAsync(2), s_handshakeTimeout); + await lease.OpenAsync().ConfigureAwait(false); + await AwaitWithTimeoutAsync(transport.WaitForOpenChannelCountAsync(2), s_handshakeTimeout).ConfigureAwait(false); - Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync()); + Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync().ConfigureAwait(false)); Assert.That(transport.CloseChannelCount, Is.EqualTo(1)); } @@ -524,9 +643,9 @@ public async Task ChannelLeaseDisposeIsIdempotentAndClosesOnce() }; RealTimeChannelLease lease = new(transport, "rt1", TimeSpan.FromMilliseconds(30)); - await lease.OpenAsync(); - await lease.DisposeAsync(); - Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync()); + await lease.OpenAsync().ConfigureAwait(false); + await lease.DisposeAsync().ConfigureAwait(false); + Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync().ConfigureAwait(false)); Assert.That(transport.CloseChannelCount, Is.EqualTo(1)); } @@ -546,10 +665,10 @@ public async Task ChannelLeaseDisposeAfterClosedSessionDoesNotThrow() }; RealTimeChannelLease lease = new(transport, "rt1", TimeSpan.FromMilliseconds(30)); - await lease.OpenAsync(); + await lease.OpenAsync().ConfigureAwait(false); - Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync()); - Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync()); + Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync().ConfigureAwait(false)); + Assert.DoesNotThrowAsync(async () => await lease.DisposeAsync().ConfigureAwait(false)); Assert.That(transport.CloseChannelCount, Is.EqualTo(1)); } @@ -566,9 +685,9 @@ public async Task ChannelLeaseDisposeAfterRefusalDoesNotClose() }; RealTimeChannelLease lease = new(transport, "rt1", TimeSpan.FromMilliseconds(30)); - await lease.OpenAsync(); - await lease.DisposeAsync(); - await lease.DisposeAsync(); + await lease.OpenAsync().ConfigureAwait(false); + await lease.DisposeAsync().ConfigureAwait(false); + await lease.DisposeAsync().ConfigureAwait(false); Assert.That(transport.CloseChannelCount, Is.Zero); } @@ -591,8 +710,8 @@ public async Task SubmitIntentRefusalThrowsAndTrySubmitReturnsRefusal() IntentDataType refusalAwareIntent = RobotIntentBuilder.Wait(2).Build(); ServiceResultException? ex = Assert.ThrowsAsync( - async () => await controller.SubmitIntentAsync(throwingIntent)); - IntentSubmissionResult result = await controller.TrySubmitIntentAsync(refusalAwareIntent); + async () => await controller.SubmitIntentAsync(throwingIntent).ConfigureAwait(false)); + IntentSubmissionResult result = await controller.TrySubmitIntentAsync(refusalAwareIntent).ConfigureAwait(false); Assert.Multiple(() => { @@ -617,9 +736,9 @@ public async Task ControllerMissionAndCancelMethodsDelegateToTransport() .HorizonStep("a", RobotIntentBuilder.Wait(1).Build()) .Build(); - MissionSubmissionResult submit = await controller.SubmitMissionAsync(mission); - MissionUpdateOutcome update = await controller.UpdateMissionAsync("m1", 5, []); - IntentCommandOutcome cancel = await controller.CancelMissionAsync("m1", StopModeEnum.QuickStop); + MissionSubmissionResult submit = await controller.SubmitMissionAsync(mission).ConfigureAwait(false); + MissionUpdateOutcome update = await controller.UpdateMissionAsync("m1", 5, []).ConfigureAwait(false); + IntentCommandOutcome cancel = await controller.CancelMissionAsync("m1", StopModeEnum.QuickStop).ConfigureAwait(false); Assert.Multiple(() => { @@ -651,7 +770,7 @@ public async Task ChannelLeaseExpiredLeaseUsesNonZeroRenewDelay() }; await using RealTimeChannelLease lease = new(transport, "rt1", TimeSpan.FromSeconds(1)); - await lease.OpenAsync(); + await lease.OpenAsync().ConfigureAwait(false); TimeSpan delay = InvokeComputeRenewDelay(lease); Assert.That(delay, Is.GreaterThan(TimeSpan.FromMilliseconds(250))); @@ -673,7 +792,7 @@ public async Task ControllerOpensRealTimeChannelLease() await using RealTimeChannelLease lease = await controller.OpenRealTimeChannelAsync( "rt1", - TimeSpan.FromSeconds(1)); + TimeSpan.FromSeconds(1)).ConfigureAwait(false); Assert.Multiple(() => { @@ -796,8 +915,10 @@ private static TimeSpan InvokeComputeRenewDelay(RealTimeChannelLease lease) return (TimeSpan)method!.Invoke(lease, [])!; } - // Generous scheduling/handshake timeout used for AwaitWithTimeoutAsync and WaitUntilAsync - // calls only. Does not affect behavioral lease durations or expiry times. + /// + /// Generous scheduling/handshake timeout used for AwaitWithTimeoutAsync and WaitUntilAsync + /// calls only. Does not affect behavioral lease durations or expiry times. + /// private static readonly TimeSpan s_handshakeTimeout = TimeSpan.FromSeconds(30); private static async Task AwaitWithTimeoutAsync(Task task, TimeSpan timeout) @@ -892,6 +1013,24 @@ private static IntentOperationSnapshot Snapshot(ExecutionStateEnum state) }; } + private static MissionSnapshot MissionSnapshot(ExecutionStateEnum state) + { + return new MissionSnapshot + { + MissionNode = new NodeId(20), + MissionId = "mission-1", + ExecutionState = state + }; + } + + private static IEnumerable TerminalExecutionStates() + { + yield return new TestCaseData(ExecutionStateEnum.Succeeded); + yield return new TestCaseData(ExecutionStateEnum.Failed); + yield return new TestCaseData(ExecutionStateEnum.Cancelled); + yield return new TestCaseData(ExecutionStateEnum.Retriable); + } + private sealed class TestClientBuilder(IServiceCollection services) : IOpcUaClientBuilder { public IServiceCollection Services { get; } = services; @@ -949,6 +1088,10 @@ private sealed class FakeRobotIntentTransport : IRobotIntentTransport, IDisposab public int ReadSnapshotCount { get; private set; } + public int ReadMissionSnapshotCount { get; private set; } + + public int ActiveSubscriptionCount => Volatile.Read(ref m_activeSubscriptionCount); + public int ReleaseCount { get; private set; } public int UpdateMissionCount { get; private set; } @@ -1230,6 +1373,16 @@ public ValueTask ReadOperationSnapshotAsync( return new ValueTask(Snapshot); } + public MissionSnapshot MissionSnapshot { get; set; } = new(); + + public ValueTask ReadMissionSnapshotAsync( + NodeId mission, + CancellationToken ct = default) + { + ReadMissionSnapshotCount++; + return new ValueTask(MissionSnapshot with { MissionNode = mission }); + } + public ValueTask ReadControlOwnerAsync(CancellationToken ct = default) { return new ValueTask(ControlOwner); @@ -1240,23 +1393,32 @@ public async IAsyncEnumerable SubscribeDataChangesAsync( [EnumeratorCancellation] CancellationToken ct = default) { SubscribeCount++; - while (!ct.IsCancellationRequested) + Interlocked.Increment(ref m_activeSubscriptionCount); + try { - await m_notificationAvailable.WaitAsync(ct).ConfigureAwait(false); - if (ChangeNotifications.TryDequeue(out RobotIntentDataChange change)) - { - yield return change; - } - else if (m_ownerNotifications.TryDequeue(out NodeId owner)) + while (!ct.IsCancellationRequested) { - yield return new RobotIntentDataChange(new NodeId(1), Variant.From(owner)); + await m_notificationAvailable.WaitAsync(ct).ConfigureAwait(false); + if (ChangeNotifications.TryDequeue(out RobotIntentDataChange change)) + { + yield return change; + } + else if (m_ownerNotifications.TryDequeue(out NodeId owner)) + { + yield return new RobotIntentDataChange(new NodeId(1), Variant.From(owner)); + } } } + finally + { + Interlocked.Decrement(ref m_activeSubscriptionCount); + } } private readonly ConcurrentQueue m_ownerNotifications = new(); private readonly SemaphoreSlim m_notificationAvailable = new(0); private readonly System.Threading.Lock m_stateLock = new(); + private int m_activeSubscriptionCount; private TaskCompletionSource? m_openChannelCountReached; private int m_openChannelCountTarget = int.MaxValue; diff --git a/tests/Opc.Ua.Robotics.Tests/IntentControllerHostTests.cs b/tests/Opc.Ua.Robotics.Tests/IntentControllerHostTests.cs index b1a5854797..802d7f4f0d 100644 --- a/tests/Opc.Ua.Robotics.Tests/IntentControllerHostTests.cs +++ b/tests/Opc.Ua.Robotics.Tests/IntentControllerHostTests.cs @@ -586,10 +586,7 @@ public async Task QueuePositionsAreRenumberedAsTheQueueDrains() Assert.That(FindOperation("c")!.QueuePosition!.Value, Is.LessThanOrEqualTo(1)); } - [TestCase(BufferModeEnum.BlendingLow)] - [TestCase(BufferModeEnum.BlendingPrevious)] - [TestCase(BufferModeEnum.BlendingNext)] - [TestCase(BufferModeEnum.BlendingHigh)] + [TestCaseSource(nameof(BlendingModes))] public async Task BlendingCompletesThePredecessorAtTheReportedPose(BufferModeEnum blendingMode) { m_executor.Gate = new SemaphoreSlim(0); @@ -885,12 +882,16 @@ public async Task RetryCreatesANewOperationAndLeavesTheOriginalTerminal() Assert.Multiple(() => { Assert.That(retry.Accepted, Is.True); - Assert.That(retry.IntentId, Is.Not.EqualTo(first.IntentId)); + Assert.That(retry.IntentId, Is.EqualTo("a#attempt-2")); Assert.That(retry.Operation, Is.Not.EqualTo(first.Operation), "a retry is a new attempt, and the history of the first survives"); }); await WaitForTerminalAsync(retry.IntentId).ConfigureAwait(false); - Assert.That(original.ExecutionState!.Value, Is.EqualTo(ExecutionStateEnum.Retriable)); + Assert.Multiple(() => + { + Assert.That(original.ExecutionState!.Value, Is.EqualTo(ExecutionStateEnum.Retriable)); + Assert.That(original.Intent!.Value!.IntentId, Is.EqualTo("a")); + }); } [Test] @@ -1183,6 +1184,25 @@ public async Task NonTerminalExecutorOutcomeFailsOperationAndPumpContinues() }); } + [Test] + public async Task FailedExecutorOutcomeWithoutReasonIsNormalized() + { + m_executor.Outcome = new IntentOutcome { State = ExecutionStateEnum.Failed }; + + m_host.SubmitIntent(m_context, null, Move("missing-reason")); + IntentOperationState operation = await WaitForTerminalAsync("missing-reason") + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(operation.ExecutionState!.Value, Is.EqualTo(ExecutionStateEnum.Failed)); + Assert.That(operation.Result!.Value!.Failure, Is.EqualTo(IntentFailureEnum.Other)); + Assert.That( + operation.Result.Value.Message.Text, + Is.EqualTo("Executor reported failure without a failure classification.")); + }); + } + [Test] public async Task QueuedMissionStepCancellationAdvancesTheMission() { @@ -1278,22 +1298,26 @@ await WaitAsync(() => [Test] public async Task ReusingTerminalMissionIdCreatesDistinctMissionNode() { - MissionAdmission first = m_host.SubmitMission(m_context, null, new MissionDataType + var firstMission = new MissionDataType { MissionId = "m1", Steps = new[] { Step("s1", 1, released: true) } - }); + }; + firstMission.Steps[0].Intent!.IntentId = "m1-first"; + MissionAdmission first = m_host.SubmitMission(m_context, null, firstMission); await WaitAsync(() => { MissionObjectState mission = FindOperationByNodeId(first.Operation)!; return mission.ExecutionState?.Value == ExecutionStateEnum.Succeeded; }).ConfigureAwait(false); - MissionAdmission second = m_host.SubmitMission(m_context, null, new MissionDataType + var secondMission = new MissionDataType { MissionId = "m1", Steps = new[] { Step("s1", 1, released: true) } - }); + }; + secondMission.Steps[0].Intent!.IntentId = "m1-second"; + MissionAdmission second = m_host.SubmitMission(m_context, null, secondMission); Assert.Multiple(() => { @@ -1426,9 +1450,9 @@ private static IntentControllerHostOptions Options( AxisCount = 6, MaxQueueDepth = 4 }; - options.Accept(RiDataTypeIds.LinearMoveIntentDataType); - options.Accept(RiDataTypeIds.JointMoveIntentDataType); - options.Accept(RiDataTypeIds.GraspIntentDataType, cancelSupported: false); + options.Accept(RiDataTypeIds.LinearMoveIntentDataType) + .Accept(RiDataTypeIds.JointMoveIntentDataType) + .Accept(RiDataTypeIds.GraspIntentDataType, cancelSupported: false); return options; } @@ -1619,6 +1643,14 @@ private static async Task WaitAsync( Assert.Fail($"timed out waiting for {conditionDescription}"); } + private static IEnumerable BlendingModes() + { + yield return new TestCaseData(BufferModeEnum.BlendingLow); + yield return new TestCaseData(BufferModeEnum.BlendingPrevious); + yield return new TestCaseData(BufferModeEnum.BlendingNext); + yield return new TestCaseData(BufferModeEnum.BlendingHigh); + } + private ServiceMessageContext m_messageContext = null!; private SystemContext m_context = null!; private IntentControllerState m_controller = null!; diff --git a/tests/Opc.Ua.Robotics.Tests/IntentMissionHostTests.cs b/tests/Opc.Ua.Robotics.Tests/IntentMissionHostTests.cs new file mode 100644 index 0000000000..dbeece1159 --- /dev/null +++ b/tests/Opc.Ua.Robotics.Tests/IntentMissionHostTests.cs @@ -0,0 +1,1369 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.RobotIntent; +using Opc.Ua.RobotIntent.Server; +using Opc.Ua.Tests; +using RiNamespaces = Opc.Ua.RobotIntent.Namespaces; + +namespace Opc.Ua.Robotics.Tests +{ + /// + /// Exercises mission browseability, retention, step IntentId correlation, + /// duplicate refusal, and authoritative failure/message. + /// + [TestFixture] + public class IntentMissionHostTests + { + [SetUp] + public void SetUp() + { + ITelemetryContext telemetry = NUnitTelemetryContext.Create(true); + m_messageContext = ServiceMessageContext.Create(telemetry); + m_messageContext.NamespaceUris.Append(RiNamespaces.RobotIntent); + m_context = new SystemContext(telemetry) + { + NamespaceUris = m_messageContext.NamespaceUris, + EncodeableFactory = m_messageContext.Factory + }; + + m_controller = new IntentControllerState(null); + m_controller.Create( + m_context, + new NodeId("Controller", 1), + new QualifiedName("Controller", 1), + new LocalizedText("Controller"), + true); + + m_executor = new ScriptedExecutor(); + m_added.Clear(); + } + + [TearDown] + public void TearDown() + { + m_host?.Dispose(); + } + + [Test] + public async Task MissionIsVisibleWhileExecuting() + { + m_executor.Gate = new SemaphoreSlim(0); + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, SimpleMission("m-visible")); + + Assert.That(admission.Accepted, Is.True, "submission should be accepted"); + Assert.That(admission.MissionId, Is.EqualTo("m-visible")); + Assert.That(admission.Operation.IsNull, Is.False, "must return a node"); + + MissionObjectState? node = FindMissionNode(admission.Operation); + Assert.That(node, Is.Not.Null, "mission node should be browsable"); + + m_executor.Gate.Release(); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? terminal = FindMissionNode(admission.Operation); + Assert.That(terminal, Is.Not.Null, "terminal mission should be retained"); + } + + [Test] + public async Task TerminalMissionsArePrunedAtRetentionBound() + { + IntentControllerHostOptions options = Options(); + options.RetainedTerminalMissions = 2; + m_host = NewHost(options); + + var admissions = new List(); + for (int i = 0; i < 4; i++) + { + MissionAdmission a = m_host.SubmitMission(m_context, null, SimpleMission($"m-{i}")); + Assert.That(a.Accepted, Is.True); + admissions.Add(a); + await WaitForCompletion(a.Operation).ConfigureAwait(false); + } + + MissionObjectState? oldest = FindMissionNode(admissions[0].Operation); + MissionObjectState? secondOldest = FindMissionNode(admissions[1].Operation); + MissionObjectState? newest = FindMissionNode(admissions[3].Operation); + + Assert.Multiple(() => + { + Assert.That(oldest, Is.Null, "oldest terminal mission should be pruned"); + Assert.That(secondOldest, Is.Null, "second-oldest terminal mission should be pruned"); + Assert.That(newest, Is.Not.Null, "newest terminal mission should be retained"); + }); + } + + [Test] + public void DuplicateActiveMissionIdIsRefused() + { + m_executor.Gate = new SemaphoreSlim(0); + m_host = NewHost(Options()); + + MissionAdmission first = m_host.SubmitMission(m_context, null, SimpleMission("dup")); + Assert.That(first.Accepted, Is.True); + + MissionAdmission second = m_host.SubmitMission(m_context, null, SimpleMission("dup")); + + Assert.Multiple(() => + { + Assert.That(second.Accepted, Is.False, "duplicate active mission should be refused"); + Assert.That(second.Message, Does.Contain("dup")); + }); + + m_executor.Gate!.Release(); + } + + [Test] + public async Task MissionStepIntentIdCorrelatesMissionAndStep() + { + m_host = NewHost(Options()); + var admitted = new List(); + m_executor.OnExecute = exec => admitted.Add(exec.IntentId); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "corr-mission", + Steps = + [ + new MissionStepDataType + { + StepId = "step-1", + SequenceId = 1, + Released = true, + Intent = Move() + }, + new MissionStepDataType + { + StepId = "step-2", + SequenceId = 2, + Released = true, + Intent = Move() + } + ] + }); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(admitted, Has.Count.EqualTo(2)); + Assert.That(admitted[0], Is.EqualTo("corr-mission/step-1")); + Assert.That(admitted[1], Is.EqualTo("corr-mission/step-2")); + }); + } + + [Test] + public async Task MissionFailurePublishesAuthoritativeFailureAndMessage() + { + m_executor.Outcome = IntentOutcome.Fail( + IntentFailureEnum.Other, "arm collision detected"); + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission( + m_context, null, SimpleMission("fail-mission")); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + Assert.That(node, Is.Not.Null); + + IntentFailureEnum failure = ReadFailureEnum(node!); + string message = ReadFailureMessage(node!); + + Assert.Multiple(() => + { + Assert.That(failure, Is.EqualTo(IntentFailureEnum.Other)); + Assert.That(message, Is.EqualTo("arm collision detected")); + }); + } + + [Test] + public async Task ExecutorExceptionPublishesExactFailureAndMessage() + { + m_executor.Exception = new InvalidOperationException("executor fault"); + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission( + m_context, + null, + SimpleMission("executor-exception")); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + + Assert.Multiple(() => + { + Assert.That(node, Is.Not.Null); + Assert.That(ReadFailureEnum(node!), Is.EqualTo(IntentFailureEnum.Other)); + Assert.That(ReadFailureMessage(node!), Is.EqualTo("executor fault")); + }); + } + + [Test] + public async Task StepAdmissionRefusalPublishesExactFailureAndMessage() + { + m_executor.Gate = new SemaphoreSlim(0); + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "step-refusal", + Steps = + [ + new MissionStepDataType + { + StepId = "first", + SequenceId = 1, + Released = true, + Intent = MoveWithId("first-step") + }, + new MissionStepDataType + { + StepId = "second", + SequenceId = 2, + Released = true, + Intent = MoveWithId("late-collision") + } + ] + }); + Assert.That(admission.Accepted, Is.True); + + var unrelated = MoveWithId("late-collision"); + unrelated.BufferMode = BufferModeEnum.Buffered; + IntentAdmission collision = m_host.SubmitIntent(m_context, null, unrelated); + Assert.That(collision.Accepted, Is.True); + + m_executor.Gate.Release(); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + + Assert.Multiple(() => + { + Assert.That(node, Is.Not.Null); + Assert.That(ReadFailureEnum(node!), Is.EqualTo(IntentFailureEnum.ParameterInvalid)); + Assert.That( + ReadFailureMessage(node!), + Is.EqualTo("IntentId 'late-collision' is already retained by another operation.")); + }); + } + + [Test] + public async Task TerminalMissionRetentionParallelsOperationRetention() + { + IntentControllerHostOptions options = Options(); + options.RetainedTerminalOperations = 4; + options.RetainedTerminalMissions = 4; + m_host = NewHost(options); + + for (int i = 0; i < 6; i++) + { + MissionAdmission a = m_host.SubmitMission(m_context, null, SimpleMission($"par-{i}")); + Assert.That(a.Accepted, Is.True); + await WaitForCompletion(a.Operation).ConfigureAwait(false); + } + + int missionCount = m_added.OfType().Count(); + Assert.That(missionCount, Is.GreaterThanOrEqualTo(4), + "at least RetainedTerminalMissions missions should survive"); + } + + [Test] + public async Task ExplicitIntentIdIsPreservedThroughExecution() + { + m_host = NewHost(Options()); + var admitted = new List(); + m_executor.OnExecute = exec => admitted.Add(exec.IntentId); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "preserve-id", + Steps = + [ + new MissionStepDataType + { + StepId = "step-a", + SequenceId = 1, + Released = true, + Intent = MoveWithId("my-explicit-id") + } + ] + }); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + Assert.That(admitted, Has.Count.EqualTo(1)); + Assert.That(admitted[0], Is.EqualTo("my-explicit-id")); + } + + [Test] + public async Task GeneratedIntentIdFollowsMissionSlashStepPattern() + { + m_host = NewHost(Options()); + var admitted = new List(); + m_executor.OnExecute = exec => admitted.Add(exec.IntentId); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "gen-mission", + Steps = + [ + new MissionStepDataType + { + StepId = "step-x", + SequenceId = 1, + Released = true, + Intent = Move() + } + ] + }); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + Assert.That(admitted, Has.Count.EqualTo(1)); + Assert.That(admitted[0], Is.EqualTo("gen-mission/step-x")); + } + + [Test] + public void DuplicateExplicitIntentIdWithinMissionIsRefused() + { + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "dup-id-mission", + Steps = + [ + new MissionStepDataType + { + StepId = "s1", + SequenceId = 1, + Released = true, + Intent = MoveWithId("same-id") + }, + new MissionStepDataType + { + StepId = "s2", + SequenceId = 2, + Released = true, + Intent = MoveWithId("same-id") + } + ] + }); + + Assert.Multiple(() => + { + Assert.That(admission.Accepted, Is.False); + Assert.That(admission.Message, Does.Contain("same-id")); + }); + } + + [Test] + public void ExplicitAndGeneratedIntentIdsArePreflightedBeforeAnyStepExecutes() + { + m_host = NewHost(Options()); + int executions = 0; + m_executor.OnExecute = _ => Interlocked.Increment(ref executions); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "two-pass", + Steps = + [ + new MissionStepDataType + { + StepId = "first", + SequenceId = 1, + Released = true, + Intent = MoveWithId("two-pass/second") + }, + new MissionStepDataType + { + StepId = "second", + SequenceId = 2, + Released = true, + Intent = Move() + } + ] + }); + + Assert.Multiple(() => + { + Assert.That(admission.Accepted, Is.False); + Assert.That(admission.Message, Does.Contain("two-pass/second")); + Assert.That(executions, Is.Zero); + }); + } + + [Test] + public async Task ExplicitIntentIdCollisionWithTerminalRetainedOperationIsRefused() + { + m_host = NewHost(Options()); + + IntentAdmission standalone = m_host.SubmitIntent( + m_context, + null, + MoveWithId("retained-terminal")); + Assert.That(standalone.Accepted, Is.True); + await WaitForIntentCompletion(standalone.Operation).ConfigureAwait(false); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "collide-terminal", + Steps = + [ + new MissionStepDataType + { + StepId = "s1", + SequenceId = 1, + Released = true, + Intent = MoveWithId("retained-terminal") + } + ] + }); + + Assert.Multiple(() => + { + Assert.That(admission.Accepted, Is.False); + Assert.That(admission.Failure, Is.EqualTo(IntentFailureEnum.ParameterInvalid)); + Assert.That(admission.Message, Does.Contain("retained-terminal")); + }); + } + + [Test] + public async Task RetryUsesStableStepIdentityAndMapsItsLatestOperation() + { + m_host = NewHost(Options()); + var executed = new List(); + int call = 0; + m_executor.OnExecute = execution => executed.Add(execution.IntentId); + m_executor.OutcomeFunc = _ => Interlocked.Increment(ref call) == 1 + ? IntentOutcome.Fail(IntentFailureEnum.Other, "retry once") + : IntentOutcome.Success; + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "retry-stable", + Steps = + [ + new MissionStepDataType + { + StepId = "step", + SequenceId = 1, + Released = true, + ErrorPolicy = ErrorPolicyEnum.Retry, + Intent = MoveWithId("stable-intent") + } + ] + }); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + MissionDataType mission = node!.Mission?.Value is MissionDataType value + ? value + : new MissionDataType(); + MissionStepDataType step = mission.Steps[0]; + + Assert.Multiple(() => + { + Assert.That(executed, Is.EqualTo(["stable-intent", "stable-intent#attempt-2"])); + Assert.That(step.Intent!.IntentId, Is.EqualTo("stable-intent#attempt-2")); + Assert.That(step.Operation.IsNull, Is.False); + Assert.That(step.Status, Is.EqualTo(ExecutionStateEnum.Succeeded)); + }); + } + + [TestCaseSource(nameof(RevisitedErrorPolicies))] + public async Task RevisitedFallbackOrCompensationStepUsesNextStableAttemptIdentity( + ErrorPolicyEnum policy, + ExecutionStateEnum expectedState) + { + m_host = NewHost(Options()); + var executed = new List(); + m_executor.OnExecute = execution => executed.Add(execution.IntentId); + m_executor.OutcomeFunc = execution => execution.IntentId switch + { + "root" => IntentOutcome.Fail(IntentFailureEnum.Other, "root fault"), + "recovery" => IntentOutcome.Fail(IntentFailureEnum.Other, "revisit recovery"), + _ => IntentOutcome.Success + }; + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "revisit", + Steps = + [ + new MissionStepDataType + { + StepId = "root", + SequenceId = 1, + Released = true, + ErrorPolicy = policy, + FallbackStepId = "recovery", + Intent = MoveWithId("root") + }, + new MissionStepDataType + { + StepId = "recovery", + SequenceId = 2, + Released = true, + ErrorPolicy = policy, + FallbackStepId = "recovery", + Intent = MoveWithId("recovery") + } + ] + }); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + MissionDataType mission = node!.Mission?.Value is MissionDataType missionData + ? missionData + : new MissionDataType(); + MissionStepDataType recovery = mission.Steps[1]; + ExecutionStateEnum missionState = node.ExecutionState?.Value is ExecutionStateEnum stateValue + ? stateValue + : ExecutionStateEnum.Accepted; + + Assert.Multiple(() => + { + Assert.That(executed, Is.EqualTo(["root", "recovery", "recovery#attempt-2"])); + Assert.That(recovery.Intent!.IntentId, Is.EqualTo("recovery#attempt-2")); + Assert.That(recovery.Operation.IsNull, Is.False); + Assert.That(recovery.Status, Is.EqualTo(expectedState)); + Assert.That( + missionState, + Is.EqualTo(policy == ErrorPolicyEnum.Compensate + ? ExecutionStateEnum.Failed + : ExecutionStateEnum.Succeeded)); + }); + } + + [Test] + public async Task FailedMissionPublishesExactStepFailureEnum() + { + m_executor.Outcome = IntentOutcome.Fail( + IntentFailureEnum.SafetyLimitExceeded, + "joint limit breached"); + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, SimpleMission("exact-fail")); + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + Assert.That(node, Is.Not.Null); + + IntentFailureEnum failure = ReadFailureEnum(node!); + string message = ReadFailureMessage(node); + + Assert.Multiple(() => + { + Assert.That(failure, Is.EqualTo(IntentFailureEnum.SafetyLimitExceeded)); + Assert.That(message, Does.Contain("joint limit breached")); + }); + } + + [Test] + public async Task FailedMissionNeverHasFailureNone() + { + m_executor.Outcome = IntentOutcome.Fail(IntentFailureEnum.None, string.Empty); + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, SimpleMission("no-none")); + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + Assert.That(node, Is.Not.Null); + ExecutionStateEnum state = node!.ExecutionState?.Value is ExecutionStateEnum s + ? s : ExecutionStateEnum.Accepted; + + Assert.Multiple(() => + { + Assert.That(state, Is.EqualTo(ExecutionStateEnum.Failed)); + IntentFailureEnum failure = ReadFailureEnum(node); + Assert.That(failure, Is.Not.EqualTo(IntentFailureEnum.None), + "A failed mission must never leave Failure=None"); + }); + } + + [Test] + public async Task SucceededMissionHasFailureNone() + { + m_executor.Outcome = IntentOutcome.Success; + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, SimpleMission("success-m")); + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + Assert.That(node, Is.Not.Null); + ExecutionStateEnum state = node!.ExecutionState?.Value is ExecutionStateEnum s + ? s : ExecutionStateEnum.Accepted; + + Assert.Multiple(() => Assert.That(state, Is.EqualTo(ExecutionStateEnum.Succeeded))); + } + + [Test] + public async Task CancelledMissionReportsCorrectState() + { + m_executor.Gate = new SemaphoreSlim(0); + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, SimpleMission("cancel-m")); + Assert.That(admission.Accepted, Is.True); + + bool cancelled = m_host.CancelMission(m_context, null, "cancel-m", StopModeEnum.QuickStop); + Assert.That(cancelled, Is.True); + + m_executor.Gate.Release(); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + Assert.That(node, Is.Not.Null); + ExecutionStateEnum state = node!.ExecutionState?.Value is ExecutionStateEnum s + ? s : ExecutionStateEnum.Accepted; + MissionDataType mission = node.Mission?.Value is MissionDataType value + ? value + : new MissionDataType(); + + Assert.Multiple(() => + { + Assert.That(state, Is.EqualTo(ExecutionStateEnum.Cancelled)); + Assert.That(mission.Steps[0].Status, Is.EqualTo(ExecutionStateEnum.Cancelled)); + Assert.That(node.CurrentStepId!.Value, Is.Empty); + }); + } + + [Test] + public async Task ErrorPolicyAbortFailsMission() + { + IntentControllerHostOptions options = Options(); + options.MaxStepRetries = 3; + m_host = NewHost(options); + m_executor.Outcome = IntentOutcome.Fail(IntentFailureEnum.Other, "step fault"); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "abort-policy", + Steps = + [ + new MissionStepDataType + { + StepId = "s1", + SequenceId = 1, + Released = true, + ErrorPolicy = ErrorPolicyEnum.Abort, + Intent = Move() + }, + new MissionStepDataType + { + StepId = "s2", + SequenceId = 2, + Released = true, + Intent = Move() + } + ] + }); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + Assert.That(node, Is.Not.Null); + ExecutionStateEnum state = node!.ExecutionState?.Value is ExecutionStateEnum s + ? s : ExecutionStateEnum.Accepted; + Assert.That(state, Is.EqualTo(ExecutionStateEnum.Failed), + "Abort policy should stop the mission after first step fails"); + } + + [Test] + public async Task ErrorPolicySkipContinuesToNextStep() + { + m_host = NewHost(Options()); + int callCount = 0; + m_executor.OnExecute = _ => callCount++; + int failOnFirst = 0; + m_executor.OutcomeFunc = exec => + { + int idx = Interlocked.Increment(ref failOnFirst); + return idx == 1 + ? IntentOutcome.Fail(IntentFailureEnum.Other, "skippable") + : IntentOutcome.Success; + }; + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "skip-policy", + Steps = + [ + new MissionStepDataType + { + StepId = "s1", + SequenceId = 1, + Released = true, + ErrorPolicy = ErrorPolicyEnum.Skip, + Intent = Move() + }, + new MissionStepDataType + { + StepId = "s2", + SequenceId = 2, + Released = true, + Intent = Move() + } + ] + }); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + Assert.That(callCount, Is.EqualTo(2), "Skip should have continued to s2"); + MissionObjectState? node = FindMissionNode(admission.Operation); + ExecutionStateEnum state = node!.ExecutionState?.Value is ExecutionStateEnum s + ? s : ExecutionStateEnum.Accepted; + Assert.That(state, Is.EqualTo(ExecutionStateEnum.Succeeded)); + } + + [Test] + public async Task StepIntentIdWrittenAtAdmissionBeforeExecution() + { + m_executor.Gate = new SemaphoreSlim(0); + m_host = NewHost(Options()); + + var mission = new MissionDataType + { + MissionId = "imm-corr", + Steps = + [ + new MissionStepDataType + { + StepId = "step-one", + SequenceId = 1, + Released = true, + Intent = Move() + } + ] + }; + + MissionAdmission admission = m_host.SubmitMission(m_context, null, mission); + Assert.That(admission.Accepted, Is.True); + + Assert.That(mission.Steps[0].Intent!.IntentId, Is.EqualTo("imm-corr/step-one"), + "IntentId should be written into the step at admission, before execution"); + + m_executor.Gate.Release(); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + } + + [Test] + public async Task MultiStepMissionTracksOperationNodesPerStep() + { + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission(m_context, null, new MissionDataType + { + MissionId = "multi-ops", + Steps = + [ + new MissionStepDataType + { + StepId = "s1", + SequenceId = 1, + Released = true, + Intent = Move() + }, + new MissionStepDataType + { + StepId = "s2", + SequenceId = 2, + Released = true, + Intent = Move() + } + ] + }); + + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + Assert.That(node, Is.Not.Null); + + ArrayOf steps = node!.Mission?.Value is MissionDataType md + ? md.Steps + : []; + Assert.That(steps.Count, Is.GreaterThanOrEqualTo(2)); + Assert.That(steps[0].Operation.IsNull, Is.False, + "First step should have an operation NodeId"); + Assert.That(steps[1].Operation.IsNull, Is.False, + "Second step should have an operation NodeId"); + } + + [Test] + public async Task GatedStepHasOperationAndStateBeforeRelease() + { + m_executor.Gate = new SemaphoreSlim(0); + var executionStarted = new TaskCompletionSource( + TaskCreationOptions.RunContinuationsAsynchronously); + m_executor.OnExecute = _ => executionStarted.TrySetResult(true); + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission( + m_context, null, SimpleMission("gated-map")); + Assert.That(admission.Accepted, Is.True); + + MissionObjectState? node = FindMissionNode(admission.Operation); + Assert.That(node, Is.Not.Null); + await executionStarted.Task.ConfigureAwait(false); + + MissionDataType md = node!.Mission?.Value is MissionDataType m + ? m : new MissionDataType(); + Assert.That(md.Steps.Count, Is.GreaterThanOrEqualTo(1)); + + MissionStepDataType firstStep = md.Steps[0]; + Assert.Multiple(() => + { + Assert.That(firstStep.Operation.IsNull, Is.False, + "Operation should be set before execution completes"); + Assert.That(firstStep.Intent!.IntentId, Is.EqualTo("gated-map/s1")); + Assert.That(firstStep.Status, Is.EqualTo(ExecutionStateEnum.Executing)); + }); + + m_executor.Gate.Release(); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + } + + [Test] + public async Task DynamicMissionsFolderUsesRobotIntentNamespace() + { + m_host = NewHost(Options()); + + MissionAdmission admission = m_host.SubmitMission( + m_context, null, SimpleMission("ns-check")); + Assert.That(admission.Accepted, Is.True); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + FolderState? missionsFolder = null; + lock (m_addedLock) + { + missionsFolder = m_added.OfType() + .FirstOrDefault(f => f.BrowseName.Name == "Missions"); + } + + if (missionsFolder != null) + { + ushort riNs = (ushort)m_messageContext.NamespaceUris.GetIndex( + RiNamespaces.RobotIntent); + MissionObjectState? mission = FindMissionNode(admission.Operation); + Assert.That( + missionsFolder.BrowseName.NamespaceIndex, + Is.EqualTo(riNs), + "Dynamically created Missions folder must use RobotIntent namespace"); + Assert.That(mission, Is.Not.Null); + Assert.That( + mission!.BrowseName.NamespaceIndex, + Is.EqualTo(m_controller.BrowseName.NamespaceIndex), + "Dynamic mission instances must use the controller instance namespace."); + } + } + + [Test] + public async Task ReusedMissionIdRetainsPriorInvocationUntilPruning() + { + IntentControllerHostOptions options = Options(); + options.RetainedTerminalMissions = 32; + m_host = NewHost(options); + + MissionAdmission first = m_host.SubmitMission( + m_context, null, SimpleMission("reuse-id", "reuse-id-first")); + Assert.That(first.Accepted, Is.True); + await WaitForCompletion(first.Operation).ConfigureAwait(false); + NodeId firstNode = first.Operation; + + MissionAdmission second = m_host.SubmitMission( + m_context, null, SimpleMission("reuse-id", "reuse-id-second")); + Assert.That(second.Accepted, Is.True); + await WaitForCompletion(second.Operation).ConfigureAwait(false); + NodeId secondNode = second.Operation; + + Assert.That(secondNode, Is.Not.EqualTo(firstNode), + "Reused MissionId should create a new node"); + + MissionObjectState? oldNode = FindMissionNode(firstNode); + Assert.That(oldNode, Is.Not.Null, + "A reused MissionId must not overwrite the prior retained invocation."); + + MissionObjectState? newNode = FindMissionNode(secondNode); + Assert.That(newNode, Is.Not.Null, + "New mission node should still be browseable"); + } + + [Test] + public async Task ReusedMissionIdWithOmittedIntentIdGeneratesANewRunIdentity() + { + m_host = NewHost(Options()); + MissionDataType firstMission = SimpleMission("reuse-generated"); + MissionDataType secondMission = SimpleMission("reuse-generated"); + + MissionAdmission first = m_host.SubmitMission(m_context, null, firstMission); + Assert.That(first.Accepted, Is.True); + await WaitForCompletion(first.Operation).ConfigureAwait(false); + + MissionAdmission second = m_host.SubmitMission(m_context, null, secondMission); + Assert.That(second.Accepted, Is.True); + await WaitForCompletion(second.Operation).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That( + firstMission.Steps[0].Intent!.IntentId, + Is.EqualTo("reuse-generated/s1")); + Assert.That( + secondMission.Steps[0].Intent!.IntentId, + Is.EqualTo("reuse-generated/s1#run-2")); + Assert.That(second.Operation, Is.Not.EqualTo(first.Operation)); + }); + } + + [Test] + public async Task ReusedMissionIdsPruneOldestTerminalInvocationsAtRetentionBound() + { + IntentControllerHostOptions options = Options(); + options.RetainedTerminalMissions = 2; + m_host = NewHost(options); + + var admissions = new List(); + for (int ii = 0; ii < 4; ii++) + { + MissionAdmission admission = m_host.SubmitMission( + m_context, + null, + SimpleMission("reused-retention", $"reused-retention-{ii}")); + Assert.That(admission.Accepted, Is.True); + admissions.Add(admission); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + } + + Assert.Multiple(() => + { + Assert.That(FindMissionNode(admissions[0].Operation), Is.Null); + Assert.That(FindMissionNode(admissions[1].Operation), Is.Null); + Assert.That(FindMissionNode(admissions[2].Operation), Is.Not.Null); + Assert.That(FindMissionNode(admissions[3].Operation), Is.Not.Null); + }); + } + + [Test] + public async Task ExplicitIntentIdCollisionWithRetainedIsRefused() + { + m_host = NewHost(Options()); + + IntentAdmission standalone = m_host.SubmitIntent( + m_context, null, MoveWithId("retained-op")); + Assert.That(standalone.Accepted, Is.True); + + MissionAdmission admission = m_host.SubmitMission( + m_context, null, new MissionDataType + { + MissionId = "collide-retained", + Steps = + [ + new MissionStepDataType + { + StepId = "s1", + SequenceId = 1, + Released = true, + Intent = MoveWithId("retained-op") + } + ] + }); + + Assert.Multiple(() => + { + Assert.That(admission.Accepted, Is.False); + Assert.That(admission.Message, Does.Contain("retained-op")); + }); + } + + [Test] + public void RefusedStepAdmissionPropagatesFailure() + { + IntentControllerHostOptions options = Options(); + m_host = NewHost(options); + + MissionAdmission admission = m_host.SubmitMission( + m_context, null, new MissionDataType + { + MissionId = "refused-step", + Steps = + [ + new MissionStepDataType + { + StepId = "s1", + SequenceId = 1, + Released = true, + Intent = MoveWithId("collide-explicit") + }, + new MissionStepDataType + { + StepId = "s2", + SequenceId = 2, + Released = true, + Intent = MoveWithId("collide-explicit") + } + ] + }); + + Assert.Multiple(() => + { + Assert.That(admission.Accepted, Is.False, + "Mission with duplicate explicit IDs should be refused"); + Assert.That( + admission.Failure, + Is.EqualTo(IntentFailureEnum.ParameterInvalid)); + Assert.That( + admission.Message, + Does.Contain("collide-explicit")); + }); + } + + [Test] + public async Task HorizonUpdateWithOmittedIntentIdsPreservesBaseAndCompletes() + { + m_executor.Gate = new SemaphoreSlim(0); + IntentControllerHostOptions options = Options(); + options.MissionHorizonSupported = true; + m_host = NewHost(options); + + MissionAdmission admission = m_host.SubmitMission( + m_context, + null, + new MissionDataType + { + MissionId = "updated-horizon", + MissionUpdateId = 1, + Steps = + [ + new MissionStepDataType + { + StepId = "base", + SequenceId = 1, + Released = true, + Intent = Move() + }, + new MissionStepDataType + { + StepId = "horizon", + SequenceId = 2, + Released = false, + Intent = Move() + } + ] + }); + Assert.That(admission.Accepted, Is.True); + + MissionUpdateOutcome update = m_host.UpdateMission( + m_context, + null, + "updated-horizon", + 2, + [ + new MissionStepDataType + { + StepId = "base", + SequenceId = 1, + Released = true, + Intent = Move() + }, + new MissionStepDataType + { + StepId = "horizon", + SequenceId = 2, + Released = true, + Intent = Move() + } + ]); + Assert.That(update.Result, Is.EqualTo(MissionUpdateResultEnum.Accepted)); + + m_executor.Gate.Release(2); + await WaitForCompletion(admission.Operation).ConfigureAwait(false); + + MissionObjectState? node = FindMissionNode(admission.Operation); + MissionDataType published = node?.Mission?.Value is MissionDataType value + ? value + : new MissionDataType(); + Assert.Multiple(() => + { + Assert.That(node, Is.Not.Null); + Assert.That(node!.ExecutionState!.Value, Is.EqualTo(ExecutionStateEnum.Succeeded)); + Assert.That(published.Steps, Has.Count.EqualTo(2)); + Assert.That(published.Steps[0].Intent!.IntentId, Is.EqualTo("updated-horizon/base")); + Assert.That(published.Steps[1].Intent!.IntentId, Is.EqualTo("updated-horizon/horizon")); + Assert.That(published.Steps[0].Operation.IsNull, Is.False); + Assert.That(published.Steps[1].Operation.IsNull, Is.False); + Assert.That(published.Steps[1].Status, Is.EqualTo(ExecutionStateEnum.Succeeded)); + }); + } + + private IntentControllerHostOptions Options() + { + var options = new IntentControllerHostOptions + { + RequireControlAuthority = false, + MissionsSupported = true, + RetainedTerminalMissions = 32 + }; + options.Accept(global::Opc.Ua.RobotIntent.DataTypeIds.LinearMoveIntentDataType); + return options; + } + + private IntentControllerHost NewHost(IntentControllerHostOptions options) + { + var host = new IntentControllerHost( + m_controller, + m_executor, + (node, ct) => + { + lock (m_addedLock) + { + m_added.Add(node); + } + return default; + }, + options, + (node, ct) => + { + lock (m_addedLock) + { + m_added.Remove(node); + } + return default; + }); + host.Start(m_context); + return host; + } + + private static MissionDataType SimpleMission(string missionId, string? intentId = null) + { + LinearMoveIntentDataType intent = Move(); + intent.IntentId = intentId ?? string.Empty; + return new MissionDataType + { + MissionId = missionId, + Steps = + [ + new MissionStepDataType + { + StepId = "s1", + SequenceId = 1, + Released = true, + Intent = intent + } + ] + }; + } + + private static LinearMoveIntentDataType Move() + { + return new LinearMoveIntentDataType + { + BufferMode = BufferModeEnum.Aborting, + Target = new Pose3DDataType + { + FrameId = "base", + Position = [1.0, 0.0, 0.0], + Orientation = [0.0, 0.0, 0.0, 1.0] + } + }; + } + + private static LinearMoveIntentDataType MoveWithId(string intentId) + { + var intent = Move(); + intent.IntentId = intentId; + return intent; + } + + private MissionObjectState? FindMissionNode(NodeId nodeId) + { + lock (m_addedLock) + { + return m_added.OfType() + .FirstOrDefault(n => n.NodeId == nodeId); + } + } + + private async Task WaitForIntentCompletion(NodeId operationNodeId) + { + for (int i = 0; i < 200; i++) + { + IntentOperationState? node; + lock (m_addedLock) + { + node = m_added.OfType() + .FirstOrDefault(candidate => candidate.NodeId == operationNodeId); + } + if (node?.ExecutionState?.Value is ExecutionStateEnum state && IntentOutcome.IsTerminal(state)) + { + return; + } + await Task.Delay(10).ConfigureAwait(false); + } + Assert.Fail("Intent did not reach terminal state within 2 seconds"); + } + + private IntentFailureEnum ReadFailureEnum(MissionObjectState node) + { + BaseObjectState? finalResult = node.FinalResultData; + if (finalResult == null) + { + return IntentFailureEnum.None; + } + if (finalResult.FindChild( + m_context, + new QualifiedName("Failure", node.BrowseName.NamespaceIndex)) + is BaseDataVariableState failureVar) + { + Variant v = failureVar.WrappedValue; + if (v.TryGetValue(out int intVal)) + { + return EnumHelper.Int32ToEnum(intVal); + } + } + return IntentFailureEnum.None; + } + + private string ReadFailureMessage(MissionObjectState node) + { + BaseObjectState? finalResult = node.FinalResultData; + if (finalResult == null) + { + return string.Empty; + } + if (finalResult.FindChild( + m_context, + new QualifiedName("Message", node.BrowseName.NamespaceIndex)) + is BaseDataVariableState messageVar) + { + Variant v = messageVar.WrappedValue; + if (v.TryGetValue(out LocalizedText text)) + { + return text.Text ?? string.Empty; + } + } + return string.Empty; + } + + private async Task WaitForCompletion(NodeId operationNodeId) + { + for (int i = 0; i < 200; i++) + { + MissionObjectState? node = FindMissionNode(operationNodeId); + if (node != null) + { + ExecutionStateEnum state = node.ExecutionState?.Value is ExecutionStateEnum s + ? s + : ExecutionStateEnum.Accepted; + if (IntentOutcome.IsTerminal(state)) + { + return; + } + } + await Task.Delay(10).ConfigureAwait(false); + } + Assert.Fail("Mission did not reach terminal state within 2 seconds"); + } + + private static IEnumerable RevisitedErrorPolicies() + { + yield return new TestCaseData( + ErrorPolicyEnum.Fallback, + ExecutionStateEnum.Succeeded); + yield return new TestCaseData( + ErrorPolicyEnum.Compensate, + ExecutionStateEnum.Succeeded); + } + + private ServiceMessageContext m_messageContext = null!; + private SystemContext m_context = null!; + private IntentControllerState m_controller = null!; + private ScriptedExecutor m_executor = null!; + private IntentControllerHost? m_host; + private readonly Lock m_addedLock = new(); + private readonly List m_added = []; + + private sealed class ScriptedExecutor : IIntentExecutor + { + public SemaphoreSlim? Gate { get; set; } + public IntentOutcome Outcome { get; set; } = IntentOutcome.Success; + public Func? OutcomeFunc { get; set; } + public Action? OnExecute { get; set; } + public Exception? Exception { get; set; } + + public async ValueTask ExecuteAsync( + IntentExecution execution, CancellationToken cancellationToken) + { + OnExecute?.Invoke(execution); + if (Exception != null) + { + throw Exception; + } + if (Gate != null) + { + await Gate.WaitAsync(cancellationToken).ConfigureAwait(false); + } + return OutcomeFunc?.Invoke(execution) ?? Outcome; + } + + public bool CanCancel(IntentExecution execution) + { + return true; + } + } + } +} diff --git a/tests/Opc.Ua.Robotics.Tests/IntentMissionHostedClientTests.cs b/tests/Opc.Ua.Robotics.Tests/IntentMissionHostedClientTests.cs new file mode 100644 index 0000000000..a80b21af33 --- /dev/null +++ b/tests/Opc.Ua.Robotics.Tests/IntentMissionHostedClientTests.cs @@ -0,0 +1,527 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +#if NET10_0 +using System; +using System.IO; +using System.Linq; +using System.Net; +using System.Net.Sockets; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using NUnit.Framework; +using Opc.Ua.Client; +using Opc.Ua.Client.Subscriptions; +using Opc.Ua.Client.Subscriptions.Streaming; +using Opc.Ua.Configuration; +using Opc.Ua.Robotics.Client.Intent; +using Opc.Ua.Robotics.Server; +using Opc.Ua.Robotics.Server.Builders; +using Opc.Ua.RobotIntent; +using Opc.Ua.RobotIntent.Server; +using Opc.Ua.Server.Hosting; + +namespace Opc.Ua.Robotics.Tests +{ + /// + /// Exercises mission discovery over a hosted OPC UA server and a real client Session. + /// + [TestFixture] + [NonParallelizable] + public sealed class IntentMissionHostedClientTests + { + [Test] + public async Task HostedMissionIsListedWhileActiveAndAfterTerminalCompletion() + { + await using var fixture = new HostedMissionFixture(); + await fixture.StartAsync().ConfigureAwait(false); + await using HostedMissionClient client = await fixture.ConnectAsync().ConfigureAwait(false); + RobotIntentClient discovery = new(client.Session, fixture.Telemetry, client.Streaming); + ArrayOf controllers = await discovery + .DiscoverControllersAsync() + .ConfigureAwait(false); + Assert.That(controllers, Has.Count.EqualTo(1)); + RobotIntentControllerClient controller = discovery.Controller(controllers[0].NodeId); + await using CommandAuthorityLease authority = await controller.RequireAuthorityAsync() + .ConfigureAwait(false); + + MissionSubmissionResult admission = await controller.SubmitMissionAsync(CreateMission("hosted-mission")) + .ConfigureAwait(false); + Assert.That(admission.Accepted, Is.True, admission.Message.Text); + await AwaitWithTimeoutAsync(fixture.WaitForExecutionStartAsync(), TimeSpan.FromSeconds(10)) + .ConfigureAwait(false); + + RobotIntentControllerState state = await controller.ReadStateAsync().ConfigureAwait(false); + ArrayOf activeMissions = await controller.ListMissionsAsync().ConfigureAwait(false); + MissionSnapshot[] activeMissionArray = activeMissions.ToArray()!; + + Assert.Multiple(() => + { + Assert.That(state.ActiveMission.Available, Is.True); + Assert.That(state.ActiveMission.Value, Is.EqualTo(admission.Operation)); + Assert.That( + activeMissionArray, + Has.One.Matches(snapshot => + snapshot.MissionId == "hosted-mission" && + snapshot.MissionNode == admission.Operation && + !RobotIntentRules.IsTerminal(snapshot.ExecutionState))); + }); + + fixture.ReleaseExecution(); + await WaitUntilAsync( + async () => + { + ArrayOf missions = await controller.ListMissionsAsync().ConfigureAwait(false); + for (int ii = 0; ii < missions.Count; ii++) + { + if (missions[ii].MissionNode == admission.Operation && + RobotIntentRules.IsTerminal(missions[ii].ExecutionState)) + { + return true; + } + } + return false; + }, + "hosted mission terminal completion").ConfigureAwait(false); + + ArrayOf terminalMissions = await controller.ListMissionsAsync().ConfigureAwait(false); + MissionSnapshot[] terminalMissionArray = terminalMissions.ToArray()!; + Assert.That( + terminalMissionArray, + Has.One.Matches(snapshot => + snapshot.MissionId == "hosted-mission" && + snapshot.MissionNode == admission.Operation && + snapshot.ExecutionState == ExecutionStateEnum.Succeeded)); + } + + [Test] + public async Task HostedFailedMissionPublishesFailureAndMessageToClient() + { + await using var fixture = new HostedMissionFixture(); + fixture.SetOutcome(IntentOutcome.Fail( + IntentFailureEnum.SafetyLimitExceeded, + "hosted safety limit")); + await fixture.StartAsync().ConfigureAwait(false); + await using HostedMissionClient client = await fixture.ConnectAsync().ConfigureAwait(false); + RobotIntentClient discovery = new(client.Session, fixture.Telemetry, client.Streaming); + ArrayOf controllers = await discovery + .DiscoverControllersAsync() + .ConfigureAwait(false); + RobotIntentControllerClient controller = discovery.Controller(controllers[0].NodeId); + await using CommandAuthorityLease authority = await controller.RequireAuthorityAsync() + .ConfigureAwait(false); + + MissionSubmissionResult admission = await controller + .SubmitMissionAsync(CreateMission("hosted-failure")) + .ConfigureAwait(false); + Assert.That(admission.Accepted, Is.True, admission.Message.Text); + await AwaitWithTimeoutAsync( + fixture.WaitForExecutionStartAsync(), + TimeSpan.FromSeconds(10)).ConfigureAwait(false); + + fixture.ReleaseExecution(); + MissionSnapshot? terminal = null; + await WaitUntilAsync( + async () => + { + ArrayOf missions = await controller + .ListMissionsAsync() + .ConfigureAwait(false); + terminal = missions.ToArray()!.FirstOrDefault(snapshot => + snapshot.MissionNode == admission.Operation && + RobotIntentRules.IsTerminal(snapshot.ExecutionState)); + return terminal != null; + }, + "hosted failed mission result").ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(terminal, Is.Not.Null); + Assert.That(terminal!.ExecutionState, Is.EqualTo(ExecutionStateEnum.Failed)); + Assert.That(terminal.Failure, Is.EqualTo(IntentFailureEnum.SafetyLimitExceeded)); + Assert.That(terminal.FailureMessage.Text, Is.EqualTo("hosted safety limit")); + }); + } + + private static MissionDataType CreateMission(string missionId) + { + return new MissionDataType + { + MissionId = missionId, + Steps = + [ + new MissionStepDataType + { + StepId = "gated", + SequenceId = 1, + Released = true, + Intent = new WaitIntentDataType + { + IntentId = $"{missionId}/gated", + Duration = 1.0 + } + } + ] + }; + } + + private static async Task AwaitWithTimeoutAsync(Task task, TimeSpan timeout) + { + if (await Task.WhenAny(task, Task.Delay(timeout)).ConfigureAwait(false) != task) + { + throw new TimeoutException(); + } + await task.ConfigureAwait(false); + } + + private static async ValueTask WaitUntilAsync( + Func> predicate, + string description) + { + DateTime deadline = DateTime.UtcNow.AddSeconds(10); + while (DateTime.UtcNow < deadline) + { + if (await predicate().ConfigureAwait(false)) + { + return; + } + await Task.Delay(25).ConfigureAwait(false); + } + Assert.Fail($"Timed out waiting for {description}."); + } + + private sealed class HostedMissionFixture : IAsyncDisposable + { + public string ServerUrl { get; private set; } = string.Empty; + + public ITelemetryContext Telemetry { get; } = DefaultTelemetry.Create( + builder => builder.SetMinimumLevel(LogLevel.Warning)); + + private GatedExecutor Executor { get; } = new(); + + public Task WaitForExecutionStartAsync() + { + return Executor.Started.Task; + } + + public void ReleaseExecution() + { + Executor.Release(); + } + + public void SetOutcome(IntentOutcome outcome) + { + Executor.Outcome = outcome; + } + + public async ValueTask StartAsync() + { + Exception? lastFailure = null; + for (int attempt = 0; attempt < 3; attempt++) + { + try + { + await StartAttemptAsync().ConfigureAwait(false); + return; + } + catch (Exception exception) + { + lastFailure = exception; + await StopHostAsync().ConfigureAwait(false); + } + } + throw new InvalidOperationException( + "The hosted mission test server did not become available.", + lastFailure); + } + + public async ValueTask ConnectAsync() + { + EndpointDescription? endpointDescription = await CoreClientUtils.SelectEndpointAsync( + m_clientConfiguration, + ServerUrl, + useSecurity: false, + Telemetry, + CancellationToken.None).ConfigureAwait(false); + Assert.That(endpointDescription, Is.Not.Null, "The hosted server endpoint must be discoverable."); + var endpoint = new ConfiguredEndpoint( + null, + endpointDescription!, + EndpointConfiguration.Create(m_clientConfiguration)); + var sessionFactory = new DefaultSessionFactory(Telemetry) + { + SubscriptionEngineFactory = DefaultSubscriptionEngineFactory.Instance + }; + ISession session = await sessionFactory.CreateAsync( + m_clientConfiguration, + endpoint, + updateBeforeConnect: false, + sessionName: "mission-hosted-client", + sessionTimeout: 60000, + identity: new UserIdentity(new AnonymousIdentityToken()), + preferredLocales: default, + ct: CancellationToken.None).ConfigureAwait(false); + if (!session.TryGetSubscriptionManager(out ISubscriptionManager? manager)) + { + throw ServiceResultException.Create( + StatusCodes.BadInvalidState, + "The hosted mission client Session did not expose a subscription manager."); + } + return new HostedMissionClient(session, new StreamingSubscription(manager)); + } + + public async ValueTask DisposeAsync() + { + await StopHostAsync().ConfigureAwait(false); + } + + private async ValueTask StartAttemptAsync() + { + int port = GetFreeTcpPort(); + ServerUrl = FormattableString.Invariant($"opc.tcp://localhost:{port}/MissionHosted"); + HostApplicationBuilder builder = Host.CreateApplicationBuilder(); + builder.Logging.ClearProviders(); + builder.Logging.SetMinimumLevel(LogLevel.Warning); + builder.Services.AddSingleton(Executor); + builder.Services + .AddOpcUa() + .AddServer(options => + { + options.ApplicationName = "MissionHostedServer"; + options.ApplicationUri = "urn:localhost:OPCFoundation:MissionHostedServer"; + options.ProductUri = "uri:opcfoundation.org:MissionHostedServer"; + options.AutoAcceptUntrustedCertificates = true; + options.EndpointUrls.Add(ServerUrl); + options.UserTokenPolicies.Add(new OpcUaUserTokenPolicy + { + TokenType = UserTokenType.Anonymous + }); + }) + .ConfigureRoles(options => options.Roles.Add(new Opc.Ua.Server.RoleDefinitionOptions + { + Name = "Operator", + Identities = + { + new Opc.Ua.Server.RoleIdentityMappingOptions + { + CriteriaType = IdentityCriteriaType.Anonymous + } + } + })) + .AddRobotIntent(options => + options.InstanceNamespaceUri = "urn:tests:robot-intent:mission-hosted") + .ConfigureRobotIntent(ConfigureRobotIntentAsync); + m_host = builder.Build(); + await m_host.StartAsync().ConfigureAwait(false); + m_clientConfiguration = await CreateClientConfigurationAsync().ConfigureAwait(false); + m_clientConfigurationReady = true; + await WaitForEndpointAsync().ConfigureAwait(false); + } + + private async ValueTask StopHostAsync() + { + if (m_host != null) + { + using var stop = new CancellationTokenSource(TimeSpan.FromSeconds(30)); + await m_host.StopAsync(stop.Token).ConfigureAwait(false); + m_host.Dispose(); + m_host = null; + } + if (m_clientConfigurationReady && + m_clientConfiguration.CertificateManager is IDisposable certificateManager) + { + certificateManager.Dispose(); + m_clientConfigurationReady = false; + } + } + + private static async ValueTask ConfigureRobotIntentAsync( + IRobotIntentBuildContext context, + CancellationToken cancellationToken) + { + await context.AddIntentControllerAsync( + "MissionController", + controller => controller + .WithOperationalMode(OperationalModeEnum.AutomaticExternal) + .WithReady(true) + .Accepts(retrySupported: true), + cancellationToken).ConfigureAwait(false); + } + + private async ValueTask CreateClientConfigurationAsync() + { + string pkiRoot = Path.Combine( + TestContext.CurrentContext.WorkDirectory, + "pki", + Guid.NewGuid().ToString("N")); + var configuration = new ApplicationConfiguration(Telemetry) + { + ApplicationName = "MissionHostedClient", + ApplicationUri = "urn:localhost:OPCFoundation:MissionHostedClient", + ApplicationType = ApplicationType.Client, + SecurityConfiguration = new SecurityConfiguration + { + ApplicationCertificate = new CertificateIdentifier + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(pkiRoot, "own"), + SubjectName = "CN=MissionHostedClient, O=OPC Foundation" + }, + TrustedIssuerCertificates = Store(Path.Combine(pkiRoot, "issuer")), + TrustedPeerCertificates = Store(Path.Combine(pkiRoot, "trusted")), + RejectedCertificateStore = Store(Path.Combine(pkiRoot, "rejected")), + AutoAcceptUntrustedCertificates = true + }, + TransportQuotas = new TransportQuotas { MaxMessageSize = 4 * 1024 * 1024 }, + ClientConfiguration = new ClientConfiguration(), + ServerConfiguration = new ServerConfiguration() + }; + await configuration.ValidateAsync(ApplicationType.Client).ConfigureAwait(false); + var application = new ApplicationInstance(configuration, Telemetry); + await application.CheckApplicationInstanceCertificatesAsync(true).ConfigureAwait(false); + configuration.CertificateManager ??= CertificateManagerFactory.Create( + configuration.SecurityConfiguration, + Telemetry); + configuration.CertificateManager.AcceptError = static (_, _) => true; + return configuration; + } + + private async ValueTask WaitForEndpointAsync() + { + Exception? lastException = null; + DateTime deadline = DateTime.UtcNow.AddSeconds(10); + while (DateTime.UtcNow < deadline) + { + try + { + EndpointDescription? endpoint = await CoreClientUtils.SelectEndpointAsync( + m_clientConfiguration, + ServerUrl, + useSecurity: false, + Telemetry, + CancellationToken.None).ConfigureAwait(false); + if (endpoint != null) + { + return; + } + } + catch (Exception exception) + { + lastException = exception; + } + await Task.Delay(100).ConfigureAwait(false); + } + throw new TimeoutException( + $"Hosted OPC UA endpoint '{ServerUrl}' did not become available. " + + $"Last error: {lastException?.Message}"); + } + + private static CertificateTrustList Store(string path) + { + return new CertificateTrustList + { + StoreType = CertificateStoreType.Directory, + StorePath = path + }; + } + + private static int GetFreeTcpPort() + { + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + int port = ((IPEndPoint)listener.LocalEndpoint).Port; + listener.Stop(); + return port; + } + + private ApplicationConfiguration m_clientConfiguration = null!; + private bool m_clientConfigurationReady; + private IHost? m_host; + + private sealed class GatedExecutor : IIntentExecutor + { + public TaskCompletionSource Started { get; } = new( + TaskCreationOptions.RunContinuationsAsynchronously); + + public IntentOutcome Outcome { get; set; } = IntentOutcome.Success; + + public ValueTask ExecuteAsync( + IntentExecution execution, + CancellationToken cancellationToken) + { + Started.TrySetResult(true); + return WaitForReleaseAsync(); + } + + public bool CanCancel(IntentExecution execution) + { + return true; + } + + public void Release() + { + m_release.TrySetResult(true); + } + + private async ValueTask WaitForReleaseAsync() + { + await m_release.Task.ConfigureAwait(false); + return Outcome; + } + + private readonly TaskCompletionSource m_release = new( + TaskCreationOptions.RunContinuationsAsynchronously); + } + } + + private sealed class HostedMissionClient( + ISession session, + IStreamingSubscription streaming) + : IAsyncDisposable + { + public ISession Session { get; } = session; + + public IStreamingSubscription Streaming { get; } = streaming; + + public async ValueTask DisposeAsync() + { + if (Session.Connected) + { + await Session.CloseAsync(1000, true).ConfigureAwait(false); + } + await Streaming.DisposeAsync().ConfigureAwait(false); + Session.Dispose(); + } + } + } +} +#endif diff --git a/tests/Opc.Ua.Robotics.Tests/Opc.Ua.Robotics.Tests.csproj b/tests/Opc.Ua.Robotics.Tests/Opc.Ua.Robotics.Tests.csproj index 226f5c007e..b6233976d0 100644 --- a/tests/Opc.Ua.Robotics.Tests/Opc.Ua.Robotics.Tests.csproj +++ b/tests/Opc.Ua.Robotics.Tests/Opc.Ua.Robotics.Tests.csproj @@ -33,8 +33,34 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/tests/Opc.Ua.Vision.Tests/OpcUaVisionClientBuilderExtensionsTests.cs b/tests/Opc.Ua.Vision.Tests/OpcUaVisionClientBuilderExtensionsTests.cs new file mode 100644 index 0000000000..f095175f3f --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/OpcUaVisionClientBuilderExtensionsTests.cs @@ -0,0 +1,138 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Moq; +using NUnit.Framework; +using Opc.Ua.Client; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Tests for . The + /// extension registers a factory over the managed-session factory + /// registered by AddClient. It is a small piece of glue but + /// it is what a Vision consumer relies on to compose the client via + /// DI. + /// + [TestFixture] + [Category("Vision")] + public sealed class OpcUaVisionClientBuilderExtensionsTests + { + [Test] + public void AddVisionClientThrowsOnNullBuilder() + { + Assert.Throws(() => + OpcUaVisionClientBuilderExtensions.AddVisionClient(null!)); + } + + [Test] + public void AddVisionClientRegistersFactorySingleton() + { + IServiceCollection services = new ServiceCollection(); + var telemetry = new Mock().Object; + services.AddSingleton(telemetry); + services.AddSingleton>>( + _ => Task.FromResult(null!)); + + var builder = new TestClientBuilder(services); + builder.AddVisionClient(); + + Assert.Multiple(() => + { + Assert.That( + services.Any(s => s.ServiceType == typeof(VisionClientFactory)), + Is.True); + Assert.That( + services.Any(s => + s.ServiceType == typeof(Func>)), + Is.True); + }); + } + + [Test] + public void AddVisionClientFactoryThrowsWhenAddClientWasNotCalled() + { + IServiceCollection services = new ServiceCollection(); + var telemetry = new Mock().Object; + services.AddSingleton(telemetry); + + var builder = new TestClientBuilder(services); + builder.AddVisionClient(); + + using ServiceProvider provider = services.BuildServiceProvider(); + Assert.Throws(() => + provider.GetRequiredService()); + } + + [Test] + public void AddVisionClientReturnsSameBuilder() + { + IServiceCollection services = new ServiceCollection(); + var telemetry = new Mock().Object; + services.AddSingleton(telemetry); + services.AddSingleton>>( + _ => Task.FromResult(null!)); + + var builder = new TestClientBuilder(services); + IOpcUaClientBuilder returned = builder.AddVisionClient(); + + Assert.That(returned, Is.SameAs(builder)); + } + + [Test] + public void AddVisionClientIsIdempotent() + { + IServiceCollection services = new ServiceCollection(); + var telemetry = new Mock().Object; + services.AddSingleton(telemetry); + services.AddSingleton>>( + _ => Task.FromResult(null!)); + + var builder = new TestClientBuilder(services); + builder.AddVisionClient(); + builder.AddVisionClient(); + + int factoryCount = services.Count(s => + s.ServiceType == typeof(VisionClientFactory)); + + Assert.That(factoryCount, Is.EqualTo(1)); + } + + private sealed class TestClientBuilder(IServiceCollection services) : IOpcUaClientBuilder + { + public IServiceCollection Services { get; } = services; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/OpenUsdCaptureLogTests.cs b/tests/Opc.Ua.Vision.Tests/OpenUsdCaptureLogTests.cs new file mode 100644 index 0000000000..a9fa91c308 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/OpenUsdCaptureLogTests.cs @@ -0,0 +1,160 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using Microsoft.Extensions.Logging; +using NUnit.Framework; +using Opc.Ua.Vision.OpenUsd; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Exercises the source-generated logging extension methods on + /// . Each method must accept its + /// declared arguments and dispatch to without + /// throwing. + /// + [TestFixture] + [Category("OpenUsd")] + public sealed class OpenUsdCaptureLogTests + { + [Test] + public void BackendSelectedDoesNotThrow() + { + var logger = new CapturingLogger(); + logger.BackendSelected("D3D12", "WARP", isSoftware: true); + + Assert.That(logger.Entries.Count, Is.GreaterThanOrEqualTo(1)); + } + + [Test] + public void BackendUnavailableDoesNotThrow() + { + var logger = new CapturingLogger(); + logger.BackendUnavailable("D3D12", "No device"); + + Assert.That(logger.Entries.Count, Is.GreaterThanOrEqualTo(1)); + } + + [Test] + public void NoBackendAvailableDoesNotThrow() + { + var logger = new CapturingLogger(); + logger.NoBackendAvailable("no gpu"); + + Assert.That(logger.Entries.Count, Is.GreaterThanOrEqualTo(1)); + } + + [Test] + public void CaptureSucceededDoesNotThrow() + { + var logger = new CapturingLogger(); + logger.CaptureSucceeded(1920, 1080, elapsedMs: 5, drawCount: 3, + meshCount: 2, backendName: "D3D12"); + + Assert.That(logger.Entries.Count, Is.GreaterThanOrEqualTo(1)); + } + + [Test] + public void BlankFrameDetectedDoesNotThrow() + { + var logger = new CapturingLogger(); + logger.BlankFrameDetected(drawCount: 0, meshCount: 0, isUniform: true); + + Assert.That(logger.Entries.Count, Is.GreaterThanOrEqualTo(1)); + } + + [Test] + public void StageOpenFailedDoesNotThrow() + { + var logger = new CapturingLogger(); + logger.StageOpenFailed("stage.usda", new InvalidOperationException("boom")); + + Assert.That(logger.Entries.Count, Is.GreaterThanOrEqualTo(1)); + } + + [Test] + public void CameraResolveFailedDoesNotThrow() + { + var logger = new CapturingLogger(); + logger.CameraResolveFailed("/World/Cam", "stage.usda", + new InvalidOperationException("boom")); + + Assert.That(logger.Entries.Count, Is.GreaterThanOrEqualTo(1)); + } + + [Test] + public void RenderFailedDoesNotThrow() + { + var logger = new CapturingLogger(); + logger.RenderFailed("D3D12", new InvalidOperationException("boom")); + + Assert.That(logger.Entries.Count, Is.GreaterThanOrEqualTo(1)); + } + + [Test] + public void EncodingFailedDoesNotThrow() + { + var logger = new CapturingLogger(); + logger.EncodingFailed(1920, 1080, new InvalidOperationException("boom")); + + Assert.That(logger.Entries.Count, Is.GreaterThanOrEqualTo(1)); + } + + private sealed class CapturingLogger : ILogger + { + public List<(LogLevel Level, EventId Id, string Message)> Entries { get; } = []; + + public IDisposable BeginScope(TState state) where TState : notnull => + NullScope.Instance; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log( + LogLevel logLevel, + EventId eventId, + TState state, + Exception? exception, + Func formatter) + { + Entries.Add((logLevel, eventId, formatter(state, exception))); + } + + private sealed class NullScope : IDisposable + { + public static readonly NullScope Instance = new(); + + public void Dispose() + { + } + } + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/OpenUsdSceneCameraCaptureServiceCollectionExtensionsTests.cs b/tests/Opc.Ua.Vision.Tests/OpenUsdSceneCameraCaptureServiceCollectionExtensionsTests.cs new file mode 100644 index 0000000000..8ae0879c50 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/OpenUsdSceneCameraCaptureServiceCollectionExtensionsTests.cs @@ -0,0 +1,121 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Linq; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using Opc.Ua.Vision.OpenUsd; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Tests for the OpenUSD DI extension. The extension registers the + /// scene camera capture provider as a singleton so a host can wire + /// it into a Vision server or a simulator. + /// + [TestFixture] + [Category("OpenUsd")] + public sealed class OpenUsdSceneCameraCaptureServiceCollectionExtensionsTests + { + [Test] + public void AddOpenUsdSceneCameraCaptureProviderThrowsOnNullServices() + { + Assert.Throws(() => + OpenUsdSceneCameraCaptureServiceCollectionExtensions + .AddOpenUsdSceneCameraCaptureProvider(null!)); + } + + [Test] + public void AddOpenUsdSceneCameraCaptureProviderInvokesConfigureDelegate() + { + IServiceCollection services = new ServiceCollection(); + bool invoked = false; + services.AddOpenUsdSceneCameraCaptureProvider(_ => invoked = true); + + using ServiceProvider provider = services.BuildServiceProvider(); + var options = provider.GetRequiredService(); + + Assert.Multiple(() => + { + Assert.That(invoked, Is.True); + Assert.That(options, Is.Not.Null); + }); + } + + [Test] + public void AddOpenUsdSceneCameraCaptureProviderRegistersSingletonProvider() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpenUsdSceneCameraCaptureProvider(); + + using ServiceProvider provider = services.BuildServiceProvider(); + var one = provider.GetRequiredService(); + var two = provider.GetRequiredService(); + + Assert.That(one, Is.SameAs(two)); + } + + [Test] + public void AddOpenUsdSceneCameraCaptureProviderIsIdempotent() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpenUsdSceneCameraCaptureProvider(); + services.AddOpenUsdSceneCameraCaptureProvider(); + + int optionsCount = services.Count(s => s.ServiceType == typeof(OpenUsdSceneCaptureOptions)); + int providerCount = services.Count(s => s.ServiceType == typeof(ISceneCameraCaptureProvider)); + + Assert.Multiple(() => + { + Assert.That(optionsCount, Is.EqualTo(1), + "TryAddSingleton must not double-register the options record."); + Assert.That(providerCount, Is.EqualTo(1), + "TryAddSingleton must not double-register the provider."); + }); + } + + [Test] + public void AddOpenUsdSceneCameraCaptureProviderWithoutConfigureRegistersDefaultOptions() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpenUsdSceneCameraCaptureProvider(); + + using ServiceProvider provider = services.BuildServiceProvider(); + var options = provider.GetRequiredService(); + var defaults = new OpenUsdSceneCaptureOptions(); + + Assert.Multiple(() => + { + Assert.That(options.MaxFrameWidth, Is.EqualTo(defaults.MaxFrameWidth)); + Assert.That(options.MaxFrameHeight, Is.EqualTo(defaults.MaxFrameHeight)); + }); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/PngEncoderTests.cs b/tests/Opc.Ua.Vision.Tests/PngEncoderTests.cs new file mode 100644 index 0000000000..a56d57d7fa --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/PngEncoderTests.cs @@ -0,0 +1,238 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.IO.Compression; +using NUnit.Framework; +using Opc.Ua.Vision.OpenUsd.Encoding; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Pins the byte-level contract of + /// - PNG signature, IHDR / IDAT / IEND chunk layout, and round-trip + /// through the standard decoder so any PNG + /// reader (including .NET's own Image) can consume the output. + /// The encoder is used for every simulated-camera frame the Vision + /// server publishes, so a regression that produces invalid PNG bytes + /// would silently break every client. + /// + [TestFixture] + public sealed class PngEncoderTests + { + [Test] + public void EncodeRgba8OutputStartsWithPngSignature() + { + byte[] rgba = MakeGradient(4, 4); + + byte[] png = PngEncoder.EncodeRgba8(4, 4, rgba); + + Assert.Multiple(() => + { + Assert.That(png.Length, Is.GreaterThan(8)); + Assert.That(png[0], Is.EqualTo(0x89)); + Assert.That(png[1], Is.EqualTo((byte)'P')); + Assert.That(png[2], Is.EqualTo((byte)'N')); + Assert.That(png[3], Is.EqualTo((byte)'G')); + Assert.That(png[4], Is.EqualTo(0x0D)); + Assert.That(png[5], Is.EqualTo(0x0A)); + Assert.That(png[6], Is.EqualTo(0x1A)); + Assert.That(png[7], Is.EqualTo(0x0A)); + }); + } + + [Test] + public void EncodeRgba8ProducesIhdrIdatIendInThatOrder() + { + byte[] png = PngEncoder.EncodeRgba8(4, 4, MakeGradient(4, 4)); + + int idxIhdr = FindChunk(png, "IHDR"); + int idxIdat = FindChunk(png, "IDAT"); + int idxIend = FindChunk(png, "IEND"); + + Assert.Multiple(() => + { + Assert.That(idxIhdr, Is.EqualTo(8), + "IHDR must be the first chunk after the PNG signature."); + Assert.That(idxIdat, Is.GreaterThan(idxIhdr)); + Assert.That(idxIend, Is.GreaterThan(idxIdat)); + }); + } + + [Test] + public void IhdrCarriesWidthHeightBitDepth8ColorType6() + { + byte[] png = PngEncoder.EncodeRgba8(7, 5, MakeGradient(7, 5)); + + int idx = FindChunk(png, "IHDR"); + int dataOffset = idx + 8; + uint w = ReadUInt32BE(png, dataOffset); + uint h = ReadUInt32BE(png, dataOffset + 4); + byte bitDepth = png[dataOffset + 8]; + byte colorType = png[dataOffset + 9]; + byte compression = png[dataOffset + 10]; + byte filter = png[dataOffset + 11]; + byte interlace = png[dataOffset + 12]; + + Assert.Multiple(() => + { + Assert.That(w, Is.EqualTo(7u)); + Assert.That(h, Is.EqualTo(5u)); + Assert.That(bitDepth, Is.EqualTo(8)); + Assert.That(colorType, Is.EqualTo(6), + "PngEncoder is RGBA-only: colour type 6 is truecolour with alpha."); + Assert.That(compression, Is.EqualTo(0)); + Assert.That(filter, Is.EqualTo(0)); + Assert.That(interlace, Is.EqualTo(0)); + }); + } + + [Test] + public void IdatDecompressesBackToFilterZeroPrefixedRowsWithOriginalPixels() + { + const int W = 3; + const int H = 2; + byte[] rgba = MakeGradient(W, H); + + byte[] png = PngEncoder.EncodeRgba8(W, H, rgba); + byte[] filtered = DecompressIdat(png); + + Assert.That(filtered.Length, Is.EqualTo(H * ((W * 4) + 1)), + "Decompressed IDAT must be H rows of (1 filter byte + W*4 pixel bytes)."); + for (int y = 0; y < H; y++) + { + int rowStart = y * ((W * 4) + 1); + Assert.That(filtered[rowStart], Is.EqualTo(0), + "PngEncoder emits filter type 0 (None) on every row."); + for (int x = 0; x < W * 4; x++) + { + int srcIdx = (y * W * 4) + x; + int dstIdx = rowStart + 1 + x; + Assert.That(filtered[dstIdx], Is.EqualTo(rgba[srcIdx]), + $"Pixel byte round-trip mismatch at y={y}, x={x}."); + } + } + } + + [Test] + public void EncodeRgba8ThrowsArgumentOutOfRangeForNonPositiveWidth() + { + Assert.That( + () => PngEncoder.EncodeRgba8(0, 1, new byte[4]), + Throws.TypeOf()); + Assert.That( + () => PngEncoder.EncodeRgba8(-1, 1, new byte[4]), + Throws.TypeOf()); + } + + [Test] + public void EncodeRgba8ThrowsArgumentOutOfRangeForNonPositiveHeight() + { + Assert.That( + () => PngEncoder.EncodeRgba8(1, 0, new byte[4]), + Throws.TypeOf()); + Assert.That( + () => PngEncoder.EncodeRgba8(1, -1, new byte[4]), + Throws.TypeOf()); + } + + [Test] + public void EncodeRgba8ThrowsArgumentExceptionForMismatchedBufferLength() + { + Assert.That( + () => PngEncoder.EncodeRgba8(4, 4, new byte[10]), + Throws.TypeOf()); + } + + private static int FindChunk(byte[] png, string fourcc) + { + if (fourcc.Length != 4) + { + throw new ArgumentException("Chunk type must be exactly 4 ASCII bytes.", nameof(fourcc)); + } + for (int i = 8; i + 8 <= png.Length;) + { + uint length = ReadUInt32BE(png, i); + if (length > (uint)int.MaxValue) + { + return -1; + } + if (png[i + 4] == (byte)fourcc[0] && + png[i + 5] == (byte)fourcc[1] && + png[i + 6] == (byte)fourcc[2] && + png[i + 7] == (byte)fourcc[3]) + { + return i; + } + i += 4 + 4 + (int)length + 4; + } + return -1; + } + + private static byte[] DecompressIdat(byte[] png) + { + int idx = FindChunk(png, "IDAT"); + Assert.That(idx, Is.GreaterThan(0), "Encoder must emit at least one IDAT chunk."); + uint len = ReadUInt32BE(png, idx); + byte[] payload = new byte[len]; + Array.Copy(png, idx + 8, payload, 0, (int)len); + using var input = new MemoryStream(payload); + using var zlib = new ZLibStream(input, CompressionMode.Decompress); + using var output = new MemoryStream(); + zlib.CopyTo(output); + return output.ToArray(); + } + + private static uint ReadUInt32BE(byte[] buf, int offset) + { + return ((uint)buf[offset] << 24) | + ((uint)buf[offset + 1] << 16) | + ((uint)buf[offset + 2] << 8) | + buf[offset + 3]; + } + + private static byte[] MakeGradient(int width, int height) + { + byte[] rgba = new byte[width * height * 4]; + for (int y = 0; y < height; y++) + { + for (int x = 0; x < width; x++) + { + int i = ((y * width) + x) * 4; + rgba[i] = (byte)(x * 17); + rgba[i + 1] = (byte)(y * 23); + rgba[i + 2] = (byte)((x + y) * 5); + rgba[i + 3] = 0xFF; + } + } + return rgba; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/SceneCameraCaptureProviderTests.cs b/tests/Opc.Ua.Vision.Tests/SceneCameraCaptureProviderTests.cs new file mode 100644 index 0000000000..9e35900aa9 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/SceneCameraCaptureProviderTests.cs @@ -0,0 +1,238 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.Vision.OpenUsd; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Exercises along the + /// paths that must hold on any host: argument validation, request + /// validation, no-graphics-device fallback, and disposal semantics. + /// The provider probes a graphics device during construction and CI + /// typically has none — the tests treat both outcomes (device found, + /// no device found) as legal and only assert on the invariants that + /// hold regardless (never throw on the no-device path, populate + /// , refuse malformed + /// requests deterministically, refuse to serve any frame once + /// disposed). + /// + [TestFixture] + public sealed class SceneCameraCaptureProviderTests + { + [Test] + public void ConstructorThrowsArgumentNullExceptionForNullOptions() + { + Assert.That( + () => new OpenUsdSceneCameraCaptureProvider(null!, telemetry: null), + Throws.TypeOf()); + } + + [Test] + public void ConstructorSucceedsWithDefaultOptionsAndProducesBackendDescriptor() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + + Assert.Multiple(() => + { + Assert.That(provider.Backend, Is.Not.Null); + Assert.That(provider.Backend.Name, Is.Not.Null); + Assert.That(provider.Backend.Name, Is.Not.Empty); + }); + } + + [Test] + public async Task CaptureAsyncRejectsRequestWithEmptyStageIdentifier() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = string.Empty, + Width = 320, + Height = 240, + Format = SceneCameraImageFormat.Png + }; + + SceneCameraCaptureResult result = await provider.CaptureAsync(request, CancellationToken.None) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Status, Is.EqualTo(SceneCameraCaptureStatus.InvalidRequest)); + Assert.That(result.Image.IsNull, Is.True); + Assert.That(result.Reason, Is.Not.Null); + }); + } + + [Test] + public async Task CaptureAsyncRejectsRequestWithZeroWidthOrHeight() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + + SceneCameraCaptureResult zeroWidth = await provider.CaptureAsync( + new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:stage", + Width = 0, + Height = 240, + Format = SceneCameraImageFormat.Png + }, CancellationToken.None).ConfigureAwait(false); + SceneCameraCaptureResult zeroHeight = await provider.CaptureAsync( + new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:stage", + Width = 320, + Height = 0, + Format = SceneCameraImageFormat.Png + }, CancellationToken.None).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(zeroWidth.Status, Is.EqualTo(SceneCameraCaptureStatus.InvalidRequest)); + Assert.That(zeroHeight.Status, Is.EqualTo(SceneCameraCaptureStatus.InvalidRequest)); + }); + } + + [Test] + public async Task CaptureAsyncRejectsFrameExceedingConfiguredMaximum() + { + var options = new OpenUsdSceneCaptureOptions { MaxFrameWidth = 64, MaxFrameHeight = 64 }; + using var provider = new OpenUsdSceneCameraCaptureProvider(options, telemetry: null); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:stage", + Width = 128, + Height = 128, + Format = SceneCameraImageFormat.Png + }; + + SceneCameraCaptureResult result = await provider.CaptureAsync(request, CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(result.Status, Is.EqualTo(SceneCameraCaptureStatus.InvalidRequest)); + } + + [Test] + public void CaptureAsyncThrowsArgumentNullExceptionForNullRequest() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + + Assert.That( + async () => await provider.CaptureAsync(null!, CancellationToken.None).ConfigureAwait(false), + Throws.TypeOf()); + } + + [Test] + public void CaptureAsyncPropagatesCancellationBeforeAnyRenderingWork() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + using var cts = new CancellationTokenSource(); + cts.Cancel(); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:stage", + Width = 32, + Height = 32 + }; + + Assert.That( + async () => await provider.CaptureAsync(request, cts.Token).ConfigureAwait(false), + Throws.InstanceOf()); + } + + [Test] + public async Task CaptureAsyncReportsNoRenderingBackendWhenDeviceProbeFailedAndNeverReturnsSuccessWithEmptyImage() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + + if (provider.Backend.IsAvailable) + { + Assert.Ignore( + "A rendering backend is available on this host; the NoRenderingBackend " + + "path only reproduces on hosts without a graphics device (typical CI). " + + "This test asserts the CI-side invariants only."); + return; + } + var request = new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:no-backend", + Width = 32, + Height = 32, + Format = SceneCameraImageFormat.Png + }; + + SceneCameraCaptureResult result = await provider.CaptureAsync(request, CancellationToken.None) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Status, Is.EqualTo(SceneCameraCaptureStatus.NoRenderingBackend), + "With no graphics device the provider must never touch native rendering code — the NoRenderingBackend path is the entire contract."); + Assert.That(result.Image.IsNull, Is.True, + "NoRenderingBackend must never return an image, even an empty one, because a caller could misread that as 'no draws' rather than 'no backend'."); + Assert.That(result.Reason, Is.Not.Null); + Assert.That(result.Backend, Is.Not.Null); + }); + } + + [Test] + public void DisposedProviderRejectsFurtherCaptureRequests() + { + var provider = new OpenUsdSceneCameraCaptureProvider(); + provider.Dispose(); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:stage", + Width = 32, + Height = 32, + Format = SceneCameraImageFormat.Png + }; + + Assert.That( + async () => await provider.CaptureAsync(request, CancellationToken.None).ConfigureAwait(false), + Throws.TypeOf()); + } + + [Test] + public void DisposeIsIdempotent() + { + var provider = new OpenUsdSceneCameraCaptureProvider(); + + Assert.DoesNotThrow(() => + { + provider.Dispose(); + provider.Dispose(); + }); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/SceneCameraCaptureTypesTests.cs b/tests/Opc.Ua.Vision.Tests/SceneCameraCaptureTypesTests.cs new file mode 100644 index 0000000000..a152023333 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/SceneCameraCaptureTypesTests.cs @@ -0,0 +1,105 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using NUnit.Framework; +using Opc.Ua.Vision.OpenUsd; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Locks the shape of the value types the OpenUsd provider hands + /// callers: a default that + /// safely signals "no rendering backend" without needing a null + /// check, and result/request records that default to something the + /// caller can reason about. + /// + [TestFixture] + public sealed class SceneCameraCaptureTypesTests + { + [Test] + public void SceneCameraCaptureBackendNoneIsAvailabilityFalseAndCarriesReason() + { + SceneCameraCaptureBackend none = SceneCameraCaptureBackend.None; + + Assert.Multiple(() => + { + Assert.That(none.Name, Is.EqualTo("None")); + Assert.That(none.IsAvailable, Is.False); + Assert.That(none.IsSoftware, Is.False); + Assert.That(none.UnavailableReason, Is.Not.Null); + }); + } + + [Test] + public void SceneCameraCaptureResultDefaultsBackendToNoneSentinel() + { + var result = new SceneCameraCaptureResult(); + + Assert.That(result.Backend, Is.SameAs(SceneCameraCaptureBackend.None)); + } + + [Test] + public void SceneCameraCaptureResultDefaultsImageToNullByteString() + { + var result = new SceneCameraCaptureResult(); + + Assert.That(result.Image.IsNull, Is.True, + "A default result must not accidentally look like an empty successful frame."); + } + + [Test] + public void SceneCameraCaptureRequestDefaultsToPngFormat() + { + var request = new SceneCameraCaptureRequest(); + + Assert.That(request.Format, Is.EqualTo(SceneCameraImageFormat.Png)); + } + + [Test] + public void SceneCameraCaptureRequestDefaultsStageIdentifierToEmptyStringNotNull() + { + var request = new SceneCameraCaptureRequest(); + + Assert.That(request.StageIdentifier, Is.EqualTo(string.Empty), + "A null StageIdentifier would defeat the InvalidRequest guard on the provider."); + } + + [Test] + public void OpenUsdSceneCaptureOptionsHasSensibleMaximumFrameSize() + { + var options = new OpenUsdSceneCaptureOptions(); + + Assert.Multiple(() => + { + Assert.That(options.MaxFrameWidth, Is.GreaterThan(0)); + Assert.That(options.MaxFrameHeight, Is.GreaterThan(0)); + }); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionBuilderIntegrationTests.cs b/tests/Opc.Ua.Vision.Tests/VisionBuilderIntegrationTests.cs new file mode 100644 index 0000000000..1b328ef768 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionBuilderIntegrationTests.cs @@ -0,0 +1,607 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Moq; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Builders; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Integration tests over the fluent Vision node builder. These + /// exercise the builder methods against a real + /// so the resulting nodes end up + /// in the address space rather than in a mocked context. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionBuilderIntegrationTests + { + [Test] + public async Task AddImageSensorExercisesEveryFluentEntryPoint() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + var mediaProvider = new Mock().Object; + VisionPose3DDataType extrinsic = CreatePose(); + VisionIntrinsicsDataType intrinsics = CreateIntrinsics(); + + context.Nodes.AddImageSensor("Camera1", sensor => sensor + .WithSensorId("SN-CAM-1") + .WithRealityKind(VisionRealityKindEnum.Physical) + .WithModality(VisionSensorModalityEnum.Area2D) + .WithManufacturer("Contoso") + .WithModel("ContosoCam 4K") + .WithSerialNumber("SN-42") + .WithDeviceUri("opc.tcp://cam-1") + .WithFrameId("cam1") + .WithResolution(1920, 1080) + .WithPixelFormat("Mono8") + .WithIntrinsics(intrinsics) + .WithOptics(o => o + .WithFocalLength(0.008) + .WithAperture(1.4) + .WithWorkingDistance(0.5) + .WithMagnification(2.0) + .WithMountType("C-Mount") + .WithLensType("Fixed")) + .WithIllumination(i => i + .WithLampType(VisionLampTypeEnum.Led) + .WithWavelength(525) + .WithRelativeIntensity(0.8) + .WithLightingMode(VisionLightingModeEnum.Continuous)) + .AddIntrinsicCalibration("Intr", calib => calib + .WithCalibrationId("intrinsic-1") + .WithIntrinsics(intrinsics) + .WithResidualError(0.25) + .WithMethod("Zhang")) + .AddExtrinsicCalibration("Extr", calib => calib + .WithCalibrationId("extrinsic-1") + .WithMount(VisionCalibrationMountEnum.EyeToHand) + .WithFrames("world", "cam1") + .WithTransform(extrinsic) + .WithResidualError(0.1)) + .AddStreamEndpoint("Rtsp", ep => ep + .WithEndpointId("stream-1") + .WithEndpointUri("rtsp://cam-1/stream") + .WithProtocol(VisionStreamProtocolEnum.Rtsp) + .WithCodec(VisionVideoCodecEnum.H264) + .WithResolution(1920, 1080) + .WithFrameRate(30.0) + .WithBitrate(8_000_000) + .WithDefaultProfileName("main")) + .AddClipEndpoint("Snap", ep => ep + .WithEndpointId("clip-1") + .WithEndpointUri("clip://cam-1/snap") + .WithClipFormat(VisionClipFormatEnum.Jpeg) + .WithQuality(90) + .WithResolution(1920, 1080) + .WithInlineDelivery(true, 1_048_576) + .WithDefaultProfileName("thumb")) + .UseMediaProvider(mediaProvider)); + + Assert.That(fixture.Manager.Root.Sensors, Is.Not.Null); + Assert.That(FindChild(fixture.Manager.Root.Sensors!, "Camera1"), + Is.Not.Null, "Camera1 must be added to the sensors folder."); + } + + [Test] + public async Task AddDepth3DSensorAppliesDepthSpecificMembers() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + context.Nodes.AddDepth3DSensor("Depth1", sensor => sensor + .WithSensorId("SN-DEP-1") + .WithModality(VisionSensorModalityEnum.Depth3D) + .WithRealityKind(VisionRealityKindEnum.Simulated) + .WithDepthRange(0.2, 5.0) + .WithDepthScale(0.001) + .WithBaseline(0.075)); + + NodeState? added = FindChild(fixture.Manager.Root.Sensors!, "Depth1"); + Assert.That(added, Is.Not.Null); + } + + [Test] + public async Task AddSensorGenericFluentSurfaceCovered() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + context.Nodes.AddSensor("Thermal1", sensor => sensor + .WithSensorId("SN-TH-1") + .WithModality(VisionSensorModalityEnum.Thermal) + .WithRealityKind(VisionRealityKindEnum.Hybrid) + .HasScenePrim(new NodeId("scene:cam1", 1)) + .MountedOn(new NodeId("mount:cam1", 1))); + + Assert.That(FindChild(fixture.Manager.Root.Sensors!, "Thermal1"), + Is.Not.Null); + } + + [Test] + public async Task AddFrameCreatesParentedFramesUsingRegistryLookup() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + VisionPose3DDataType worldPose = CreatePose(); + VisionPose3DDataType camPose = CreatePose(); + + context.Nodes + .AddFrame("World", f => f + .WithFrameId("world") + .WithRole(VisionFrameRoleEnum.World) + .WithTransform(worldPose)) + .AddFrame("Cam1", f => f + .WithFrameId("cam1") + .WithRole(VisionFrameRoleEnum.Camera) + .WithParent("world") + .WithTransform(camPose)); + + Assert.That(fixture.Manager.Root.Frames, Is.Not.Null); + Assert.That(FindChild(fixture.Manager.Root.Frames!, "World"), + Is.Not.Null); + Assert.That(FindChild(fixture.Manager.Root.Frames!, "Cam1"), + Is.Not.Null); + } + + [Test] + public async Task AddFrameAcceptsExplicitParentNodeId() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + context.Nodes.AddFrame("World", f => f + .WithFrameId("world") + .WithRole(VisionFrameRoleEnum.World) + .WithTransform(CreatePose())); + + NodeState worldNode = FindChild(fixture.Manager.Root.Frames!, "World")!; + context.Nodes.AddFrame("Tool0", f => f + .WithFrameId("tool0") + .WithRole(VisionFrameRoleEnum.Tool) + .WithParent(worldNode.NodeId) + .WithTransform(CreatePose())); + + Assert.That(FindChild(fixture.Manager.Root.Frames!, "Tool0"), + Is.Not.Null); + } + + [Test] + public async Task AddFrameWithEmptyFrameIdThrowsBadConfigurationError() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + ServiceResultException ex = Assert.Throws(() => + context.Nodes.AddFrame("Nameless", f => f + .WithRole(VisionFrameRoleEnum.World)))!; + Assert.That(ex.StatusCode, Is.EqualTo(StatusCodes.BadConfigurationError)); + } + + [Test] + public async Task AddPipelineExercisesEveryPipelineFluentEntryPoint() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + var inferenceProvider = new Mock().Object; + var feedbackSink = new Mock().Object; + + context.Nodes.AddImageSensor("Cam1", s => s + .WithSensorId("SN-CAM") + .WithModality(VisionSensorModalityEnum.Area2D)); + + NodeState sensorNode = FindChild(fixture.Manager.Root.Sensors!, "Cam1")!; + + context.Nodes.AddPipeline("Pipe1", p => p + .WithPipelineId("pipeline-1") + .WithSensor(sensorNode.NodeId) + .WithDeployment(new NodeId("deployment:pipeline-1", 1)) + .ProducedBy(new NodeId("producer:workcell1", 1)) + .UseInferenceProvider(inferenceProvider, onServer: true) + .UseFeedbackSink(feedbackSink)); + + Assert.That(fixture.Manager.Root.Pipelines, Is.Not.Null); + Assert.That(FindChild(fixture.Manager.Root.Pipelines!, "Pipe1"), + Is.Not.Null); + } + + [Test] + public async Task AddPipelineHonoursOffServerInferenceFacet() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + var inferenceProvider = new Mock().Object; + + context.Nodes.AddPipeline("Off", p => p + .WithPipelineId("off-server") + .UseInferenceProvider(inferenceProvider, onServer: false)); + + Assert.That(FindChild(fixture.Manager.Root.Pipelines!, "Off"), + Is.Not.Null); + } + + [Test] + public async Task AddImageSensorRejectsEmptyBrowseName() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddImageSensor(string.Empty, _ => { })); + } + + [Test] + public async Task AddImageSensorRejectsNullConfigureDelegate() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddImageSensor("Cam1", null!)); + } + + [Test] + public async Task AddDepth3DSensorRejectsEmptyBrowseName() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddDepth3DSensor(string.Empty, _ => { })); + } + + [Test] + public async Task AddDepth3DSensorRejectsNullConfigureDelegate() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddDepth3DSensor("D1", null!)); + } + + [Test] + public async Task AddSensorRejectsEmptyBrowseName() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddSensor(string.Empty, _ => { })); + } + + [Test] + public async Task AddSensorRejectsNullConfigureDelegate() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddSensor("S1", null!)); + } + + [Test] + public async Task AddFrameRejectsEmptyBrowseName() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddFrame(string.Empty, _ => { })); + } + + [Test] + public async Task AddFrameRejectsNullConfigureDelegate() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddFrame("F1", null!)); + } + + [Test] + public async Task AddPipelineRejectsEmptyBrowseName() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddPipeline(string.Empty, _ => { })); + } + + [Test] + public async Task AddPipelineRejectsNullConfigureDelegate() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddPipeline("P1", null!)); + } + + [Test] + public async Task WithOpticsRejectsNullConfigureDelegate() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddSensor("S1", s => s.WithOptics(null!))); + } + + [Test] + public async Task WithIlluminationRejectsNullConfigureDelegate() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddSensor("S1", s => s.WithIllumination(null!))); + } + + [Test] + public async Task UseMediaProviderRejectsNullProvider() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddSensor("S1", s => s.UseMediaProvider(null!))); + } + + [Test] + public async Task AddIntrinsicCalibrationRejectsEmptyBrowseName() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddSensor("S1", s => + s.AddIntrinsicCalibration(string.Empty, _ => { }))); + } + + [Test] + public async Task AddExtrinsicCalibrationRejectsEmptyBrowseName() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddSensor("S1", s => + s.AddExtrinsicCalibration(string.Empty, _ => { }))); + } + + [Test] + public async Task AddStreamEndpointRejectsEmptyBrowseName() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddSensor("S1", s => + s.AddStreamEndpoint(string.Empty, _ => { }))); + } + + [Test] + public async Task AddClipEndpointRejectsEmptyBrowseName() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddSensor("S1", s => + s.AddClipEndpoint(string.Empty, _ => { }))); + } + + [Test] + public async Task WithTransformRejectsNullTransform() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.Nodes.AddFrame("F1", f => f + .WithFrameId("f1") + .WithTransform(null!))); + } + + [Test] + public async Task AddStreamEndpointWithMjpegAddsCorrectFacet() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + context.Nodes.AddSensor("Cam1", s => s + .WithSensorId("SN-CAM") + .WithModality(VisionSensorModalityEnum.Area2D) + .AddStreamEndpoint("Mjpeg", ep => ep + .WithEndpointId("mjpeg-1") + .WithEndpointUri("http://cam-1/mjpeg") + .WithProtocol(VisionStreamProtocolEnum.Mjpeg) + .WithCodec(VisionVideoCodecEnum.Mjpeg))); + + Assert.That(FindChild(fixture.Manager.Root.Sensors!, "Cam1"), + Is.Not.Null); + } + + [Test] + public async Task AddClipEndpointWithoutInlineDeliveryIsAllowed() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + context.Nodes.AddSensor("Cam1", s => s + .WithSensorId("SN-CAM") + .WithModality(VisionSensorModalityEnum.Area2D) + .AddClipEndpoint("Snap", ep => ep + .WithEndpointId("snap-1") + .WithClipFormat(VisionClipFormatEnum.Png) + .WithInlineDelivery(false, 0) + .WithDefaultProfileName("noinline"))); + + Assert.That(FindChild(fixture.Manager.Root.Sensors!, "Cam1"), + Is.Not.Null); + } + + [Test] + public async Task AddImageSensorHasScenePrimAndMountedOnAreCovered() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + context.Nodes.AddImageSensor("Cam1", s => s + .WithSensorId("SN-CAM") + .WithModality(VisionSensorModalityEnum.Area2D) + .HasScenePrim(new NodeId("scene:prim1", 1)) + .MountedOn(new NodeId("mount:cam1", 1))); + + Assert.That(FindChild(fixture.Manager.Root.Sensors!, "Cam1"), + Is.Not.Null); + } + + [Test] + public async Task AddImageSensorIgnoresNullScenePrimAndMount() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + context.Nodes.AddImageSensor("Cam1", s => s + .WithSensorId("SN-CAM") + .HasScenePrim(NodeId.Null) + .MountedOn(NodeId.Null)); + + Assert.That(FindChild(fixture.Manager.Root.Sensors!, "Cam1"), + Is.Not.Null); + } + + [Test] + public async Task BuildContextExposesVisionAndInstanceNamespaceIndexes() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Multiple(() => + { + Assert.That(context.VisionNamespaceIndex, Is.GreaterThan((ushort)0)); + Assert.That(context.InstanceNamespaceIndex, Is.GreaterThan((ushort)0)); + Assert.That(context.Manager, Is.SameAs(fixture.Manager)); + Assert.That(context.Root, Is.SameAs(fixture.Manager.Root)); + Assert.That(context.Context, Is.Not.Null); + Assert.That(context.CancellationToken, Is.EqualTo(CancellationToken.None)); + }); + } + + [Test] + public async Task GetRequiredServiceThrowsWhenNoServiceProviderConfigured() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + Assert.Throws(() => + context.GetRequiredService()); + } + + private static VisionPose3DDataType CreatePose() + { + return new VisionPose3DDataType + { + FrameId = "world", + Position = new[] { 0.0, 0.0, 0.0 }, + Orientation = new[] { 0.0, 0.0, 0.0, 1.0 }, + Covariance = ArrayOf.Empty + }; + } + + private static VisionIntrinsicsDataType CreateIntrinsics() + { + return new VisionIntrinsicsDataType + { + Fx = 1400.0, + Fy = 1400.0, + Cx = 960.0, + Cy = 540.0, + Skew = 0.0 + }; + } + + private static NodeState? FindChild(NodeState parent, string browseName) + { + var children = new System.Collections.Generic.List(); + parent.GetChildren(null!, children); + for (int ii = 0; ii < children.Count; ii++) + { + if (children[ii].BrowseName.Name == browseName) + { + return children[ii]; + } + } + return null; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionClientCoverageTests.cs b/tests/Opc.Ua.Vision.Tests/VisionClientCoverageTests.cs new file mode 100644 index 0000000000..d6c8d7de74 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionClientCoverageTests.cs @@ -0,0 +1,493 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Runtime.CompilerServices; +using System.Threading; +using System.Threading.Tasks; +using Moq; +using NUnit.Framework; +using Opc.Ua.Client; +using Opc.Ua.Client.Subscriptions; +using Opc.Ua.Client.Subscriptions.Streaming; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Fills the gaps flagged by the vision-coverage baseline: exercises + /// along the + /// happy path (48 lines that no other test entered), the + /// record surface, and the + /// three Observe entry points that were only + /// covered on their argument-guard path. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionClientCoverageTests + { + [Test] + public async Task ReadExtrinsicCalibrationReturnsSnapshotWithMountFramesAndTransform() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + var pose = new VisionPose3DDataType + { + FrameId = "flange", + Position = new double[] { 0.10, 0.20, 0.30 }, + Orientation = new double[] { 0.0, 0.0, 0.0, 1.0 } + }; + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.CalibrationId, + new(2700u, 3), "ex-1"); + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.PerformedAt, + new(2701u, 3), new DateTimeUtc(new DateTime(2024, 6, 1, 12, 0, 0, DateTimeKind.Utc))); + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.Valid, + new(2702u, 3), true); + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.ResidualError, + new(2703u, 3), 0.08); + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.Method, + new(2704u, 3), "HandEye-Tsai"); + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.Mount, + new(2705u, 3), (int)VisionCalibrationMountEnum.EyeInHand); + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.SourceFrame, + new(2706u, 3), harness.SensorNodeId); + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.TargetFrame, + new(2707u, 3), harness.FrameNodeId); + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.Transform, + new(2708u, 3), Variant.FromStructure(pose)); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionExtrinsicCalibrationSnapshot snapshot = await sensor + .ReadExtrinsicCalibrationAsync(harness.ExtrinsicCalibrationNodeId) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(snapshot.NodeId, Is.EqualTo(harness.ExtrinsicCalibrationNodeId)); + Assert.That(snapshot.CalibrationId, Is.EqualTo("ex-1")); + Assert.That(snapshot.Valid, Is.True); + Assert.That(snapshot.ResidualError, Is.EqualTo(0.08)); + Assert.That(snapshot.Method, Is.EqualTo("HandEye-Tsai")); + Assert.That(snapshot.Mount, Is.EqualTo(VisionCalibrationMountEnum.EyeInHand)); + Assert.That(snapshot.SourceFrameId, Is.EqualTo(harness.SensorNodeId)); + Assert.That(snapshot.TargetFrameId, Is.EqualTo(harness.FrameNodeId)); + Assert.That(snapshot.Transform, Is.Not.Null); + Assert.That(snapshot.Transform!.FrameId, Is.EqualTo("flange")); + }); + } + + [Test] + public async Task ReadExtrinsicCalibrationLeavesOptionalMembersDefaultWhenTheyResolveToNull() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + harness.AddValueChild(harness.ExtrinsicCalibrationNodeId, BrowseNames.CalibrationId, + new(2710u, 3), "ex-min"); + // No PerformedAt/Valid/ResidualError/Method/Mount/Frames/Transform bindings. + // BrowsePathResults for those names must come back BadNoMatch, so the ArrayOf + // slots are Null and each TakeXxx helper skips the value read entirely. + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionExtrinsicCalibrationSnapshot snapshot = await sensor + .ReadExtrinsicCalibrationAsync(harness.ExtrinsicCalibrationNodeId) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(snapshot.CalibrationId, Is.EqualTo("ex-min"), + "the one bound member must survive the missing-optional path"); + Assert.That(snapshot.Valid, Is.False, + "TakeBool must return the struct default when the member is absent, not throw"); + Assert.That(snapshot.ResidualError, Is.EqualTo(0.0)); + Assert.That(snapshot.Method, Is.Null); + Assert.That(snapshot.Mount, Is.EqualTo(default(VisionCalibrationMountEnum))); + Assert.That(snapshot.Transform, Is.Null); + // SourceFrameId / TargetFrameId are INullable structs — assert .IsNull rather than + // Is.Null (see the coverage instructions). + Assert.That(snapshot.SourceFrameId.IsNull, Is.True); + Assert.That(snapshot.TargetFrameId.IsNull, Is.True); + }); + } + + [Test] + public void VisionExtrinsicCalibrationSnapshotRecordEqualityIsStructural() + { + var nodeId = new NodeId(9001u, 3); + var pose = new VisionPose3DDataType { FrameId = "tcp" }; + var left = new VisionExtrinsicCalibrationSnapshot + { + NodeId = nodeId, + CalibrationId = "cal", + PerformedAt = new DateTimeUtc(new DateTime(2024, 2, 3, 4, 5, 6, DateTimeKind.Utc)), + Valid = true, + ResidualError = 0.42, + Method = "Tsai", + Mount = VisionCalibrationMountEnum.EyeToHand, + SourceFrameId = new NodeId(1u, 3), + TargetFrameId = new NodeId(2u, 3), + Transform = pose + }; + var right = left with { }; + + Assert.Multiple(() => + { + Assert.That(right, Is.EqualTo(left), + "sealed record `with { }` must yield a structurally equal snapshot"); + Assert.That(right.GetHashCode(), Is.EqualTo(left.GetHashCode())); + Assert.That(right, Is.Not.SameAs(left)); + Assert.That(right.NodeId, Is.EqualTo(nodeId)); + Assert.That(right.Transform, Is.SameAs(pose)); + Assert.That(right.SourceFrameId, Is.EqualTo(new NodeId(1u, 3))); + Assert.That(right.TargetFrameId, Is.EqualTo(new NodeId(2u, 3))); + }); + } + + [Test] + public void ObserveDetectionsAsyncRejectsNullStreamingWithArgumentNullException() + { + var harness = new VisionSessionHarness(); + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + + Assert.That(() => reader.ObserveDetectionsAsync(null!), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("streaming")); + } + + [Test] + public void ObserveInspectionAsyncRejectsNullStreamingWithArgumentNullException() + { + var harness = new VisionSessionHarness(); + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + + Assert.That(() => reader.ObserveInspectionAsync(null!), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("streaming")); + } + + [Test] + public void ObserveDetectionsAsyncThrowsBadNotFoundWhenResultDoesNotExposeDetections() + { + var harness = new VisionSessionHarness(); + var streaming = new Mock().Object; + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + + var ex = Assert.ThrowsAsync(async () => + { + await foreach (var _ in reader.ObserveDetectionsAsync(streaming) + .ConfigureAwait(false)) + { + } + }); + + Assert.That((uint)ex!.StatusCode, Is.EqualTo(StatusCodes.BadNotFound)); + } + + [Test] + public void ObserveInspectionAsyncThrowsBadNotFoundWhenResultDoesNotExposeCharacteristics() + { + var harness = new VisionSessionHarness(); + var streaming = new Mock().Object; + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + + var ex = Assert.ThrowsAsync(async () => + { + await foreach (var _ in reader.ObserveInspectionAsync(streaming) + .ConfigureAwait(false)) + { + } + }); + + Assert.That((uint)ex!.StatusCode, Is.EqualTo(StatusCodes.BadNotFound)); + } + + [Test] + public void ObserveSegmentationAsyncThrowsBadNotFoundWhenResultDoesNotExposeMask() + { + var harness = new VisionSessionHarness(); + var streaming = new Mock().Object; + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + + var ex = Assert.ThrowsAsync(async () => + { + await foreach (var _ in reader.ObserveSegmentationAsync(streaming) + .ConfigureAwait(false)) + { + } + }); + + Assert.That((uint)ex!.StatusCode, Is.EqualTo(StatusCodes.BadNotFound)); + } + + [Test] + public async Task ObserveDetectionsAsyncCompletesGracefullyWhenSubscribeCompletesWithNoNotifications() + { + var harness = new VisionSessionHarness(); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.Detections, + new(3210u, 3), new Variant(ArrayOf.Empty)); + + IReadOnlyList? monitored = null; + var streaming = new Mock(); + streaming + .Setup(s => s.SubscribeDataChangesAsync( + It.IsAny>(), + It.IsAny(), + It.IsAny())) + .Returns, + Opc.Ua.Client.Subscriptions.MonitoredItems.MonitoredItemOptions?, + CancellationToken>((ids, _, ct) => + { + monitored = ids; + return EmptyStreamAsync(ct); + }); + + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + int received = 0; + await foreach (var _ in reader.ObserveDetectionsAsync(streaming.Object) + .ConfigureAwait(false)) + { + received++; + } + + Assert.Multiple(() => + { + Assert.That(received, Is.EqualTo(0), + "an empty upstream stream must yield zero snapshots — the reader " + + "may not fabricate data"); + Assert.That(monitored, Is.Not.Null); + Assert.That(monitored!.Count, Is.EqualTo(1)); + Assert.That(monitored[0], Is.EqualTo(new NodeId(3210u, 3)), + "the observe iterator must have resolved the Detections child NodeId"); + }); + } + + [Test] + public async Task ObserveInspectionAsyncCompletesGracefullyWhenSubscribeCompletesWithNoNotifications() + { + var harness = new VisionSessionHarness(); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.Characteristics, + new(3220u, 3), new Variant(ArrayOf.Empty)); + + IReadOnlyList? monitored = null; + var streaming = new Mock(); + streaming + .Setup(s => s.SubscribeDataChangesAsync( + It.IsAny>(), + It.IsAny(), + It.IsAny())) + .Returns, + Opc.Ua.Client.Subscriptions.MonitoredItems.MonitoredItemOptions?, + CancellationToken>((ids, _, ct) => + { + monitored = ids; + return EmptyStreamAsync(ct); + }); + + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + int received = 0; + await foreach (var _ in reader.ObserveInspectionAsync(streaming.Object) + .ConfigureAwait(false)) + { + received++; + } + + Assert.That(received, Is.EqualTo(0)); + Assert.That(monitored, Is.Not.Null); + Assert.That(monitored![0], Is.EqualTo(new NodeId(3220u, 3))); + } + + [Test] + public async Task ObserveSegmentationAsyncCompletesGracefullyWhenSubscribeCompletesWithNoNotifications() + { + var harness = new VisionSessionHarness(); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.Mask, + new(3230u, 3), new Variant(ByteString.Empty)); + + IReadOnlyList? monitored = null; + var streaming = new Mock(); + streaming + .Setup(s => s.SubscribeDataChangesAsync( + It.IsAny>(), + It.IsAny(), + It.IsAny())) + .Returns, + Opc.Ua.Client.Subscriptions.MonitoredItems.MonitoredItemOptions?, + CancellationToken>((ids, _, ct) => + { + monitored = ids; + return EmptyStreamAsync(ct); + }); + + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + int received = 0; + await foreach (var _ in reader.ObserveSegmentationAsync(streaming.Object) + .ConfigureAwait(false)) + { + received++; + } + + Assert.That(received, Is.EqualTo(0)); + Assert.That(monitored, Is.Not.Null); + Assert.That(monitored![0], Is.EqualTo(new NodeId(3230u, 3))); + } + + [Test] + public async Task GetStreamEndpointAsyncReturnsSessionFromServer() + { + var harness = new VisionSessionHarness(); + var session = new VisionStreamSessionDataType + { + SessionToken = new ByteString(new byte[] { 42, 43 }), + Uri = "rtsp://cam.local/live", + Protocol = VisionStreamProtocolEnum.Rtsp, + ExpiresAt = new DateTimeUtc(new DateTime(2025, 1, 1, 0, 0, 0, DateTimeKind.Utc)) + }; + harness.ConfigureCall(StatusCodes.Good, + Variant.FromStructure(session), + new Variant(harness.StreamEndpointNodeId)); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + VisionStreamSessionDataType result = await media.GetStreamEndpointAsync( + harness.StreamEndpointNodeId, + "default", + VisionStreamProtocolEnum.Rtsp).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result, Is.Not.Null); + Assert.That(result.Uri, Is.EqualTo("rtsp://cam.local/live")); + Assert.That(result.Protocol, Is.EqualTo(VisionStreamProtocolEnum.Rtsp)); + Assert.That(result.SessionToken.ToArray(), Is.EqualTo(new byte[] { 42, 43 }), + "the token bytes must survive the round-trip through the proxy layer"); + }); + } + + [Test] + public async Task ReadLatestClipMetadataAsyncReturnsMetadataWhenPresent() + { + var harness = new VisionSessionHarness(); + var metadata = new VisionImageReferenceDataType + { + Uri = "opc.ua://server/clips/latest.json", + DigestAlgorithm = "SHA-256", + PixelFormat = "Mono8", + Width = 640, + Height = 480 + }; + harness.AddValueChild(harness.ClipEndpointNodeId, BrowseNames.LatestClipMetadata, + new(2401u, 3), Variant.FromStructure(metadata)); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + VisionImageReferenceDataType? result = await media + .ReadLatestClipMetadataAsync(harness.ClipEndpointNodeId) + .ConfigureAwait(false); + + Assert.That(result, Is.Not.Null); + Assert.That(result!.Uri, Is.EqualTo("opc.ua://server/clips/latest.json")); + Assert.That(result.PixelFormat, Is.EqualTo("Mono8")); + Assert.That(result.Width, Is.EqualTo(640u)); + Assert.That(result.Height, Is.EqualTo(480u)); + } + + [Test] + public async Task ReadLatestClipMetadataAsyncReturnsNullWhenAbsent() + { + var harness = new VisionSessionHarness(); + // No metadata node bound — TryReadStructureAsync short-circuits on a null NodeId. + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + VisionImageReferenceDataType? result = await media + .ReadLatestClipMetadataAsync(harness.ClipEndpointNodeId) + .ConfigureAwait(false); + + Assert.That(result, Is.Null, + "when the LatestClipMetadata child is absent the reader must return null " + + "rather than fabricate an empty descriptor"); + } + + private static async IAsyncEnumerable EmptyStreamAsync( + [EnumeratorCancellation] CancellationToken cancellationToken) + { + await Task.CompletedTask.ConfigureAwait(false); + cancellationToken.ThrowIfCancellationRequested(); + yield break; + } + + [Test] + public void VisionClientFactoryCreateAsyncInvokesTheInjectedSessionFactoryAndPropagatesCancellation() + { + using var cts = new CancellationTokenSource(); + cts.Cancel(); + var telemetry = new Mock().Object; + CancellationToken observedToken = default; + int invocations = 0; + Func> sessionFactory = ct => + { + Interlocked.Increment(ref invocations); + observedToken = ct; + ct.ThrowIfCancellationRequested(); + return Task.FromResult(null!); + }; + var factory = new VisionClientFactory(sessionFactory, telemetry); + + Assert.That( + async () => await factory.CreateAsync(cts.Token).ConfigureAwait(false), + Throws.InstanceOf()); + Assert.Multiple(() => + { + Assert.That(invocations, Is.EqualTo(1), + "CreateAsync must forward straight to the session factory once"); + Assert.That(observedToken, Is.EqualTo(cts.Token), + "the exact caller token must be threaded through — not default"); + }); + } + + [Test] + public void VisionClientFactoryConstructorRejectsNullSessionFactory() + { + var telemetry = new Mock().Object; + + Assert.That(() => new VisionClientFactory(null!, telemetry), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("sessionFactory")); + } + + [Test] + public void VisionClientFactoryConstructorRejectsNullTelemetry() + { + Func> sessionFactory = + _ => Task.FromResult(null!); + + Assert.That(() => new VisionClientFactory(sessionFactory, null!), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("telemetry")); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionClientDiscoveryTests.cs b/tests/Opc.Ua.Vision.Tests/VisionClientDiscoveryTests.cs new file mode 100644 index 0000000000..b73d7863fc --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionClientDiscoveryTests.cs @@ -0,0 +1,214 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Discovery and enumeration tests over with a + /// populated Vision address space. Complements + /// , which pins the empty-namespace + /// short-circuit path. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionClientDiscoveryTests + { + [Test] + public void IsVisionNamespaceAvailableReturnsTrueWhenNamespacePresent() + { + var harness = new VisionSessionHarness(); + + Assert.That(harness.Client.IsVisionNamespaceAvailable, Is.True); + } + + [Test] + public void VisionRootIdResolvesToWellKnownVisionObject() + { + var harness = new VisionSessionHarness(); + + NodeId root = harness.Client.VisionRootId; + + Assert.That(root.IsNull, Is.False); + Assert.That(root, Is.EqualTo(harness.VisionRootId)); + } + + [Test] + public void SensorsFolderIdResolvesToVisionSensorsFolder() + { + var harness = new VisionSessionHarness(); + + NodeId sensors = harness.Client.SensorsFolderId; + + Assert.That(sensors.IsNull, Is.False); + Assert.That(sensors, Is.EqualTo(harness.SensorsFolderId)); + } + + [Test] + public async Task GetPipelinesFolderIdReturnsBrowseTargetWhenPresent() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + + NodeId pipelines = await harness.Client.GetPipelinesFolderIdAsync() + .ConfigureAwait(false); + + Assert.That(pipelines, Is.EqualTo(harness.PipelinesFolderId)); + } + + [Test] + public async Task GetFramesFolderIdReturnsBrowseTargetWhenPresent() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + + NodeId frames = await harness.Client.GetFramesFolderIdAsync() + .ConfigureAwait(false); + + Assert.That(frames, Is.EqualTo(harness.FramesFolderId)); + } + + [Test] + public async Task DiscoverSensorsReturnsBrowsedSensorInstances() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType, "Cam1"); + + ArrayOf nodes = await harness.Client.DiscoverSensorsAsync() + .ConfigureAwait(false); + + Assert.That(nodes.Count, Is.EqualTo(1)); + Assert.That(nodes[0], Is.EqualTo(harness.SensorNodeId)); + } + + [Test] + public async Task DiscoverPipelinesReturnsBrowsedPipelineInstances() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline("Pipe1"); + + ArrayOf nodes = await harness.Client.DiscoverPipelinesAsync() + .ConfigureAwait(false); + + Assert.That(nodes.Count, Is.EqualTo(1)); + Assert.That(nodes[0], Is.EqualTo(harness.PipelineNodeId)); + } + + [Test] + public async Task DiscoverFramesReturnsBrowsedCoordinateFrameInstances() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddFrame("F1"); + + ArrayOf nodes = await harness.Client.DiscoverFramesAsync() + .ConfigureAwait(false); + + Assert.That(nodes.Count, Is.EqualTo(1)); + Assert.That(nodes[0], Is.EqualTo(harness.FrameNodeId)); + } + + [Test] + public async Task EnumerateSensorsYieldsEntriesWithBrowseNameAndTypeDefinition() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType, "Cam1"); + + var entries = new List(); + await foreach (VisionNodeEntry entry in harness.Client.EnumerateSensorsAsync()) + { + entries.Add(entry); + } + + Assert.That(entries.Count, Is.EqualTo(1)); + Assert.That(entries[0].NodeId, Is.EqualTo(harness.SensorNodeId)); + Assert.That(entries[0].BrowseName.Name, Is.EqualTo("Cam1")); + Assert.That(entries[0].DisplayName.Text, Is.EqualTo("Cam1")); + } + + [Test] + public async Task EnumeratePipelinesYieldsEntriesWithTypeDefinition() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline("Pipe1"); + + var entries = new List(); + await foreach (VisionNodeEntry entry in harness.Client.EnumeratePipelinesAsync()) + { + entries.Add(entry); + } + + Assert.That(entries.Count, Is.EqualTo(1)); + Assert.That(entries[0].NodeId, Is.EqualTo(harness.PipelineNodeId)); + } + + [Test] + public async Task EnumerateFramesYieldsEntriesWithTypeDefinition() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddFrame("F1"); + + var entries = new List(); + await foreach (VisionNodeEntry entry in harness.Client.EnumerateFramesAsync()) + { + entries.Add(entry); + } + + Assert.That(entries.Count, Is.EqualTo(1)); + Assert.That(entries[0].NodeId, Is.EqualTo(harness.FrameNodeId)); + } + + [Test] + public async Task DiscoverSensorsReturnsEmptyWhenSensorTypeMismatched() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType, "Cam1"); + harness.NodeCache + .Setup(c => c.IsTypeOfAsync( + Moq.It.IsAny(), + Moq.It.IsAny(), + Moq.It.IsAny())) + .Returns(new System.Threading.Tasks.ValueTask(false)); + + ArrayOf nodes = await harness.Client.DiscoverSensorsAsync() + .ConfigureAwait(false); + + Assert.That(nodes.Count, Is.EqualTo(0), + "IsTypeOfAsync returning false must filter out non-Vision instances."); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionClientFacadeTests.cs b/tests/Opc.Ua.Vision.Tests/VisionClientFacadeTests.cs new file mode 100644 index 0000000000..6f9e8bd80e --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionClientFacadeTests.cs @@ -0,0 +1,359 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Moq; +using NUnit.Framework; +using Opc.Ua.Client; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Covers the namespace short-circuit + /// (server has no Vision namespace ⇒ every enumeration returns empty + /// and root NodeIds report ), the sub-client + /// factory methods on , and the argument + /// guards on every facade constructor reached from the client factory + /// methods (, + /// and friends). + /// + [TestFixture] + public sealed class VisionClientFacadeTests + { + [Test] + public void ConstructorThrowsArgumentNullExceptionForNullSession() + { + Assert.That( + () => new VisionClient(null!, new Mock().Object), + Throws.TypeOf()); + } + + [Test] + public void ConstructorThrowsArgumentNullExceptionForNullTelemetry() + { + Mock session = NewSessionMock(); + + Assert.That( + () => new VisionClient(session.Object, null!), + Throws.TypeOf()); + } + + [Test] + public void IsVisionNamespaceAvailableReturnsFalseWhenSessionDoesNotHaveVisionNamespace() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + Assert.That(client.IsVisionNamespaceAvailable, Is.False); + } + + [Test] + public void VisionRootIdIsNullWhenSessionDoesNotHaveVisionNamespace() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + Assert.That(client.VisionRootId.IsNull, Is.True); + } + + [Test] + public void SensorsFolderIdIsNullWhenSessionDoesNotHaveVisionNamespace() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + Assert.That(client.SensorsFolderId.IsNull, Is.True); + } + + [Test] + public async Task DiscoverSensorsReturnsEmptyWhenSessionDoesNotHaveVisionNamespace() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + ArrayOf nodes = await client.DiscoverSensorsAsync().ConfigureAwait(false); + + Assert.That(nodes.Count, Is.EqualTo(0)); + } + + [Test] + public async Task GetPipelinesFolderIdIsNullWhenSessionDoesNotHaveVisionNamespace() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + NodeId nodeId = await client.GetPipelinesFolderIdAsync().ConfigureAwait(false); + + Assert.That(nodeId.IsNull, Is.True); + } + + [Test] + public async Task GetFramesFolderIdIsNullWhenSessionDoesNotHaveVisionNamespace() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + NodeId nodeId = await client.GetFramesFolderIdAsync().ConfigureAwait(false); + + Assert.That(nodeId.IsNull, Is.True); + } + + [Test] + public async Task EnumerateSensorsYieldsNoEntriesWhenSessionDoesNotHaveVisionNamespace() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + var entries = new List(); + await foreach (VisionNodeEntry entry in client.EnumerateSensorsAsync()) + { + entries.Add(entry); + } + + Assert.That(entries, Is.Empty); + } + + [Test] + public async Task DiscoverPipelinesReturnsEmptyWhenSessionDoesNotHaveVisionNamespace() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + ArrayOf nodes = await client.DiscoverPipelinesAsync().ConfigureAwait(false); + + Assert.That(nodes.Count, Is.EqualTo(0)); + } + + [Test] + public async Task DiscoverFramesReturnsEmptyWhenSessionDoesNotHaveVisionNamespace() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + ArrayOf nodes = await client.DiscoverFramesAsync().ConfigureAwait(false); + + Assert.That(nodes.Count, Is.EqualTo(0)); + } + + [Test] + public void SessionAndTelemetryPropertiesReturnConstructorArguments() + { + Mock session = NewSessionMock(); + var telemetry = new Mock().Object; + + var client = new VisionClient(session.Object, telemetry); + + Assert.Multiple(() => + { + Assert.That(client.Session, Is.SameAs(session.Object)); + Assert.That(client.Telemetry, Is.SameAs(telemetry)); + }); + } + + [Test] + public void SensorFactoryRejectsNullNodeIdWithArgumentException() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + Assert.That(() => client.Sensor(NodeId.Null), + Throws.TypeOf()); + } + + [Test] + public void MediaFactoryRejectsNullNodeIdWithArgumentException() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + Assert.That(() => client.Media(NodeId.Null), + Throws.TypeOf()); + } + + [Test] + public void PipelineFactoryRejectsNullNodeIdWithArgumentException() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + Assert.That(() => client.Pipeline(NodeId.Null), + Throws.TypeOf()); + } + + [Test] + public void FeedbackFactoryRejectsNullNodeIdWithArgumentException() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + Assert.That(() => client.Feedback(NodeId.Null), + Throws.TypeOf()); + } + + [Test] + public void ResultFactoryRejectsNullNodeIdWithArgumentException() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + Assert.That(() => client.Result(NodeId.Null), + Throws.TypeOf()); + } + + [Test] + public void SensorFactoryReturnsClientBoundToRequestedNodeId() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + NodeId sensorId = new(1234, 3); + + VisionSensorClient sensor = client.Sensor(sensorId); + + Assert.That(sensor.SensorNodeId, Is.EqualTo(sensorId)); + } + + [Test] + public void MediaFactoryReturnsClientBoundToRequestedNodeId() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + NodeId mediaId = new(555, 3); + + VisionMediaClient media = client.Media(mediaId); + + Assert.That(media.MediaNodeId, Is.EqualTo(mediaId)); + } + + [Test] + public void FramesFactoryReturnsGraphInstance() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + + VisionFrameGraph graph = client.Frames(); + + Assert.That(graph, Is.Not.Null); + } + + [Test] + public void FrameGraphComposeAsyncThrowsArgumentNullExceptionForNullPose() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + VisionFrameGraph graph = client.Frames(); + + Assert.That( + () => graph.ComposeAsync(null!, new NodeId(1, 3), new NodeId(2, 3)), + Throws.TypeOf()); + } + + [Test] + public void FrameGraphComposeAsyncThrowsArgumentExceptionWhenFromFrameIsNull() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + VisionFrameGraph graph = client.Frames(); + var pose = new VisionPose3DDataType + { + FrameId = "a", + Position = new double[] { 0, 0, 0 }, + Orientation = new double[] { 0, 0, 0, 1 }, + Covariance = ArrayOf.Empty + }; + + Assert.That( + () => graph.ComposeAsync(pose, NodeId.Null, new NodeId(2, 3)), + Throws.TypeOf()); + } + + [Test] + public void FrameGraphComposeAsyncThrowsArgumentExceptionWhenToFrameIsNull() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + VisionFrameGraph graph = client.Frames(); + var pose = new VisionPose3DDataType + { + FrameId = "a", + Position = new double[] { 0, 0, 0 }, + Orientation = new double[] { 0, 0, 0, 1 }, + Covariance = ArrayOf.Empty + }; + + Assert.That( + () => graph.ComposeAsync(pose, new NodeId(1, 3), NodeId.Null), + Throws.TypeOf()); + } + + [Test] + public void FrameGraphComposeTransformAsyncRejectsBothNullFrameIds() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + VisionFrameGraph graph = client.Frames(); + + Assert.Multiple(() => + { + Assert.That( + () => graph.ComposeTransformAsync(NodeId.Null, new NodeId(2, 3)), + Throws.TypeOf()); + Assert.That( + () => graph.ComposeTransformAsync(new NodeId(1, 3), NodeId.Null), + Throws.TypeOf()); + }); + } + + [Test] + public void FrameGraphReadAsyncThrowsArgumentExceptionForNullNodeId() + { + VisionClient client = BuildClientWithoutVisionNamespace(); + VisionFrameGraph graph = client.Frames(); + + Assert.That(() => graph.ReadAsync(NodeId.Null), + Throws.TypeOf()); + } + + [Test] + public async Task VisionClientRegistersVisionEncodeableTypesOnConstruction() + { + Mock session = NewSessionMock(); + var telemetry = new Mock().Object; + + _ = new VisionClient(session.Object, telemetry); + + var factory = session.Object.Factory; + var probe = new VisionPose3DDataType(); + Assert.That(factory.TryGetEncodeableType(probe.BinaryEncodingId, out _), Is.True, + "VisionClient must register the Vision encodeable types with the session so it can decode Vision structures."); + await Task.CompletedTask.ConfigureAwait(false); + } + + private static VisionClient BuildClientWithoutVisionNamespace() + { + Mock session = NewSessionMock(); + var telemetry = new Mock().Object; + return new VisionClient(session.Object, telemetry); + } + + private static Mock NewSessionMock() + { + var session = new Mock(); + var telemetry = new Mock().Object; + var messageContext = ServiceMessageContext.Create(telemetry); + session.SetupGet(s => s.NamespaceUris).Returns(new NamespaceTable()); + session.SetupGet(s => s.MessageContext).Returns(messageContext); + session.SetupGet(s => s.Factory).Returns(messageContext.Factory); + return session; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionConformanceUrisTests.cs b/tests/Opc.Ua.Vision.Tests/VisionConformanceUrisTests.cs new file mode 100644 index 0000000000..67fdd398af --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionConformanceUrisTests.cs @@ -0,0 +1,213 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Reflection; +using NUnit.Framework; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Locks down the surface: every + /// facet name has a matching URI, + /// enumerates them in the declared order without dropping any, and the + /// internal TryGetFacetUri only ever accepts the VIS- prefix. + /// + [TestFixture] + public sealed class VisionConformanceUrisTests + { + [Test] + public void FacetBaseAndProfileBaseAreTheAdvertisedUris() + { + Assert.Multiple(() => + { + Assert.That(VisionConformanceUris.FacetBase, + Is.EqualTo("http://opcfoundation.org/UA-Profile/Vision/Facet/")); + Assert.That(VisionConformanceUris.ProfileBase, + Is.EqualTo("http://opcfoundation.org/UA-Profile/Vision/Server/")); + }); + } + + [Test] + public void AllFacetsEnumeratesEveryNameConstantExactlyOnceAndInDeclarationOrder() + { + IReadOnlyList allFacets = MaterializeAllFacets(); + IReadOnlyList declared = ReflectFacetNames(); + + Assert.Multiple(() => + { + Assert.That(allFacets, Is.EquivalentTo(declared), + "AllFacets must list every VisionConformanceUris.FacetNames constant exactly once; " + + "if this fails a name was added without extending the ordered AllFacets array."); + Assert.That(allFacets.Distinct(StringComparer.Ordinal).Count(), Is.EqualTo(allFacets.Count), + "AllFacets must not contain duplicates."); + }); + } + + [Test] + public void EveryFacetNameHasAMatchingFacetUri() + { + IReadOnlyList declared = ReflectFacetNames(); + IReadOnlyDictionary uriByShort = ReflectFacetUrisByShortName(); + foreach (string name in declared) + { + Assert.That(uriByShort.ContainsKey(name), + $"Facet name '{name}' is declared under FacetNames but no matching FacetsUri constant exposes '{VisionConformanceUris.FacetBase}'."); + Assert.That(uriByShort[name], Is.EqualTo(VisionConformanceUris.FacetBase + name["VIS-".Length..]), + $"The URI constant for '{name}' must be FacetBase + '{name["VIS-".Length..]}'."); + } + } + + [Test] + public void TryGetFacetUriBuildsUriFromVisPrefixedShortName() + { + bool ok = InvokeTryGetFacetUri(VisionConformanceUris.FacetNames.Base, out string uri); + + Assert.Multiple(() => + { + Assert.That(ok, Is.True); + Assert.That(uri, Is.EqualTo(VisionConformanceUris.Facets.Base)); + }); + } + + [Test] + public void TryGetFacetUriRejectsInputsWithoutVisPrefix() + { + bool ok = InvokeTryGetFacetUri("Foo-Base", out string uri); + + Assert.Multiple(() => + { + Assert.That(ok, Is.False); + Assert.That(uri, Is.EqualTo(string.Empty)); + }); + } + + [Test] + public void TryGetFacetUriRejectsNullOrEmptyInput() + { + bool okNull = InvokeTryGetFacetUri(null!, out string uriNull); + bool okEmpty = InvokeTryGetFacetUri(string.Empty, out string uriEmpty); + + Assert.Multiple(() => + { + Assert.That(okNull, Is.False); + Assert.That(uriNull, Is.EqualTo(string.Empty)); + Assert.That(okEmpty, Is.False); + Assert.That(uriEmpty, Is.EqualTo(string.Empty)); + }); + } + + [Test] + public void ProfileUrisAreDistinctAndUseTheProfileBaseConstant() + { + var uris = new[] + { + VisionConformanceUris.Profiles.Basic, + VisionConformanceUris.Profiles.Inspection, + VisionConformanceUris.Profiles.Detection, + VisionConformanceUris.Profiles.Inference + }; + + Assert.Multiple(() => + { + foreach (string uri in uris) + { + Assert.That(uri, Does.StartWith(VisionConformanceUris.ProfileBase)); + } + Assert.That(uris.Distinct(StringComparer.Ordinal).Count(), Is.EqualTo(uris.Length)); + }); + } + + private static IReadOnlyList MaterializeAllFacets() + { + var result = new List(); + ArrayOf facets = VisionConformanceUris.AllFacets; + for (int i = 0; i < facets.Count; i++) + { + result.Add(facets[i]); + } + return result; + } + + private static IReadOnlyList ReflectFacetNames() + { + var result = new List(); + foreach (FieldInfo field in typeof(VisionConformanceUris.FacetNames) + .GetFields(BindingFlags.Public | BindingFlags.Static) + .Where(f => f.IsLiteral && !f.IsInitOnly)) + { + object? raw = field.GetRawConstantValue(); + if (raw is string s) + { + result.Add(s); + } + } + return result; + } + + private static IReadOnlyDictionary ReflectFacetUrisByShortName() + { + var byName = typeof(VisionConformanceUris.FacetNames) + .GetFields(BindingFlags.Public | BindingFlags.Static) + .Where(f => f.IsLiteral && !f.IsInitOnly) + .ToDictionary(f => f.Name, f => (string)f.GetRawConstantValue()!, StringComparer.Ordinal); + var byUri = typeof(VisionConformanceUris.Facets) + .GetFields(BindingFlags.Public | BindingFlags.Static) + .Where(f => f.IsLiteral && !f.IsInitOnly) + .ToDictionary(f => f.Name, f => (string)f.GetRawConstantValue()!, StringComparer.Ordinal); + var result = new Dictionary(StringComparer.Ordinal); + foreach (KeyValuePair pair in byName) + { + if (byUri.TryGetValue(pair.Key, out string? uri)) + { + result[pair.Value] = uri; + } + } + return result; + } + + private static bool InvokeTryGetFacetUri(string name, out string facetUri) + { + MethodInfo? method = typeof(VisionConformanceUris).GetMethod( + "TryGetFacetUri", + BindingFlags.NonPublic | BindingFlags.Static, + binder: null, + types: new[] { typeof(string), typeof(string).MakeByRefType() }, + modifiers: null); + Assert.That(method, Is.Not.Null, "VisionConformanceUris.TryGetFacetUri must exist."); + object?[] args = new object?[] { name, string.Empty }; + bool ok = (bool)method!.Invoke(null, args)!; + facetUri = (string)args[1]!; + return ok; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionCoordinateFrameMathTests.cs b/tests/Opc.Ua.Vision.Tests/VisionCoordinateFrameMathTests.cs new file mode 100644 index 0000000000..14534e48ef --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionCoordinateFrameMathTests.cs @@ -0,0 +1,477 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Nails down §5.12 pose composition — quaternion order (x, y, z, w), + /// positions in metres, empty-covariance sentinel, and the + /// frame-precedence rule (the composed pose reports its parent's + /// FrameId). Uses precomputed numeric oracles so a silent sign / axis + /// swap in the rotation kernel cannot slip through as a "did not throw" + /// pass. + /// + [TestFixture] + public sealed class VisionCoordinateFrameMathTests + { + private const double Tol = 1e-10; + + [Test] + public void IdentityReturnsPoseWithZeroPositionUnitQuaternionAndEmptyCovariance() + { + VisionPose3DDataType identity = VisionCoordinateFrameMath.Identity("base"); + + Assert.Multiple(() => + { + Assert.That(identity.FrameId, Is.EqualTo("base")); + Assert.That(identity.Position.Count, Is.EqualTo(3)); + Assert.That(identity.Position[0], Is.EqualTo(0.0)); + Assert.That(identity.Position[1], Is.EqualTo(0.0)); + Assert.That(identity.Position[2], Is.EqualTo(0.0)); + Assert.That(identity.Orientation.Count, Is.EqualTo(4)); + Assert.That(identity.Orientation[0], Is.EqualTo(0.0)); + Assert.That(identity.Orientation[1], Is.EqualTo(0.0)); + Assert.That(identity.Orientation[2], Is.EqualTo(0.0)); + Assert.That(identity.Orientation[3], Is.EqualTo(1.0)); + Assert.That(identity.Covariance.Count, Is.EqualTo(0)); + }); + } + + [Test] + public void IdentityWithNullFrameIdCollapsesToEmptyString() + { + VisionPose3DDataType identity = VisionCoordinateFrameMath.Identity(null!); + + Assert.That(identity.FrameId, Is.EqualTo(string.Empty)); + } + + [Test] + public void ComposeWithIdentityChildReturnsParent() + { + VisionPose3DDataType parent = MakePose( + "world", + new[] { 1.0, 2.0, 3.0 }, + UnitQuaternion(0.0, 0.0, Math.PI / 6.0)); + VisionPose3DDataType child = VisionCoordinateFrameMath.Identity("child"); + + VisionPose3DDataType composed = VisionCoordinateFrameMath.Compose(parent, child); + + AssertPositionEqual(composed, 1.0, 2.0, 3.0); + AssertOrientationEqual(composed, parent.Orientation[0], parent.Orientation[1], parent.Orientation[2], parent.Orientation[3]); + Assert.That(composed.FrameId, Is.EqualTo("world")); + Assert.That(composed.Covariance.Count, Is.EqualTo(0)); + } + + [Test] + public void ComposeAppliesParentRotationToChildTranslation() + { + VisionPose3DDataType parent = MakePose( + "world", + new[] { 10.0, 0.0, 0.0 }, + UnitQuaternion(0.0, 0.0, Math.PI / 2.0)); + VisionPose3DDataType child = MakePose( + "arm", + new[] { 1.0, 0.0, 0.0 }, + new[] { 0.0, 0.0, 0.0, 1.0 }); + + VisionPose3DDataType composed = VisionCoordinateFrameMath.Compose(parent, child); + + AssertPositionEqual(composed, 10.0, 1.0, 0.0); + AssertOrientationEqual(composed, 0.0, 0.0, Math.Sin(Math.PI / 4.0), Math.Cos(Math.PI / 4.0)); + Assert.That(composed.FrameId, Is.EqualTo("world"), "The frame precedence rule requires the composed pose to inherit the parent's FrameId."); + } + + [Test] + public void ComposeThreeStagesCameraToFlangeToBaseAgreesWithManualMultiplication() + { + VisionPose3DDataType baseToWorld = MakePose( + "world", + new[] { 0.0, 0.0, 0.0 }, + new[] { 0.0, 0.0, 0.0, 1.0 }); + VisionPose3DDataType flangeInBase = MakePose( + "base", + new[] { 0.5, 0.0, 0.75 }, + UnitQuaternion(0.0, 0.0, Math.PI / 4.0)); + VisionPose3DDataType cameraInFlange = MakePose( + "flange", + new[] { 0.02, 0.0, 0.1 }, + UnitQuaternion(Math.PI / 12.0, 0.0, 0.0)); + + VisionPose3DDataType flangeInWorld = VisionCoordinateFrameMath.Compose(baseToWorld, flangeInBase); + VisionPose3DDataType cameraInWorldViaFlange = VisionCoordinateFrameMath.Compose(flangeInWorld, cameraInFlange); + VisionPose3DDataType cameraInBase = VisionCoordinateFrameMath.Compose(flangeInBase, cameraInFlange); + VisionPose3DDataType cameraInWorldDirect = VisionCoordinateFrameMath.Compose(baseToWorld, cameraInBase); + + Assert.Multiple(() => + { + Assert.That(cameraInWorldViaFlange.Position[0], Is.EqualTo(cameraInWorldDirect.Position[0]).Within(Tol)); + Assert.That(cameraInWorldViaFlange.Position[1], Is.EqualTo(cameraInWorldDirect.Position[1]).Within(Tol)); + Assert.That(cameraInWorldViaFlange.Position[2], Is.EqualTo(cameraInWorldDirect.Position[2]).Within(Tol)); + Assert.That(cameraInWorldViaFlange.Orientation[0], Is.EqualTo(cameraInWorldDirect.Orientation[0]).Within(Tol)); + Assert.That(cameraInWorldViaFlange.Orientation[1], Is.EqualTo(cameraInWorldDirect.Orientation[1]).Within(Tol)); + Assert.That(cameraInWorldViaFlange.Orientation[2], Is.EqualTo(cameraInWorldDirect.Orientation[2]).Within(Tol)); + Assert.That(cameraInWorldViaFlange.Orientation[3], Is.EqualTo(cameraInWorldDirect.Orientation[3]).Within(Tol)); + }); + } + + [Test] + public void ComposeCameraToFlangeToToolCentrePointYieldsRealNumbers() + { + VisionPose3DDataType tcpInFlange = MakePose( + "flange", + new[] { 0.0, 0.0, 0.20 }, + UnitQuaternion(0.0, Math.PI, 0.0)); + VisionPose3DDataType flangeInBase = MakePose( + "base", + new[] { 0.6, 0.1, 0.4 }, + UnitQuaternion(0.0, 0.0, Math.PI / 3.0)); + + VisionPose3DDataType tcpInBase = VisionCoordinateFrameMath.Compose(flangeInBase, tcpInFlange); + + Assert.Multiple(() => + { + Assert.That(IsFinite(tcpInBase.Position[0]), Is.True); + Assert.That(IsFinite(tcpInBase.Position[1]), Is.True); + Assert.That(IsFinite(tcpInBase.Position[2]), Is.True); + Assert.That(IsFinite(tcpInBase.Orientation[0]), Is.True); + Assert.That(IsFinite(tcpInBase.Orientation[1]), Is.True); + Assert.That(IsFinite(tcpInBase.Orientation[2]), Is.True); + Assert.That(IsFinite(tcpInBase.Orientation[3]), Is.True); + Assert.That(tcpInBase.Position[2], Is.EqualTo(0.6).Within(Tol), + "TCP is 0.2 m along the flange +Z, and after a π rotation about Y that vector points into +Z of base, so ExpectedZ = 0.4 + 0.2 = 0.6."); + }); + } + + [Test] + public void ComposeRenormalisesNonUnitQuaternionAndPreservesAxisDirection() + { + VisionPose3DDataType parent = MakePose( + "world", + new[] { 0.0, 0.0, 0.0 }, + new[] { 0.0, 0.0, 4.0, 4.0 }); + VisionPose3DDataType child = VisionCoordinateFrameMath.Identity("child"); + + VisionPose3DDataType composed = VisionCoordinateFrameMath.Compose(parent, child); + + double norm = Math.Sqrt( + (composed.Orientation[0] * composed.Orientation[0]) + + (composed.Orientation[1] * composed.Orientation[1]) + + (composed.Orientation[2] * composed.Orientation[2]) + + (composed.Orientation[3] * composed.Orientation[3])); + Assert.Multiple(() => + { + Assert.That(norm, Is.EqualTo(1.0).Within(Tol)); + Assert.That(composed.Orientation[2], Is.EqualTo(Math.Sqrt(0.5)).Within(Tol)); + Assert.That(composed.Orientation[3], Is.EqualTo(Math.Sqrt(0.5)).Within(Tol)); + }); + } + + [Test] + public void InvertFollowedByComposeReturnsIdentityWithinTolerance() + { + VisionPose3DDataType pose = MakePose( + "camera", + new[] { 0.3, -0.2, 0.5 }, + UnitQuaternion(Math.PI / 5.0, Math.PI / 7.0, Math.PI / 3.0)); + + VisionPose3DDataType inverse = VisionCoordinateFrameMath.Invert(pose); + VisionPose3DDataType composed = VisionCoordinateFrameMath.Compose(pose, inverse); + + AssertPositionEqual(composed, 0.0, 0.0, 0.0); + AssertOrientationEqual(composed, 0.0, 0.0, 0.0, 1.0); + } + + [Test] + public void ComposeThrowsArgumentExceptionWhenPositionLengthIsWrong() + { + VisionPose3DDataType broken = new VisionPose3DDataType + { + FrameId = "broken", + Position = new double[] { 1.0, 2.0 }, + Orientation = new double[] { 0.0, 0.0, 0.0, 1.0 }, + Covariance = ArrayOf.Empty + }; + VisionPose3DDataType identity = VisionCoordinateFrameMath.Identity("root"); + + Assert.That( + () => VisionCoordinateFrameMath.Compose(broken, identity), + Throws.TypeOf()); + } + + [Test] + public void ComposeThrowsArgumentExceptionWhenOrientationLengthIsWrong() + { + VisionPose3DDataType broken = new VisionPose3DDataType + { + FrameId = "broken", + Position = new double[] { 0.0, 0.0, 0.0 }, + Orientation = new double[] { 0.0, 0.0, 1.0 }, + Covariance = ArrayOf.Empty + }; + VisionPose3DDataType identity = VisionCoordinateFrameMath.Identity("root"); + + Assert.That( + () => VisionCoordinateFrameMath.Compose(broken, identity), + Throws.TypeOf()); + } + + [Test] + public void ComposeRefusesAZeroNormQuaternionRatherThanSubstitutingIdentity() + { + VisionPose3DDataType zeroRotation = new VisionPose3DDataType + { + FrameId = "sensor", + Position = new double[] { 0.1, 0.2, 0.3 }, + Orientation = new double[] { 0.0, 0.0, 0.0, 0.0 }, + Covariance = ArrayOf.Empty + }; + VisionPose3DDataType identity = VisionCoordinateFrameMath.Identity("root"); + + // Treating a zero quaternion as identity would compose a pose that looks plausible + // and points the wrong way. For a grasp that is worse than a refusal. + ServiceResultException exception = Assert.Throws( + () => VisionCoordinateFrameMath.Compose(zeroRotation, identity))!; + Assert.That(exception.StatusCode, Is.EqualTo((StatusCode)StatusCodes.BadInvalidArgument)); + } + + [Test] + public void ComposeAlwaysResetsCovarianceToEmptyArraySentinel() + { + VisionPose3DDataType a = MakePose("world", new[] { 0.1, 0.2, 0.3 }, new[] { 0.0, 0.0, 0.0, 1.0 }); + VisionPose3DDataType b = MakePose("child", new[] { 0.0, 0.0, 0.0 }, new[] { 0.0, 0.0, 0.0, 1.0 }); + a.Covariance = new double[36]; + b.Covariance = new double[36]; + + VisionPose3DDataType composed = VisionCoordinateFrameMath.Compose(a, b); + + Assert.That(composed.Covariance.Count, Is.EqualTo(0)); + } + + [Test] + public void TransformFromToOnIdenticalFrameReturnsIdentityInTargetFrame() + { + var frames = new Dictionary(StringComparer.Ordinal) + { + ["base"] = new( + "base", + VisionFrameRoleEnum.Base, + string.Empty, + VisionCoordinateFrameMath.Identity("base")) + }; + + VisionPose3DDataType pose = VisionCoordinateFrameMath.TransformFromTo(frames, "base", "base"); + + AssertPositionEqual(pose, 0.0, 0.0, 0.0); + AssertOrientationEqual(pose, 0.0, 0.0, 0.0, 1.0); + Assert.That(pose.FrameId, Is.EqualTo("base")); + } + + [Test] + public void TransformFromToWalksTreeAndComposesTransforms() + { + VisionPose3DDataType flangeInBase = MakePose( + "base", + new[] { 1.0, 0.0, 0.0 }, + new[] { 0.0, 0.0, 0.0, 1.0 }); + VisionPose3DDataType cameraInFlange = MakePose( + "flange", + new[] { 0.0, 0.0, 0.2 }, + new[] { 0.0, 0.0, 0.0, 1.0 }); + var frames = new Dictionary(StringComparer.Ordinal) + { + ["base"] = new("base", VisionFrameRoleEnum.Base, string.Empty, VisionCoordinateFrameMath.Identity("base")), + ["flange"] = new("flange", VisionFrameRoleEnum.MechanicalInterface, "base", flangeInBase), + ["camera"] = new("camera", VisionFrameRoleEnum.Camera, "flange", cameraInFlange) + }; + + VisionPose3DDataType poseCameraInBase = VisionCoordinateFrameMath.TransformFromTo(frames, "camera", "base"); + + AssertPositionEqual(poseCameraInBase, 1.0, 0.0, 0.2); + AssertOrientationEqual(poseCameraInBase, 0.0, 0.0, 0.0, 1.0); + Assert.That(poseCameraInBase.FrameId, Is.EqualTo("base")); + } + + [Test] + public void TransformFromToDetectsCycleAndThrowsBadInvalidArgument() + { + var identity = VisionCoordinateFrameMath.Identity(string.Empty); + var frames = new Dictionary(StringComparer.Ordinal) + { + ["a"] = new("a", VisionFrameRoleEnum.Base, "b", identity), + ["b"] = new("b", VisionFrameRoleEnum.Base, "a", identity) + }; + + ServiceResultException ex = Assert.Throws( + () => VisionCoordinateFrameMath.TransformFromTo(frames, "a", "b"))!; + + Assert.That(ex.StatusCode, Is.EqualTo(StatusCodes.BadInvalidArgument)); + } + + [Test] + public void TransformFromToWithUnknownFrameThrowsBadNodeIdUnknown() + { + var identity = VisionCoordinateFrameMath.Identity(string.Empty); + var frames = new Dictionary(StringComparer.Ordinal) + { + ["base"] = new("base", VisionFrameRoleEnum.Base, string.Empty, identity) + }; + + ServiceResultException ex = Assert.Throws( + () => VisionCoordinateFrameMath.TransformFromTo(frames, "ghost", "base"))!; + + Assert.That(ex.StatusCode, Is.EqualTo(StatusCodes.BadNodeIdUnknown)); + } + + [Test] + public void TransformFromToWithMissingParentPointerReportsBadNodeIdUnknown() + { + var identity = VisionCoordinateFrameMath.Identity(string.Empty); + var frames = new Dictionary(StringComparer.Ordinal) + { + ["camera"] = new("camera", VisionFrameRoleEnum.Camera, "missing-parent", identity), + ["base"] = new("base", VisionFrameRoleEnum.Base, string.Empty, identity) + }; + + ServiceResultException ex = Assert.Throws( + () => VisionCoordinateFrameMath.TransformFromTo(frames, "camera", "base"))!; + + Assert.That(ex.StatusCode, Is.EqualTo(StatusCodes.BadNodeIdUnknown)); + } + + [Test] + public void TransformFromToWithEmptyFrameIdArgumentThrowsBadInvalidArgument() + { + var frames = new Dictionary(StringComparer.Ordinal); + + ServiceResultException ex1 = Assert.Throws( + () => VisionCoordinateFrameMath.TransformFromTo(frames, string.Empty, "base"))!; + ServiceResultException ex2 = Assert.Throws( + () => VisionCoordinateFrameMath.TransformFromTo(frames, "base", string.Empty))!; + + Assert.Multiple(() => + { + Assert.That(ex1.StatusCode, Is.EqualTo(StatusCodes.BadInvalidArgument)); + Assert.That(ex2.StatusCode, Is.EqualTo(StatusCodes.BadInvalidArgument)); + }); + } + + [Test] + public void TransformFromToThrowsArgumentNullExceptionForNullFramesDictionary() + { + Assert.That( + () => VisionCoordinateFrameMath.TransformFromTo(null!, "a", "b"), + Throws.TypeOf()); + } + + [Test] + public void InvertThrowsArgumentExceptionOnBadPoseShape() + { + VisionPose3DDataType broken = new VisionPose3DDataType + { + FrameId = "broken", + Position = new double[] { 0.0, 0.0 }, + Orientation = new double[] { 0.0, 0.0, 0.0, 1.0 }, + Covariance = ArrayOf.Empty + }; + + Assert.That(() => VisionCoordinateFrameMath.Invert(broken), + Throws.TypeOf()); + } + + [Test] + public void PositionAndOrientationLengthConstantsMatchSpecification() + { + Assert.Multiple(() => + { + Assert.That(VisionCoordinateFrameMath.PositionLength, Is.EqualTo(3)); + Assert.That(VisionCoordinateFrameMath.OrientationLength, Is.EqualTo(4)); + }); + } + + private static VisionPose3DDataType MakePose(string frameId, double[] position, double[] quaternion) + { + return new VisionPose3DDataType + { + FrameId = frameId, + Position = position, + Orientation = quaternion, + Covariance = ArrayOf.Empty + }; + } + + private static bool IsFinite(double value) + { + return !double.IsNaN(value) && !double.IsInfinity(value); + } + + private static double[] UnitQuaternion(double rollX, double pitchY, double yawZ) + { + double cx = Math.Cos(rollX / 2.0); + double sx = Math.Sin(rollX / 2.0); + double cy = Math.Cos(pitchY / 2.0); + double sy = Math.Sin(pitchY / 2.0); + double cz = Math.Cos(yawZ / 2.0); + double sz = Math.Sin(yawZ / 2.0); + return new double[] + { + (sx * cy * cz) - (cx * sy * sz), + (cx * sy * cz) + (sx * cy * sz), + (cx * cy * sz) - (sx * sy * cz), + (cx * cy * cz) + (sx * sy * sz) + }; + } + + private static void AssertPositionEqual(VisionPose3DDataType pose, double x, double y, double z) + { + Assert.Multiple(() => + { + Assert.That(pose.Position[0], Is.EqualTo(x).Within(Tol)); + Assert.That(pose.Position[1], Is.EqualTo(y).Within(Tol)); + Assert.That(pose.Position[2], Is.EqualTo(z).Within(Tol)); + }); + } + + private static void AssertOrientationEqual(VisionPose3DDataType pose, double x, double y, double z, double w) + { + Assert.Multiple(() => + { + Assert.That(pose.Orientation[0], Is.EqualTo(x).Within(Tol)); + Assert.That(pose.Orientation[1], Is.EqualTo(y).Within(Tol)); + Assert.That(pose.Orientation[2], Is.EqualTo(z).Within(Tol)); + Assert.That(pose.Orientation[3], Is.EqualTo(w).Within(Tol)); + }); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionFacetCalculatorTests.cs b/tests/Opc.Ua.Vision.Tests/VisionFacetCalculatorTests.cs new file mode 100644 index 0000000000..9bad937756 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionFacetCalculatorTests.cs @@ -0,0 +1,344 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Reflection; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Pins facet derivation from the . Robot + /// Intent shipped 31 URIs no registered node backed — the drift is a + /// build-time failure here because the test iterates every facet name + /// constant via reflection and asserts publication tracks registry + /// content exactly, so an extra name added to the URI list without a + /// registry backing fails immediately. + /// + [TestFixture] + public sealed class VisionFacetCalculatorTests + { + [Test] + public void ComputeThrowsArgumentNullExceptionForNullRegistry() + { + Assert.That(() => VisionFacetCalculator.Compute(null!), + Throws.TypeOf()); + } + + [Test] + public void ComputeReturnsEmptyWhenNoSensorOrPipelineIsRegistered() + { + var registry = new VisionRegistry(); + + ArrayOf facets = VisionFacetCalculator.Compute(registry); + + Assert.That(facets.Count, Is.EqualTo(0)); + } + + [Test] + public void ComputeUnionsFacetsFromEverySensorAndPipeline() + { + var registry = new VisionRegistry(); + registry.AddSensor(NewSensor("cam-1", new NodeId(101), VisionConformanceUris.FacetNames.Base, VisionConformanceUris.FacetNames.MediaJpeg)); + registry.AddSensor(NewSensor("cam-2", new NodeId(102), VisionConformanceUris.FacetNames.SensorParams)); + registry.AddPipeline(NewPipeline("pipe-1", new NodeId(201), VisionConformanceUris.FacetNames.InferenceOnServer)); + + List facets = ToList(VisionFacetCalculator.Compute(registry)); + + Assert.Multiple(() => + { + Assert.That(facets, Contains.Item(VisionConformanceUris.FacetNames.Base)); + Assert.That(facets, Contains.Item(VisionConformanceUris.FacetNames.MediaJpeg)); + Assert.That(facets, Contains.Item(VisionConformanceUris.FacetNames.SensorParams)); + Assert.That(facets, Contains.Item(VisionConformanceUris.FacetNames.InferenceOnServer)); + Assert.That(facets.Count, Is.EqualTo(4)); + }); + } + + [Test] + public void ComputeReturnsFacetsInOrdinalSortedOrder() + { + var registry = new VisionRegistry(); + registry.AddSensor(NewSensor("cam-1", new NodeId(101), + VisionConformanceUris.FacetNames.MediaRtsp, + VisionConformanceUris.FacetNames.Base, + VisionConformanceUris.FacetNames.Feedback, + VisionConformanceUris.FacetNames.MediaJpeg)); + + ArrayOf facets = VisionFacetCalculator.Compute(registry); + + var expected = new[] + { + VisionConformanceUris.FacetNames.MediaRtsp, + VisionConformanceUris.FacetNames.Base, + VisionConformanceUris.FacetNames.Feedback, + VisionConformanceUris.FacetNames.MediaJpeg + }.OrderBy(x => x, StringComparer.Ordinal).ToArray(); + var actual = new List(); + for (int i = 0; i < facets.Count; i++) + { + actual.Add(facets[i]); + } + Assert.That(actual, Is.EqualTo(expected)); + } + + [Test] + public void ComputeDeduplicatesFacetsSharedAcrossSensors() + { + var registry = new VisionRegistry(); + registry.AddSensor(NewSensor("a", new NodeId(1), VisionConformanceUris.FacetNames.Base)); + registry.AddSensor(NewSensor("b", new NodeId(2), VisionConformanceUris.FacetNames.Base)); + + ArrayOf facets = VisionFacetCalculator.Compute(registry); + + Assert.That(facets.Count, Is.EqualTo(1)); + } + + [Test] + public void PublishedFacetsAreExactlyThoseBackedByRegistryContent() + { + var registry = new VisionRegistry(); + var declaredFacetNames = new List(); + foreach (FieldInfo field in typeof(VisionConformanceUris.FacetNames) + .GetFields(BindingFlags.Public | BindingFlags.Static) + .Where(f => f.IsLiteral && !f.IsInitOnly)) + { + object? raw = field.GetRawConstantValue(); + if (raw is string name) + { + declaredFacetNames.Add(name); + } + } + Assert.That(declaredFacetNames.Count, Is.GreaterThan(0), + "The reflection sweep must find at least one facet-name constant."); + + for (int i = 0; i < declaredFacetNames.Count; i++) + { + registry.AddSensor(NewSensor("cam" + i, new NodeId((uint)(1000 + i)), declaredFacetNames[i])); + } + + ArrayOf facets = VisionFacetCalculator.Compute(registry); + + var published = new List(); + for (int i = 0; i < facets.Count; i++) + { + published.Add(facets[i]); + } + var expected = declaredFacetNames.OrderBy(x => x, StringComparer.Ordinal).ToList(); + Assert.That(published, Is.EqualTo(expected), + "Every facet the specification declares must be published when a registry entry backs it; " + + "any drift between VisionConformanceUris.FacetNames and what the calculator emits fails here."); + } + + [Test] + public void ComputeIgnoresFacetsThatAreNotBackedByAnyRegistryEntry() + { + var registry = new VisionRegistry(); + registry.AddSensor(NewSensor("cam", new NodeId(1), VisionConformanceUris.FacetNames.Base)); + + List facets = ToList(VisionFacetCalculator.Compute(registry)); + + Assert.Multiple(() => + { + Assert.That(facets, Does.Not.Contain(VisionConformanceUris.FacetNames.MediaJpeg)); + Assert.That(facets, Does.Not.Contain(VisionConformanceUris.FacetNames.Feedback)); + Assert.That(facets, Does.Not.Contain(VisionConformanceUris.FacetNames.InferenceOnServer)); + }); + } + + [Test] + public void ComputePassesThroughFacetsRegisteredThatAreNotInTheDeclaredFacetNameConstants() + { + const string custom = "http://example.com/vision/CustomFacet"; + var registry = new VisionRegistry(); + registry.AddSensor(NewSensor("cam", new NodeId(1), custom)); + + List facets = ToList(VisionFacetCalculator.Compute(registry)); + + Assert.That(facets, Contains.Item(custom), + "The calculator must be a pass-through of registered facets; a hard-coded whitelist here is the exact drift pattern Robot Intent shipped, so a silent filter must fail this test."); + } + + [Test] + public void ComputeProfilesPublishesBasicProfileWhenBaseJpegAndRtspAreAllPresent() + { + ArrayOf facets = new string[] + { + VisionConformanceUris.FacetNames.Base, + VisionConformanceUris.FacetNames.MediaJpeg, + VisionConformanceUris.FacetNames.MediaRtsp + }.ToArrayOf(); + + List profiles = ToList(VisionFacetCalculator.ComputeProfiles(facets)); + + Assert.That(profiles, Contains.Item(VisionConformanceUris.Profiles.Basic)); + } + + [Test] + public void ComputeProfilesDoesNotPublishBasicWhenAnyRequirementIsMissing() + { + ArrayOf facets = new string[] + { + VisionConformanceUris.FacetNames.Base, + VisionConformanceUris.FacetNames.MediaJpeg + }.ToArrayOf(); + + List profiles = ToList(VisionFacetCalculator.ComputeProfiles(facets)); + + Assert.That(profiles, Does.Not.Contain(VisionConformanceUris.Profiles.Basic)); + } + + [Test] + public void ComputeProfilesPublishesInspectionProfileWhenInspectionAndFeedbackFacetsPresent() + { + ArrayOf facets = new string[] + { + VisionConformanceUris.FacetNames.ResultInspection, + VisionConformanceUris.FacetNames.Feedback + }.ToArrayOf(); + + List profiles = ToList(VisionFacetCalculator.ComputeProfiles(facets)); + + Assert.That(profiles, Contains.Item(VisionConformanceUris.Profiles.Inspection)); + } + + [Test] + public void ComputeProfilesPublishesDetectionProfileWhenDetectionAndFeedbackFacetsPresent() + { + ArrayOf facets = new string[] + { + VisionConformanceUris.FacetNames.ResultDetection, + VisionConformanceUris.FacetNames.Feedback + }.ToArrayOf(); + + List profiles = ToList(VisionFacetCalculator.ComputeProfiles(facets)); + + Assert.That(profiles, Contains.Item(VisionConformanceUris.Profiles.Detection)); + } + + [Test] + public void ComputeProfilesRequiresFeedbackForInspectionAndDetection() + { + ArrayOf withoutFeedback = new string[] + { + VisionConformanceUris.FacetNames.ResultInspection, + VisionConformanceUris.FacetNames.ResultDetection + }.ToArrayOf(); + + List profiles = ToList(VisionFacetCalculator.ComputeProfiles(withoutFeedback)); + + Assert.Multiple(() => + { + Assert.That(profiles, Does.Not.Contain(VisionConformanceUris.Profiles.Inspection)); + Assert.That(profiles, Does.Not.Contain(VisionConformanceUris.Profiles.Detection)); + }); + } + + [Test] + public void ComputeProfilesPublishesInferenceProfileWhenInferenceOnServerFacetIsPresent() + { + ArrayOf facets = new string[] + { + VisionConformanceUris.FacetNames.InferenceOnServer + }.ToArrayOf(); + + List profiles = ToList(VisionFacetCalculator.ComputeProfiles(facets)); + + Assert.That(profiles, Contains.Item(VisionConformanceUris.Profiles.Inference)); + } + + [Test] + public void ComputeProfilesReturnsEmptyForEmptyFacetInput() + { + ArrayOf profiles = VisionFacetCalculator.ComputeProfiles(ArrayOf.Empty); + + Assert.That(profiles.Count, Is.EqualTo(0)); + } + + [Test] + public void ComputeProfilesIgnoresNullAndEmptyFacetStringsInInput() + { + ArrayOf facets = new string[] + { + string.Empty, + null!, + VisionConformanceUris.FacetNames.InferenceOnServer + }.ToArrayOf(); + + List profiles = ToList(VisionFacetCalculator.ComputeProfiles(facets)); + + Assert.That(profiles, Contains.Item(VisionConformanceUris.Profiles.Inference)); + } + + private static List ToList(ArrayOf array) + { + var list = new List(array.Count); + for (int i = 0; i < array.Count; i++) + { + list.Add(array[i]); + } + return list; + } + + private static SensorRegistration NewSensor(string browseName, NodeId nodeId, params string[] facets) + { + var facetSet = new HashSet(StringComparer.Ordinal); + foreach (string facet in facets) + { + facetSet.Add(facet); + } + return new SensorRegistration( + browseName, + nodeId, + new VisionSensorState(null), + VisionSensorModalityEnum.Area2D, + VisionRealityKindEnum.Physical, + facetSet, + mediaProvider: null); + } + + private static PipelineRegistration NewPipeline(string browseName, NodeId nodeId, params string[] facets) + { + var facetSet = new HashSet(StringComparer.Ordinal); + foreach (string facet in facets) + { + facetSet.Add(facet); + } + return new PipelineRegistration( + browseName, + nodeId, + new InferencePipelineState(null), + facetSet); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionFeedbackDispatcherTests.cs b/tests/Opc.Ua.Vision.Tests/VisionFeedbackDispatcherTests.cs new file mode 100644 index 0000000000..53e5bebe1e --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionFeedbackDispatcherTests.cs @@ -0,0 +1,562 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Moq; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Pins the feedback publication path. §9 feedback is how an + /// off-Server model publishes results the Server itself did not + /// compute — every rejection code the dispatcher returns is a + /// public contract: + /// + /// when + /// no is bound to the pipeline + /// (missing configuration, not a client fault). + /// The sink's own when + /// the sink runs — good or bad, verbatim. + /// when + /// the sink throws a non-cancellation exception. + /// Propagates + /// unchanged so cooperative + /// cancellation of the caller's context is honoured end-to-end. + /// + /// + [TestFixture] + public sealed class VisionFeedbackDispatcherTests + { + [Test] + public async Task SubmitDetectionsWhenFeedbackSinkIsNullReturnsBadNotSupported() + { + var harness = new FeedbackHarness(pipelineId: 601, feedbackSink: null); + + SubmitDetectionsMethodStateResult result = await harness.InvokeSubmitDetections( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + detections: ArrayOf.Empty, + frameReference: new VisionImageReferenceDataType(), + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadNotSupported)); + } + + [Test] + public async Task SubmitDetectionsForwardsGoodResultFromSinkVerbatimAndCarriesRequestArguments() + { + var sink = new Mock(MockBehavior.Strict); + VisionSubmitDetectionsRequest? captured = null; + sink.Setup(s => s.SubmitDetectionsAsync(It.IsAny(), It.IsAny())) + .Returns((req, _) => + { + captured = req; + return new ValueTask(ServiceResult.Good); + }); + var harness = new FeedbackHarness(pipelineId: 602, feedbackSink: sink.Object); + var frameRef = new VisionImageReferenceDataType { Uri = "opc.tcp://cam/frame/42" }; + ByteString inline = ByteString.From(new byte[] { 1, 2, 3, 4 }); + ArrayOf detections = OneDetection(); + + SubmitDetectionsMethodStateResult result = await harness.InvokeSubmitDetections( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + detections: detections, + frameReference: frameRef, + inlineImage: inline).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(captured, Is.Not.Null); + Assert.That(captured!.Pipeline, Is.EqualTo(harness.PipelineNodeId)); + Assert.That(captured!.Purpose, Is.EqualTo(VisionFeedbackPurposeEnum.Reconciliation)); + Assert.That(captured!.InlineImage, Is.EqualTo(inline)); + }); + } + + [Test] + public async Task SubmitDetectionsReturnsSinkFailureCodeVerbatim() + { + var sink = new Mock(); + sink.Setup(s => s.SubmitDetectionsAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new ServiceResult(StatusCodes.BadInvalidArgument)); + var harness = new FeedbackHarness(pipelineId: 603, feedbackSink: sink.Object); + + SubmitDetectionsMethodStateResult result = await harness.InvokeSubmitDetections( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + detections: ArrayOf.Empty, + frameReference: new VisionImageReferenceDataType(), + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadInvalidArgument), + "The dispatcher must not rewrite a sink's own failure code; the sink is the domain owner."); + } + + [Test] + public async Task SubmitDetectionsWhenSinkThrowsGeneralExceptionReturnsBadInternalError() + { + var sink = new Mock(); + sink.Setup(s => s.SubmitDetectionsAsync(It.IsAny(), It.IsAny())) + .ThrowsAsync(new InvalidOperationException("boom")); + var harness = new FeedbackHarness(pipelineId: 604, feedbackSink: sink.Object); + + SubmitDetectionsMethodStateResult result = await harness.InvokeSubmitDetections( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + detections: OneDetection(), + frameReference: new VisionImageReferenceDataType(), + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadInternalError)); + } + + [Test] + public void SubmitDetectionsPropagatesOperationCanceledExceptionFromSink() + { + var sink = new Mock(); + sink.Setup(s => s.SubmitDetectionsAsync(It.IsAny(), It.IsAny())) + .ThrowsAsync(new OperationCanceledException()); + var harness = new FeedbackHarness(pipelineId: 605, feedbackSink: sink.Object); + + Assert.That( + async () => await harness.InvokeSubmitDetections( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + detections: OneDetection(), + frameReference: new VisionImageReferenceDataType(), + inlineImage: default).ConfigureAwait(false), + Throws.InstanceOf(), + "The dispatcher must let OperationCanceledException flow through so cooperative cancellation is honoured."); + } + + [Test] + public async Task SubmitCorrectionWhenFeedbackSinkIsNullReturnsBadNotSupported() + { + var harness = new FeedbackHarness(pipelineId: 606, feedbackSink: null); + + SubmitCorrectionMethodStateResult result = await harness.InvokeSubmitCorrection( + resultId: "r-1", + purpose: VisionFeedbackPurposeEnum.Overlay, + correctedDetections: ArrayOf.Empty, + correctedCharacteristics: ArrayOf.Empty, + reason: new LocalizedText("en", "test"), + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadNotSupported)); + } + + [Test] + public async Task SubmitCorrectionForwardsRequestArgumentsToSink() + { + var sink = new Mock(); + VisionSubmitCorrectionRequest? captured = null; + sink.Setup(s => s.SubmitCorrectionAsync(It.IsAny(), It.IsAny())) + .Returns((req, _) => + { + captured = req; + return new ValueTask(ServiceResult.Good); + }); + var harness = new FeedbackHarness(pipelineId: 607, feedbackSink: sink.Object); + + SubmitCorrectionMethodStateResult result = await harness.InvokeSubmitCorrection( + resultId: "r-42", + purpose: VisionFeedbackPurposeEnum.GroundTruthLabel, + correctedDetections: OneDetection(), + correctedCharacteristics: ArrayOf.Empty, + reason: new LocalizedText("en", "manual correction"), + inlineImage: default).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(captured, Is.Not.Null); + Assert.That(captured!.Pipeline, Is.EqualTo(harness.PipelineNodeId)); + Assert.That(captured!.ResultId, Is.EqualTo("r-42")); + Assert.That(captured!.Purpose, Is.EqualTo(VisionFeedbackPurposeEnum.GroundTruthLabel)); + }); + } + + [Test] + public async Task SubmitCorrectionRefusesAMissingResultIdRatherThanInventingOne() + { + var sink = new Mock(); + VisionSubmitCorrectionRequest? captured = null; + sink.Setup(s => s.SubmitCorrectionAsync(It.IsAny(), It.IsAny())) + .Returns((req, _) => + { + captured = req; + return new ValueTask(ServiceResult.Good); + }); + var harness = new FeedbackHarness(pipelineId: 608, feedbackSink: sink.Object); + + SubmitCorrectionMethodStateResult result = await harness.InvokeSubmitCorrection( + resultId: null!, + purpose: VisionFeedbackPurposeEnum.Reconciliation, + correctedDetections: ArrayOf.Empty, + correctedCharacteristics: ArrayOf.Empty, + reason: new LocalizedText("en", "no result-id supplied"), + inlineImage: default).ConfigureAwait(false); + + // ResultId names the result being corrected. Substituting an empty string would ask + // the sink to correct an unnamed result, so the dispatcher refuses instead and the + // sink is never called. + Assert.Multiple(() => + { + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo((StatusCode)StatusCodes.BadInvalidArgument)); + Assert.That(captured, Is.Null); + }); + } + + [Test] + public async Task SubmitCorrectionReturnsSinkFailureVerbatim() + { + var sink = new Mock(); + sink.Setup(s => s.SubmitCorrectionAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new ServiceResult(StatusCodes.BadUserAccessDenied)); + var harness = new FeedbackHarness(pipelineId: 609, feedbackSink: sink.Object); + + SubmitCorrectionMethodStateResult result = await harness.InvokeSubmitCorrection( + resultId: "r-1", + purpose: VisionFeedbackPurposeEnum.Reconciliation, + correctedDetections: OneDetection(), + correctedCharacteristics: ArrayOf.Empty, + reason: default, + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadUserAccessDenied)); + } + + [Test] + public async Task SubmitCorrectionWhenSinkThrowsGeneralExceptionReturnsBadInternalError() + { + var sink = new Mock(); + sink.Setup(s => s.SubmitCorrectionAsync(It.IsAny(), It.IsAny())) + .ThrowsAsync(new InvalidOperationException("sink failure")); + var harness = new FeedbackHarness(pipelineId: 610, feedbackSink: sink.Object); + + SubmitCorrectionMethodStateResult result = await harness.InvokeSubmitCorrection( + resultId: "r-1", + purpose: VisionFeedbackPurposeEnum.Reconciliation, + correctedDetections: OneDetection(), + correctedCharacteristics: ArrayOf.Empty, + reason: default, + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadInternalError)); + } + + [Test] + public async Task AttachFeedbackMethodsIsSafeWhenIndividualMethodsAreMissing() + { + var harness = new FeedbackHarness(pipelineId: 611, feedbackSink: null, feedbackWithSubmitOnly: true); + + SubmitDetectionsMethodStateResult result = await harness.InvokeSubmitDetections( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + detections: ArrayOf.Empty, + frameReference: new VisionImageReferenceDataType(), + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadNotSupported), + "AttachFeedbackMethods must tolerate a partial feedback surface — a missing SubmitCorrection method must not cause an NRE when SubmitDetections is invoked."); + } + + [Test] + public async Task SubmitDetectionsRefusesAnEmptyDetectionArrayWithoutTheFlag() + { + var sink = new Mock(MockBehavior.Strict); + var harness = new FeedbackHarness(pipelineId: 612, feedbackSink: sink.Object); + + // Part 9.5 pairs the array with the flag: an empty array without + // SceneIsEmpty is a lost payload, not an observation. The strict mock + // proves the sink is never consulted - the dispatcher owns this rule. + SubmitDetectionsMethodStateResult result = await harness.InvokeSubmitDetections( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + detections: ArrayOf.Empty, + frameReference: new VisionImageReferenceDataType(), + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInvalidArgument)); + } + + [Test] + public async Task SubmitDetectionsAcceptsAnEmptyDetectionArrayWhenSceneIsEmpty() + { + var sink = new Mock(); + VisionSubmitDetectionsRequest? captured = null; + sink.Setup(s => s.SubmitDetectionsAsync(It.IsAny(), It.IsAny())) + .Returns((req, _) => + { + captured = req; + return new ValueTask(ServiceResult.Good); + }); + var harness = new FeedbackHarness(pipelineId: 615, feedbackSink: sink.Object); + + // "I examined this frame and there is nothing in it" is the terminating + // condition of a bin-picking task and a valid negative training label. + SubmitDetectionsMethodStateResult result = await harness.InvokeSubmitDetections( + purpose: VisionFeedbackPurposeEnum.GroundTruthLabel, + detections: ArrayOf.Empty, + frameReference: new VisionImageReferenceDataType(), + inlineImage: default, + sceneIsEmpty: true).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(captured, Is.Not.Null); + Assert.That(captured!.SceneIsEmpty, Is.True, + "The sink has to see the flag; it is what distinguishes the " + + "observation from an empty submission it should discard."); + }); + } + + [Test] + public async Task SubmitDetectionsRefusesSceneIsEmptyWithDetectionsAttached() + { + var sink = new Mock(MockBehavior.Strict); + var harness = new FeedbackHarness(pipelineId: 616, feedbackSink: sink.Object); + + // Asserting the frame is empty while attaching what was found in it says + // two contradictory things about one frame. + SubmitDetectionsMethodStateResult result = await harness.InvokeSubmitDetections( + purpose: VisionFeedbackPurposeEnum.GroundTruthLabel, + detections: OneDetection(), + frameReference: new VisionImageReferenceDataType(), + inlineImage: default, + sceneIsEmpty: true).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInvalidArgument)); + } + + [Test] + public async Task SubmitCorrectionRefusesWhenNeitherCorrectedArrayIsPopulated() + { + var sink = new Mock(MockBehavior.Strict); + var harness = new FeedbackHarness(pipelineId: 613, feedbackSink: sink.Object); + + // Part 9.5 asks for at most one non-empty array, and both empty means + // something only when RetractAll says so. + SubmitCorrectionMethodStateResult result = await harness.InvokeSubmitCorrection( + resultId: "r-1", + purpose: VisionFeedbackPurposeEnum.GroundTruthLabel, + correctedDetections: ArrayOf.Empty, + correctedCharacteristics: ArrayOf.Empty, + reason: default, + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInvalidArgument)); + } + + [Test] + public async Task SubmitCorrectionAcceptsBothArraysEmptyWhenRetractAllIsSet() + { + var sink = new Mock(); + VisionSubmitCorrectionRequest? captured = null; + sink.Setup(s => s.SubmitCorrectionAsync(It.IsAny(), It.IsAny())) + .Returns((req, _) => + { + captured = req; + return new ValueTask(ServiceResult.Good); + }); + var harness = new FeedbackHarness(pipelineId: 617, feedbackSink: sink.Object); + + // The false-positive retraction: correcting a result down to nothing. + // It is the error class an operator is most able to label with + // confidence, and it was inexpressible before this flag existed. + SubmitCorrectionMethodStateResult result = await harness.InvokeSubmitCorrection( + resultId: "r-1", + purpose: VisionFeedbackPurposeEnum.GroundTruthLabel, + correctedDetections: ArrayOf.Empty, + correctedCharacteristics: ArrayOf.Empty, + reason: default, + inlineImage: default, + retractAll: true).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(captured, Is.Not.Null); + Assert.That(captured!.RetractAll, Is.True); + }); + } + + [Test] + public async Task SubmitCorrectionRefusesRetractAllWithAReplacementAttached() + { + var sink = new Mock(MockBehavior.Strict); + var harness = new FeedbackHarness(pipelineId: 618, feedbackSink: sink.Object); + + // RetractAll asserts the result should contain nothing at all, so + // carrying a replacement contradicts it. + SubmitCorrectionMethodStateResult result = await harness.InvokeSubmitCorrection( + resultId: "r-1", + purpose: VisionFeedbackPurposeEnum.GroundTruthLabel, + correctedDetections: OneDetection(), + correctedCharacteristics: ArrayOf.Empty, + reason: default, + inlineImage: default, + retractAll: true).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInvalidArgument)); + } + + [Test] + public async Task SubmitCorrectionRefusesWhenBothCorrectedArraysArePopulated() + { + var sink = new Mock(MockBehavior.Strict); + var harness = new FeedbackHarness(pipelineId: 614, feedbackSink: sink.Object); + + SubmitCorrectionMethodStateResult result = await harness.InvokeSubmitCorrection( + resultId: "r-1", + purpose: VisionFeedbackPurposeEnum.GroundTruthLabel, + correctedDetections: OneDetection(), + correctedCharacteristics: OneCharacteristic(), + reason: default, + inlineImage: default).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInvalidArgument)); + } + + private static ArrayOf OneDetection() + { + return new[] + { + new VisionDetectionDataType { ClassLabel = "Part", Confidence = 0.9 } + }.ToArrayOf(); + } + + private static ArrayOf OneCharacteristic() + { + return new[] + { + new VisionCharacteristicDataType { Name = "Diameter" } + }.ToArrayOf(); + } + + private sealed class FeedbackHarness + { + public FeedbackHarness( + uint pipelineId, + IVisionFeedbackSink? feedbackSink, + bool feedbackWithSubmitOnly = false) + { + PipelineNodeId = new NodeId(pipelineId, 4); + var pipeline = new InferencePipelineState(null); + var feedback = new VisionFeedbackState(null) + { + SubmitDetections = new SubmitDetectionsMethodState(null) + }; + if (!feedbackWithSubmitOnly) + { + feedback.SubmitCorrection = new SubmitCorrectionMethodState(null); + feedback.SubmitInspectionResult = new SubmitInspectionResultMethodState(null); + feedback.SubmitImageReference = new SubmitImageReferenceMethodState(null); + } + + var registration = new PipelineRegistration( + "pipe", + PipelineNodeId, + pipeline, + new HashSet(StringComparer.Ordinal)) + { + FeedbackSink = feedbackSink + }; + + m_registry = new VisionRegistry(); + m_registry.AddPipeline(registration); + var dispatcher = new VisionMethodDispatcher(m_registry, NullLogger.Instance); + dispatcher.AttachFeedbackMethods(PipelineNodeId, feedback); + m_submitDetections = feedback.SubmitDetections!.OnCallAsync; + m_submitCorrection = feedback.SubmitCorrection?.OnCallAsync; + + Assert.That(m_submitDetections, Is.Not.Null, + "AttachFeedbackMethods must wire an OnCallAsync handler onto SubmitDetections."); + } + + public NodeId PipelineNodeId { get; } + + public async Task InvokeSubmitDetections( + VisionFeedbackPurposeEnum purpose, + ArrayOf detections, + VisionImageReferenceDataType frameReference, + ByteString inlineImage, + bool sceneIsEmpty = false) + { + return await m_submitDetections!( + null!, + null!, + PipelineNodeId, + purpose, + detections, + frameReference, + inlineImage, + sceneIsEmpty, + CancellationToken.None).ConfigureAwait(false); + } + + public async Task InvokeSubmitCorrection( + string resultId, + VisionFeedbackPurposeEnum purpose, + ArrayOf correctedDetections, + ArrayOf correctedCharacteristics, + LocalizedText reason, + ByteString inlineImage, + bool retractAll = false) + { + Assert.That(m_submitCorrection, Is.Not.Null, + "This helper requires a full feedback surface; construct FeedbackHarness with feedbackWithSubmitOnly=false."); + return await m_submitCorrection!( + null!, + null!, + PipelineNodeId, + resultId, + purpose, + correctedDetections, + correctedCharacteristics, + reason, + inlineImage, + retractAll, + CancellationToken.None).ConfigureAwait(false); + } + + private readonly VisionRegistry m_registry; + private readonly SubmitDetectionsMethodStateMethodAsyncCallHandler? m_submitDetections; + private readonly SubmitCorrectionMethodStateMethodAsyncCallHandler? m_submitCorrection; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionFeedbackSubmitInspectionAndImageReferenceTests.cs b/tests/Opc.Ua.Vision.Tests/VisionFeedbackSubmitInspectionAndImageReferenceTests.cs new file mode 100644 index 0000000000..656d96aa58 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionFeedbackSubmitInspectionAndImageReferenceTests.cs @@ -0,0 +1,362 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Moq; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Pins the SubmitInspectionResult and SubmitImageReference dispatch + /// paths, mirroring the guarantees that + /// pins for SubmitDetections and SubmitCorrection. + /// + /// + /// The dispatcher's job for both methods is a thin transformation from + /// the generated OPC UA delegate signature to the provider's + /// contract, with these public-visible + /// invariants: + /// + /// when no + /// feedback sink is bound. + /// The sink's own when + /// the sink runs — good or bad, verbatim. + /// when + /// the sink throws a non-cancellation exception. + /// is + /// re-thrown so cooperative cancellation is honoured end-to-end. + /// The caller's arguments are forwarded to the sink + /// verbatim so the sink sees the same request the client sent. + /// + /// The resultId-may-be-null branch of these two handlers is + /// intentionally not asserted here: see the coverage report for the + /// "answers instead of refuses" finding. + /// + [TestFixture] + public sealed class VisionFeedbackSubmitInspectionAndImageReferenceTests + { + [Test] + public async Task SubmitInspectionWhenFeedbackSinkIsNullReturnsBadNotSupported() + { + var harness = new InspectionAndImageRefHarness( + pipelineId: 801, + feedbackSink: null); + + SubmitInspectionResultMethodStateResult result = await harness.InvokeSubmitInspection( + resultId: "r-1", + evaluation: VisionResultEvaluationEnum.Ok, + characteristics: ArrayOf.Empty).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadNotSupported), + "Without a feedback sink the dispatcher must refuse the call with BadNotSupported — " + + "this is a configuration gap, not a client fault."); + } + + [Test] + public async Task SubmitInspectionForwardsRequestArgumentsToSink() + { + var sink = new Mock(MockBehavior.Strict); + VisionSubmitInspectionResultRequest? captured = null; + sink.Setup(s => s.SubmitInspectionResultAsync( + It.IsAny(), It.IsAny())) + .Returns((req, _) => + { + captured = req; + return new ValueTask(ServiceResult.Good); + }); + var harness = new InspectionAndImageRefHarness( + pipelineId: 802, + feedbackSink: sink.Object); + var characteristics = new ArrayOf( + new[] { new VisionCharacteristicDataType { CharacteristicId = "measure/length" } }); + + SubmitInspectionResultMethodStateResult result = await harness.InvokeSubmitInspection( + resultId: "insp-42", + evaluation: VisionResultEvaluationEnum.NotOk, + characteristics: characteristics).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True, + "A Good ServiceResult from the sink must be forwarded to the caller unchanged."); + Assert.That(captured, Is.Not.Null); + Assert.That(captured!.Pipeline, Is.EqualTo(harness.PipelineNodeId), + "The pipeline NodeId the delegate was wired for must be forwarded to the sink."); + Assert.That(captured!.ResultId, Is.EqualTo("insp-42"), + "The caller-supplied ResultId must be forwarded to the sink verbatim."); + Assert.That(captured!.Evaluation, Is.EqualTo(VisionResultEvaluationEnum.NotOk), + "The caller-supplied Evaluation must be forwarded to the sink verbatim."); + }); + } + + [Test] + public async Task SubmitInspectionReturnsSinkFailureCodeVerbatim() + { + var sink = new Mock(MockBehavior.Strict); + sink.Setup(s => s.SubmitInspectionResultAsync( + It.IsAny(), It.IsAny())) + .ReturnsAsync(new ServiceResult(StatusCodes.BadUserAccessDenied)); + var harness = new InspectionAndImageRefHarness( + pipelineId: 803, + feedbackSink: sink.Object); + + SubmitInspectionResultMethodStateResult result = await harness.InvokeSubmitInspection( + resultId: "r-1", + evaluation: VisionResultEvaluationEnum.Ok, + characteristics: ArrayOf.Empty).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadUserAccessDenied), + "The dispatcher must not rewrite a sink's own failure code; the sink is the domain owner."); + } + + [Test] + public async Task SubmitInspectionWhenSinkThrowsGeneralExceptionReturnsBadInternalError() + { + var sink = new Mock(MockBehavior.Strict); + sink.Setup(s => s.SubmitInspectionResultAsync( + It.IsAny(), It.IsAny())) + .ThrowsAsync(new InvalidOperationException("sink failure")); + var harness = new InspectionAndImageRefHarness( + pipelineId: 804, + feedbackSink: sink.Object); + + SubmitInspectionResultMethodStateResult result = await harness.InvokeSubmitInspection( + resultId: "r-1", + evaluation: VisionResultEvaluationEnum.Ok, + characteristics: ArrayOf.Empty).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadInternalError), + "A sink exception must be mapped to BadInternalError so the caller sees a clean failure code."); + } + + [Test] + public void SubmitInspectionPropagatesOperationCanceledExceptionFromSink() + { + var sink = new Mock(MockBehavior.Strict); + sink.Setup(s => s.SubmitInspectionResultAsync( + It.IsAny(), It.IsAny())) + .ThrowsAsync(new OperationCanceledException()); + var harness = new InspectionAndImageRefHarness( + pipelineId: 805, + feedbackSink: sink.Object); + + Assert.That(async () => await harness.InvokeSubmitInspection( + resultId: "r-1", + evaluation: VisionResultEvaluationEnum.Ok, + characteristics: ArrayOf.Empty).ConfigureAwait(false), + Throws.InstanceOf(), + "The dispatcher must let OperationCanceledException flow through so cooperative cancellation is honoured."); + } + + [Test] + public async Task SubmitImageReferenceWhenFeedbackSinkIsNullReturnsBadNotSupported() + { + var harness = new InspectionAndImageRefHarness( + pipelineId: 811, + feedbackSink: null); + + SubmitImageReferenceMethodStateResult result = await harness.InvokeSubmitImageReference( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + image: new VisionImageReferenceDataType(), + resultId: "img-1").ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadNotSupported)); + } + + [Test] + public async Task SubmitImageReferenceForwardsRequestArgumentsToSink() + { + var sink = new Mock(MockBehavior.Strict); + VisionSubmitImageReferenceRequest? captured = null; + sink.Setup(s => s.SubmitImageReferenceAsync( + It.IsAny(), It.IsAny())) + .Returns((req, _) => + { + captured = req; + return new ValueTask(ServiceResult.Good); + }); + var harness = new InspectionAndImageRefHarness( + pipelineId: 812, + feedbackSink: sink.Object); + var image = new VisionImageReferenceDataType { Uri = "opc.tcp://cam/frame/99" }; + + SubmitImageReferenceMethodStateResult result = await harness.InvokeSubmitImageReference( + purpose: VisionFeedbackPurposeEnum.Overlay, + image: image, + resultId: "img-42").ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True, + "A Good ServiceResult from the sink must be forwarded to the caller unchanged."); + Assert.That(captured, Is.Not.Null); + Assert.That(captured!.Pipeline, Is.EqualTo(harness.PipelineNodeId), + "The pipeline NodeId the delegate was wired for must be forwarded to the sink."); + Assert.That(captured!.Purpose, Is.EqualTo(VisionFeedbackPurposeEnum.Overlay), + "The caller-supplied Purpose must be forwarded to the sink verbatim."); + Assert.That(captured!.Image.Uri, Is.EqualTo(image.Uri), + "The caller-supplied image reference must be forwarded to the sink verbatim."); + Assert.That(captured!.ResultId, Is.EqualTo("img-42"), + "The caller-supplied ResultId must be forwarded to the sink verbatim."); + }); + } + + [Test] + public async Task SubmitImageReferenceReturnsSinkFailureCodeVerbatim() + { + var sink = new Mock(MockBehavior.Strict); + sink.Setup(s => s.SubmitImageReferenceAsync( + It.IsAny(), It.IsAny())) + .ReturnsAsync(new ServiceResult(StatusCodes.BadInvalidArgument)); + var harness = new InspectionAndImageRefHarness( + pipelineId: 813, + feedbackSink: sink.Object); + + SubmitImageReferenceMethodStateResult result = await harness.InvokeSubmitImageReference( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + image: new VisionImageReferenceDataType(), + resultId: "r-1").ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadInvalidArgument)); + } + + [Test] + public async Task SubmitImageReferenceWhenSinkThrowsGeneralExceptionReturnsBadInternalError() + { + var sink = new Mock(MockBehavior.Strict); + sink.Setup(s => s.SubmitImageReferenceAsync( + It.IsAny(), It.IsAny())) + .ThrowsAsync(new InvalidOperationException("sink failure")); + var harness = new InspectionAndImageRefHarness( + pipelineId: 814, + feedbackSink: sink.Object); + + SubmitImageReferenceMethodStateResult result = await harness.InvokeSubmitImageReference( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + image: new VisionImageReferenceDataType(), + resultId: "r-1").ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadInternalError)); + } + + [Test] + public void SubmitImageReferencePropagatesOperationCanceledExceptionFromSink() + { + var sink = new Mock(MockBehavior.Strict); + sink.Setup(s => s.SubmitImageReferenceAsync( + It.IsAny(), It.IsAny())) + .ThrowsAsync(new OperationCanceledException()); + var harness = new InspectionAndImageRefHarness( + pipelineId: 815, + feedbackSink: sink.Object); + + Assert.That(async () => await harness.InvokeSubmitImageReference( + purpose: VisionFeedbackPurposeEnum.Reconciliation, + image: new VisionImageReferenceDataType(), + resultId: "r-1").ConfigureAwait(false), + Throws.InstanceOf()); + } + + private sealed class InspectionAndImageRefHarness + { + public InspectionAndImageRefHarness( + uint pipelineId, + IVisionFeedbackSink? feedbackSink) + { + PipelineNodeId = new NodeId(pipelineId, 4); + var pipeline = new InferencePipelineState(null); + var feedback = new VisionFeedbackState(null) + { + SubmitDetections = new SubmitDetectionsMethodState(null), + SubmitCorrection = new SubmitCorrectionMethodState(null), + SubmitInspectionResult = new SubmitInspectionResultMethodState(null), + SubmitImageReference = new SubmitImageReferenceMethodState(null) + }; + var registration = new PipelineRegistration( + "pipe", + PipelineNodeId, + pipeline, + new HashSet(StringComparer.Ordinal)) + { + FeedbackSink = feedbackSink + }; + var registry = new VisionRegistry(); + registry.AddPipeline(registration); + var dispatcher = new VisionMethodDispatcher(registry, NullLogger.Instance); + dispatcher.AttachFeedbackMethods(PipelineNodeId, feedback); + m_submitInspection = feedback.SubmitInspectionResult!.OnCallAsync; + m_submitImageReference = feedback.SubmitImageReference!.OnCallAsync; + Assert.That(m_submitInspection, Is.Not.Null); + Assert.That(m_submitImageReference, Is.Not.Null); + } + + public NodeId PipelineNodeId { get; } + + public async Task InvokeSubmitInspection( + string resultId, + VisionResultEvaluationEnum evaluation, + ArrayOf characteristics) + { + return await m_submitInspection!( + null!, + null!, + PipelineNodeId, + resultId, + evaluation, + characteristics, + CancellationToken.None).ConfigureAwait(false); + } + + public async Task InvokeSubmitImageReference( + VisionFeedbackPurposeEnum purpose, + VisionImageReferenceDataType image, + string resultId) + { + return await m_submitImageReference!( + null!, + null!, + PipelineNodeId, + purpose, + image, + resultId, + CancellationToken.None).ConfigureAwait(false); + } + + private readonly SubmitInspectionResultMethodStateMethodAsyncCallHandler? m_submitInspection; + private readonly SubmitImageReferenceMethodStateMethodAsyncCallHandler? m_submitImageReference; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionHostingCoverageTests.cs b/tests/Opc.Ua.Vision.Tests/VisionHostingCoverageTests.cs new file mode 100644 index 0000000000..2cddb1b857 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionHostingCoverageTests.cs @@ -0,0 +1,478 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using NUnit.Framework; +using Opc.Ua.Server; +using Opc.Ua.Server.Hosting; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Hosting; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Coverage for the generic overloads + /// AddVisionMediaProvider<T>, + /// AddVisionInferenceProvider<T> and + /// AddVisionFeedbackSink<T>, plus argument guards, plus + /// wiring and + /// against a real server + /// fixture. + /// + [TestFixture] + [Category("Vision")] + [Category("Hosting")] + public sealed class VisionHostingCoverageTests + { + [Test] + public void AddVisionMediaProviderGenericRegistersConcreteProviderAndRegistration() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .AddVisionMediaProvider("Sensor1"); + + using ServiceProvider provider = services.BuildServiceProvider(); + var registration = provider + .GetServices().FirstOrDefault(); + var concrete = provider.GetService(); + + Assert.Multiple(() => + { + Assert.That(concrete, Is.Not.Null, + "the generic overload must register TProvider so DI can inject its dependencies"); + Assert.That(registration, Is.Not.Null); + Assert.That(registration!.SensorBrowseName, Is.EqualTo("Sensor1")); + Assert.That(registration.Provider, Is.SameAs(concrete), + "the wrapping registration must resolve TProvider through DI, not new it up"); + }); + } + + [Test] + public void AddVisionMediaProviderGenericThrowsOnNullBuilder() + { + Assert.That( + () => OpcUaServerVisionBuilderExtensions + .AddVisionMediaProvider(null!, "Sensor1"), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("builder")); + } + + [Test] + public void AddVisionMediaProviderGenericThrowsOnEmptyBrowseName() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + + Assert.That( + () => builder.AddVisionMediaProvider(string.Empty), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("sensorBrowseName")); + } + + [Test] + public void AddVisionInferenceProviderGenericRegistersConcreteProviderAndRegistration() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .AddVisionInferenceProvider("Pipeline1"); + + using ServiceProvider provider = services.BuildServiceProvider(); + var registration = provider + .GetServices().FirstOrDefault(); + var concrete = provider.GetService(); + + Assert.Multiple(() => + { + Assert.That(concrete, Is.Not.Null); + Assert.That(registration, Is.Not.Null); + Assert.That(registration!.PipelineBrowseName, Is.EqualTo("Pipeline1")); + Assert.That(registration.Provider, Is.SameAs(concrete)); + Assert.That(registration.OnServer, Is.True, + "the generic overload must default to onServer=true, matching the §8.2 default facet"); + }); + } + + [Test] + public void AddVisionInferenceProviderGenericRespectsOnServerFalseFlag() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .AddVisionInferenceProvider("Pipeline1", onServer: false); + + using ServiceProvider provider = services.BuildServiceProvider(); + var registration = provider + .GetServices().First(); + + Assert.That(registration.OnServer, Is.False); + } + + [Test] + public void AddVisionInferenceProviderGenericThrowsOnNullBuilder() + { + Assert.That( + () => OpcUaServerVisionBuilderExtensions + .AddVisionInferenceProvider(null!, "Pipeline1"), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("builder")); + } + + [Test] + public void AddVisionInferenceProviderGenericThrowsOnEmptyBrowseName() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + + Assert.That( + () => builder.AddVisionInferenceProvider(string.Empty), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("pipelineBrowseName")); + } + + [Test] + public void AddVisionFeedbackSinkGenericRegistersConcreteSinkAndRegistration() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .AddVisionFeedbackSink("Pipeline1"); + + using ServiceProvider provider = services.BuildServiceProvider(); + var registration = provider + .GetServices().FirstOrDefault(); + var concrete = provider.GetService(); + + Assert.Multiple(() => + { + Assert.That(concrete, Is.Not.Null); + Assert.That(registration, Is.Not.Null); + Assert.That(registration!.PipelineBrowseName, Is.EqualTo("Pipeline1")); + Assert.That(registration.Sink, Is.SameAs(concrete)); + }); + } + + [Test] + public void AddVisionFeedbackSinkGenericThrowsOnNullBuilder() + { + Assert.That( + () => OpcUaServerVisionBuilderExtensions + .AddVisionFeedbackSink(null!, "Pipeline1"), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("builder")); + } + + [Test] + public void AddVisionFeedbackSinkGenericThrowsOnEmptyBrowseName() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + + Assert.That( + () => builder.AddVisionFeedbackSink(string.Empty), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("pipelineBrowseName")); + } + + [Test] + public void AddVisionRegistersHostedFactoryAndNodeManagerRegistration() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision(); + + using ServiceProvider provider = services.BuildServiceProvider(); + VisionNodeManagerFactory standalone = provider + .GetRequiredService(); + VisionHostedNodeManagerFactory hosted = provider + .GetRequiredService(); + var registrations = provider.GetServices(); + + Assert.Multiple(() => + { + Assert.That(standalone.NamespacesUris.Count, Is.GreaterThanOrEqualTo(1)); + Assert.That(hosted.NamespacesUris.Count, Is.GreaterThanOrEqualTo(1), + "the hosted factory must report the Vision namespace so the server registers it"); + Assert.That(registrations.Any(), Is.True, + "AddVision must attach a hosted node-manager registration to the server"); + }); + } + + [Test] + public async Task VisionNodeManagerFactoryCreateAsyncReturnsAsyncNodeManagerAgainstRealServer() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + var factory = new VisionNodeManagerFactory(); + + IAsyncNodeManager manager = await factory.CreateAsync( + fixture.Server.CurrentInstance, + fixture.Configuration, + CancellationToken.None).ConfigureAwait(false); + try + { + Assert.That(manager, Is.Not.Null); + Assert.That(manager, Is.InstanceOf(), + "the standalone factory must build a VisionNodeManager"); + } + finally + { + if (manager is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + } + } + + [Test] + public async Task VisionHostedNodeManagerFactoryCreateAsyncReturnsAsyncNodeManagerAgainstRealServer() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision(); + using ServiceProvider serviceProvider = services.BuildServiceProvider(); + VisionHostedNodeManagerFactory factory = serviceProvider + .GetRequiredService(); + + IAsyncNodeManager manager = await factory.CreateAsync( + fixture.Server.CurrentInstance, + fixture.Configuration, + CancellationToken.None).ConfigureAwait(false); + try + { + Assert.That(manager, Is.Not.Null); + Assert.That(manager, Is.InstanceOf()); + } + finally + { + if (manager is IAsyncDisposable asyncDisposable) + { + await asyncDisposable.DisposeAsync().ConfigureAwait(false); + } + } + } + + [Test] + public async Task VisionPostSetupRunnerInvokesConfiguratorsWhoseTargetTypeMatchesTheManager() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + int invocations = 0; + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .ConfigureVision(_ => Interlocked.Increment(ref invocations)); + using ServiceProvider serviceProvider = services.BuildServiceProvider(); + IVisionPostSetupRunner runner = serviceProvider + .GetRequiredService(); + var options = new VisionServerOptions(); + + await runner.RunAsync( + fixture.Manager, + fixture.Manager.Root, + options, + CancellationToken.None).ConfigureAwait(false); + + Assert.That(invocations, Is.EqualTo(1), + "the runner must have invoked the matching configurator exactly once"); + } + + [Test] + public async Task VisionPostSetupRunnerIsANoOpForNonVisionNodeManager() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + int invocations = 0; + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .ConfigureVision(_ => Interlocked.Increment(ref invocations)); + using ServiceProvider serviceProvider = services.BuildServiceProvider(); + IVisionPostSetupRunner runner = serviceProvider + .GetRequiredService(); + using var otherManager = new CustomNodeManager2Stub( + fixture.Server.CurrentInstance, fixture.Configuration); + + await runner.RunAsync( + otherManager, + fixture.Manager.Root, + new VisionServerOptions(), + CancellationToken.None).ConfigureAwait(false); + + Assert.That(invocations, Is.EqualTo(0), + "a non-Vision manager must not trigger any Vision configurator"); + } + + [Test] + public void VisionPostSetupRunnerRejectsNullManager() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision(); + using ServiceProvider serviceProvider = services.BuildServiceProvider(); + IVisionPostSetupRunner runner = serviceProvider + .GetRequiredService(); + + Assert.That( + async () => await runner.RunAsync( + null!, + new VisionRootState(null), + new VisionServerOptions(), + CancellationToken.None).ConfigureAwait(false), + Throws.InstanceOf() + .With.Property("ParamName").EqualTo("manager")); + } + + [System.Diagnostics.CodeAnalysis.SuppressMessage( + "Performance", "CA1812:Avoid uninstantiated internal classes", + Justification = "Instantiated by Microsoft.Extensions.DependencyInjection via AddVisionMediaProvider().")] + private sealed class StubMediaProvider : IVisionMediaProvider + { + public ValueTask GetClipAsync( + VisionClipRequest request, CancellationToken cancellationToken) + { + return new ValueTask(new VisionClipResult( + ServiceResult.Good, + new VisionImageReferenceDataType(), + default, + ByteString.Empty)); + } + + public ValueTask GetStreamAsync( + VisionStreamRequest request, CancellationToken cancellationToken) + { + return new ValueTask(new VisionStreamLease( + ServiceResult.Good, new VisionStreamSessionDataType(), NodeId.Null)); + } + + public ValueTask ReleaseStreamAsync( + ByteString sessionToken, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask ConfigureStreamAsync( + VisionStreamConfigurationRequest request, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask SelectEndpointAsync( + NodeId streamEndpoint, NodeId clipEndpoint, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + } + + [System.Diagnostics.CodeAnalysis.SuppressMessage( + "Performance", "CA1812:Avoid uninstantiated internal classes", + Justification = "Instantiated by Microsoft.Extensions.DependencyInjection via AddVisionInferenceProvider().")] + private sealed class StubInferenceProvider : IVisionInferenceProvider + { + public ValueTask RunInferenceAsync( + VisionInferenceRunRequest request, CancellationToken cancellationToken) + { + return new ValueTask( + new VisionInferenceRunResult(ServiceResult.Good, string.Empty)); + } + + public ValueTask StartContinuousAsync( + NodeId pipeline, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask StopAsync( + NodeId pipeline, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + } + + [System.Diagnostics.CodeAnalysis.SuppressMessage( + "Performance", "CA1812:Avoid uninstantiated internal classes", + Justification = "Instantiated by Microsoft.Extensions.DependencyInjection via AddVisionFeedbackSink().")] + private sealed class StubFeedbackSink : IVisionFeedbackSink + { + public ValueTask SubmitDetectionsAsync( + VisionSubmitDetectionsRequest request, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask SubmitInspectionResultAsync( + VisionSubmitInspectionResultRequest request, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask SubmitCorrectionAsync( + VisionSubmitCorrectionRequest request, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask SubmitImageReferenceAsync( + VisionSubmitImageReferenceRequest request, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + } + + private sealed class CustomNodeManager2Stub : AsyncCustomNodeManager + { + public CustomNodeManager2Stub( + IServerInternal server, ApplicationConfiguration configuration) + : base(server, configuration, server.Telemetry.CreateLogger(), + new[] { "urn:test:stub" }) + { + } + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionHostingTests.cs b/tests/Opc.Ua.Vision.Tests/VisionHostingTests.cs new file mode 100644 index 0000000000..be1ad271c9 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionHostingTests.cs @@ -0,0 +1,395 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.DependencyInjection; +using Moq; +using NUnit.Framework; +using Opc.Ua.Server; +using Opc.Ua.Server.Fluent; +using Opc.Ua.Server.Hosting; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Hosting; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Tests for the Vision hosting extensions on + /// . These validate DI wiring and + /// argument guards without booting a full server. + /// + [TestFixture] + [Category("Vision")] + [Category("Hosting")] + public sealed class VisionHostingTests + { + [Test] + public void AddVisionThrowsOnNullBuilder() + { + Assert.Throws(() => + OpcUaServerVisionBuilderExtensions.AddVision(null!)); + } + + [Test] + public void AddVisionRegistersModelProviderAndPostSetupRunner() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision(); + + using ServiceProvider provider = services.BuildServiceProvider(); + var modelProviders = provider.GetServices(); + IVisionPostSetupRunner runner = + provider.GetRequiredService(); + + Assert.That(modelProviders, Is.Not.Empty); + Assert.That(runner, Is.Not.Null); + } + + [Test] + public void AddVisionAppliesConfigurationDelegate() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision(options => options.InstanceNamespaceUri = "urn:test:vision"); + + using ServiceProvider provider = services.BuildServiceProvider(); + VisionServerOptions options = provider + .GetRequiredService>() + .Value; + + Assert.That(options.InstanceNamespaceUri, Is.EqualTo("urn:test:vision")); + } + + [Test] + public void AddVisionMediaProviderThrowsOnNullBuilder() + { + var provider = new Mock().Object; + + Assert.Throws(() => + OpcUaServerVisionBuilderExtensions.AddVisionMediaProvider( + null!, "Sensor1", provider)); + } + + [Test] + public void AddVisionMediaProviderInstanceThrowsOnEmptyBrowseName() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + var provider = new Mock().Object; + + Assert.Throws(() => + builder.AddVisionMediaProvider(string.Empty, provider)); + } + + [Test] + public void AddVisionMediaProviderInstanceThrowsOnNullProvider() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + + Assert.Throws(() => + builder.AddVisionMediaProvider("Sensor1", (IVisionMediaProvider)null!)); + } + + [Test] + public void AddVisionMediaProviderInstanceRegistersRegistration() + { + IServiceCollection services = new ServiceCollection(); + var mediaProvider = new Mock().Object; + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .AddVisionMediaProvider("Sensor1", mediaProvider); + + using ServiceProvider provider = services.BuildServiceProvider(); + var registration = provider + .GetServices().FirstOrDefault(); + + Assert.That(registration, Is.Not.Null); + Assert.That(registration!.SensorBrowseName, Is.EqualTo("Sensor1")); + Assert.That(registration.Provider, Is.SameAs(mediaProvider)); + } + + [Test] + public void AddVisionInferenceProviderThrowsOnNullBuilder() + { + var provider = new Mock().Object; + + Assert.Throws(() => + OpcUaServerVisionBuilderExtensions.AddVisionInferenceProvider( + null!, "Pipeline1", provider)); + } + + [Test] + public void AddVisionInferenceProviderInstanceThrowsOnEmptyBrowseName() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + var provider = new Mock().Object; + + Assert.Throws(() => + builder.AddVisionInferenceProvider(string.Empty, provider)); + } + + [Test] + public void AddVisionInferenceProviderInstanceThrowsOnNullProvider() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + + Assert.Throws(() => + builder.AddVisionInferenceProvider( + "Pipeline1", (IVisionInferenceProvider)null!)); + } + + [Test] + public void AddVisionInferenceProviderInstanceRegistersRegistration() + { + IServiceCollection services = new ServiceCollection(); + var inferenceProvider = new Mock().Object; + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .AddVisionInferenceProvider("Pipeline1", inferenceProvider); + + using ServiceProvider provider = services.BuildServiceProvider(); + var registration = provider + .GetServices().FirstOrDefault(); + + Assert.That(registration, Is.Not.Null); + Assert.That(registration!.PipelineBrowseName, Is.EqualTo("Pipeline1")); + Assert.That(registration.Provider, Is.SameAs(inferenceProvider)); + Assert.That(registration.OnServer, Is.True); + } + + [Test] + public void AddVisionInferenceProviderRegistersOffServerWhenFlagFalse() + { + IServiceCollection services = new ServiceCollection(); + var inferenceProvider = new Mock().Object; + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .AddVisionInferenceProvider("Pipeline1", inferenceProvider, + onServer: false); + + using ServiceProvider provider = services.BuildServiceProvider(); + var registration = provider + .GetServices().First(); + + Assert.That(registration.OnServer, Is.False); + } + + [Test] + public void AddVisionFeedbackSinkThrowsOnNullBuilder() + { + var sink = new Mock().Object; + + Assert.Throws(() => + OpcUaServerVisionBuilderExtensions.AddVisionFeedbackSink( + null!, "Pipeline1", sink)); + } + + [Test] + public void AddVisionFeedbackSinkInstanceThrowsOnEmptyBrowseName() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + var sink = new Mock().Object; + + Assert.Throws(() => + builder.AddVisionFeedbackSink(string.Empty, sink)); + } + + [Test] + public void AddVisionFeedbackSinkInstanceThrowsOnNullSink() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + + Assert.Throws(() => + builder.AddVisionFeedbackSink("Pipeline1", (IVisionFeedbackSink)null!)); + } + + [Test] + public void AddVisionFeedbackSinkInstanceRegistersRegistration() + { + IServiceCollection services = new ServiceCollection(); + var feedbackSink = new Mock().Object; + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .AddVisionFeedbackSink("Pipeline1", feedbackSink); + + using ServiceProvider provider = services.BuildServiceProvider(); + var registration = provider + .GetServices().FirstOrDefault(); + + Assert.That(registration, Is.Not.Null); + Assert.That(registration!.PipelineBrowseName, Is.EqualTo("Pipeline1")); + Assert.That(registration.Sink, Is.SameAs(feedbackSink)); + } + + [Test] + public void ConfigureVisionThrowsOnNullDelegate() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + + Assert.Throws(() => + builder.ConfigureVision((Action)null!)); + } + + [Test] + public void ConfigureVisionAcceptsSyncDelegate() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .ConfigureVision(_ => { }); + + using ServiceProvider provider = services.BuildServiceProvider(); + var configurators = provider.GetServices(); + + Assert.That(configurators, Is.Not.Empty); + } + + [Test] + public void ConfigureVisionAcceptsAsyncDelegate() + { + IServiceCollection services = new ServiceCollection(); + services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test") + .AddVision() + .ConfigureVision((_, _) => default); + + using ServiceProvider provider = services.BuildServiceProvider(); + var configurators = provider.GetServices(); + + Assert.That(configurators, Is.Not.Empty); + } + + [Test] + public void ConfigureVisionForThrowsOnUnsupportedNodeManagerType() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + + Assert.Throws(() => + builder.ConfigureVisionFor((_, _) => default)); + } + + [Test] + public void ConfigureVisionForThrowsOnNullBuilder() + { + Assert.Throws(() => + OpcUaServerVisionBuilderExtensions.ConfigureVisionFor( + null!, (_, _) => default)); + } + + [Test] + public void ConfigureVisionForThrowsOnNullDelegate() + { + IServiceCollection services = new ServiceCollection(); + IOpcUaServerBuilder builder = services.AddOpcUa() + .AddServer(o => o.ApplicationName = "test"); + + Assert.Throws(() => + builder.ConfigureVisionFor(null!)); + } + + [Test] + public void VisionNodeManagerFactoryDefaultCtorReportsVisionNamespace() + { + var factory = new VisionNodeManagerFactory(); + + ArrayOf namespaces = factory.NamespacesUris; + + Assert.That(namespaces.Count, Is.GreaterThanOrEqualTo(1)); + bool containsVision = false; + for (int ii = 0; ii < namespaces.Count; ii++) + { + if (namespaces[ii] == global::Opc.Ua.Vision.Namespaces.Vision) + { + containsVision = true; + break; + } + } + Assert.That(containsVision, Is.True); + } + + [Test] + public void VisionNodeManagerFactoryReportsInstanceNamespaceInAdditionToVision() + { + var options = new VisionServerOptions + { + InstanceNamespaceUri = "urn:test:vision:instance" + }; + var providers = new IVisionModelProvider[] { new VisionModelProvider() } + .ToArrayOf(); + var factory = new VisionNodeManagerFactory(providers, options); + + ArrayOf namespaces = factory.NamespacesUris; + + bool hasInstance = false; + for (int ii = 0; ii < namespaces.Count; ii++) + { + if (namespaces[ii] == "urn:test:vision:instance") + { + hasInstance = true; + break; + } + } + Assert.That(hasInstance, Is.True); + } + + private abstract class UnsupportedNodeManager : AsyncCustomNodeManager + { + protected UnsupportedNodeManager(IServerInternal server) + : base(server, "urn:test:unsupported") + { + } + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionInferenceServiceTests.cs b/tests/Opc.Ua.Vision.Tests/VisionInferenceServiceTests.cs new file mode 100644 index 0000000000..e65754aebd --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionInferenceServiceTests.cs @@ -0,0 +1,827 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Moq; +using NUnit.Framework; +using Opc.Ua.Client; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Tests for , , + /// and the summary/handle-only detail semantics. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionInferenceServiceTests + { + [Test] + public void InferenceFactoryReturnsNonNullService() + { + var harness = new VisionSessionHarness(); + VisionInferenceService service = harness.Client.Inference(); + Assert.That(service, Is.Not.Null); + } + + [Test] + public async Task RunOneShotReturnsHandleOnlyWithoutReadingPayload() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionInferenceResult result = await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.HandleOnly, + VisionExpectedResultKind.Auto, + maxItems: 10).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.ResultId, Is.EqualTo("result-42")); + Assert.That(result.Resolved, Is.True); + Assert.That(result.RequestedPipelineName, Is.EqualTo("Pipeline1")); + Assert.That(result.RequestedPipelineNodeId, Is.EqualTo(harness.PipelineNodeId)); + Assert.That(result.DetectionSummary, Is.Null, + "HandleOnly must not read a summary."); + Assert.That(result.InspectionSummary, Is.Null); + Assert.That(result.SegmentationSummary, Is.Null); + }); + } + + [Test] + public async Task RunOneShotDetectionSummaryBoundsItems() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + SetupDetectionResultChildren(harness); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionInferenceResult result = await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Auto, + maxItems: 3).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.ResultKind, Is.EqualTo(VisionResultKind.Detection)); + Assert.That(result.DetectionSummary, Is.Not.Null); + Assert.That(result.DetectionSummary!.Items.Count, Is.LessThanOrEqualTo(3)); + }); + } + + [Test] + public async Task RunOneShotDetectionSummaryReadsAllFieldsWhenUnbounded() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + SetupDetectionResultChildren(harness); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionInferenceResult result = await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Auto, + maxItems: 100).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.ResultId, Is.EqualTo("result-42")); + Assert.That(result.Resolved, Is.True); + Assert.That(result.ResultKind, Is.EqualTo(VisionResultKind.Detection)); + Assert.That(result.DetectionSummary, Is.Not.Null); + Assert.That(result.DetectionSummary!.FrameId, Is.EqualTo("world")); + Assert.That(result.DetectionSummary.ModelVersionUsed, Is.EqualTo("v1.0")); + Assert.That(result.RequestedPipelineNodeId, Is.EqualTo(harness.PipelineNodeId)); + Assert.That(result.PipelineId, Is.EqualTo(new NodeId(3002u, 3))); + Assert.That(result.SensorId, Is.EqualTo(harness.SensorNodeId)); + Assert.That(result.FrameId, Is.EqualTo("world"), + "Provenance FrameId propagated to result."); + Assert.That(result.ModelVersionUsed, Is.EqualTo("v1.0"), + "Provenance ModelVersionUsed propagated to result."); + }); + } + + [Test] + public void RunOneShotThrowsOnExpectedKindMismatchWhenAuthoritative() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + + var ex = Assert.ThrowsAsync(async () => + await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Inspection, + maxItems: 10).ConfigureAwait(false)); + + Assert.That(ex!.Message, Does.Contain("Expected result kind 'Inspection'")); + } + + [Test] + public async Task RunOneShotAcceptsMatchingExpectedKind() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + SetupDetectionResultChildren(harness); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionInferenceResult result = await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Detection, + maxItems: 10).ConfigureAwait(false); + + Assert.That(result.ResultKind, Is.EqualTo(VisionResultKind.Detection)); + } + + [Test] + public void RunOneShotRejectsNullPipeline() + { + var harness = new VisionSessionHarness(); + VisionInferenceService service = harness.Client.Inference(); + + Assert.ThrowsAsync(async () => + await service.RunOneShotAsync( + null!, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Auto, + maxItems: 10).ConfigureAwait(false)); + } + + [Test] + public void RunOneShotRejectsNegativeMaxItems() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + + Assert.ThrowsAsync(async () => + await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Auto, + maxItems: -1).ConfigureAwait(false)); + } + + [Test] + public void RunOneShotRejectsMaxItemsAbove100() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + + Assert.ThrowsAsync(async () => + await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Auto, + maxItems: 101).ConfigureAwait(false)); + } + + [Test] + public void RunOneShotRejectsUndefinedExpectedKind() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + + Assert.ThrowsAsync(async () => + await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + (VisionExpectedResultKind)99, + maxItems: 10).ConfigureAwait(false)); + } + + [Test] + public void RunOneShotRejectsUndefinedDetail() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + + Assert.ThrowsAsync(async () => + await service.RunOneShotAsync( + pipeline, + "Pipeline1", + (VisionResultDetail)99, + VisionExpectedResultKind.Auto, + maxItems: 10).ConfigureAwait(false)); + } + + [Test] + public async Task RunOneShotUnresolvedResultReturnsHandleWithResolvedFalse() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.ConfigureCall(StatusCodes.Good, new Variant("result-orphan")); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionInferenceResult result = await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Auto, + maxItems: 10).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.ResultId, Is.EqualTo("result-orphan")); + Assert.That(result.Resolved, Is.False); + Assert.That(result.ResultNodeId.IsNull, Is.True); + Assert.That(result.ResultKind, Is.EqualTo(VisionResultKind.Unknown)); + Assert.That(result.DetectionSummary, Is.Null); + }); + } + + [Test] + public async Task RunOneShotUnresolvedResultReturnsHandleEvenWithExpectedKind() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.ConfigureCall(StatusCodes.Good, new Variant("result-orphan")); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionInferenceResult result = await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Detection, + maxItems: 10).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Resolved, Is.False, + "Unresolved result should not throw even with expectedKind set."); + Assert.That(result.ResultKind, Is.EqualTo(VisionResultKind.Unknown)); + Assert.That(result.DetectionSummary, Is.Null, + "Unresolved result returns handle-only."); + }); + } + + [Test] + public async Task RunOneShotThrowsWhenResolvedKindCannotBeDeterminedForConcreteExpectedKind() + { + var harness = new VisionSessionHarness(); + SetupMinimalDetectionPipeline(harness); + harness.NodeCache.Reset(); + harness.NodeCache + .Setup(c => c.IsTypeOfAsync( + It.IsAny(), It.IsAny(), It.IsAny())) + .Returns((NodeId _, NodeId type, CancellationToken _) => + new ValueTask( + type == new NodeId( + ObjectTypes.VisionResultType, + harness.VisionNamespaceIndex))); + + VisionInferenceService service = harness.Client.Inference(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionResultKind kind = await service.DetermineResultKindAsync( + harness.InferenceResultNodeId).ConfigureAwait(false); + Assert.That(kind, Is.EqualTo(VisionResultKind.Unknown)); + VisionInferenceResult autoResult = await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.HandleOnly, + VisionExpectedResultKind.Auto, + maxItems: 10).ConfigureAwait(false); + Assert.That(autoResult.Resolved, Is.True); + Assert.That(autoResult.ResultKind, Is.EqualTo(VisionResultKind.Unknown)); + + var ex = Assert.ThrowsAsync(async () => + await service.RunOneShotAsync( + pipeline, + "Pipeline1", + VisionResultDetail.Summary, + VisionExpectedResultKind.Detection, + maxItems: 10).ConfigureAwait(false)); + + Assert.That(ex!.Message, Does.Contain("Cannot determine result kind")); + } + + [Test] + public async Task DetermineResultKindReturnsDetection() + { + var harness = new VisionSessionHarness(); + SetupResultTypeDefinition(harness, ObjectTypes.DetectionResultType); + + VisionInferenceService service = harness.Client.Inference(); + VisionResultKind kind = await service.DetermineResultKindAsync( + harness.InferenceResultNodeId).ConfigureAwait(false); + + Assert.That(kind, Is.EqualTo(VisionResultKind.Detection)); + } + + [Test] + public async Task DetermineResultKindReturnsUnknownForNullNodeId() + { + var harness = new VisionSessionHarness(); + + VisionInferenceService service = harness.Client.Inference(); + VisionResultKind kind = await service.DetermineResultKindAsync( + NodeId.Null).ConfigureAwait(false); + + Assert.That(kind, Is.EqualTo(VisionResultKind.Unknown)); + } + + [Test] + public async Task DetermineResultKindDetectsSubtype() + { + var harness = new VisionSessionHarness(); + const uint vendorSubtype = 99999; + harness.AddBrowse(harness.InferenceResultNodeId, + [new ReferenceDescription + { + NodeId = new ExpandedNodeId( + new NodeId(vendorSubtype, harness.VisionNamespaceIndex)), + BrowseName = new QualifiedName("VendorDetectionResultType", + harness.VisionNamespaceIndex), + DisplayName = new LocalizedText("VendorDetectionResultType"), + NodeClass = NodeClass.ObjectType, + TypeDefinition = ExpandedNodeId.Null, + ReferenceTypeId = Opc.Ua.ReferenceTypeIds.HasTypeDefinition, + IsForward = true + }]); + + VisionInferenceService service = harness.Client.Inference(); + VisionResultKind kind = await service.DetermineResultKindAsync( + harness.InferenceResultNodeId).ConfigureAwait(false); + + Assert.That(kind, Is.EqualTo(VisionResultKind.Detection), + "IsTypeOfAsync returns true for subtypes, so vendor-derived " + + "types should resolve to the base Vision kind."); + } + + [Test] + public async Task DetermineResultKindReturnsUnknownForZeroRefs() + { + var harness = new VisionSessionHarness(); + harness.AddBrowse(harness.InferenceResultNodeId, []); + + VisionInferenceService service = harness.Client.Inference(); + VisionResultKind kind = await service.DetermineResultKindAsync( + harness.InferenceResultNodeId).ConfigureAwait(false); + + Assert.That(kind, Is.EqualTo(VisionResultKind.Unknown)); + } + + [Test] + public void DetermineResultKindThrowsOnMultipleTypeDefinitions() + { + var harness = new VisionSessionHarness(); + harness.AddBrowse(harness.InferenceResultNodeId, + [ + new ReferenceDescription + { + NodeId = new ExpandedNodeId( + new NodeId(ObjectTypes.DetectionResultType, + harness.VisionNamespaceIndex)), + BrowseName = new QualifiedName("DetectionResultType", + harness.VisionNamespaceIndex), + DisplayName = new LocalizedText("DetectionResultType"), + NodeClass = NodeClass.ObjectType, + TypeDefinition = ExpandedNodeId.Null, + ReferenceTypeId = Opc.Ua.ReferenceTypeIds.HasTypeDefinition, + IsForward = true + }, + new ReferenceDescription + { + NodeId = new ExpandedNodeId( + new NodeId(ObjectTypes.InspectionResultType, + harness.VisionNamespaceIndex)), + BrowseName = new QualifiedName("InspectionResultType", + harness.VisionNamespaceIndex), + DisplayName = new LocalizedText("InspectionResultType"), + NodeClass = NodeClass.ObjectType, + TypeDefinition = ExpandedNodeId.Null, + ReferenceTypeId = Opc.Ua.ReferenceTypeIds.HasTypeDefinition, + IsForward = true + } + ]); + + VisionInferenceService service = harness.Client.Inference(); + Assert.ThrowsAsync(async () => + await service.DetermineResultKindAsync(harness.InferenceResultNodeId) + .ConfigureAwait(false)); + } + + [Test] + public void VisionResultKindEnumHasExpectedValues() + { + Assert.Multiple(() => + { + Assert.That((int)VisionResultKind.Unknown, Is.EqualTo(0)); + Assert.That((int)VisionResultKind.Detection, Is.EqualTo(1)); + Assert.That((int)VisionResultKind.Inspection, Is.EqualTo(2)); + Assert.That((int)VisionResultKind.Segmentation, Is.EqualTo(3)); + }); + } + + [Test] + public void VisionExpectedResultKindEnumHasExpectedValues() + { + Assert.Multiple(() => + { + Assert.That((int)VisionExpectedResultKind.Auto, Is.EqualTo(0)); + Assert.That((int)VisionExpectedResultKind.Detection, Is.EqualTo(1)); + Assert.That((int)VisionExpectedResultKind.Inspection, Is.EqualTo(2)); + Assert.That((int)VisionExpectedResultKind.Segmentation, Is.EqualTo(3)); + }); + } + + [Test] + public void VisionResultDetailEnumHasExpectedValues() + { + Assert.Multiple(() => + { + Assert.That((int)VisionResultDetail.Summary, Is.EqualTo(0)); + Assert.That((int)VisionResultDetail.HandleOnly, Is.EqualTo(1)); + }); + } + + [Test] + public void VisionDetectionSummaryRecordRoundTrips() + { + var summary = new VisionDetectionSummary + { + CreationTime = new DateTimeUtc(new DateTime(2024, 6, 1, 12, 0, 0, DateTimeKind.Utc)), + ModelVersionUsed = "v2.1", + FrameId = "world", + TotalDetections = 3, + Items = new VisionDetectionItem[] + { + new() + { + DetectionId = "det-1", + ClassLabel = "RedCube", + ClassId = 1, + Confidence = 0.95, + HasPose = false + } + }.ToArrayOf() + }; + + Assert.Multiple(() => + { + Assert.That(summary.TotalDetections, Is.EqualTo(3)); + Assert.That(summary.Items.Count, Is.EqualTo(1)); + Assert.That(summary.Items[0].DetectionId, Is.EqualTo("det-1")); + Assert.That(summary.Items[0].ClassLabel, Is.EqualTo("RedCube")); + Assert.That(summary.Items[0].Confidence, Is.EqualTo(0.95)); + Assert.That(summary.Items[0].HasPose, Is.False); + Assert.That(summary.Items[0].Pose, Is.Null); + Assert.That(summary.ModelVersionUsed, Is.EqualTo("v2.1")); + Assert.That(summary.FrameId, Is.EqualTo("world")); + }); + } + + [Test] + public void VisionDetectionSummaryRetainsFullPoseForDirectConsumers() + { + var pose = new VisionPose3DDataType + { + FrameId = "world" + }; + var summary = new VisionDetectionSummary + { + Items = new VisionDetectionItem[] + { + new() + { + DetectionId = "det-pose", + HasPose = true, + Pose = pose + } + }.ToArrayOf() + }; + + Assert.That(summary.Items[0].Pose, Is.SameAs(pose)); + } + + [Test] + public void VisionInspectionSummaryRecordRoundTrips() + { + var summary = new VisionInspectionSummary + { + CreationTime = new DateTimeUtc(new DateTime(2024, 6, 1, 12, 0, 0, DateTimeKind.Utc)), + Evaluation = VisionResultEvaluationEnum.Ok, + PartId = "part-A", + RecipeId = "recipe-1", + TotalCharacteristics = 2, + Items = new VisionCharacteristicItem[] + { + new() + { + Name = "diameter", + Status = VisionToleranceStatusEnum.InTolerance, + Deviation = 0.01 + } + }.ToArrayOf() + }; + + Assert.Multiple(() => + { + Assert.That(summary.Evaluation, Is.EqualTo(VisionResultEvaluationEnum.Ok)); + Assert.That(summary.PartId, Is.EqualTo("part-A")); + Assert.That(summary.RecipeId, Is.EqualTo("recipe-1")); + Assert.That(summary.TotalCharacteristics, Is.EqualTo(2)); + Assert.That(summary.Items.Count, Is.EqualTo(1)); + Assert.That(summary.Items[0].Name, Is.EqualTo("diameter")); + }); + } + + [Test] + public void VisionSegmentationSummaryRecordRoundTrips() + { + var summary = new VisionSegmentationSummary + { + CreationTime = new DateTimeUtc(new DateTime(2024, 6, 1, 12, 0, 0, DateTimeKind.Utc)), + LabelClasses = new[] { "background", "part" }.ToArrayOf(), + MaskWidth = 640, + MaskHeight = 480, + MaskFormat = "Mono8" + }; + + Assert.Multiple(() => + { + Assert.That(summary.LabelClasses.Count, Is.EqualTo(2)); + Assert.That(summary.MaskWidth, Is.EqualTo(640)); + Assert.That(summary.MaskHeight, Is.EqualTo(480)); + Assert.That(summary.MaskFormat, Is.EqualTo("Mono8")); + }); + } + + [Test] + public void VisionInferenceResultRecordRequiresResultIdAndRequestedPipelineNodeId() + { + var result = new VisionInferenceResult + { + ResultId = "test-result", + RequestedPipelineNodeId = new NodeId(1u, 2) + }; + + Assert.Multiple(() => + { + Assert.That(result.ResultId, Is.EqualTo("test-result")); + Assert.That(result.RequestedPipelineNodeId, Is.EqualTo(new NodeId(1u, 2))); + Assert.That(result.Resolved, Is.False); + Assert.That(result.ResultKind, Is.EqualTo(VisionResultKind.Unknown)); + Assert.That(result.PipelineId.IsNull, Is.True); + Assert.That(result.SensorId.IsNull, Is.True); + Assert.That(result.ModelVersionUsed, Is.Null); + Assert.That(result.FrameId, Is.Null); + Assert.That(result.DetectionSummary, Is.Null); + Assert.That(result.InspectionSummary, Is.Null); + Assert.That(result.SegmentationSummary, Is.Null); + }); + } + + [Test] + public async Task ResolvePipelineByNameReturnsMatchingEntry() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline("BinPickingPipeline"); + + VisionNodeEntry entry = await harness.Client.ResolvePipelineAsync( + "BinPickingPipeline").ConfigureAwait(false); + + Assert.That(entry.BrowseName.Name, Is.EqualTo("BinPickingPipeline")); + } + + [Test] + public async Task ResolvePipelineByNodeIdReturnsMatchingEntry() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline("Pipeline1"); + + VisionNodeEntry entry = await harness.Client.ResolvePipelineAsync( + harness.PipelineNodeId.ToString()).ConfigureAwait(false); + + Assert.That(entry.NodeId, Is.EqualTo(harness.PipelineNodeId)); + } + + [Test] + public async Task ResolvePipelineByDisplayNameReturnsEntry() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline("MyPipeline"); + + VisionNodeEntry entry = await harness.Client.ResolvePipelineAsync( + "MyPipeline").ConfigureAwait(false); + + Assert.That(entry.DisplayName.Text, Is.EqualTo("MyPipeline")); + } + + [Test] + public async Task ResolvePipelineTrimsWhitespace() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline("BinPicking"); + + VisionNodeEntry entry = await harness.Client.ResolvePipelineAsync( + " BinPicking ").ConfigureAwait(false); + + Assert.That(entry.BrowseName.Name, Is.EqualTo("BinPicking")); + } + + [Test] + public void ResolvePipelineIsCaseSensitive() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline("BinPicking"); + + Assert.ThrowsAsync(async () => + await harness.Client.ResolvePipelineAsync("binpicking") + .ConfigureAwait(false)); + } + + [Test] + public void ResolvePipelineRejectsNullOrWhitespace() + { + var harness = new VisionSessionHarness(); + + Assert.ThrowsAsync(async () => + await harness.Client.ResolvePipelineAsync(string.Empty) + .ConfigureAwait(false)); + + Assert.ThrowsAsync(async () => + await harness.Client.ResolvePipelineAsync(" ") + .ConfigureAwait(false)); + } + + [Test] + public void ResolvePipelineThrowsWhenNotFound() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline("Alpha"); + + var ex = Assert.ThrowsAsync(async () => + await harness.Client.ResolvePipelineAsync("Beta") + .ConfigureAwait(false)); + + Assert.That(ex!.Message, Does.Contain("not found")); + Assert.That(ex.Message, Does.Contain("Alpha")); + } + + [Test] + public void ResolvePipelineErrorsListBrowseNameDisplayNameAndNodeId() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + ReferenceDescription entry = harness.Ref( + harness.PipelineNodeId, + "PipelineBrowseName", + ObjectTypes.InferencePipelineType); + entry.DisplayName = new LocalizedText("Pipeline Display Name"); + harness.AddBrowse(harness.PipelinesFolderId, [entry]); + + var ex = Assert.ThrowsAsync(async () => + await harness.Client.ResolvePipelineAsync("missing").ConfigureAwait(false)); + + Assert.Multiple(() => + { + Assert.That(ex!.Message, Does.Contain("BrowseName='PipelineBrowseName'")); + Assert.That(ex.Message, Does.Contain("DisplayName='Pipeline Display Name'")); + Assert.That(ex.Message, Does.Contain($"NodeId='{harness.PipelineNodeId}'")); + }); + } + + [Test] + public void ResolvePipelineThrowsOnAmbiguity() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + var secondPipelineId = new NodeId(3001u, 3); + harness.AddBrowse(harness.PipelinesFolderId, + [ + harness.Ref(harness.PipelineNodeId, "Dup", + ObjectTypes.InferencePipelineType), + harness.Ref(secondPipelineId, "Dup", + ObjectTypes.InferencePipelineType) + ]); + + var ex = Assert.ThrowsAsync(async () => + await harness.Client.ResolvePipelineAsync("Dup") + .ConfigureAwait(false)); + + Assert.That(ex!.Message, Does.Contain("Ambiguous")); + } + + private static void SetupMinimalDetectionPipeline(VisionSessionHarness harness) + { + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.ConfigureCall(StatusCodes.Good, new Variant("result-42")); + + harness.AddChild(harness.PipelineNodeId, BrowseNames.Results, + harness.ResultsFolderId); + harness.AddBrowse(harness.ResultsFolderId, + [harness.Ref(harness.InferenceResultNodeId, "result-42", + ObjectTypes.DetectionResultType)]); + + SetupResultTypeDefinition(harness, ObjectTypes.DetectionResultType); + } + + private static void SetupDetectionResultChildren( + VisionSessionHarness harness) + { + harness.AddValueChild(harness.InferenceResultNodeId, BrowseNames.ResultId, + new(5000u, 3), "result-42"); + harness.AddValueChild(harness.InferenceResultNodeId, BrowseNames.CreationTime, + new(5001u, 3), new DateTimeUtc(new DateTime(2024, 1, 1, 0, 0, 0, DateTimeKind.Utc))); + harness.AddValueChild(harness.InferenceResultNodeId, BrowseNames.Sensor, + new(5004u, 3), harness.SensorNodeId); + harness.AddValueChild(harness.InferenceResultNodeId, BrowseNames.Pipeline, + new(5005u, 3), new NodeId(3002u, 3)); + harness.AddValueChild(harness.InferenceResultNodeId, BrowseNames.FrameId, + new(5002u, 3), "world"); + harness.AddValueChild(harness.InferenceResultNodeId, BrowseNames.ModelVersionUsed, + new(5003u, 3), "v1.0"); + } + + private static void SetupResultTypeDefinition( + VisionSessionHarness harness, uint typeId) + { + harness.AddBrowse(harness.InferenceResultNodeId, + [new ReferenceDescription + { + NodeId = new ExpandedNodeId( + new NodeId(typeId, harness.VisionNamespaceIndex)), + BrowseName = new QualifiedName("DetectionResultType", + harness.VisionNamespaceIndex), + DisplayName = new LocalizedText("DetectionResultType"), + NodeClass = NodeClass.ObjectType, + TypeDefinition = ExpandedNodeId.Null, + ReferenceTypeId = Opc.Ua.ReferenceTypeIds.HasTypeDefinition, + IsForward = true + }]); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionLearningJobBindingTests.cs b/tests/Opc.Ua.Vision.Tests/VisionLearningJobBindingTests.cs new file mode 100644 index 0000000000..5b96ade2a8 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionLearningJobBindingTests.cs @@ -0,0 +1,149 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.Vision.Client; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Builders; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Tests the optional LearningJob binding on Vision inference pipelines. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionLearningJobBindingTests + { + [Test] + public async Task WithLearningJobCreatesTypedBrowsablePropertyWithValue() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + var learningJobNodeId = new NodeId("learning-job-1", 3); + NodeId learningJobPropertyId = NodeId.Null; + + await fixture.Manager.ConfigureVisionAsync(context => + { + context.Nodes.AddPipeline("Detector", p => p + .WithPipelineId("detector") + .WithLearningJob(learningJobNodeId)); + + NodeState pipeline = FindChild(context.Root.Pipelines!, "Detector"); + learningJobPropertyId = FindChild(pipeline, BrowseNames.LearningJob).NodeId; + }).ConfigureAwait(false); + + NodeState? registered = fixture.Manager.FindPredefinedNode(learningJobPropertyId); + + Assert.Multiple(() => + { + Assert.That(registered, Is.Not.Null, + "the LearningJob property must be reachable by its own NodeId " + + "so a client can resolve and read it."); + Assert.That(registered, Is.InstanceOf()); + }); + + var learningJobProperty = (BaseVariableState)registered!; + Assert.Multiple(() => + { + Assert.That(learningJobProperty.Value, Is.EqualTo(learningJobNodeId)); + Assert.That(learningJobProperty.ReferenceTypeId, + Is.EqualTo(global::Opc.Ua.ReferenceTypeIds.HasProperty), + "LearningJob is a property of the pipeline, not a component or external reference."); + Assert.That(learningJobProperty.TypeDefinitionId, + Is.EqualTo(global::Opc.Ua.VariableTypeIds.PropertyType), + "clients filtering by PropertyType must not silently skip the generated child."); + }); + } + + [Test] + public async Task PipelineBuiltWithoutLearningJobLeavesOptionalChildAbsent() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + InferencePipelineState? pipeline = null; + + await fixture.Manager.ConfigureVisionAsync(context => + { + context.Nodes.AddPipeline("Detector", p => p + .WithPipelineId("detector")); + + pipeline = (InferencePipelineState)FindChild(context.Root.Pipelines!, "Detector"); + }).ConfigureAwait(false); + + Assert.That(pipeline, Is.Not.Null); + Assert.That(TryFindChild(pipeline!, BrowseNames.LearningJob, out NodeState? learningJob), Is.False, + "LearningJob is optional and should be absent unless the host binds one."); + Assert.That(learningJob, Is.Null); + } + + [Test] + public async Task PipelineClientReadsLearningJobId() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + var learningJobNodeId = new NodeId("learning-job-1", 3); + harness.AddValueChild(harness.PipelineNodeId, BrowseNames.LearningJob, + new NodeId(3014u, 3), learningJobNodeId); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionPipelineSnapshot snapshot = await pipeline.ReadAsync().ConfigureAwait(false); + + Assert.That(snapshot.LearningJobId, Is.EqualTo(learningJobNodeId)); + } + + private static NodeState FindChild(NodeState parent, string browseName) + { + Assert.That(TryFindChild(parent, browseName, out NodeState? match), Is.True, + $"'{browseName}' must exist below '{parent.BrowseName.Name}'."); + return match!; + } + + private static bool TryFindChild(NodeState parent, string browseName, out NodeState? match) + { + var children = new List(); + parent.GetChildren(null!, children); + for (int ii = 0; ii < children.Count; ii++) + { + if (children[ii].BrowseName.Name == browseName) + { + match = children[ii]; + return true; + } + } + match = null; + return false; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionMediaClientTests.cs b/tests/Opc.Ua.Vision.Tests/VisionMediaClientTests.cs new file mode 100644 index 0000000000..5a413945f1 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionMediaClientTests.cs @@ -0,0 +1,320 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Tests for — enumerations, clip and + /// stream endpoint calls, ReadLatestClip §6.4 status classification. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionMediaClientTests + { + [Test] + public async Task EnumerateClipEndpointsYieldsFromClipEndpointsFolder() + { + var harness = new VisionSessionHarness(); + harness.AddChild(harness.MediaNodeId, BrowseNames.ClipEndpoints, + harness.ClipEndpointsFolderId); + harness.AddBrowse(harness.ClipEndpointsFolderId, + [harness.Ref(harness.ClipEndpointNodeId, "ClipA", + ObjectTypes.ClipEndpointType)]); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + var entries = new List(); + await foreach (VisionNodeEntry entry in media.EnumerateClipEndpointsAsync()) + { + entries.Add(entry); + } + + Assert.That(entries.Count, Is.EqualTo(1)); + Assert.That(entries[0].NodeId, Is.EqualTo(harness.ClipEndpointNodeId)); + } + + [Test] + public async Task EnumerateStreamEndpointsYieldsFromStreamEndpointsFolder() + { + var harness = new VisionSessionHarness(); + harness.AddChild(harness.MediaNodeId, BrowseNames.StreamEndpoints, + harness.StreamEndpointsFolderId); + harness.AddBrowse(harness.StreamEndpointsFolderId, + [harness.Ref(harness.StreamEndpointNodeId, "StreamA", + ObjectTypes.StreamEndpointType)]); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + var entries = new List(); + await foreach (VisionNodeEntry entry in media.EnumerateStreamEndpointsAsync()) + { + entries.Add(entry); + } + + Assert.That(entries.Count, Is.EqualTo(1)); + Assert.That(entries[0].NodeId, Is.EqualTo(harness.StreamEndpointNodeId)); + } + + [Test] + public async Task EnumerateClipEndpointsYieldsNothingWhenFolderAbsent() + { + var harness = new VisionSessionHarness(); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + var entries = new List(); + await foreach (VisionNodeEntry entry in media.EnumerateClipEndpointsAsync()) + { + entries.Add(entry); + } + + Assert.That(entries.Count, Is.EqualTo(0)); + } + + [Test] + public async Task GetClipReturnsPopulatedResultOnGoodCall() + { + var harness = new VisionSessionHarness(); + var descriptor = new VisionImageReferenceDataType + { + Uri = "opc.ua://server/clips/latest" + }; + harness.ConfigureCall(StatusCodes.Good, + Variant.FromStructure(descriptor), + new Variant(harness.ClipEndpointNodeId), + new Variant(ByteString.Empty)); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + VisionClipResult result = await media.GetClipAsync( + harness.ClipEndpointNodeId, + "res-1", + default, + VisionClipFormatEnum.Jpeg, + requestInline: false).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.EndpointNodeId, Is.EqualTo(harness.ClipEndpointNodeId)); + Assert.That(result.Image, Is.Not.Null); + Assert.That(result.Image.Uri, Is.EqualTo("opc.ua://server/clips/latest")); + }); + } + + [Test] + public void ConfigureStreamEndpointRejectsNullEndpointNodeId() + { + var harness = new VisionSessionHarness(); + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + + var ex = Assert.ThrowsAsync(async () => + await media.ConfigureStreamEndpointAsync( + NodeId.Null, + VisionVideoCodecEnum.H264, + 1920, 1080, 30.0, 8_000_000).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("streamEndpointNodeId")); + } + + [Test] + public void GetStreamEndpointRejectsNullProfileName() + { + var harness = new VisionSessionHarness(); + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + + var ex = Assert.ThrowsAsync(async () => + await media.GetStreamEndpointAsync( + harness.StreamEndpointNodeId, + null!, + VisionStreamProtocolEnum.Rtsp).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("profileName")); + } + + [Test] + public async Task ConfigureStreamEndpointDoesNotThrowOnGoodCall() + { + var harness = new VisionSessionHarness(); + harness.ConfigureCall(StatusCodes.Good); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + + Assert.DoesNotThrowAsync(async () => + await media.ConfigureStreamEndpointAsync( + harness.StreamEndpointNodeId, + VisionVideoCodecEnum.H264, + 1920, 1080, 30.0, 8_000_000).ConfigureAwait(false)); + await Task.CompletedTask.ConfigureAwait(false); + } + + [Test] + public async Task ReleaseStreamEndpointDoesNotThrowOnGoodCall() + { + var harness = new VisionSessionHarness(); + harness.ConfigureCall(StatusCodes.Good); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + + Assert.DoesNotThrowAsync(async () => + await media.ReleaseStreamEndpointAsync( + ByteString.Empty).ConfigureAwait(false)); + await Task.CompletedTask.ConfigureAwait(false); + } + + [Test] + public async Task SelectEndpointDoesNotThrowOnGoodCall() + { + var harness = new VisionSessionHarness(); + harness.ConfigureCall(StatusCodes.Good); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + + Assert.DoesNotThrowAsync(async () => + await media.SelectEndpointAsync( + harness.StreamEndpointNodeId, + harness.ClipEndpointNodeId).ConfigureAwait(false)); + await Task.CompletedTask.ConfigureAwait(false); + } + + [Test] + public void ReadLatestClipRejectsNullEndpointNodeId() + { + var harness = new VisionSessionHarness(); + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + + var ex = Assert.ThrowsAsync(async () => + await media.ReadLatestClipAsync(NodeId.Null).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("clipEndpointNodeId")); + } + + [Test] + public async Task ReadLatestClipReturnsAvailableWhenStatusGood() + { + var harness = new VisionSessionHarness(); + harness.AddValueChild(harness.ClipEndpointNodeId, BrowseNames.LatestClip, + new(2400u, 3), new Variant(new ByteString(new byte[] { 1, 2, 3 }))); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + VisionInlineClipReading reading = await media.ReadLatestClipAsync( + harness.ClipEndpointNodeId).ConfigureAwait(false); + + Assert.That(reading.State, Is.EqualTo(VisionInlineClipState.Available)); + } + + [Test] + public async Task ReadLatestClipReturnsNotYetAvailableForBadNoDataAvailable() + { + var harness = new VisionSessionHarness(); + harness.AddChild(harness.ClipEndpointNodeId, BrowseNames.LatestClip, + new NodeId(2400u, 3)); + harness.AddValueStatus(new NodeId(2400u, 3), + StatusCodes.BadNoDataAvailable); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + VisionInlineClipReading reading = await media.ReadLatestClipAsync( + harness.ClipEndpointNodeId).ConfigureAwait(false); + + Assert.That(reading.State, Is.EqualTo(VisionInlineClipState.NotYetAvailable)); + } + + [Test] + public async Task ReadLatestClipReturnsInlineDisabledForBadNotSupported() + { + var harness = new VisionSessionHarness(); + harness.AddChild(harness.ClipEndpointNodeId, BrowseNames.LatestClip, + new NodeId(2400u, 3)); + harness.AddValueStatus(new NodeId(2400u, 3), + StatusCodes.BadNotSupported); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + VisionInlineClipReading reading = await media.ReadLatestClipAsync( + harness.ClipEndpointNodeId).ConfigureAwait(false); + + Assert.That(reading.State, Is.EqualTo(VisionInlineClipState.InlineDisabled)); + } + + [Test] + public async Task ReadLatestClipReturnsOverflowForBadEncodingLimitsExceeded() + { + var harness = new VisionSessionHarness(); + harness.AddChild(harness.ClipEndpointNodeId, BrowseNames.LatestClip, + new NodeId(2400u, 3)); + harness.AddValueStatus(new NodeId(2400u, 3), + StatusCodes.BadEncodingLimitsExceeded); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + VisionInlineClipReading reading = await media.ReadLatestClipAsync( + harness.ClipEndpointNodeId).ConfigureAwait(false); + + Assert.That(reading.State, Is.EqualTo(VisionInlineClipState.Overflow)); + } + + [Test] + public async Task ReadLatestClipReturnsFaultedForOtherBadStatuses() + { + var harness = new VisionSessionHarness(); + harness.AddChild(harness.ClipEndpointNodeId, BrowseNames.LatestClip, + new NodeId(2400u, 3)); + harness.AddValueStatus(new NodeId(2400u, 3), + StatusCodes.BadDeviceFailure); + + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + VisionInlineClipReading reading = await media.ReadLatestClipAsync( + harness.ClipEndpointNodeId).ConfigureAwait(false); + + Assert.That(reading.State, Is.EqualTo(VisionInlineClipState.Faulted)); + } + + [Test] + public void ReadLatestClipMetadataRejectsNullEndpointNodeId() + { + var harness = new VisionSessionHarness(); + VisionMediaClient media = harness.Client.Media(harness.MediaNodeId); + + var ex = Assert.ThrowsAsync(async () => + await media.ReadLatestClipMetadataAsync(NodeId.Null) + .ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("clipEndpointNodeId")); + } + + [Test] + public void ConstructorRejectsNullMediaNodeId() + { + var harness = new VisionSessionHarness(); + + Assert.Throws(() => + harness.Client.Media(NodeId.Null)); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionMediaGatingTests.cs b/tests/Opc.Ua.Vision.Tests/VisionMediaGatingTests.cs new file mode 100644 index 0000000000..07d4c809cb --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionMediaGatingTests.cs @@ -0,0 +1,412 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Moq; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Pins the §6.4 inline-clip gating rule end-to-end through the + /// server's . The dispatcher is + /// the only place these rules are enforced: LatestClip must + /// report when + /// InlineDeliveryEnabled is false, and inline bytes + /// that exceed MaxInlineClipSize must be nulled with + /// . Both rules + /// must fire before the provider is consulted for the gating case + /// and after the provider returns for the overflow case. + /// + [TestFixture] + public sealed class VisionMediaGatingTests + { + [Test] + public async Task GetClipWithRequestInlineTrueAndInlineDeliveryDisabledReturnsBadNotSupportedBeforeProviderIsCalled() + { + var mediaProvider = new Mock(MockBehavior.Strict); + var harness = new MediaHarness( + sensorId: 101, endpointId: 501, inlineDeliveryEnabled: false, maxInlineClipSize: 4096, + mediaProvider.Object); + + GetClipMethodStateResult result = await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "any", + requestInline: true).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadNotSupported)); + mediaProvider.Verify( + p => p.GetClipAsync(It.IsAny(), It.IsAny()), + Times.Never, + "Inline gating must short-circuit before consulting the media provider."); + }); + } + + [Test] + public async Task GetClipWithRequestInlineFalseIsAllowedEvenWhenInlineDeliveryDisabled() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetClipAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new VisionClipResult( + ServiceResult.Good, + new VisionImageReferenceDataType { Uri = "urn:test:clip" }, + default, + default)); + var harness = new MediaHarness( + sensorId: 102, endpointId: 502, inlineDeliveryEnabled: false, maxInlineClipSize: 4096, + mediaProvider.Object); + + GetClipMethodStateResult result = await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "any", + requestInline: false).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(result.InlineImage.IsNull, Is.True, + "A caller that did not request inline delivery must not receive inline bytes on the way out."); + }); + } + + [Test] + public async Task GetClipWithInlineEnabledAndPayloadWithinLimitReturnsInlineBytes() + { + byte[] payload = new byte[512]; + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetClipAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new VisionClipResult( + ServiceResult.Good, + new VisionImageReferenceDataType { Uri = "urn:test:clip" }, + default, + ByteString.From(payload))); + var harness = new MediaHarness( + sensorId: 103, endpointId: 503, inlineDeliveryEnabled: true, maxInlineClipSize: 4096, + mediaProvider.Object); + + GetClipMethodStateResult result = await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "any", + requestInline: true).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(result.InlineImage.IsNull, Is.False); + Assert.That(result.InlineImage.Length, Is.EqualTo(512)); + }); + } + + [Test] + public async Task GetClipWithInlineEnabledButPayloadExceedingLimitReturnsBadEncodingLimitsAndNullsInlineImage() + { + byte[] payload = new byte[8192]; + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetClipAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new VisionClipResult( + ServiceResult.Good, + new VisionImageReferenceDataType { Uri = "urn:test:big-clip" }, + default, + ByteString.From(payload))); + var harness = new MediaHarness( + sensorId: 104, endpointId: 504, inlineDeliveryEnabled: true, maxInlineClipSize: 4096, + mediaProvider.Object); + + GetClipMethodStateResult result = await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "any", + requestInline: true).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo(StatusCodes.BadEncodingLimitsExceeded)); + Assert.That(result.InlineImage.IsNull, Is.True, + "Overflow must null the inline image so a naive client does not read partial bytes."); + Assert.That(result.Image, Is.Not.Null, + "The out-of-band image reference must survive the overflow so callers can still fetch the clip through the URI channel."); + }); + } + + [Test] + public async Task GetClipWithNoMediaProviderReturnsBadNotSupported() + { + var harness = new MediaHarness( + sensorId: 105, endpointId: 505, inlineDeliveryEnabled: true, maxInlineClipSize: 4096, + mediaProvider: null); + + GetClipMethodStateResult result = await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "any", + requestInline: true).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo(StatusCodes.BadNotSupported)); + } + + [Test] + public async Task GetClipWhenProviderThrowsUnexpectedExceptionReturnsBadInternalError() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetClipAsync(It.IsAny(), It.IsAny())) + .ThrowsAsync(new InvalidOperationException("boom")); + var harness = new MediaHarness( + sensorId: 106, endpointId: 506, inlineDeliveryEnabled: true, maxInlineClipSize: 4096, + mediaProvider.Object); + + GetClipMethodStateResult result = await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "any", + requestInline: true).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo(StatusCodes.BadInternalError)); + } + + [Test] + public void GetClipWhenProviderThrowsOperationCanceledPropagates() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetClipAsync(It.IsAny(), It.IsAny())) + .ThrowsAsync(new OperationCanceledException()); + var harness = new MediaHarness( + sensorId: 107, endpointId: 507, inlineDeliveryEnabled: true, maxInlineClipSize: 4096, + mediaProvider.Object); + + Assert.That( + async () => await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "any", + requestInline: true).ConfigureAwait(false), + Throws.InstanceOf()); + } + + [Test] + public async Task GetClipPublishesTheEncodedFrameOnLatestClipAndItsDescriptorOnLatestClipMetadata() + { + byte[] payload = [1, 2, 3, 4, 5, 6, 7, 8]; + var descriptor = new VisionImageReferenceDataType + { + Uri = "opcua-inline://cell/frames/42", + Width = 612u, + Height = 512u + }; + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetClipAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new VisionClipResult( + ServiceResult.Good, descriptor, default, ByteString.From(payload))); + var harness = new MediaHarness( + sensorId: 108, endpointId: 508, inlineDeliveryEnabled: true, maxInlineClipSize: 4096, + mediaProvider.Object); + + Assert.That(harness.Clip.LatestClip!.StatusCode, Is.EqualTo((StatusCode)StatusCodes.Good), + "Precondition: the harness starts with an unpublished LatestClip."); + + await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "42", + requestInline: true).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(harness.Clip.LatestClip!.Value.IsNull, Is.False, + "A clip the Server has just encoded is by definition the latest one; leaving LatestClip " + + "unwritten makes a consumer that reads the published frame first wait forever."); + Assert.That(harness.Clip.LatestClip!.Value.Length, Is.EqualTo(payload.Length)); + Assert.That(StatusCode.IsGood(harness.Clip.LatestClip!.StatusCode), Is.True); + Assert.That(harness.Clip.LatestClipMetadata!.Value, Is.Not.Null); + Assert.That(harness.Clip.LatestClipMetadata!.Value.Uri, Is.EqualTo("opcua-inline://cell/frames/42"), + "The descriptor beside the published frame is how a consumer learns which image the " + + "detections are expressed in."); + Assert.That(harness.Clip.LatestClipMetadata!.Value.Width, Is.EqualTo(612u)); + Assert.That(StatusCode.IsGood(harness.Clip.LatestClipMetadata!.StatusCode), Is.True); + }); + } + + [Test] + public async Task GetClipDoesNotPublishLatestClipWhenThePayloadOverflowsTheInlineLimit() + { + byte[] payload = new byte[8192]; + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetClipAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new VisionClipResult( + ServiceResult.Good, + new VisionImageReferenceDataType { Uri = "urn:test:big-clip" }, + default, + ByteString.From(payload))); + var harness = new MediaHarness( + sensorId: 109, endpointId: 509, inlineDeliveryEnabled: true, maxInlineClipSize: 4096, + mediaProvider.Object); + + await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "any", + requestInline: true).ConfigureAwait(false); + + Assert.That(harness.Clip.LatestClip!.Value.IsNull, Is.True, + "A clip the Server refused to deliver must not be published as the latest one."); + } + + [Test] + public async Task GetClipDoesNotPublishLatestClipWhenInlineDeliveryIsDisabled() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetClipAsync(It.IsAny(), It.IsAny())) + .ReturnsAsync(new VisionClipResult( + ServiceResult.Good, + new VisionImageReferenceDataType { Uri = "urn:test:clip" }, + default, + ByteString.From([1, 2, 3, 4]))); + var harness = new MediaHarness( + sensorId: 110, endpointId: 510, inlineDeliveryEnabled: false, maxInlineClipSize: 4096, + mediaProvider.Object); + + await harness.InvokeGetClip( + endpoint: harness.EndpointNodeId, + resultId: "any", + requestInline: false).ConfigureAwait(false); + + Assert.That(harness.Clip.LatestClip!.Value.IsNull, Is.True, + "LatestClip is the inline channel, so a Server with inline delivery off must leave it alone."); + } + + [Test] + public void ClipEndpointExposesLatestClipMetadataAlongsideLatestClipInlineDeliveryEnabledAndMaxInlineSize() + { + var clip = new ClipEndpointState(null) + { + InlineDeliveryEnabled = PropertyState.With(null!, false), + MaxInlineClipSize = PropertyState.With(null!, 1024u), + LatestClip = BaseDataVariableState.With(null!), + LatestClipMetadata = BaseDataVariableState.With>(null!) + }; + + Assert.Multiple(() => + { + Assert.That(clip.InlineDeliveryEnabled, Is.Not.Null); + Assert.That(clip.MaxInlineClipSize, Is.Not.Null); + Assert.That(clip.LatestClip, Is.Not.Null, + "The inline byte channel must remain present on the type surface even when it is administratively disabled."); + Assert.That(clip.LatestClipMetadata, Is.Not.Null, + "The metadata channel is the fallback that clients read when inline delivery is off."); + }); + } + + private sealed class MediaHarness + { + public MediaHarness( + uint sensorId, + uint endpointId, + bool inlineDeliveryEnabled, + uint maxInlineClipSize, + IVisionMediaProvider? mediaProvider) + { + SensorNodeId = new NodeId(sensorId, 4); + EndpointNodeId = new NodeId(endpointId, 4); + var sensor = new VisionSensorState(null); + var media = new VisionMediaManagementState(null) + { + GetClip = new GetClipMethodState(null) + }; + var clip = new ClipEndpointState(null) + { + NodeId = EndpointNodeId, + InlineDeliveryEnabled = PropertyState.With(null!, inlineDeliveryEnabled), + MaxInlineClipSize = PropertyState.With(null!, maxInlineClipSize), + LatestClip = BaseDataVariableState.With(null!), + LatestClipMetadata = BaseDataVariableState + .With>(null!) + }; + + var registration = new SensorRegistration( + "cam", + SensorNodeId, + sensor, + VisionSensorModalityEnum.Area2D, + VisionRealityKindEnum.Physical, + new HashSet(StringComparer.Ordinal), + mediaProvider); + registration.ClipEndpoints.Add(clip); + sensor.Media = media; + Clip = clip; + + m_registry = new VisionRegistry(); + m_registry.AddSensor(registration); + var dispatcher = new VisionMethodDispatcher(m_registry, NullLogger.Instance); + dispatcher.AttachMediaMethods(SensorNodeId, media); + m_getClip = media.GetClip.OnCallAsync; + Assert.That(m_getClip, Is.Not.Null, + "AttachMediaMethods must wire an OnCallAsync handler onto GetClip."); + Media = media; + } + + public NodeId SensorNodeId { get; } + + public NodeId EndpointNodeId { get; } + + public VisionMediaManagementState Media { get; } + + public ClipEndpointState Clip { get; } + + public async Task InvokeGetClip( + NodeId endpoint, string resultId, bool requestInline) + { + return await m_getClip!( + null!, + Media.GetClip!, + SensorNodeId, + endpoint, + resultId, + DateTimeUtc.Now, + VisionClipFormatEnum.Png, + requestInline, + CancellationToken.None).ConfigureAwait(false); + } + + private readonly VisionRegistry m_registry; + private readonly GetClipMethodStateMethodAsyncCallHandler? m_getClip; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionMediaInlineClassificationTests.cs b/tests/Opc.Ua.Vision.Tests/VisionMediaInlineClassificationTests.cs new file mode 100644 index 0000000000..fdaee034fa --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionMediaInlineClassificationTests.cs @@ -0,0 +1,162 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Reflection; +using NUnit.Framework; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Locks the §6.4 inline-clip status-code classification a media + /// client applies when reading LatestClip. A server that has + /// InlineDeliveryEnabled = false writes + /// , and the client must + /// surface this as + /// rather than a generic + /// — the metadata endpoint remains readable regardless. Any refactor + /// that reshuffles the four members (LatestClip, LatestClipMetadata, + /// InlineDeliveryEnabled, MaxInlineClipSize) into different classes + /// still owes callers this exact status mapping. + /// + [TestFixture] + public sealed class VisionMediaInlineClassificationTests + { + [Test] + public void BadNotSupportedMapsToInlineDisabled() + { + VisionInlineClipState state = ClassifyInlineState(StatusCodes.BadNotSupported); + + Assert.That(state, Is.EqualTo(VisionInlineClipState.InlineDisabled)); + } + + [Test] + public void BadNoDataAvailableMapsToNotYetAvailable() + { + VisionInlineClipState state = ClassifyInlineState(StatusCodes.BadNoDataAvailable); + + Assert.That(state, Is.EqualTo(VisionInlineClipState.NotYetAvailable)); + } + + [Test] + public void BadEncodingLimitsExceededMapsToOverflow() + { + VisionInlineClipState state = ClassifyInlineState(StatusCodes.BadEncodingLimitsExceeded); + + Assert.That(state, Is.EqualTo(VisionInlineClipState.Overflow)); + } + + [Test] + public void OtherBadStatusMapsToFaulted() + { + VisionInlineClipState state = ClassifyInlineState(StatusCodes.BadInternalError); + + Assert.That(state, Is.EqualTo(VisionInlineClipState.Faulted)); + } + + [Test] + public void UncertainStatusMapsToFaulted() + { + VisionInlineClipState state = ClassifyInlineState(StatusCodes.UncertainNoCommunicationLastUsableValue); + + Assert.That(state, Is.EqualTo(VisionInlineClipState.Faulted)); + } + + [Test] + public void ArbitraryBadStatusMapsToFaultedNotToOneOfTheDedicatedStates() + { + VisionInlineClipState state = ClassifyInlineState(StatusCodes.BadTimeout); + + Assert.Multiple(() => + { + Assert.That(state, Is.EqualTo(VisionInlineClipState.Faulted)); + Assert.That(state, Is.Not.EqualTo(VisionInlineClipState.InlineDisabled)); + Assert.That(state, Is.Not.EqualTo(VisionInlineClipState.NotYetAvailable)); + Assert.That(state, Is.Not.EqualTo(VisionInlineClipState.Overflow)); + }); + } + + [Test] + public void InlineClipReadingCarriesStatusCodeAndByteStringWithoutInterferingWithMetadata() + { + var bytes = ByteString.From(new byte[] { 1, 2, 3 }); + var meta = new VisionImageReferenceDataType + { + Uri = "urn:test:image" + }; + StatusCode status = StatusCodes.Good; + + var reading = new VisionInlineClipReading(bytes, meta, status, VisionInlineClipState.Available); + + Assert.Multiple(() => + { + Assert.That(reading.Bytes.Length, Is.EqualTo(3)); + Assert.That(reading.State, Is.EqualTo(VisionInlineClipState.Available)); + Assert.That(reading.StatusCode, Is.EqualTo(status)); + Assert.That(reading.Metadata, Is.Not.Null); + Assert.That(reading.Metadata!.Uri, Is.EqualTo("urn:test:image")); + }); + } + + [Test] + public void InlineClipReadingKeepsMetadataAvailableEvenWhenInlineIsDisabled() + { + var meta = new VisionImageReferenceDataType + { + Uri = "urn:test:meta-only" + }; + var reading = new VisionInlineClipReading( + ByteString.Empty, + meta, + StatusCodes.BadNotSupported, + VisionInlineClipState.InlineDisabled); + + Assert.Multiple(() => + { + Assert.That(reading.State, Is.EqualTo(VisionInlineClipState.InlineDisabled)); + Assert.That(reading.Metadata, Is.Not.Null, + "The metadata channel must stay readable even when the inline byte channel reports Bad_NotSupported."); + Assert.That(reading.Metadata!.Uri, Is.EqualTo("urn:test:meta-only")); + }); + } + + private static VisionInlineClipState ClassifyInlineState(StatusCode statusCode) + { + MethodInfo? method = typeof(VisionMediaClient).GetMethod( + "ClassifyInlineState", + BindingFlags.NonPublic | BindingFlags.Static, + binder: null, + types: new[] { typeof(StatusCode) }, + modifiers: null); + Assert.That(method, Is.Not.Null, + "VisionMediaClient.ClassifyInlineState must exist and take a StatusCode. If this reflection lookup fails the mapping cannot be enforced."); + return (VisionInlineClipState)method!.Invoke(null, new object[] { statusCode })!; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionMethodDispatcherPipelineHandlerTests.cs b/tests/Opc.Ua.Vision.Tests/VisionMethodDispatcherPipelineHandlerTests.cs new file mode 100644 index 0000000000..c58133934e --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionMethodDispatcherPipelineHandlerTests.cs @@ -0,0 +1,544 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Moq; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Pins the inference pipeline method dispatch path. Every rejection + /// code the dispatcher returns is a public contract: + /// + /// when + /// the pipeline NodeId is not registered (the method was invoked on a + /// pipeline the registry never saw). + /// when + /// the pipeline is registered but no + /// is bound (missing configuration, not a client fault). + /// The provider's own when + /// the provider runs — good or bad, verbatim. + /// when + /// the provider throws a non-cancellation exception. + /// Propagates + /// unchanged so cooperative + /// cancellation of the caller's context is honoured end-to-end. + /// + /// + [TestFixture] + public sealed class VisionMethodDispatcherPipelineHandlerTests + { + [Test] + public async Task RunInferenceReturnsBadNodeIdUnknownWhenPipelineIsNotRegistered() + { + var registeredPipelineId = new NodeId(701, 4); + var orphanPipelineId = new NodeId(702, 4); + var harness = new PipelineHarness( + pipelineNodeId: registeredPipelineId, + inferenceProvider: null, + attachOrphanNodeId: orphanPipelineId); + + RunInferenceMethodStateResult result = await harness.InvokeRunInference( + new DateTimeUtc(new DateTime(2024, 6, 15, 10, 0, 0, DateTimeKind.Utc))).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadNodeIdUnknown), + "RunInference must refuse a call whose pipeline NodeId is not in the registry — " + + "the delegate was wired for a NodeId no PipelineRegistration ever claimed."); + } + + [Test] + public async Task RunInferenceReturnsBadNotSupportedWhenInferenceProviderIsNull() + { + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(703, 4), + inferenceProvider: null); + + RunInferenceMethodStateResult result = await harness.InvokeRunInference( + new DateTimeUtc(new DateTime(2024, 6, 15, 10, 0, 0, DateTimeKind.Utc))).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadNotSupported), + "Without an inference provider the dispatcher must refuse the call with BadNotSupported — " + + "this is a configuration gap, not a client fault."); + } + + [Test] + public async Task RunInferenceForwardsProviderResultServiceCodeAndResultIdOnSuccess() + { + var provider = new Mock(MockBehavior.Strict); + provider.Setup(p => p.RunInferenceAsync(It.IsAny(), It.IsAny())) + .Returns(new ValueTask( + new VisionInferenceRunResult(ServiceResult.Good, "run-42"))); + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(704, 4), + inferenceProvider: provider.Object); + + RunInferenceMethodStateResult result = await harness.InvokeRunInference( + new DateTimeUtc(new DateTime(2024, 6, 15, 10, 0, 0, DateTimeKind.Utc))).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True, + "A Good ServiceResult from the provider must be forwarded to the caller unchanged."); + Assert.That(result.ResultId, Is.EqualTo("run-42"), + "The provider-supplied ResultId is the single output the caller can use to look up " + + "the produced inference in the Results folder; it must be forwarded verbatim."); + }); + } + + [Test] + public async Task RunInferenceForwardsSensorAndDeploymentNodeIdsFromPipelineToProvider() + { + var sensorNodeId = new NodeId(801, 4); + var deploymentNodeId = new NodeId(802, 4); + var pipelineNodeId = new NodeId(705, 4); + VisionInferenceRunRequest captured = default; + bool wasCalled = false; + var provider = new Mock(MockBehavior.Strict); + provider.Setup(p => p.RunInferenceAsync(It.IsAny(), It.IsAny())) + .Returns((req, _) => + { + captured = req; + wasCalled = true; + return new ValueTask( + new VisionInferenceRunResult(ServiceResult.Good, "id")); + }); + var harness = new PipelineHarness( + pipelineNodeId: pipelineNodeId, + inferenceProvider: provider.Object, + sensorNodeId: sensorNodeId, + deploymentNodeId: deploymentNodeId); + var timestamp = new DateTimeUtc(new DateTime(2024, 6, 15, 10, 0, 0, DateTimeKind.Utc)); + + await harness.InvokeRunInference(timestamp).ConfigureAwait(false); + + Assert.That(wasCalled, Is.True, + "The provider must have been invoked so the dispatcher-produced request is observable."); + Assert.Multiple(() => + { + Assert.That(captured.Pipeline, Is.EqualTo(pipelineNodeId), + "The pipeline NodeId the delegate was wired for must be forwarded to the provider."); + Assert.That(captured.Sensor, Is.EqualTo(sensorNodeId), + "The pipeline's Sensor property value must be read and forwarded so the provider " + + "knows which sensor to render from."); + Assert.That(captured.Deployment, Is.EqualTo(deploymentNodeId), + "The pipeline's Deployment property value must be forwarded so the provider " + + "knows which deployment to run."); + Assert.That(captured.Timestamp, Is.EqualTo(timestamp), + "The caller-supplied timestamp must be forwarded to the provider unchanged."); + }); + } + + [Test] + public async Task RunInferenceReturnsNodeIdNullSensorAndDeploymentWhenPropertiesAreMissing() + { + var pipelineNodeId = new NodeId(706, 4); + VisionInferenceRunRequest captured = default; + bool wasCalled = false; + var provider = new Mock(MockBehavior.Strict); + provider.Setup(p => p.RunInferenceAsync(It.IsAny(), It.IsAny())) + .Returns((req, _) => + { + captured = req; + wasCalled = true; + return new ValueTask( + new VisionInferenceRunResult(ServiceResult.Good, "id")); + }); + var harness = new PipelineHarness( + pipelineNodeId: pipelineNodeId, + inferenceProvider: provider.Object); + + await harness.InvokeRunInference(default).ConfigureAwait(false); + + Assert.That(wasCalled, Is.True); + Assert.Multiple(() => + { + Assert.That(captured.Sensor.IsNull, Is.True, + "When the pipeline has no Sensor property, ReadPipelineSensor must return NodeId.Null — " + + "propagating a random value would silently associate the run with a sensor the caller never named."); + Assert.That(captured.Deployment.IsNull, Is.True, + "When the pipeline has no Deployment property, ReadPipelineDeployment must return NodeId.Null."); + }); + } + + [Test] + public async Task RunInferenceReturnsBadInternalErrorWhenProviderThrowsNonCancellationException() + { + var provider = new Mock(MockBehavior.Strict); + provider.Setup(p => p.RunInferenceAsync(It.IsAny(), It.IsAny())) + .Throws(new InvalidOperationException("provider blew up")); + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(707, 4), + inferenceProvider: provider.Object); + + RunInferenceMethodStateResult result = await harness.InvokeRunInference(default) + .ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, Is.EqualTo(StatusCodes.BadInternalError), + "A provider exception must not tear the server down — the dispatcher must map it to BadInternalError " + + "so the caller sees a clean failure code instead of a stack trace propagating out of the method call."); + } + + [Test] + public void RunInferencePropagatesOperationCanceledExceptionFromProvider() + { + var provider = new Mock(MockBehavior.Strict); + provider.Setup(p => p.RunInferenceAsync(It.IsAny(), It.IsAny())) + .Throws(new OperationCanceledException("cancelled from provider")); + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(708, 4), + inferenceProvider: provider.Object); + + Assert.That(async () => await harness.InvokeRunInference(default).ConfigureAwait(false), + Throws.InstanceOf(), + "OperationCanceledException from the provider must be rethrown unchanged so the caller's " + + "cooperative cancellation is honoured; the dispatcher must not swallow it into BadInternalError."); + } + + [Test] + public async Task StartContinuousReturnsBadNodeIdUnknownWhenPipelineIsNotRegistered() + { + var registeredPipelineId = new NodeId(710, 4); + var orphanPipelineId = new NodeId(711, 4); + var harness = new PipelineHarness( + pipelineNodeId: registeredPipelineId, + inferenceProvider: null, + attachOrphanNodeId: orphanPipelineId); + + ServiceResult result = await harness.InvokeStartContinuous().ConfigureAwait(false); + + Assert.That(result.StatusCode, Is.EqualTo(StatusCodes.BadNodeIdUnknown), + "StartContinuous must refuse a call whose pipeline NodeId is not in the registry."); + } + + [Test] + public async Task StartContinuousReturnsBadNotSupportedWhenInferenceProviderIsNull() + { + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(712, 4), + inferenceProvider: null); + + ServiceResult result = await harness.InvokeStartContinuous().ConfigureAwait(false); + + Assert.That(result.StatusCode, Is.EqualTo(StatusCodes.BadNotSupported), + "Without an inference provider the dispatcher must refuse the call with BadNotSupported."); + } + + [Test] + public async Task StartContinuousForwardsProviderResultOnSuccess() + { + var provider = new Mock(MockBehavior.Strict); + NodeId capturedNodeId = default; + bool wasCalled = false; + provider.Setup(p => p.StartContinuousAsync(It.IsAny(), It.IsAny())) + .Returns((nodeId, _) => + { + capturedNodeId = nodeId; + wasCalled = true; + return new ValueTask(ServiceResult.Good); + }); + var pipelineNodeId = new NodeId(713, 4); + var harness = new PipelineHarness( + pipelineNodeId: pipelineNodeId, + inferenceProvider: provider.Object); + + ServiceResult result = await harness.InvokeStartContinuous().ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result), Is.True, + "A Good ServiceResult from the provider must be forwarded to the caller unchanged."); + Assert.That(wasCalled, Is.True, + "The provider must have been invoked so the forwarded NodeId is observable."); + Assert.That(capturedNodeId, Is.EqualTo(pipelineNodeId), + "The pipeline NodeId the delegate was wired for must be forwarded to the provider so it " + + "knows which pipeline to start."); + }); + } + + [Test] + public async Task StartContinuousReturnsBadInternalErrorWhenProviderThrowsNonCancellationException() + { + var provider = new Mock(MockBehavior.Strict); + provider.Setup(p => p.StartContinuousAsync(It.IsAny(), It.IsAny())) + .Throws(new InvalidOperationException("provider blew up")); + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(714, 4), + inferenceProvider: provider.Object); + + ServiceResult result = await harness.InvokeStartContinuous().ConfigureAwait(false); + + Assert.That(result.StatusCode, Is.EqualTo(StatusCodes.BadInternalError), + "A provider exception must be mapped to BadInternalError so the method call returns cleanly."); + } + + [Test] + public void StartContinuousPropagatesOperationCanceledExceptionFromProvider() + { + var provider = new Mock(MockBehavior.Strict); + provider.Setup(p => p.StartContinuousAsync(It.IsAny(), It.IsAny())) + .Throws(new OperationCanceledException("cancelled from provider")); + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(715, 4), + inferenceProvider: provider.Object); + + Assert.That(async () => await harness.InvokeStartContinuous().ConfigureAwait(false), + Throws.InstanceOf(), + "OperationCanceledException from the provider must be rethrown unchanged."); + } + + [Test] + public async Task StopReturnsBadNodeIdUnknownWhenPipelineIsNotRegistered() + { + var registeredPipelineId = new NodeId(720, 4); + var orphanPipelineId = new NodeId(721, 4); + var harness = new PipelineHarness( + pipelineNodeId: registeredPipelineId, + inferenceProvider: null, + attachOrphanNodeId: orphanPipelineId); + + ServiceResult result = await harness.InvokeStop().ConfigureAwait(false); + + Assert.That(result.StatusCode, Is.EqualTo(StatusCodes.BadNodeIdUnknown), + "Stop must refuse a call whose pipeline NodeId is not in the registry."); + } + + [Test] + public async Task StopReturnsBadNotSupportedWhenInferenceProviderIsNull() + { + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(722, 4), + inferenceProvider: null); + + ServiceResult result = await harness.InvokeStop().ConfigureAwait(false); + + Assert.That(result.StatusCode, Is.EqualTo(StatusCodes.BadNotSupported), + "Without an inference provider the dispatcher must refuse the call with BadNotSupported."); + } + + [Test] + public async Task StopForwardsProviderResultOnSuccess() + { + var provider = new Mock(MockBehavior.Strict); + NodeId capturedNodeId = default; + bool wasCalled = false; + provider.Setup(p => p.StopAsync(It.IsAny(), It.IsAny())) + .Returns((nodeId, _) => + { + capturedNodeId = nodeId; + wasCalled = true; + return new ValueTask(ServiceResult.Good); + }); + var pipelineNodeId = new NodeId(723, 4); + var harness = new PipelineHarness( + pipelineNodeId: pipelineNodeId, + inferenceProvider: provider.Object); + + ServiceResult result = await harness.InvokeStop().ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result), Is.True, + "A Good ServiceResult from the provider must be forwarded to the caller unchanged."); + Assert.That(wasCalled, Is.True, + "The provider must have been invoked so the forwarded NodeId is observable."); + Assert.That(capturedNodeId, Is.EqualTo(pipelineNodeId), + "The pipeline NodeId the delegate was wired for must be forwarded to the provider so it " + + "knows which pipeline to stop."); + }); + } + + [Test] + public async Task StopReturnsBadInternalErrorWhenProviderThrowsNonCancellationException() + { + var provider = new Mock(MockBehavior.Strict); + provider.Setup(p => p.StopAsync(It.IsAny(), It.IsAny())) + .Throws(new InvalidOperationException("provider blew up")); + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(724, 4), + inferenceProvider: provider.Object); + + ServiceResult result = await harness.InvokeStop().ConfigureAwait(false); + + Assert.That(result.StatusCode, Is.EqualTo(StatusCodes.BadInternalError), + "A provider exception must be mapped to BadInternalError."); + } + + [Test] + public void StopPropagatesOperationCanceledExceptionFromProvider() + { + var provider = new Mock(MockBehavior.Strict); + provider.Setup(p => p.StopAsync(It.IsAny(), It.IsAny())) + .Throws(new OperationCanceledException("cancelled from provider")); + var harness = new PipelineHarness( + pipelineNodeId: new NodeId(725, 4), + inferenceProvider: provider.Object); + + Assert.That(async () => await harness.InvokeStop().ConfigureAwait(false), + Throws.InstanceOf(), + "OperationCanceledException from the provider must be rethrown unchanged."); + } + + [Test] + public void AttachPipelineMethodsIsSafeWhenIndividualMethodsAreMissing() + { + // A pipeline surface built without all three method children must not cause the + // dispatcher to throw at attach time — the InferencePipelineState type's method + // children are all optional per its generated declaration. + var pipeline = new InferencePipelineState(null) + { + RunInference = new RunInferenceMethodState(null) + // StartContinuous and Stop deliberately left null. + }; + var registration = new PipelineRegistration( + "pipe", new NodeId(730, 4), pipeline, new HashSet(StringComparer.Ordinal)); + var registry = new VisionRegistry(); + registry.AddPipeline(registration); + var dispatcher = new VisionMethodDispatcher(registry, NullLogger.Instance); + + Assert.DoesNotThrow(() => dispatcher.AttachPipelineMethods(registration.NodeId, pipeline), + "AttachPipelineMethods must tolerate a partial pipeline surface — a missing " + + "StartContinuous or Stop method must not cause an NRE when only RunInference is wired."); + Assert.That(pipeline.RunInference!.OnCallAsync, Is.Not.Null, + "The RunInference handler must still be wired even when the other two methods are missing."); + } + + private sealed class PipelineHarness + { + /// + /// Builds a harness that registers a pipeline at + /// and attaches the delegate + /// against the same NodeId, so calling the delegate finds the + /// registration and the ordinary path runs. When + /// is non-Null, the + /// delegate is instead attached against that orphan NodeId + /// while the registration remains at + /// . That combination + /// exercises the BadNodeIdUnknown branch — the closure looks up + /// the orphan NodeId in the registry and finds nothing. + /// + public PipelineHarness( + NodeId pipelineNodeId, + IVisionInferenceProvider? inferenceProvider, + NodeId sensorNodeId = default, + NodeId deploymentNodeId = default, + NodeId attachOrphanNodeId = default) + { + PipelineNodeId = pipelineNodeId; + NodeId attachNodeId = attachOrphanNodeId.IsNull ? pipelineNodeId : attachOrphanNodeId; + var pipeline = new InferencePipelineState(null) + { + RunInference = new RunInferenceMethodState(null), + StartContinuous = new MethodState(null), + Stop = new MethodState(null) + }; + if (!sensorNodeId.IsNull) + { + var sensor = PropertyState.With(pipeline); + sensor.Value = sensorNodeId; + pipeline.Sensor = sensor; + } + if (!deploymentNodeId.IsNull) + { + var deployment = PropertyState.With(pipeline); + deployment.Value = deploymentNodeId; + pipeline.Deployment = deployment; + } + var registration = new PipelineRegistration( + "pipe", + pipelineNodeId, + pipeline, + new HashSet(StringComparer.Ordinal)) + { + InferenceProvider = inferenceProvider + }; + m_registry = new VisionRegistry(); + m_registry.AddPipeline(registration); + var dispatcher = new VisionMethodDispatcher(m_registry, NullLogger.Instance); + // Attach against attachNodeId — normally the same as the registered NodeId, + // but tests may supply an unregistered NodeId to exercise BadNodeIdUnknown. + dispatcher.AttachPipelineMethods(attachNodeId, pipeline); + m_runInference = pipeline.RunInference!.OnCallAsync; + m_startContinuous = pipeline.StartContinuous!.OnCallMethod2Async; + m_stop = pipeline.Stop!.OnCallMethod2Async; + + Assert.That(m_runInference, Is.Not.Null); + Assert.That(m_startContinuous, Is.Not.Null); + Assert.That(m_stop, Is.Not.Null); + } + + public NodeId PipelineNodeId { get; } + + public async Task InvokeRunInference(DateTimeUtc timestamp) + { + return await m_runInference!( + null!, + null!, + PipelineNodeId, + timestamp, + CancellationToken.None).ConfigureAwait(false); + } + + public async Task InvokeStartContinuous() + { + var outputs = new List(); + return await m_startContinuous!( + null!, + null!, + PipelineNodeId, + ArrayOf.Empty, + outputs, + CancellationToken.None).ConfigureAwait(false); + } + + public async Task InvokeStop() + { + var outputs = new List(); + return await m_stop!( + null!, + null!, + PipelineNodeId, + ArrayOf.Empty, + outputs, + CancellationToken.None).ConfigureAwait(false); + } + + private readonly VisionRegistry m_registry; + private readonly RunInferenceMethodStateMethodAsyncCallHandler? m_runInference; + private readonly GenericMethodCalledEventHandler2Async? m_startContinuous; + private readonly GenericMethodCalledEventHandler2Async? m_stop; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionMethodDispatcherStreamEndpointTests.cs b/tests/Opc.Ua.Vision.Tests/VisionMethodDispatcherStreamEndpointTests.cs new file mode 100644 index 0000000000..7b1587a080 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionMethodDispatcherStreamEndpointTests.cs @@ -0,0 +1,488 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using Moq; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Closes the coverage gap on the stream-endpoint side of + /// : GetStreamEndpoint, + /// ReleaseStreamEndpoint, ConfigureStreamEndpoint and + /// SelectEndpoint — every branch (missing provider, provider throws + /// OperationCanceledException, provider throws general exception, + /// good result forwarding, and the SelectEndpoint side effect of + /// updating the preferred-endpoint properties) is asserted. + /// + [TestFixture] + public sealed class VisionMethodDispatcherStreamEndpointTests + { + [Test] + public async Task GetStreamEndpointReturnsBadNotSupportedWhenNoMediaProviderIsRegistered() + { + var harness = new StreamHarness(mediaProvider: null); + + GetStreamEndpointMethodStateResult result = await harness.InvokeGetStreamEndpoint( + harness.EndpointNodeId, + "default", + VisionStreamProtocolEnum.Rtsp).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadNotSupported)); + } + + [Test] + public async Task GetStreamEndpointForwardsLeaseFromProviderOnSuccess() + { + var lease = new VisionStreamLease( + ServiceResult.Good, + new VisionStreamSessionDataType + { + SessionToken = new ByteString(new byte[] { 9, 9 }), + Uri = "rtsp://cam.local/main", + Protocol = VisionStreamProtocolEnum.Rtsp, + ExpiresAt = new DateTimeUtc(new DateTime(2025, 5, 5, 0, 0, 0, DateTimeKind.Utc)) + }, + new NodeId(9999u, 4)); + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetStreamAsync( + It.IsAny(), + It.IsAny())) + .ReturnsAsync(lease); + var harness = new StreamHarness(mediaProvider.Object); + + GetStreamEndpointMethodStateResult result = await harness.InvokeGetStreamEndpoint( + harness.EndpointNodeId, + "high", + VisionStreamProtocolEnum.Rtsp).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(result.Session, Is.Not.Null); + Assert.That(result.Session.Uri, Is.EqualTo("rtsp://cam.local/main")); + Assert.That(result.EndpointOut, Is.EqualTo(new NodeId(9999u, 4)), + "the dispatcher must not rewrite the resolved endpoint the provider returned"); + }); + mediaProvider.Verify(p => p.GetStreamAsync( + It.Is(r => + r.Endpoint == harness.EndpointNodeId && + r.ProfileName == "high" && + r.PreferredProtocol == VisionStreamProtocolEnum.Rtsp), + It.IsAny()), Times.Once); + } + + [Test] + public async Task GetStreamEndpointReturnsBadInternalErrorWhenProviderThrowsUnexpected() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetStreamAsync( + It.IsAny(), + It.IsAny())) + .ThrowsAsync(new InvalidOperationException("provider blew up")); + var harness = new StreamHarness(mediaProvider.Object); + + GetStreamEndpointMethodStateResult result = await harness.InvokeGetStreamEndpoint( + harness.EndpointNodeId, + "default", + VisionStreamProtocolEnum.Rtsp).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInternalError)); + } + + [Test] + public void GetStreamEndpointPropagatesOperationCanceled() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.GetStreamAsync( + It.IsAny(), + It.IsAny())) + .ThrowsAsync(new OperationCanceledException()); + var harness = new StreamHarness(mediaProvider.Object); + + Assert.That( + async () => await harness.InvokeGetStreamEndpoint( + harness.EndpointNodeId, + "default", + VisionStreamProtocolEnum.Rtsp).ConfigureAwait(false), + Throws.InstanceOf(), + "OperationCanceled must propagate — a caller who cancelled the call must " + + "not receive a fabricated status code"); + } + + [Test] + public async Task ReleaseStreamEndpointReturnsBadNotSupportedWhenNoMediaProviderIsRegistered() + { + var harness = new StreamHarness(mediaProvider: null); + + ReleaseStreamEndpointMethodStateResult result = await harness.InvokeReleaseStreamEndpoint( + new ByteString(new byte[] { 1, 2, 3 })).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadNotSupported)); + } + + [Test] + public async Task ReleaseStreamEndpointForwardsGoodResultFromProvider() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.ReleaseStreamAsync( + It.IsAny(), + It.IsAny())) + .ReturnsAsync(ServiceResult.Good); + var harness = new StreamHarness(mediaProvider.Object); + + ReleaseStreamEndpointMethodStateResult result = await harness.InvokeReleaseStreamEndpoint( + new ByteString(new byte[] { 1 })).ConfigureAwait(false); + + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + mediaProvider.Verify(p => p.ReleaseStreamAsync( + It.IsAny(), It.IsAny()), Times.Once); + } + + [Test] + public async Task ReleaseStreamEndpointReturnsBadInternalErrorWhenProviderThrowsUnexpected() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.ReleaseStreamAsync( + It.IsAny(), + It.IsAny())) + .ThrowsAsync(new InvalidOperationException("boom")); + var harness = new StreamHarness(mediaProvider.Object); + + ReleaseStreamEndpointMethodStateResult result = await harness.InvokeReleaseStreamEndpoint( + ByteString.Empty).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInternalError)); + } + + [Test] + public void ReleaseStreamEndpointPropagatesOperationCanceled() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.ReleaseStreamAsync( + It.IsAny(), + It.IsAny())) + .ThrowsAsync(new OperationCanceledException()); + var harness = new StreamHarness(mediaProvider.Object); + + Assert.That( + async () => await harness.InvokeReleaseStreamEndpoint( + ByteString.Empty).ConfigureAwait(false), + Throws.InstanceOf()); + } + + [Test] + public async Task ConfigureStreamEndpointReturnsBadNotSupportedWhenNoMediaProviderIsRegistered() + { + var harness = new StreamHarness(mediaProvider: null); + + ConfigureStreamEndpointMethodStateResult result = await harness.InvokeConfigureStreamEndpoint( + harness.EndpointNodeId, + VisionVideoCodecEnum.H264, + 1920, 1080, 30.0, 8_000_000).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadNotSupported)); + } + + [Test] + public async Task ConfigureStreamEndpointForwardsConfigurationRequestToProvider() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.ConfigureStreamAsync( + It.IsAny(), + It.IsAny())) + .ReturnsAsync(ServiceResult.Good); + var harness = new StreamHarness(mediaProvider.Object); + + ConfigureStreamEndpointMethodStateResult result = await harness.InvokeConfigureStreamEndpoint( + harness.EndpointNodeId, + VisionVideoCodecEnum.H264, + 1920, 1080, 30.0, 8_000_000).ConfigureAwait(false); + + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + mediaProvider.Verify(p => p.ConfigureStreamAsync( + It.Is(r => + r.Endpoint == harness.EndpointNodeId && + r.Codec == VisionVideoCodecEnum.H264 && + r.Width == 1920 && + r.Height == 1080 && + r.FrameRate == 30.0 && + r.Bitrate == 8_000_000), + It.IsAny()), Times.Once); + } + + [Test] + public async Task ConfigureStreamEndpointReturnsBadInternalErrorWhenProviderThrowsUnexpected() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.ConfigureStreamAsync( + It.IsAny(), + It.IsAny())) + .ThrowsAsync(new InvalidOperationException("boom")); + var harness = new StreamHarness(mediaProvider.Object); + + ConfigureStreamEndpointMethodStateResult result = await harness.InvokeConfigureStreamEndpoint( + harness.EndpointNodeId, + VisionVideoCodecEnum.H265, + 1280, 720, 25.0, 4_000_000).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInternalError)); + } + + [Test] + public void ConfigureStreamEndpointPropagatesOperationCanceled() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.ConfigureStreamAsync( + It.IsAny(), + It.IsAny())) + .ThrowsAsync(new OperationCanceledException()); + var harness = new StreamHarness(mediaProvider.Object); + + Assert.That( + async () => await harness.InvokeConfigureStreamEndpoint( + harness.EndpointNodeId, + VisionVideoCodecEnum.H264, + 1920, 1080, 30.0, 8_000_000).ConfigureAwait(false), + Throws.InstanceOf()); + } + + [Test] + public async Task SelectEndpointReturnsBadNotSupportedWhenNoMediaProviderIsRegistered() + { + var harness = new StreamHarness(mediaProvider: null); + + SelectEndpointMethodStateResult result = await harness.InvokeSelectEndpoint( + harness.EndpointNodeId, harness.ClipEndpointNodeId).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadNotSupported)); + } + + [Test] + public async Task SelectEndpointUpdatesPreferredEndpointsWhenGoodResult() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.SelectEndpointAsync( + It.IsAny(), + It.IsAny(), + It.IsAny())) + .ReturnsAsync(ServiceResult.Good); + var harness = new StreamHarness(mediaProvider.Object); + + SelectEndpointMethodStateResult result = await harness.InvokeSelectEndpoint( + harness.EndpointNodeId, harness.ClipEndpointNodeId).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(ServiceResult.IsGood(result.ServiceResult), Is.True); + Assert.That(harness.Media.PreferredStreamEndpoint!.Value, + Is.EqualTo(harness.EndpointNodeId), + "a good SelectEndpoint must have written the new preferred stream endpoint"); + Assert.That(harness.Media.PreferredClipEndpoint!.Value, + Is.EqualTo(harness.ClipEndpointNodeId), + "a good SelectEndpoint must have written the new preferred clip endpoint"); + }); + } + + [Test] + public async Task SelectEndpointReturnsBadInternalErrorWhenProviderThrowsUnexpected() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.SelectEndpointAsync( + It.IsAny(), + It.IsAny(), + It.IsAny())) + .ThrowsAsync(new InvalidOperationException("boom")); + var harness = new StreamHarness(mediaProvider.Object); + + SelectEndpointMethodStateResult result = await harness.InvokeSelectEndpoint( + harness.EndpointNodeId, harness.ClipEndpointNodeId).ConfigureAwait(false); + + Assert.That(result.ServiceResult.StatusCode, + Is.EqualTo((StatusCode)StatusCodes.BadInternalError)); + } + + [Test] + public void SelectEndpointPropagatesOperationCanceled() + { + var mediaProvider = new Mock(); + mediaProvider + .Setup(p => p.SelectEndpointAsync( + It.IsAny(), + It.IsAny(), + It.IsAny())) + .ThrowsAsync(new OperationCanceledException()); + var harness = new StreamHarness(mediaProvider.Object); + + Assert.That( + async () => await harness.InvokeSelectEndpoint( + harness.EndpointNodeId, harness.ClipEndpointNodeId).ConfigureAwait(false), + Throws.InstanceOf()); + } + + private sealed class StreamHarness + { + public StreamHarness(IVisionMediaProvider? mediaProvider) + { + SensorNodeId = new NodeId(701u, 4); + EndpointNodeId = new NodeId(801u, 4); + ClipEndpointNodeId = new NodeId(802u, 4); + var sensor = new VisionSensorState(null); + Media = new VisionMediaManagementState(null) + { + GetStreamEndpoint = new GetStreamEndpointMethodState(null), + ReleaseStreamEndpoint = new ReleaseStreamEndpointMethodState(null), + ConfigureStreamEndpoint = new ConfigureStreamEndpointMethodState(null), + SelectEndpoint = new SelectEndpointMethodState(null), + PreferredStreamEndpoint = PropertyState.With( + null!, NodeId.Null), + PreferredClipEndpoint = PropertyState.With( + null!, NodeId.Null) + }; + sensor.Media = Media; + var registration = new SensorRegistration( + "cam", + SensorNodeId, + sensor, + VisionSensorModalityEnum.Area2D, + VisionRealityKindEnum.Physical, + new HashSet(StringComparer.Ordinal), + mediaProvider); + m_registry = new VisionRegistry(); + m_registry.AddSensor(registration); + var dispatcher = new VisionMethodDispatcher(m_registry, NullLogger.Instance); + dispatcher.AttachMediaMethods(SensorNodeId, Media); + m_getStream = Media.GetStreamEndpoint.OnCallAsync; + m_releaseStream = Media.ReleaseStreamEndpoint.OnCallAsync; + m_configureStream = Media.ConfigureStreamEndpoint.OnCallAsync; + m_selectEndpoint = Media.SelectEndpoint.OnCallAsync; + Assert.Multiple(() => + { + Assert.That(m_getStream, Is.Not.Null, + "AttachMediaMethods must wire a GetStreamEndpoint handler"); + Assert.That(m_releaseStream, Is.Not.Null); + Assert.That(m_configureStream, Is.Not.Null); + Assert.That(m_selectEndpoint, Is.Not.Null); + }); + } + + public NodeId SensorNodeId { get; } + + public NodeId EndpointNodeId { get; } + + public NodeId ClipEndpointNodeId { get; } + + public VisionMediaManagementState Media { get; } + + public async Task InvokeGetStreamEndpoint( + NodeId endpoint, string profileName, VisionStreamProtocolEnum protocol) + { + return await m_getStream!( + null!, + Media.GetStreamEndpoint!, + SensorNodeId, + endpoint, + profileName, + protocol, + CancellationToken.None).ConfigureAwait(false); + } + + public async Task InvokeReleaseStreamEndpoint( + ByteString sessionToken) + { + return await m_releaseStream!( + null!, + Media.ReleaseStreamEndpoint!, + SensorNodeId, + sessionToken, + CancellationToken.None).ConfigureAwait(false); + } + + public async Task InvokeConfigureStreamEndpoint( + NodeId endpoint, VisionVideoCodecEnum codec, + uint width, uint height, double frameRate, uint bitrate) + { + return await m_configureStream!( + null!, + Media.ConfigureStreamEndpoint!, + SensorNodeId, + endpoint, + codec, + width, + height, + frameRate, + bitrate, + CancellationToken.None).ConfigureAwait(false); + } + + public async Task InvokeSelectEndpoint( + NodeId streamEndpoint, NodeId clipEndpoint) + { + return await m_selectEndpoint!( + null!, + Media.SelectEndpoint!, + SensorNodeId, + streamEndpoint, + clipEndpoint, + CancellationToken.None).ConfigureAwait(false); + } + + private readonly VisionRegistry m_registry; + private readonly GetStreamEndpointMethodStateMethodAsyncCallHandler? m_getStream; + private readonly ReleaseStreamEndpointMethodStateMethodAsyncCallHandler? m_releaseStream; + private readonly ConfigureStreamEndpointMethodStateMethodAsyncCallHandler? m_configureStream; + private readonly SelectEndpointMethodStateMethodAsyncCallHandler? m_selectEndpoint; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionMethodSurfaceTests.cs b/tests/Opc.Ua.Vision.Tests/VisionMethodSurfaceTests.cs new file mode 100644 index 0000000000..8cbc4f8803 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionMethodSurfaceTests.cs @@ -0,0 +1,290 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using System.Threading.Tasks; +using Moq; +using NUnit.Framework; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Pins the wire contract of the Vision Methods. + /// + /// + /// A Method is only callable if three things hold: the Method node + /// exists on the instance, it carries an InputArguments + /// Property so the stack knows how many arguments to expect, and its + /// MethodDeclarationId names the type's declaration so a client + /// calling with the type-declaration MethodId resolves to it. None of + /// the three held before, and no unit test could see it, because the + /// dispatcher tests invoke the handler delegates directly and never go + /// through MethodState.Call. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionMethodSurfaceTests + { + [Test] + public async Task PipelineWithAnInferenceProviderExposesTheInferenceMethods() + { + InferencePipelineState pipeline = await BuildPipelineAsync( + withInference: true, withFeedback: false).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(pipeline.RunInference, Is.Not.Null, + "A pipeline that can run inference must expose RunInference."); + Assert.That(pipeline.StartContinuous, Is.Not.Null); + Assert.That(pipeline.Stop, Is.Not.Null); + Assert.That(pipeline.Results, Is.Not.Null, + "RunInference publishes into Results, so the folder must exist."); + }); + } + + [Test] + public async Task RunInferenceDeclaresItsArgumentsSoACallIsNotRefused() + { + InferencePipelineState pipeline = await BuildPipelineAsync( + withInference: true, withFeedback: false).ConfigureAwait(false); + RunInferenceMethodState method = pipeline.RunInference!; + + Assert.Multiple(() => + { + Assert.That(method.InputArguments, Is.Not.Null, + "Without InputArguments the stack expects zero arguments and " + + "refuses the call with BadTooManyArguments."); + Assert.That(method.InputArguments!.Value.Count, Is.EqualTo(1)); + Assert.That(method.InputArguments!.Value[0].Name, Is.EqualTo("Timestamp")); + Assert.That(method.OutputArguments, Is.Not.Null); + Assert.That(method.OutputArguments!.Value.Count, Is.EqualTo(1)); + Assert.That(method.OutputArguments!.Value[0].Name, Is.EqualTo("ResultId")); + }); + } + + [Test] + public async Task RunInferenceNamesTheTypeDeclarationSoAClientCallResolves() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + InferencePipelineState pipeline = await BuildPipelineAsync( + fixture, withInference: true, withFeedback: false).ConfigureAwait(false); + + NodeId expected = ExpandedNodeId.ToNodeId( + MethodIds.InferencePipelineType_RunInference, + fixture.Manager.SystemContext.NamespaceUris); + + Assert.That(pipeline.RunInference!.MethodDeclarationId, Is.EqualTo(expected), + "A client calls with the type-declaration MethodId; NodeState.FindMethod " + + "only matches it against MethodDeclarationId."); + } + + [Test] + public async Task PipelineWithAFeedbackSinkExposesTheFourFeedbackMethods() + { + InferencePipelineState pipeline = await BuildPipelineAsync( + withInference: false, withFeedback: true).ConfigureAwait(false); + + Assert.That(pipeline.Feedback, Is.Not.Null, + "A pipeline with a feedback sink must expose the Feedback object."); + VisionFeedbackState feedback = pipeline.Feedback!; + Assert.Multiple(() => + { + Assert.That(feedback.SubmitDetections, Is.Not.Null); + Assert.That(feedback.SubmitInspectionResult, Is.Not.Null); + Assert.That(feedback.SubmitCorrection, Is.Not.Null); + Assert.That(feedback.SubmitImageReference, Is.Not.Null); + }); + } + + [Test] + public async Task SubmitCorrectionDeclaresAllSevenArgumentsInOrder() + { + InferencePipelineState pipeline = await BuildPipelineAsync( + withInference: false, withFeedback: true).ConfigureAwait(false); + SubmitCorrectionMethodState method = pipeline.Feedback!.SubmitCorrection!; + + var names = new List(); + for (int ii = 0; ii < method.InputArguments!.Value.Count; ii++) + { + names.Add(method.InputArguments!.Value[ii].Name ?? string.Empty); + } + + Assert.That(names, Is.EqualTo(new[] + { + "ResultId", + "Purpose", + "CorrectedDetections", + "CorrectedCharacteristics", + "Reason", + "InlineImage", + "RetractAll" + }).AsCollection, "The order is positional on the wire, so it must match the spec."); + } + + [Test] + public async Task SubmitDetectionsDeclaresTheSceneIsEmptyFlagLast() + { + InferencePipelineState pipeline = await BuildPipelineAsync( + withInference: false, withFeedback: true).ConfigureAwait(false); + SubmitDetectionsMethodState method = pipeline.Feedback!.SubmitDetections!; + + var names = new List(); + for (int ii = 0; ii < method.InputArguments!.Value.Count; ii++) + { + names.Add(method.InputArguments!.Value[ii].Name ?? string.Empty); + } + + Assert.That(names, Is.EqualTo(new[] + { + "Purpose", + "Detections", + "FrameReference", + "InlineImage", + "SceneIsEmpty" + }).AsCollection, + "SceneIsEmpty is what makes an empty Detections array a real " + + "observation, so it has to reach the wire in the position the " + + "specification gives it."); + } + + [Test] + public async Task PipelineWithoutProvidersExposesNoMethods() + { + InferencePipelineState pipeline = await BuildPipelineAsync( + withInference: false, withFeedback: false).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(pipeline.RunInference, Is.Null, + "A pipeline with no inference provider must not advertise RunInference."); + Assert.That(pipeline.Feedback, Is.Null, + "A pipeline with no feedback sink must not advertise Feedback."); + }); + } + + [Test] + public async Task EveryChildTheBuilderCreatesCarriesAReferenceType() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + InferencePipelineState pipeline = await BuildPipelineAsync( + fixture, withInference: true, withFeedback: true).ConfigureAwait(false); + + // A child referenced by nothing cannot be browsed from its parent + // and a browse path cannot be translated to it, so the client sees + // an object whose optional children do not exist. + AssertAllChildrenReferenced(pipeline); + } + + [Test] + public async Task SensorWithAMediaProviderExposesTheMediaMethods() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + await fixture.Manager.ConfigureVisionAsync(context => + { + context.Nodes.AddImageSensor("Camera", s => s + .WithSensorId("SN-CAM") + .WithModality(VisionSensorModalityEnum.Area2D) + .UseMediaProvider(new Mock().Object)); + }).ConfigureAwait(false); + + var sensors = new List(); + fixture.Manager.Root.Sensors!.GetChildren(null!, sensors); + var sensor = (ImageSensorState)sensors[0]; + + Assert.That(sensor.Media, Is.Not.Null, + "A sensor with a media provider must expose the Media object."); + VisionMediaManagementState media = sensor.Media!; + Assert.Multiple(() => + { + Assert.That(media.GetStreamEndpoint, Is.Not.Null); + Assert.That(media.ReleaseStreamEndpoint, Is.Not.Null); + Assert.That(media.ConfigureStreamEndpoint, Is.Not.Null); + Assert.That(media.SelectEndpoint, Is.Not.Null); + Assert.That(media.GetClip, Is.Not.Null); + Assert.That(media.GetClip!.InputArguments!.Value.Count, Is.EqualTo(5)); + Assert.That(media.GetClip!.OutputArguments!.Value.Count, Is.EqualTo(3)); + }); + } + + private static void AssertAllChildrenReferenced(NodeState node) + { + var children = new List(); + node.GetChildren(null!, children); + for (int ii = 0; ii < children.Count; ii++) + { + Assert.That(children[ii].ReferenceTypeId.IsNull, Is.False, + $"'{children[ii].BrowseName.Name}' below '{node.BrowseName.Name}' " + + "has no reference type, so nothing can reach it."); + AssertAllChildrenReferenced(children[ii]); + } + } + + private static async Task BuildPipelineAsync( + bool withInference, + bool withFeedback) + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + return await BuildPipelineAsync(fixture, withInference, withFeedback) + .ConfigureAwait(false); + } + + private static async Task BuildPipelineAsync( + VisionServerFixture fixture, + bool withInference, + bool withFeedback) + { + await fixture.Manager.ConfigureVisionAsync(context => + { + context.Nodes.AddPipeline("Detector", p => + { + p.WithPipelineId("detector"); + if (withInference) + { + p.UseInferenceProvider( + new Mock().Object, onServer: true); + } + if (withFeedback) + { + p.UseFeedbackSink(new Mock().Object); + } + }); + }).ConfigureAwait(false); + + var pipelines = new List(); + fixture.Manager.Root.Pipelines!.GetChildren(null!, pipelines); + return (InferencePipelineState)pipelines[0]; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionNodeManagerIntegrationTests.cs b/tests/Opc.Ua.Vision.Tests/VisionNodeManagerIntegrationTests.cs new file mode 100644 index 0000000000..e7ab6366f7 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionNodeManagerIntegrationTests.cs @@ -0,0 +1,187 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Builders; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Integration tests over a real + /// booted inside a . + /// These exercise the address-space bootstrap, the fluent build + /// context, and the small policy surfaces on the manager itself. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionNodeManagerIntegrationTests + { + [Test] + public async Task CreateAddressSpaceExposesVisionRootUnderServer() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + Assert.That(fixture.Manager.Root, Is.Not.Null); + Assert.That(fixture.Manager.Root.BrowseName.Name, Is.EqualTo("Vision")); + Assert.That(fixture.Manager.Root.Sensors, Is.Not.Null, + "The mandatory Sensors folder must exist under Vision."); + } + + [Test] + public async Task ConformanceUnitsIsEmptyByDefault() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + Assert.That(fixture.Manager.ConformanceUnits.Count, Is.EqualTo(0)); + } + + [Test] + public async Task ServerProfilesIsEmptyWhenNoSensorsOrPipelinesAreAdded() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + Assert.That(fixture.Manager.ServerProfiles.Count, Is.EqualTo(0), + "No facets are exposed on an empty Vision address space."); + } + + [Test] + public async Task CreateVisionBuildContextExposesVisionRootAndInstanceIndex() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + IVisionBuildContext context = fixture.Manager.CreateVisionBuildContext(); + + Assert.Multiple(() => + { + Assert.That(context.Root, Is.SameAs(fixture.Manager.Root)); + Assert.That(context.InstanceNamespaceIndex, Is.GreaterThan(0)); + Assert.That(context.VisionNamespaceIndex, Is.GreaterThan(0)); + Assert.That(context.InstanceNamespaceIndex, Is.Not.EqualTo(context.VisionNamespaceIndex)); + Assert.That(context.Nodes, Is.Not.Null); + }); + } + + [Test] + public async Task NewNodeIdReturnsExistingNodeIdWhenNodeAlreadyHasOne() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + var existing = new BaseObjectState(null) + { + NodeId = new NodeId("Existing", fixture.Manager.NamespaceIndex), + SymbolicName = "Existing" + }; + + NodeId result = fixture.Manager.New(fixture.Manager.SystemContext, existing); + + Assert.That(result, Is.EqualTo(existing.NodeId)); + } + + [Test] + public async Task NewNodeIdSynthesisesGuidWhenNoParentAndNoExistingNodeId() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + var orphan = new BaseObjectState(null) + { + NodeId = NodeId.Null, + SymbolicName = "Orphan" + }; + + NodeId result = fixture.Manager.New(fixture.Manager.SystemContext, orphan); + + Assert.That(result.IsNull, Is.False); + Assert.That(result.IdType, Is.EqualTo(IdType.Guid)); + } + + [Test] + public async Task NewNodeIdBuildsChildPathWhenParentIsPresent() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + var parent = new BaseObjectState(null) + { + NodeId = new NodeId("Parent", fixture.Manager.NamespaceIndex), + SymbolicName = "Parent" + }; + var child = new BaseObjectState(parent) + { + NodeId = NodeId.Null, + SymbolicName = "Child" + }; + + NodeId result = fixture.Manager.New(fixture.Manager.SystemContext, child); + + Assert.That(result.IdType, Is.EqualTo(IdType.String)); + Assert.That(result.IdentifierAsString, Does.Contain("Parent")); + Assert.That(result.IdentifierAsString, Does.Contain("Child")); + } + + [Test] + public async Task AddingImageSensorPublishesFacetProfilesOnServer() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + IVisionBuildContext context = fixture.CreateBuildContext(); + + context.Nodes.AddImageSensor("Cam1", sensor => sensor + .WithSensorId("cam-1") + .WithModality(VisionSensorModalityEnum.Area2D) + .WithRealityKind(VisionRealityKindEnum.Physical) + .WithResolution(640u, 480u) + .WithPixelFormat("Mono8")); + + fixture.Manager.PublishServerProfiles(); + + ArrayOf profiles = fixture.Manager.ServerProfiles; + Assert.That(profiles.Count, Is.GreaterThan(0), + "Adding a sensor must derive at least one facet on the server profile array."); + } + + [Test] + public async Task DisposeAsyncCanBeCalledMultipleTimesWithoutThrowing() + { + var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + await fixture.DisposeAsync().ConfigureAwait(false); + Assert.DoesNotThrowAsync(async () => await fixture.DisposeAsync().ConfigureAwait(false)); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionNodeRegistrationTests.cs b/tests/Opc.Ua.Vision.Tests/VisionNodeRegistrationTests.cs new file mode 100644 index 0000000000..b545474d96 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionNodeRegistrationTests.cs @@ -0,0 +1,248 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Builders; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Pins the address-space registration contract. The node manager adds + /// the Vision root to its index before configurators run, so everything + /// the fluent builder grafts on afterwards has to be registered too. + /// Browsing forward from a parent walks NodeState.Children in + /// memory and therefore succeeds either way; only a lookup by the node's + /// own — which is how an ordinary client and the MCP + /// discovery tools navigate — can tell the difference. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionNodeRegistrationTests + { + [Test] + public async Task ConfigureVisionRegistersEveryNodeTheBuilderCreated() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + NodeId sensorId = NodeId.Null; + NodeId frameId = NodeId.Null; + NodeId pipelineId = NodeId.Null; + + await fixture.Manager.ConfigureVisionAsync(context => + { + context.Nodes.AddImageSensor("Camera", s => s + .WithSensorId("SN-CAM") + .WithModality(VisionSensorModalityEnum.Area2D)); + context.Nodes.AddFrame("World", f => f + .WithFrameId("world") + .WithRole(VisionFrameRoleEnum.World)); + context.Nodes.AddPipeline("Detector", p => p + .WithPipelineId("detector")); + + sensorId = FindChild(context.Root.Sensors!, "Camera").NodeId; + frameId = FindChild(context.Root.Frames!, "World").NodeId; + pipelineId = FindChild(context.Root.Pipelines!, "Detector").NodeId; + }).ConfigureAwait(false); + + VisionRootState root = fixture.Manager.Root; + Assert.Multiple(() => + { + Assert.That(fixture.Manager.FindPredefinedNode(root.Sensors!.NodeId), + Is.Not.Null, "the Sensors folder must be reachable by its own NodeId"); + Assert.That(fixture.Manager.FindPredefinedNode(root.Frames!.NodeId), + Is.Not.Null, "the Frames folder must be reachable by its own NodeId"); + Assert.That(fixture.Manager.FindPredefinedNode(root.Pipelines!.NodeId), + Is.Not.Null, "the Pipelines folder must be reachable by its own NodeId"); + Assert.That(fixture.Manager.FindPredefinedNode(sensorId), + Is.Not.Null, "the sensor must be reachable by its own NodeId"); + Assert.That(fixture.Manager.FindPredefinedNode(frameId), + Is.Not.Null, "the frame must be reachable by its own NodeId"); + Assert.That(fixture.Manager.FindPredefinedNode(pipelineId), + Is.Not.Null, "the pipeline must be reachable by its own NodeId"); + }); + } + + [Test] + public async Task ConfigureVisionRegistersNodesNestedBelowASensor() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + NodeId calibrationsId = NodeId.Null; + NodeId handEyeId = NodeId.Null; + + await fixture.Manager.ConfigureVisionAsync(context => + { + context.Nodes.AddImageSensor("Camera", s => s + .WithSensorId("SN-CAM") + .WithModality(VisionSensorModalityEnum.Area2D) + .AddExtrinsicCalibration("HandEye", c => c + .WithCalibrationId("hand-eye") + .WithMount(VisionCalibrationMountEnum.EyeInHand) + .WithFrames("flange", "camera_eih"))); + + NodeState sensor = FindChild(context.Root.Sensors!, "Camera"); + NodeState calibrations = FindChild(sensor, "Calibrations"); + calibrationsId = calibrations.NodeId; + handEyeId = FindChild(calibrations, "HandEye").NodeId; + }).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(fixture.Manager.FindPredefinedNode(calibrationsId), + Is.Not.Null, "the Calibrations folder must be reachable by its own NodeId"); + Assert.That(fixture.Manager.FindPredefinedNode(handEyeId), + Is.Not.Null, "the calibration must be reachable by its own NodeId"); + }); + } + + [Test] + public async Task ConfigureVisionRegistersAPipelinesFeedbackAndResultsChildren() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + NodeId feedbackId = NodeId.Null; + NodeId resultsId = NodeId.Null; + NodeId submitDetectionsId = NodeId.Null; + + await fixture.Manager.ConfigureVisionAsync(context => + { + context.Nodes.AddPipeline("Detector", p => p + .WithPipelineId("detector") + .UseInferenceProvider(new StubInferenceProvider()) + .UseFeedbackSink(new StubFeedbackSink())); + + NodeState pipeline = FindChild(context.Root.Pipelines!, "Detector"); + NodeState feedback = FindChild(pipeline, "Feedback"); + feedbackId = feedback.NodeId; + resultsId = FindChild(pipeline, "Results").NodeId; + submitDetectionsId = FindChild(feedback, "SubmitDetections").NodeId; + }).ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(fixture.Manager.FindPredefinedNode(feedbackId), + Is.Not.Null, + "the Feedback object is created after the pipeline is added to its parent, " + + "so it must still be registered by its own NodeId - a client resolves it " + + "with TranslateBrowsePathsToNodeIds, which yields no target for an " + + "unregistered node even though Browse still lists it."); + Assert.That(fixture.Manager.FindPredefinedNode(resultsId), + Is.Not.Null, "the Results folder must be reachable by its own NodeId"); + Assert.That(fixture.Manager.FindPredefinedNode(submitDetectionsId), + Is.Not.Null, "a Feedback method must be reachable by its own NodeId"); + }); + } + + [Test] + public async Task ConfigureVisionRejectsNullConfigureDelegate() + { + await using var fixture = new VisionServerFixture(); + await fixture.StartAsync().ConfigureAwait(false); + + Assert.That( + async () => await fixture.Manager.ConfigureVisionAsync(null!) + .ConfigureAwait(false), + Throws.ArgumentNullException); + } + + private static NodeState FindChild(NodeState parent, string browseName) + { + NodeState? match = null; + var children = new System.Collections.Generic.List(); + parent.GetChildren(null!, children); + for (int ii = 0; ii < children.Count; ii++) + { + if (children[ii].BrowseName.Name == browseName) + { + match = children[ii]; + break; + } + } + Assert.That(match, Is.Not.Null, + $"'{browseName}' must exist below '{parent.BrowseName.Name}'."); + return match!; + } + + private sealed class StubInferenceProvider : IVisionInferenceProvider + { + public ValueTask RunInferenceAsync( + VisionInferenceRunRequest request, CancellationToken cancellationToken) + { + return new ValueTask( + new VisionInferenceRunResult(ServiceResult.Good, string.Empty)); + } + + public ValueTask StartContinuousAsync( + NodeId pipeline, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask StopAsync( + NodeId pipeline, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + } + + private sealed class StubFeedbackSink : IVisionFeedbackSink + { + public ValueTask SubmitDetectionsAsync( + VisionSubmitDetectionsRequest request, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask SubmitInspectionResultAsync( + VisionSubmitInspectionResultRequest request, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask SubmitCorrectionAsync( + VisionSubmitCorrectionRequest request, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + + public ValueTask SubmitImageReferenceAsync( + VisionSubmitImageReferenceRequest request, CancellationToken cancellationToken) + { + return new ValueTask(ServiceResult.Good); + } + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionOpenUsdCoverageTests.cs b/tests/Opc.Ua.Vision.Tests/VisionOpenUsdCoverageTests.cs new file mode 100644 index 0000000000..9184a80da7 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionOpenUsdCoverageTests.cs @@ -0,0 +1,402 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Microsoft.Extensions.Logging.Abstractions; +using NUnit.Framework; +using Opc.Ua.Vision.OpenUsd; +using Opc.Ua.Vision.OpenUsd.Rendering; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Coverage tests for + /// paths that are not exercised by + /// and for the internal guard clauses. + /// + /// + /// Three families of paths in this provider are host-topology-dependent + /// and therefore cannot be honestly covered by a unit test on this host: + /// + /// The else branch of the constructor at + /// OpenUsdSceneCameraCaptureProvider.cs lines 96-106 (device + /// probe failed → m_device = null) is only reachable on hosts + /// where every backend probe fails; on this Windows host D3D12 always + /// succeeds. The complementary + /// method NoBackend at lines 366-378 is only reached from that + /// branch, so it inherits the same restriction. See + /// + /// for the CI-side invariants. + /// The full success path of CaptureCore + /// (rendering RGBA, PNG-encoding, returning Succeeded) requires + /// a well-formed USD stage and OpenUSD plugin tree, and belongs in + /// integration tests. + /// Every reachable path inside CaptureCore + /// (StageOpenFailed, CameraResolveFailed, RenderFailed, + /// BlankFrame, EncodingFailed) is unreachable from managed + /// test code today: UsdStage.Open tears the test host down inside + /// native code for both an existing malformed .usda file and a + /// syntactically valid minimal .usda stage. This is a second + /// instance of the same "answers instead of refusing" pattern the + /// ValidateRequest comment already calls out for missing files — + /// filed as a defect below rather than covered here. + /// Similarly, DeviceSelector.FormatException + /// is only invoked when a backend probe throws; on this host every probe + /// succeeds, so the helper is unreachable from a unit test. + /// + /// + [TestFixture] + [Category("OpenUsd")] + public sealed class VisionOpenUsdCoverageTests + { + [Test] + public void ConstructorReturnsNullPluginPathWhenConfiguredPluginPathDoesNotExist() + { + string nonExistent = Path.Combine( + TestContext.CurrentContext.TestDirectory, + "not-a-real-plugin-dir-" + Guid.NewGuid().ToString("N")); + var options = new OpenUsdSceneCaptureOptions { PluginPath = nonExistent }; + + using var provider = new OpenUsdSceneCameraCaptureProvider(options, telemetry: null); + + Assert.That(provider.PluginPath, Is.Null, + "A configured PluginPath that does not exist must resolve to null so " + + "the provider falls back to the auto-discovery path."); + } + + [Test] + public void ConstructorResolvesConfiguredPluginPathWhenDirectoryExists() + { + string tempPluginDir = Path.Combine( + TestContext.CurrentContext.TestDirectory, + "temp-plugin-dir-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(tempPluginDir); + try + { + var options = new OpenUsdSceneCaptureOptions { PluginPath = tempPluginDir }; + + using var provider = new OpenUsdSceneCameraCaptureProvider(options, telemetry: null); + + Assert.That(provider.PluginPath, Is.EqualTo(tempPluginDir), + "A configured PluginPath that exists on disk must be returned verbatim."); + } + finally + { + try + { + Directory.Delete(tempPluginDir, recursive: true); + } + catch (IOException) + { + } + catch (UnauthorizedAccessException) + { + } + } + } + + [Test] + public async Task CaptureAsyncRefusesLocalStagePathThatNamesNoReadableFile() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + string missingPath = Path.Combine( + TestContext.CurrentContext.TestDirectory, + "does-not-exist-" + Guid.NewGuid().ToString("N") + ".usda"); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = missingPath, + Width = 64, + Height = 64, + Format = SceneCameraImageFormat.Png + }; + + SceneCameraCaptureResult result = await provider.CaptureAsync(request, CancellationToken.None) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Status, Is.EqualTo(SceneCameraCaptureStatus.InvalidRequest), + "A missing local file must be refused in managed code so the native " + + "resolver never sees it; otherwise UsdStage.Open can tear the process down."); + Assert.That(result.Reason, Is.Not.Null); + Assert.That(result.Reason, Does.Contain("does not name a readable file")); + Assert.That(result.Image.IsNull, Is.True); + }); + } + + [Test] + public async Task CaptureAsyncRefusesUnknownImageFormat() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:stage", + Width = 64, + Height = 64, + Format = (SceneCameraImageFormat)999 + }; + + SceneCameraCaptureResult result = await provider.CaptureAsync(request, CancellationToken.None) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Status, Is.EqualTo(SceneCameraCaptureStatus.InvalidRequest)); + Assert.That(result.Reason, Is.Not.Null); + Assert.That(result.Reason, Does.Contain("not supported")); + Assert.That(result.Image.IsNull, Is.True); + }); + } + + [Test] + public async Task CaptureResultEchoesRequestSuppliedTimestampOnFailurePaths() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + var ts = new DateTime(2024, 6, 15, 12, 34, 56, DateTimeKind.Utc); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = string.Empty, + Width = 32, + Height = 32, + Format = SceneCameraImageFormat.Png, + TimestampUtc = ts + }; + + SceneCameraCaptureResult result = await provider.CaptureAsync(request, CancellationToken.None) + .ConfigureAwait(false); + + Assert.That(result.TimestampUtc, Is.EqualTo(ts), + "A caller-supplied TimestampUtc must be echoed back on failure results " + + "so the caller can correlate the failure with the request it made."); + } + + [Test] + public async Task CaptureResultCarriesBackendDescriptorOnEveryOutcome() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = string.Empty, + Width = 32, + Height = 32, + Format = SceneCameraImageFormat.Png + }; + + SceneCameraCaptureResult result = await provider.CaptureAsync(request, CancellationToken.None) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(result.Backend, Is.Not.Null); + Assert.That(result.Backend, Is.SameAs(provider.Backend), + "Every result must carry the provider's Backend descriptor unchanged so " + + "the caller can correlate a failure with the graphics backend it used."); + }); + } + + [Test] + public void SceneCameraCaptureRequestExposesEveryInitOnlyPropertyForCallers() + { + var ts = new DateTime(2024, 1, 1, 0, 0, 0, DateTimeKind.Utc); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:cam", + PrimPath = "/World/Camera", + Width = 640, + Height = 480, + TimeCode = 42.5, + Format = SceneCameraImageFormat.Png, + TimestampUtc = ts + }; + + Assert.Multiple(() => + { + Assert.That(request.StageIdentifier, Is.EqualTo("urn:test:cam")); + Assert.That(request.PrimPath, Is.EqualTo("/World/Camera")); + Assert.That(request.Width, Is.EqualTo(640)); + Assert.That(request.Height, Is.EqualTo(480)); + Assert.That(request.TimeCode, Is.EqualTo(42.5)); + Assert.That(request.Format, Is.EqualTo(SceneCameraImageFormat.Png)); + Assert.That(request.TimestampUtc, Is.EqualTo(ts)); + }); + } + + [Test] + public void ConstructorSucceedsWithPreferSoftwareOptionAndProducesBackendDescriptor() + { + using var provider = new OpenUsdSceneCameraCaptureProvider( + new OpenUsdSceneCaptureOptions { PreferSoftware = true }, + telemetry: null); + + Assert.Multiple(() => + { + Assert.That(provider.Backend, Is.Not.Null); + Assert.That(provider.Backend.Name, Is.Not.Empty, + "PreferSoftware=true changes the D3D12 probe order on Windows but must " + + "still resolve to a named backend descriptor."); + }); + } + + [Test] + public void ConstructorSucceedsWithAllowSoftwareFallbackFalseAndProducesBackendDescriptor() + { + using var provider = new OpenUsdSceneCameraCaptureProvider( + new OpenUsdSceneCaptureOptions { AllowSoftwareFallback = false }, + telemetry: null); + + Assert.Multiple(() => + { + Assert.That(provider.Backend, Is.Not.Null); + Assert.That(provider.Backend.Name, Is.Not.Empty, + "AllowSoftwareFallback=false drops the WARP probe on Windows but must " + + "still resolve to a named backend descriptor when hardware D3D12 is available."); + }); + } + + [Test] + public void DeviceSelectorTrySelectDeviceThrowsArgumentNullExceptionForNullOptions() + { + Assert.That(() => + DeviceSelector.TrySelectDevice( + null!, + NullLogger.Instance, + out _, + out _), + Throws.TypeOf()); + } + + [Test] + public void DeviceSelectorTrySelectDeviceThrowsArgumentNullExceptionForNullLogger() + { + Assert.That(() => + DeviceSelector.TrySelectDevice( + new OpenUsdSceneCaptureOptions(), + null!, + out _, + out _), + Throws.TypeOf()); + } + + [Test] + public void DeviceSelectorTrySelectDevicePopulatesBackendDescriptorWhenAnyProbeSucceeds() + { + bool selected = DeviceSelector.TrySelectDevice( + new OpenUsdSceneCaptureOptions(), + NullLogger.Instance, + out SelectedSilkDevice device, + out string reason); + + if (!selected) + { + Assert.That(reason, Is.Not.Empty, + "TrySelectDevice must fill the aggregate reason when every probe fails, " + + "so the caller can surface it in a diagnostic."); + return; + } + try + { + Assert.Multiple(() => + { + Assert.That(device.Device, Is.Not.Null); + Assert.That(device.Backend, Is.Not.Null); + Assert.That(device.Backend.IsAvailable, Is.True); + Assert.That(device.Backend.Name, Is.Not.Empty); + Assert.That(reason, Is.Empty, + "aggregateReason must be empty on the success path so the caller " + + "does not log a stale unavailable message."); + }); + } + finally + { + device.Device.Dispose(); + } + } + + [Test] + public async Task CaptureAsyncPropagatesCancellationRequestedBeforeCall() + { + using var provider = new OpenUsdSceneCameraCaptureProvider(); + using var cts = new CancellationTokenSource(); + cts.Cancel(); + var request = new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:stage", + Width = 32, + Height = 32, + Format = SceneCameraImageFormat.Png + }; + + OperationCanceledException? thrown = null; + try + { + await provider.CaptureAsync(request, cts.Token).ConfigureAwait(false); + } + catch (OperationCanceledException ex) + { + thrown = ex; + } + Assert.That(thrown, Is.Not.Null, + "A pre-cancelled token must cancel the capture before touching the render pipeline; " + + "the check runs before ValidateRequest so the stage identifier value is irrelevant."); + } + + [Test] + public async Task DisposedProviderDisposeIsIdempotentAndFurtherCapturesRejected() + { + var provider = new OpenUsdSceneCameraCaptureProvider(); + provider.Dispose(); + Assert.DoesNotThrow(provider.Dispose, + "Dispose must be idempotent - a defensive host may call it twice."); + + var request = new SceneCameraCaptureRequest + { + StageIdentifier = "urn:test:stage", + Width = 32, + Height = 32, + Format = SceneCameraImageFormat.Png + }; + + ObjectDisposedException? thrown = null; + try + { + await provider.CaptureAsync(request, CancellationToken.None).ConfigureAwait(false); + } + catch (ObjectDisposedException ex) + { + thrown = ex; + } + Assert.That(thrown, Is.Not.Null, + "A capture request after Dispose must be refused with ObjectDisposedException, " + + "not swallowed and answered as a failure."); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionPipelineClientTests.cs b/tests/Opc.Ua.Vision.Tests/VisionPipelineClientTests.cs new file mode 100644 index 0000000000..4cfd7d9b42 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionPipelineClientTests.cs @@ -0,0 +1,420 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Tests for and + /// through the harness. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionPipelineClientTests + { + [Test] + public async Task ReadReturnsSnapshotWithPipelineIdAndState() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.AddValueChild(harness.PipelineNodeId, BrowseNames.PipelineId, + new(3010u, 3), "pipeline-1"); + harness.AddValueChild(harness.PipelineNodeId, BrowseNames.State, + new(3011u, 3), (int)VisionEndpointStateEnum.Active); + harness.AddValueChild(harness.PipelineNodeId, BrowseNames.Continuous, + new(3012u, 3), true); + harness.AddValueChild(harness.PipelineNodeId, BrowseNames.Sensor, + new(3013u, 3), harness.SensorNodeId); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionPipelineSnapshot snapshot = await pipeline.ReadAsync() + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(snapshot.PipelineId, Is.EqualTo("pipeline-1")); + Assert.That(snapshot.State, Is.EqualTo(VisionEndpointStateEnum.Active)); + Assert.That(snapshot.Continuous, Is.True); + Assert.That(snapshot.SensorId, Is.EqualTo(harness.SensorNodeId)); + Assert.That(snapshot.NodeId, Is.EqualTo(harness.PipelineNodeId)); + }); + } + + [Test] + public async Task ReadStateReturnsCurrentState() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.AddValueChild(harness.PipelineNodeId, BrowseNames.State, + new(3020u, 3), (int)VisionEndpointStateEnum.Ready); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionEndpointStateEnum state = await pipeline.ReadStateAsync() + .ConfigureAwait(false); + + Assert.That(state, Is.EqualTo(VisionEndpointStateEnum.Ready)); + } + + [Test] + public async Task ReadStateReturnsDefaultWhenStateNodeAbsent() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionEndpointStateEnum state = await pipeline.ReadStateAsync() + .ConfigureAwait(false); + + Assert.That(state, Is.EqualTo(default(VisionEndpointStateEnum))); + } + + [Test] + public async Task RunInferenceReturnsServerResultId() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.ConfigureCall(StatusCodes.Good, new Variant("result-42")); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + string resultId = await pipeline.RunInferenceAsync().ConfigureAwait(false); + + Assert.That(resultId, Is.EqualTo("result-42")); + } + + [Test] + public async Task StartContinuousDoesNotThrowOnGoodCall() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.ConfigureCall(StatusCodes.Good); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + + Assert.DoesNotThrowAsync(async () => + await pipeline.StartContinuousAsync().ConfigureAwait(false)); + } + + [Test] + public async Task StopDoesNotThrowOnGoodCall() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.ConfigureCall(StatusCodes.Good); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + + Assert.DoesNotThrowAsync(async () => + await pipeline.StopAsync().ConfigureAwait(false)); + } + + [Test] + public async Task EnumerateResultsYieldsInferenceResultsFromFolder() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.AddChild(harness.PipelineNodeId, BrowseNames.Results, + harness.ResultsFolderId); + harness.AddBrowse(harness.ResultsFolderId, + [harness.Ref(harness.InferenceResultNodeId, "R1", + ObjectTypes.DetectionResultType)]); + + var entries = new List(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + await foreach (VisionNodeEntry entry in pipeline.EnumerateResultsAsync()) + { + entries.Add(entry); + } + + Assert.That(entries.Count, Is.EqualTo(1)); + Assert.That(entries[0].NodeId, Is.EqualTo(harness.InferenceResultNodeId)); + } + + [Test] + public async Task EnumerateResultsYieldsNothingWhenResultsFolderAbsent() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + + var entries = new List(); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + await foreach (VisionNodeEntry entry in pipeline.EnumerateResultsAsync()) + { + entries.Add(entry); + } + + Assert.That(entries.Count, Is.EqualTo(0)); + } + + [Test] + public async Task OpenFeedbackReturnsNullWhenNoFeedbackObject() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionFeedbackClient? feedback = await pipeline.OpenFeedbackAsync() + .ConfigureAwait(false); + + Assert.That(feedback, Is.Null); + } + + [Test] + public async Task OpenFeedbackReturnsClientWhenFeedbackObjectPresent() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.AddChild(harness.PipelineNodeId, BrowseNames.Feedback, + harness.FeedbackNodeId); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionFeedbackClient? feedback = await pipeline.OpenFeedbackAsync() + .ConfigureAwait(false); + + Assert.That(feedback, Is.Not.Null); + Assert.That(feedback!.FeedbackNodeId, Is.EqualTo(harness.FeedbackNodeId)); + } + + [Test] + public void ConstructorRejectsNullPipelineNodeId() + { + var harness = new VisionSessionHarness(); + + Assert.Throws(() => + harness.Client.Pipeline(NodeId.Null)); + } + } + + /// + /// Tests for pre-flight argument + /// validation. Post-flight validation (server refusal) is covered by the + /// dispatcher-level tests. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionFeedbackClientTests + { + [Test] + public async Task SubmitDetectionsRejectsEmptyDetections() + { + VisionFeedbackClient feedback = await BuildFeedbackAsync().ConfigureAwait(false); + + // Part 9.5 pairs the array with the flag: an empty array without + // SceneIsEmpty is a lost payload rather than an observation, so the + // wrapper refuses it before the call leaves the client. + var ex = Assert.ThrowsAsync(async () => + await feedback.SubmitDetectionsAsync( + VisionFeedbackPurposeEnum.Overlay, + ArrayOf.Empty, + null, + ByteString.Empty).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("detections")); + } + + [Test] + public async Task SubmitDetectionsRejectsSceneIsEmptyWithDetectionsAttached() + { + VisionFeedbackClient feedback = await BuildFeedbackAsync().ConfigureAwait(false); + + var ex = Assert.ThrowsAsync(async () => + await feedback.SubmitDetectionsAsync( + VisionFeedbackPurposeEnum.GroundTruthLabel, + new[] { new VisionDetectionDataType { ClassLabel = "Part" } }.ToArrayOf(), + null, + ByteString.Empty, + sceneIsEmpty: true).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("detections")); + } + + [Test] + public async Task SubmitCorrectionRejectsRetractAllWithAReplacementAttached() + { + VisionFeedbackClient feedback = await BuildFeedbackAsync().ConfigureAwait(false); + + var ex = Assert.ThrowsAsync(async () => + await feedback.SubmitCorrectionAsync( + "r-1", + VisionFeedbackPurposeEnum.GroundTruthLabel, + new[] { new VisionDetectionDataType { ClassLabel = "Part" } }.ToArrayOf(), + ArrayOf.Empty, + default, + ByteString.Empty, + retractAll: true).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("correctedDetections")); + } + + [Test] + public async Task SubmitInspectionResultRejectsEmptyResultId() + { + VisionFeedbackClient feedback = await BuildFeedbackAsync().ConfigureAwait(false); + var characteristics = new List { new() }.ToArrayOf(); + + var ex = Assert.ThrowsAsync(async () => + await feedback.SubmitInspectionResultAsync( + string.Empty, + VisionResultEvaluationEnum.Ok, + characteristics).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("resultId")); + } + + [Test] + public async Task SubmitInspectionResultRejectsEmptyCharacteristics() + { + VisionFeedbackClient feedback = await BuildFeedbackAsync().ConfigureAwait(false); + + var ex = Assert.ThrowsAsync(async () => + await feedback.SubmitInspectionResultAsync( + "r-1", + VisionResultEvaluationEnum.Ok, + ArrayOf.Empty).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("characteristics")); + } + + [Test] + public async Task SubmitCorrectionRejectsEmptyResultId() + { + VisionFeedbackClient feedback = await BuildFeedbackAsync().ConfigureAwait(false); + var detections = new List { new() }.ToArrayOf(); + + var ex = Assert.ThrowsAsync(async () => + await feedback.SubmitCorrectionAsync( + string.Empty, + VisionFeedbackPurposeEnum.GroundTruthLabel, + detections, + ArrayOf.Empty, + LocalizedText.Null, + ByteString.Empty).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("resultId")); + } + + [Test] + public async Task SubmitCorrectionRejectsBothDetectionsAndCharacteristicsSupplied() + { + VisionFeedbackClient feedback = await BuildFeedbackAsync().ConfigureAwait(false); + var detections = new List { new() }.ToArrayOf(); + var characteristics = new List { new() }.ToArrayOf(); + + Assert.ThrowsAsync(async () => + await feedback.SubmitCorrectionAsync( + "r-1", + VisionFeedbackPurposeEnum.GroundTruthLabel, + detections, + characteristics, + LocalizedText.Null, + ByteString.Empty).ConfigureAwait(false)); + } + + [Test] + public async Task SubmitCorrectionRejectsBothDetectionsAndCharacteristicsEmpty() + { + VisionFeedbackClient feedback = await BuildFeedbackAsync().ConfigureAwait(false); + + Assert.ThrowsAsync(async () => + await feedback.SubmitCorrectionAsync( + "r-1", + VisionFeedbackPurposeEnum.GroundTruthLabel, + ArrayOf.Empty, + ArrayOf.Empty, + LocalizedText.Null, + ByteString.Empty).ConfigureAwait(false)); + } + + [Test] + public async Task SubmitImageReferenceRejectsNullImage() + { + VisionFeedbackClient feedback = await BuildFeedbackAsync().ConfigureAwait(false); + + var ex = Assert.ThrowsAsync(async () => + await feedback.SubmitImageReferenceAsync( + VisionFeedbackPurposeEnum.GroundTruthLabel, + null!, + "r-1").ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("image")); + } + + [Test] + public async Task SubmitDetectionsSuccessfulForwardsCallToServer() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.AddChild(harness.PipelineNodeId, BrowseNames.Feedback, + harness.FeedbackNodeId); + harness.ConfigureCall(StatusCodes.Good); + + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionFeedbackClient? feedback = await pipeline.OpenFeedbackAsync() + .ConfigureAwait(false); + Assert.That(feedback, Is.Not.Null); + var detections = new List { new() }.ToArrayOf(); + + Assert.DoesNotThrowAsync(async () => + await feedback!.SubmitDetectionsAsync( + VisionFeedbackPurposeEnum.Overlay, + detections, + null, + ByteString.Empty).ConfigureAwait(false)); + } + + private static async Task BuildFeedbackAsync() + { + var harness = new VisionSessionHarness(); + harness.ConfigureVisionFolders(); + harness.AddPipeline(); + harness.AddChild(harness.PipelineNodeId, BrowseNames.Feedback, + harness.FeedbackNodeId); + VisionPipelineClient pipeline = harness.Client.Pipeline(harness.PipelineNodeId); + VisionFeedbackClient? feedback = await pipeline.OpenFeedbackAsync() + .ConfigureAwait(false); + return feedback!; + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionProviderContractRecordsTests.cs b/tests/Opc.Ua.Vision.Tests/VisionProviderContractRecordsTests.cs new file mode 100644 index 0000000000..4d2f4d2a4c --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionProviderContractRecordsTests.cs @@ -0,0 +1,267 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using Moq; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Exercises the simple request/result value records exposed to + /// providers and sinks. These records are part of the public API + /// surface; their positional-property behaviour must remain intact. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionProviderContractRecordsTests + { + [Test] + public void VisionInferenceRunRequestRoundTripsPositionalProperties() + { + var pipeline = new NodeId(1, 1); + var sensor = new NodeId(2, 1); + var deployment = new NodeId(3, 1); + DateTimeUtc timestamp = DateTimeUtc.From(new DateTime(2025, 1, 1, 12, 0, 0, DateTimeKind.Utc)); + + var request = new VisionInferenceRunRequest(pipeline, sensor, deployment, timestamp); + + Assert.Multiple(() => + { + Assert.That(request.Pipeline, Is.EqualTo(pipeline)); + Assert.That(request.Sensor, Is.EqualTo(sensor)); + Assert.That(request.Deployment, Is.EqualTo(deployment)); + Assert.That(request.Timestamp, Is.EqualTo(timestamp)); + }); + } + + [Test] + public void VisionInferenceRunRequestEqualityIsStructural() + { + var a = new VisionInferenceRunRequest( + new NodeId(1, 1), new NodeId(2, 1), new NodeId(3, 1), + DateTimeUtc.From(new DateTime(2025, 1, 1, 12, 0, 0, DateTimeKind.Utc))); + var b = new VisionInferenceRunRequest( + new NodeId(1, 1), new NodeId(2, 1), new NodeId(3, 1), + DateTimeUtc.From(new DateTime(2025, 1, 1, 12, 0, 0, DateTimeKind.Utc))); + + Assert.That(a, Is.EqualTo(b)); + Assert.That(a.GetHashCode(), Is.EqualTo(b.GetHashCode())); + } + + [Test] + public void VisionInferenceRunResultRoundTripsPositionalProperties() + { + ServiceResult sr = ServiceResult.Good; + var result = new VisionInferenceRunResult(sr, "result-42"); + + Assert.Multiple(() => + { + Assert.That(result.ServiceResult, Is.EqualTo(sr)); + Assert.That(result.ResultId, Is.EqualTo("result-42")); + }); + } + + [Test] + public void VisionStreamRequestRoundTripsPositionalProperties() + { + var endpoint = new NodeId(10, 1); + var request = new VisionStreamRequest(endpoint, "profile-1", VisionStreamProtocolEnum.Rtsp); + + Assert.Multiple(() => + { + Assert.That(request.Endpoint, Is.EqualTo(endpoint)); + Assert.That(request.ProfileName, Is.EqualTo("profile-1")); + Assert.That(request.PreferredProtocol, Is.EqualTo(VisionStreamProtocolEnum.Rtsp)); + }); + } + + [Test] + public void VisionStreamLeaseRoundTripsPositionalProperties() + { + ServiceResult sr = ServiceResult.Good; + var session = new VisionStreamSessionDataType + { + Uri = "rtsp://x", + SessionToken = ByteString.Empty + }; + var endpointOut = new NodeId(1, 1); + + var lease = new VisionStreamLease(sr, session, endpointOut); + + Assert.Multiple(() => + { + Assert.That(lease.ServiceResult, Is.EqualTo(sr)); + Assert.That(lease.Session, Is.SameAs(session)); + Assert.That(lease.EndpointOut, Is.EqualTo(endpointOut)); + }); + } + + [Test] + public void VisionStreamConfigurationRequestRoundTripsPositionalProperties() + { + var endpoint = new NodeId(10, 1); + var request = new VisionStreamConfigurationRequest( + endpoint, VisionVideoCodecEnum.H264, 1920, 1080, 30.0, 8_000_000); + + Assert.Multiple(() => + { + Assert.That(request.Endpoint, Is.EqualTo(endpoint)); + Assert.That(request.Codec, Is.EqualTo(VisionVideoCodecEnum.H264)); + Assert.That(request.Width, Is.EqualTo((uint)1920)); + Assert.That(request.Height, Is.EqualTo((uint)1080)); + Assert.That(request.FrameRate, Is.EqualTo(30.0)); + Assert.That(request.Bitrate, Is.EqualTo((uint)8_000_000)); + }); + } + + [Test] + public void VisionClipRequestRoundTripsPositionalProperties() + { + var endpoint = new NodeId(10, 1); + DateTimeUtc timestamp = DateTimeUtc.From(new DateTime(2025, 1, 1, 12, 0, 0, DateTimeKind.Utc)); + + var request = new VisionClipRequest( + endpoint, "result-1", timestamp, VisionClipFormatEnum.Jpeg, RequestInline: true); + + Assert.Multiple(() => + { + Assert.That(request.Endpoint, Is.EqualTo(endpoint)); + Assert.That(request.ResultId, Is.EqualTo("result-1")); + Assert.That(request.Timestamp, Is.EqualTo(timestamp)); + Assert.That(request.Format, Is.EqualTo(VisionClipFormatEnum.Jpeg)); + Assert.That(request.RequestInline, Is.True); + }); + } + + [Test] + public void VisionClipResultRoundTripsPositionalProperties() + { + ServiceResult sr = ServiceResult.Good; + var image = new VisionImageReferenceDataType(); + var endpointOut = new NodeId(1, 1); + ByteString inline = ByteString.From(new byte[] { 0x01, 0x02 }); + + var result = new VisionClipResult(sr, image, endpointOut, inline); + + Assert.Multiple(() => + { + Assert.That(result.ServiceResult, Is.EqualTo(sr)); + Assert.That(result.Image, Is.SameAs(image)); + Assert.That(result.EndpointOut, Is.EqualTo(endpointOut)); + Assert.That(result.InlineImage, Is.EqualTo(inline)); + }); + } + + [Test] + public void VisionSubmitDetectionsRequestRoundTripsPositionalProperties() + { + var pipeline = new NodeId(10, 1); + ArrayOf detections = + new List().ToArrayOf(); + var frameRef = new VisionImageReferenceDataType(); + ByteString inline = ByteString.Empty; + + var request = new VisionSubmitDetectionsRequest( + pipeline, VisionFeedbackPurposeEnum.Overlay, detections, frameRef, inline); + + Assert.Multiple(() => + { + Assert.That(request.Pipeline, Is.EqualTo(pipeline)); + Assert.That(request.Purpose, Is.EqualTo(VisionFeedbackPurposeEnum.Overlay)); + Assert.That(request.FrameReference, Is.SameAs(frameRef)); + Assert.That(request.InlineImage, Is.EqualTo(inline)); + }); + } + + [Test] + public void VisionSubmitInspectionResultRequestRoundTripsPositionalProperties() + { + var pipeline = new NodeId(10, 1); + ArrayOf characteristics = + new List().ToArrayOf(); + + var request = new VisionSubmitInspectionResultRequest( + pipeline, "result-1", VisionResultEvaluationEnum.Ok, characteristics); + + Assert.Multiple(() => + { + Assert.That(request.Pipeline, Is.EqualTo(pipeline)); + Assert.That(request.ResultId, Is.EqualTo("result-1")); + Assert.That(request.Evaluation, Is.EqualTo(VisionResultEvaluationEnum.Ok)); + }); + } + + [Test] + public void VisionSubmitCorrectionRequestRoundTripsPositionalProperties() + { + var pipeline = new NodeId(10, 1); + ArrayOf dets = + new List().ToArrayOf(); + ArrayOf chars = + new List().ToArrayOf(); + var reason = new LocalizedText("en", "bad"); + ByteString inline = ByteString.Empty; + + var request = new VisionSubmitCorrectionRequest( + pipeline, "result-1", VisionFeedbackPurposeEnum.GroundTruthLabel, + dets, chars, reason, inline); + + Assert.Multiple(() => + { + Assert.That(request.Pipeline, Is.EqualTo(pipeline)); + Assert.That(request.ResultId, Is.EqualTo("result-1")); + Assert.That(request.Purpose, Is.EqualTo(VisionFeedbackPurposeEnum.GroundTruthLabel)); + Assert.That(request.Reason, Is.EqualTo(reason)); + Assert.That(request.InlineImage, Is.EqualTo(inline)); + }); + } + + [Test] + public void VisionSubmitImageReferenceRequestRoundTripsPositionalProperties() + { + var pipeline = new NodeId(10, 1); + var image = new VisionImageReferenceDataType(); + + var request = new VisionSubmitImageReferenceRequest( + pipeline, VisionFeedbackPurposeEnum.Overlay, image, "result-1"); + + Assert.Multiple(() => + { + Assert.That(request.Pipeline, Is.EqualTo(pipeline)); + Assert.That(request.Purpose, Is.EqualTo(VisionFeedbackPurposeEnum.Overlay)); + Assert.That(request.Image, Is.SameAs(image)); + Assert.That(request.ResultId, Is.EqualTo("result-1")); + }); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionRegistryTests.cs b/tests/Opc.Ua.Vision.Tests/VisionRegistryTests.cs new file mode 100644 index 0000000000..db38646c99 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionRegistryTests.cs @@ -0,0 +1,281 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Pins the shape of the internal Vision registry that backs facet + /// computation, frame math, and method dispatch. The registry is the + /// single source of truth the node manager consults after materialising + /// nodes from the address space; every "does anyone have this facet?", + /// "resolve this frame id", and "route this method call" query + /// eventually hits one of these lookups. + /// + [TestFixture] + public sealed class VisionRegistryTests + { + [Test] + public void AddSensorMakesItLookupableByBrowseNameAndNodeId() + { + var registry = new VisionRegistry(); + SensorRegistration reg = NewSensor("cam1", 101); + + registry.AddSensor(reg); + + Assert.Multiple(() => + { + Assert.That(registry.TryGetSensor("cam1", out SensorRegistration? byName), Is.True); + Assert.That(byName, Is.SameAs(reg)); + Assert.That(registry.TryGetSensor(new NodeId(101, 4), out SensorRegistration? byId), Is.True); + Assert.That(byId, Is.SameAs(reg)); + }); + } + + [Test] + public void TryGetSensorReturnsFalseForUnknownBrowseName() + { + var registry = new VisionRegistry(); + + Assert.That(registry.TryGetSensor("nope", out _), Is.False); + } + + [Test] + public void TryGetSensorWithNullNodeIdReturnsFalse() + { + var registry = new VisionRegistry(); + registry.AddSensor(NewSensor("cam1", 101)); + NodeId nullId = default; + + Assert.That(registry.TryGetSensor(nullId, out _), Is.False, + "A null NodeId must not throw and must not return a sensor."); + } + + [Test] + public void AddPipelineMakesItLookupableByBrowseNameAndNodeId() + { + var registry = new VisionRegistry(); + PipelineRegistration reg = NewPipeline("pipe1", 201); + + registry.AddPipeline(reg); + + Assert.Multiple(() => + { + Assert.That(registry.TryGetPipeline("pipe1", out PipelineRegistration? byName), Is.True); + Assert.That(byName, Is.SameAs(reg)); + Assert.That(registry.TryGetPipeline(new NodeId(201, 4), out PipelineRegistration? byId), Is.True); + Assert.That(byId, Is.SameAs(reg)); + }); + } + + [Test] + public void TryGetPipelineWithNullNodeIdReturnsFalse() + { + var registry = new VisionRegistry(); + registry.AddPipeline(NewPipeline("pipe1", 201)); + NodeId nullId = default; + + Assert.That(registry.TryGetPipeline(nullId, out _), Is.False); + } + + [Test] + public void AnySensorHasFacetIsTrueOnlyWhenAnySensorContainsThatFacet() + { + var registry = new VisionRegistry(); + var facets = new HashSet { VisionConformanceUris.FacetNames.Optics }; + registry.AddSensor(NewSensor("cam1", 101, facets)); + + Assert.Multiple(() => + { + Assert.That(registry.AnySensorHasFacet(VisionConformanceUris.FacetNames.Optics), Is.True); + Assert.That(registry.AnySensorHasFacet(VisionConformanceUris.FacetNames.MediaInline), Is.False); + Assert.That(registry.AnySensorHasFacet("nonexistent-facet"), Is.False); + }); + } + + [Test] + public void AnyPipelineHasFacetIsTrueOnlyWhenAnyPipelineContainsThatFacet() + { + var registry = new VisionRegistry(); + var facets = new HashSet { VisionConformanceUris.FacetNames.InferenceOnServer }; + registry.AddPipeline(NewPipeline("pipe1", 201, facets)); + + Assert.Multiple(() => + { + Assert.That(registry.AnyPipelineHasFacet(VisionConformanceUris.FacetNames.InferenceOnServer), Is.True); + Assert.That(registry.AnyPipelineHasFacet(VisionConformanceUris.FacetNames.Feedback), Is.False); + }); + } + + [Test] + public void AddFrameMakesItLookupableByBrowseNameAndFrameId() + { + var registry = new VisionRegistry(); + FrameRegistration reg = NewFrame("baseFrame", 301, frameId: "base", parentFrameId: null); + + registry.AddFrame(reg); + + Assert.Multiple(() => + { + Assert.That(registry.TryGetFrame("baseFrame", out FrameRegistration? byName), Is.True); + Assert.That(byName, Is.SameAs(reg)); + Assert.That(registry.TryGetFrameByFrameId("base", out FrameRegistration? byFrameId), Is.True); + Assert.That(byFrameId, Is.SameAs(reg)); + Assert.That(registry.TryFindFrameByFrameId("base"), Is.SameAs(reg)); + }); + } + + [Test] + public void TryGetFrameForUnknownFrameIdReturnsFalseWithoutThrowing() + { + var registry = new VisionRegistry(); + + Assert.Multiple(() => + { + Assert.That(registry.TryGetFrameByFrameId("nope", out FrameRegistration? reg), Is.False); + Assert.That(reg, Is.Null); + Assert.That(registry.TryFindFrameByFrameId("nope"), Is.Null); + }); + } + + [Test] + public void TryGetFrameHandlesNullBrowseNameAsEmptyStringLookup() + { + var registry = new VisionRegistry(); + + Assert.That(registry.TryGetFrame(null!, out FrameRegistration? reg), Is.False); + Assert.That(reg, Is.Null); + } + + [Test] + public void ToFrameSnapshotsKeysByFrameIdAndCarriesTransform() + { + var registry = new VisionRegistry(); + registry.AddFrame(NewFrame("baseFrame", 301, frameId: "base", parentFrameId: null)); + registry.AddFrame(NewFrame("tcpFrame", 302, frameId: "tcp", parentFrameId: "base")); + + IReadOnlyDictionary snapshots + = registry.ToFrameSnapshots(); + + Assert.Multiple(() => + { + Assert.That(snapshots, Has.Count.EqualTo(2)); + Assert.That(snapshots.ContainsKey("base"), Is.True); + Assert.That(snapshots.ContainsKey("tcp"), Is.True); + Assert.That(snapshots["tcp"].ParentFrameId, Is.EqualTo("base")); + Assert.That(snapshots["base"].ParentFrameId, Is.Empty, + "Frame with a null parent must surface as an empty ParentFrameId, not null."); + }); + } + + [Test] + public void SensorAndPipelineRegistrationsAreEnumerableViaTheReadOnlyDictionaries() + { + var registry = new VisionRegistry(); + registry.AddSensor(NewSensor("cam1", 101)); + registry.AddSensor(NewSensor("cam2", 102)); + registry.AddPipeline(NewPipeline("pipe1", 201)); + + Assert.Multiple(() => + { + Assert.That(registry.Sensors, Has.Count.EqualTo(2)); + Assert.That(registry.SensorsByNodeId, Has.Count.EqualTo(2)); + Assert.That(registry.Pipelines, Has.Count.EqualTo(1)); + Assert.That(registry.PipelinesByNodeId, Has.Count.EqualTo(1)); + }); + } + + [Test] + public void ResolveDeferredExtrinsicsIsSafeToCallWithNothingDeferred() + { + var registry = new VisionRegistry(); + + Assert.DoesNotThrow(() => registry.ResolveDeferredExtrinsics()); + } + + [Test] + public void AddDeferredExtrinsicResolutionAcceptsNullCalibrationWithoutThrowing() + { + var registry = new VisionRegistry(); + + Assert.DoesNotThrow( + () => registry.AddDeferredExtrinsicResolution(null!, "sourceFrame", "targetFrame")); + } + + private static SensorRegistration NewSensor( + string browseName, uint id, HashSet? facets = null) + { + var sensor = new VisionSensorState(null); + return new SensorRegistration( + browseName, + new NodeId(id, 4), + sensor, + VisionSensorModalityEnum.Area2D, + VisionRealityKindEnum.Physical, + facets ?? new HashSet(System.StringComparer.Ordinal), + mediaProvider: null); + } + + private static PipelineRegistration NewPipeline( + string browseName, uint id, HashSet? facets = null) + { + var pipeline = new InferencePipelineState(null); + return new PipelineRegistration( + browseName, + new NodeId(id, 4), + pipeline, + facets ?? new HashSet(System.StringComparer.Ordinal)); + } + + private static FrameRegistration NewFrame( + string browseName, uint id, string frameId, string? parentFrameId) + { + var frame = new CoordinateFrameState(null); + var transform = new VisionPose3DDataType + { + FrameId = frameId, + Position = new double[] { 0, 0, 0 }, + Orientation = new double[] { 0, 0, 0, 1 }, + Covariance = System.Array.Empty() + }; + return new FrameRegistration( + browseName, + new NodeId(id, 4), + frameId, + VisionFrameRoleEnum.Base, + parentFrameId, + transform, + frame); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionResultAndFrameGraphTests.cs b/tests/Opc.Ua.Vision.Tests/VisionResultAndFrameGraphTests.cs new file mode 100644 index 0000000000..4ab17131f5 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionResultAndFrameGraphTests.cs @@ -0,0 +1,331 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Tests for — snapshot reads for + /// detection, inspection and segmentation results. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionResultReaderTests + { + [Test] + public async Task ReadInspectionReturnsPopulatedSnapshot() + { + var harness = new VisionSessionHarness(); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.ResultId, + new(3210u, 3), "insp-1"); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.Evaluation, + new(3211u, 3), (int)VisionResultEvaluationEnum.Ok); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.PartId, + new(3212u, 3), "part-A"); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.RecipeId, + new(3213u, 3), "recipe-1"); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.CreationTime, + new(3214u, 3), new DateTimeUtc(new DateTime(2024, 1, 1))); + + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + VisionInspectionResultSnapshot snapshot = await reader.ReadInspectionAsync() + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(snapshot.ResultId, Is.EqualTo("insp-1")); + Assert.That(snapshot.Evaluation, Is.EqualTo(VisionResultEvaluationEnum.Ok)); + Assert.That(snapshot.PartId, Is.EqualTo("part-A")); + Assert.That(snapshot.RecipeId, Is.EqualTo("recipe-1")); + Assert.That(snapshot.NodeId, Is.EqualTo(harness.ResultNodeId)); + }); + } + + [Test] + public async Task ReadDetectionReturnsPopulatedSnapshot() + { + var harness = new VisionSessionHarness(); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.ResultId, + new(3220u, 3), "det-1"); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.FrameId, + new(3221u, 3), "frame-1"); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.CreationTime, + new(3222u, 3), new DateTimeUtc(new DateTime(2024, 1, 1))); + + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + VisionDetectionResultSnapshot snapshot = await reader.ReadDetectionAsync() + .ConfigureAwait(false); + + Assert.That(snapshot.ResultId, Is.EqualTo("det-1")); + Assert.That(snapshot.FrameId, Is.EqualTo("frame-1")); + } + + [Test] + public async Task ReadSegmentationReturnsPopulatedSnapshot() + { + var harness = new VisionSessionHarness(); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.ResultId, + new(3230u, 3), "seg-1"); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.CreationTime, + new(3231u, 3), new DateTimeUtc(new DateTime(2024, 1, 1))); + harness.AddValueChild(harness.ResultNodeId, BrowseNames.LabelClasses, + new(3232u, 3), new Variant(new[] { "background", "part" })); + + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + VisionSegmentationResultSnapshot snapshot = await reader.ReadSegmentationAsync() + .ConfigureAwait(false); + + Assert.That(snapshot.ResultId, Is.EqualTo("seg-1")); + Assert.That(snapshot.LabelClasses.Count, Is.EqualTo(2)); + Assert.That(snapshot.LabelClasses[0], Is.EqualTo("background")); + } + + [Test] + public void ConstructorRejectsNullResultNodeId() + { + var harness = new VisionSessionHarness(); + + Assert.Throws(() => + harness.Client.Result(NodeId.Null)); + } + + [Test] + public void ObserveDetectionsRejectsNullStreaming() + { + var harness = new VisionSessionHarness(); + VisionResultReader reader = harness.Client.Result(harness.ResultNodeId); + + Assert.Throws(() => + reader.ObserveDetectionsAsync(null!)); + } + } + + /// + /// Tests for — frame read, compose, + /// walk-to-root guards. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionFrameGraphTests + { + [Test] + public async Task ReadReturnsSnapshotWithFrameIdAndTransform() + { + var harness = new VisionSessionHarness(); + var identityTransform = new VisionPose3DDataType + { + FrameId = "root", + Position = new[] { 0.0, 0.0, 0.0 }.ToArrayOf(), + Orientation = new[] { 0.0, 0.0, 0.0, 1.0 }.ToArrayOf() + }; + harness.AddValueChild(harness.FrameNodeId, BrowseNames.FrameId, + new(4010u, 3), "root"); + harness.AddValueChild(harness.FrameNodeId, BrowseNames.Role, + new(4011u, 3), (int)VisionFrameRoleEnum.World); + harness.AddValueChild(harness.FrameNodeId, BrowseNames.Transform, + new(4013u, 3), Variant.FromStructure(identityTransform)); + + VisionFrameGraph graph = harness.Client.Frames(); + VisionFrameSnapshot snapshot = await graph.ReadAsync(harness.FrameNodeId) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(snapshot.FrameId, Is.EqualTo("root")); + Assert.That(snapshot.Role, Is.EqualTo(VisionFrameRoleEnum.World)); + Assert.That(snapshot.NodeId, Is.EqualTo(harness.FrameNodeId)); + Assert.That(snapshot.Transform, Is.Not.Null); + }); + } + + [Test] + public void ReadRejectsNullFrameNodeId() + { + var harness = new VisionSessionHarness(); + VisionFrameGraph graph = harness.Client.Frames(); + + var ex = Assert.ThrowsAsync(async () => + await graph.ReadAsync(NodeId.Null).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("frameNodeId")); + } + + [Test] + public void ComposeRejectsNullPose() + { + var harness = new VisionSessionHarness(); + VisionFrameGraph graph = harness.Client.Frames(); + + var ex = Assert.ThrowsAsync(async () => + await graph.ComposeAsync(null!, harness.FrameNodeId, + harness.FrameNodeId).ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("pose")); + } + + [Test] + public void ComposeRejectsNullFromFrameId() + { + var harness = new VisionSessionHarness(); + VisionFrameGraph graph = harness.Client.Frames(); + var pose = new VisionPose3DDataType(); + + var ex = Assert.ThrowsAsync(async () => + await graph.ComposeAsync(pose, NodeId.Null, harness.FrameNodeId) + .ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("fromFrameId")); + } + + [Test] + public void ComposeRejectsNullToFrameId() + { + var harness = new VisionSessionHarness(); + VisionFrameGraph graph = harness.Client.Frames(); + var pose = new VisionPose3DDataType(); + + var ex = Assert.ThrowsAsync(async () => + await graph.ComposeAsync(pose, harness.FrameNodeId, NodeId.Null) + .ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("toFrameId")); + } + + [Test] + public void ComposeTransformRejectsNullFromFrameId() + { + var harness = new VisionSessionHarness(); + VisionFrameGraph graph = harness.Client.Frames(); + + var ex = Assert.ThrowsAsync(async () => + await graph.ComposeTransformAsync(NodeId.Null, harness.FrameNodeId) + .ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("fromFrameId")); + } + + [Test] + public void ComposeTransformRejectsNullToFrameId() + { + var harness = new VisionSessionHarness(); + VisionFrameGraph graph = harness.Client.Frames(); + + var ex = Assert.ThrowsAsync(async () => + await graph.ComposeTransformAsync(harness.FrameNodeId, NodeId.Null) + .ConfigureAwait(false)); + + Assert.That(ex!.ParamName, Is.EqualTo("toFrameId")); + } + + [Test] + public async Task ComposeTransformReturnsIdentityWhenFromEqualsTo() + { + var harness = new VisionSessionHarness(); + harness.AddValueChild(harness.FrameNodeId, BrowseNames.FrameId, + new(4020u, 3), "root"); + + VisionFrameGraph graph = harness.Client.Frames(); + VisionPose3DDataType transform = await graph.ComposeTransformAsync( + harness.FrameNodeId, harness.FrameNodeId).ConfigureAwait(false); + + Assert.That(transform, Is.Not.Null); + Assert.That(transform.Orientation.Count, Is.GreaterThanOrEqualTo(4), + "An identity quaternion has four components."); + } + } + + /// + /// Tests for and + /// . + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionClientFactoryTests + { + [Test] + public void ConstructorRejectsNullSessionFactory() + { + var telemetry = new Moq.Mock().Object; + + var ex = Assert.Throws(() => + new VisionClientFactory(null!, telemetry)); + + Assert.That(ex!.ParamName, Is.EqualTo("sessionFactory")); + } + + [Test] + public void ConstructorRejectsNullTelemetry() + { + var ex = Assert.Throws(() => + new VisionClientFactory( + _ => Task.FromResult(null!), + null!)); + + Assert.That(ex!.ParamName, Is.EqualTo("telemetry")); + } + + [Test] + public void SessionVisionRejectsNullSession() + { + var telemetry = new Moq.Mock().Object; + + var ex = Assert.Throws(() => + SessionVisionExtensions.Vision(null!, telemetry)); + + Assert.That(ex!.ParamName, Is.EqualTo("session")); + } + + [Test] + public void SessionVisionRejectsNullTelemetry() + { + var harness = new VisionSessionHarness(); + + var ex = Assert.Throws(() => + harness.Session.Object.Vision(null!)); + + Assert.That(ex!.ParamName, Is.EqualTo("telemetry")); + } + + [Test] + public void SessionVisionReturnsVisionClientBoundToSession() + { + var harness = new VisionSessionHarness(); + + VisionClient client = harness.Session.Object.Vision(harness.Telemetry); + + Assert.That(client, Is.Not.Null); + Assert.That(client.Session, Is.SameAs(harness.Session.Object)); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionSensorClientTests.cs b/tests/Opc.Ua.Vision.Tests/VisionSensorClientTests.cs new file mode 100644 index 0000000000..46233941af --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionSensorClientTests.cs @@ -0,0 +1,406 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using System.Threading.Tasks; +using NUnit.Framework; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Tests for through the harness. + /// Every test drives the client via the public front door + /// (Client.Sensor(id)) so the underlying + /// VisionSensorTypeClient proxy code is exercised as well. + /// + [TestFixture] + [Category("Vision")] + public sealed class VisionSensorClientTests + { + [Test] + public async Task ReadIdentityReturnsAllPopulatedMembers() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.SensorId, + new(2500u, 3), "cam-1"); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.RealityKind, + new(2501u, 3), (int)VisionRealityKindEnum.Physical); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.Modality, + new(2502u, 3), (int)VisionSensorModalityEnum.Area2D); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.Manufacturer, + new(2503u, 3), new LocalizedText("ACME")); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.Model, + new(2504u, 3), new LocalizedText("Model X")); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.SerialNumber, + new(2505u, 3), "SN-1"); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.DeviceUri, + new(2506u, 3), "urn:acme:camera:1"); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.FrameId, + new(2507u, 3), "cam-1"); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionSensorIdentity identity = await sensor.ReadIdentityAsync() + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(identity.SensorId, Is.EqualTo("cam-1")); + Assert.That(identity.RealityKind, Is.EqualTo(VisionRealityKindEnum.Physical)); + Assert.That(identity.Modality, Is.EqualTo(VisionSensorModalityEnum.Area2D)); + Assert.That(identity.Manufacturer.Text, Is.EqualTo("ACME")); + Assert.That(identity.Model.Text, Is.EqualTo("Model X")); + Assert.That(identity.SerialNumber, Is.EqualTo("SN-1")); + Assert.That(identity.DeviceUri, Is.EqualTo("urn:acme:camera:1")); + Assert.That(identity.FrameId, Is.EqualTo("cam-1")); + Assert.That(identity.NodeId, Is.EqualTo(harness.SensorNodeId)); + }); + } + + [Test] + public async Task ReadImageMembersReturnsNullWhenCoreMembersAbsent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionImageSensorSnapshot? snapshot = await sensor.ReadImageMembersAsync() + .ConfigureAwait(false); + + Assert.That(snapshot, Is.Null, + "Sensor with no Width, Height or PixelFormat is not an image sensor."); + } + + [Test] + public async Task ReadImageMembersReturnsPopulatedSnapshotWhenPresent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.Width, + new(2510u, 3), 1920u); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.Height, + new(2511u, 3), 1080u); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.PixelFormat, + new(2512u, 3), "Mono8"); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.ExposureTime, + new(2513u, 3), 0.001); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.Gain, + new(2514u, 3), 1.5); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.AcquisitionFrameRate, + new(2515u, 3), 30.0); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionImageSensorSnapshot? snapshot = await sensor.ReadImageMembersAsync() + .ConfigureAwait(false); + + Assert.That(snapshot, Is.Not.Null); + Assert.Multiple(() => + { + Assert.That(snapshot!.Width, Is.EqualTo(1920u)); + Assert.That(snapshot.Height, Is.EqualTo(1080u)); + Assert.That(snapshot.PixelFormat, Is.EqualTo("Mono8")); + Assert.That(snapshot.ExposureTime, Is.EqualTo(0.001)); + Assert.That(snapshot.Gain, Is.EqualTo(1.5)); + Assert.That(snapshot.AcquisitionFrameRate, Is.EqualTo(30.0)); + }); + } + + [Test] + public async Task ReadDepthMembersReturnsNullWhenNoDepthMembersPresent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.Depth3DSensorType); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionDepth3DSensorSnapshot? snapshot = await sensor.ReadDepthMembersAsync() + .ConfigureAwait(false); + + Assert.That(snapshot, Is.Null); + } + + [Test] + public async Task ReadDepthMembersReturnsPopulatedSnapshotWhenPresent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.Depth3DSensorType); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.MinDepth, + new(2520u, 3), 0.1); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.MaxDepth, + new(2521u, 3), 5.0); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.DepthScale, + new(2522u, 3), 0.001); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.Baseline, + new(2523u, 3), 0.05); + harness.AddValueChild(harness.SensorNodeId, BrowseNames.PointsPerFrame, + new(2524u, 3), 640u * 480u); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionDepth3DSensorSnapshot? snapshot = await sensor.ReadDepthMembersAsync() + .ConfigureAwait(false); + + Assert.That(snapshot, Is.Not.Null); + Assert.Multiple(() => + { + Assert.That(snapshot!.MinDepth, Is.EqualTo(0.1)); + Assert.That(snapshot.MaxDepth, Is.EqualTo(5.0)); + Assert.That(snapshot.DepthScale, Is.EqualTo(0.001)); + Assert.That(snapshot.Baseline, Is.EqualTo(0.05)); + Assert.That(snapshot.PointsPerFrame, Is.EqualTo(640u * 480u)); + }); + } + + [Test] + public async Task ReadOpticsReturnsNullWhenNotPresent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionOpticsSnapshot? optics = await sensor.ReadOpticsAsync() + .ConfigureAwait(false); + + Assert.That(optics, Is.Null); + } + + [Test] + public async Task ReadOpticsReadsMembersWhenPresent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + harness.AddChild(harness.SensorNodeId, BrowseNames.Optics, harness.OpticsNodeId); + harness.AddValueChild(harness.OpticsNodeId, BrowseNames.FocalLength, + new(2530u, 3), 8.0); + harness.AddValueChild(harness.OpticsNodeId, BrowseNames.Aperture, + new(2531u, 3), 2.8); + harness.AddValueChild(harness.OpticsNodeId, BrowseNames.MinimumWorkingDistance, + new(2532u, 3), 0.3); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionOpticsSnapshot? optics = await sensor.ReadOpticsAsync() + .ConfigureAwait(false); + + Assert.That(optics, Is.Not.Null); + Assert.Multiple(() => + { + Assert.That(optics!.NodeId, Is.EqualTo(harness.OpticsNodeId)); + Assert.That(optics.FocalLength, Is.EqualTo(8.0)); + Assert.That(optics.Aperture, Is.EqualTo(2.8)); + Assert.That(optics.WorkingDistance, Is.EqualTo(0.3)); + }); + } + + [Test] + public async Task ReadIlluminationReturnsNullWhenNotPresent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionIlluminationSnapshot? illumination = await sensor.ReadIlluminationAsync() + .ConfigureAwait(false); + + Assert.That(illumination, Is.Null); + } + + [Test] + public async Task ReadIlluminationReadsMembersWhenPresent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + harness.AddChild(harness.SensorNodeId, BrowseNames.Illumination, + harness.IlluminationNodeId); + harness.AddValueChild(harness.IlluminationNodeId, BrowseNames.Wavelength, + new(2540u, 3), 850.0); + harness.AddValueChild(harness.IlluminationNodeId, BrowseNames.RelativeIntensity, + new(2541u, 3), 0.9); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionIlluminationSnapshot? illumination = await sensor.ReadIlluminationAsync() + .ConfigureAwait(false); + + Assert.That(illumination, Is.Not.Null); + Assert.Multiple(() => + { + Assert.That(illumination!.NodeId, Is.EqualTo(harness.IlluminationNodeId)); + Assert.That(illumination.Wavelength, Is.EqualTo(850.0)); + Assert.That(illumination.RelativeIntensity, Is.EqualTo(0.9)); + }); + } + + [Test] + public async Task GetMountedFrameIdReturnsNullNodeIdWhenNotMounted() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + NodeId mount = await sensor.GetMountedFrameIdAsync().ConfigureAwait(false); + + Assert.That(mount.IsNull, Is.True); + } + + [Test] + public async Task GetMountedFrameIdReturnsFrameWhenMountedOnReferenceExists() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + harness.AppendBrowse(harness.SensorNodeId, new ReferenceDescription + { + NodeId = new ExpandedNodeId(harness.FrameNodeId), + BrowseName = new QualifiedName("MyMount", harness.VisionNamespaceIndex), + DisplayName = new LocalizedText("MyMount"), + NodeClass = NodeClass.Object, + TypeDefinition = new ExpandedNodeId( + new NodeId(ObjectTypes.CoordinateFrameType, harness.VisionNamespaceIndex)), + ReferenceTypeId = new NodeId(ReferenceTypes.MountedOn, harness.VisionNamespaceIndex), + IsForward = true + }); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + NodeId mount = await sensor.GetMountedFrameIdAsync().ConfigureAwait(false); + + Assert.That(mount, Is.EqualTo(harness.FrameNodeId)); + } + + [Test] + public async Task EnumerateCalibrationsYieldsCalibrationsFromNestedFolder() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + harness.AddChild(harness.SensorNodeId, BrowseNames.Calibrations, + harness.CalibrationsFolderId); + harness.AddBrowse(harness.CalibrationsFolderId, + [harness.Ref(harness.IntrinsicCalibrationNodeId, "Intrinsic", + ObjectTypes.IntrinsicCalibrationType)]); + + var entries = new List(); + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + await foreach (VisionNodeEntry entry in sensor.EnumerateCalibrationsAsync()) + { + entries.Add(entry); + } + + Assert.That(entries.Count, Is.EqualTo(1)); + Assert.That(entries[0].NodeId, Is.EqualTo(harness.IntrinsicCalibrationNodeId)); + } + + [Test] + public async Task ReadIntrinsicCalibrationThrowsForNullNodeId() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + + Assert.ThrowsAsync(async () => + await sensor.ReadIntrinsicCalibrationAsync(NodeId.Null) + .ConfigureAwait(false)); + } + + [Test] + public async Task ReadIntrinsicCalibrationReturnsPopulatedSnapshot() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + harness.AddValueChild(harness.IntrinsicCalibrationNodeId, BrowseNames.CalibrationId, + new(2600u, 3), "cal-1"); + harness.AddValueChild(harness.IntrinsicCalibrationNodeId, BrowseNames.PerformedAt, + new(2601u, 3), new DateTimeUtc(new System.DateTime(2024, 1, 1))); + harness.AddValueChild(harness.IntrinsicCalibrationNodeId, BrowseNames.Valid, + new(2602u, 3), true); + harness.AddValueChild(harness.IntrinsicCalibrationNodeId, BrowseNames.ResidualError, + new(2603u, 3), 0.15); + harness.AddValueChild(harness.IntrinsicCalibrationNodeId, BrowseNames.Method, + new(2604u, 3), "Zhang"); + + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + VisionIntrinsicCalibrationSnapshot snapshot = await sensor + .ReadIntrinsicCalibrationAsync(harness.IntrinsicCalibrationNodeId) + .ConfigureAwait(false); + + Assert.Multiple(() => + { + Assert.That(snapshot.CalibrationId, Is.EqualTo("cal-1")); + Assert.That(snapshot.Valid, Is.True); + Assert.That(snapshot.ResidualError, Is.EqualTo(0.15)); + Assert.That(snapshot.Method, Is.EqualTo("Zhang")); + Assert.That(snapshot.NodeId, Is.EqualTo(harness.IntrinsicCalibrationNodeId)); + }); + } + + [Test] + public async Task ReadExtrinsicCalibrationThrowsForNullNodeId() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + + Assert.ThrowsAsync(async () => + await sensor.ReadExtrinsicCalibrationAsync(NodeId.Null) + .ConfigureAwait(false)); + } + + [Test] + public async Task OpenMediaReturnsNullWhenNoMediaObjectPresent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + + VisionMediaClient? media = await sensor.OpenMediaAsync() + .ConfigureAwait(false); + + Assert.That(media, Is.Null); + } + + [Test] + public async Task OpenMediaReturnsClientWhenMediaObjectPresent() + { + var harness = new VisionSessionHarness(); + harness.AddSensor(ObjectTypes.ImageSensorType); + harness.AddChild(harness.SensorNodeId, BrowseNames.Media, harness.MediaNodeId); + VisionSensorClient sensor = harness.Client.Sensor(harness.SensorNodeId); + + VisionMediaClient? media = await sensor.OpenMediaAsync() + .ConfigureAwait(false); + + Assert.That(media, Is.Not.Null); + Assert.That(media!.MediaNodeId, Is.EqualTo(harness.MediaNodeId)); + } + + [Test] + public void ConstructorRejectsNullSensorNodeId() + { + var harness = new VisionSessionHarness(); + + Assert.Throws(() => harness.Client.Sensor(NodeId.Null)); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionServerFixture.cs b/tests/Opc.Ua.Vision.Tests/VisionServerFixture.cs new file mode 100644 index 0000000000..3d3c2357e1 --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionServerFixture.cs @@ -0,0 +1,93 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using Opc.Ua.Server; +using Opc.Ua.Server.TestFramework; +using Opc.Ua.Vision.Server; +using Opc.Ua.Vision.Server.Builders; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Boots a StandardServer instance and attaches a + /// so tests can exercise the fluent + /// build context and node-manager lifecycle against a real address + /// space rather than through a bag of mocks. + /// + internal sealed class VisionServerFixture : IAsyncDisposable + { + private ServerFixture? m_fixture; + + public StandardServer Server { get; private set; } = null!; + + public ApplicationConfiguration Configuration => + m_fixture?.Config ?? + throw new InvalidOperationException("The Vision server fixture is not started."); + + public VisionNodeManager Manager { get; private set; } = null!; + + public IVisionBuildContext CreateBuildContext() + { + return Manager.CreateVisionBuildContext(); + } + + public async Task StartAsync() + { + m_fixture = new ServerFixture( + telemetry => new StandardServer(telemetry)) + { + AutoAccept = true, + SecurityNone = true + }; + Server = await m_fixture.StartAsync().ConfigureAwait(false); + + Manager = new VisionNodeManager( + Server.CurrentInstance, + m_fixture.Config); + var externalReferences = new Dictionary>(); + await Manager.CreateAddressSpaceAsync(externalReferences) + .ConfigureAwait(false); + } + + public async ValueTask DisposeAsync() + { + if (Manager != null) + { + await Manager.DisposeAsync().ConfigureAwait(false); + } + if (m_fixture != null) + { + await m_fixture.StopAsync().ConfigureAwait(false); + } + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionServerOptionsTests.cs b/tests/Opc.Ua.Vision.Tests/VisionServerOptionsTests.cs new file mode 100644 index 0000000000..f8ca58c4ee --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionServerOptionsTests.cs @@ -0,0 +1,149 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using NUnit.Framework; +using Opc.Ua.Vision.Server; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Covers — the five failure + /// branches (blank URI, non-absolute URI, blank version, URI clashing + /// with the OPC UA base namespace, URI clashing with the Vision + /// companion namespace) and the happy path. + /// + [TestFixture] + public sealed class VisionServerOptionsTests + { + [Test] + public void DefaultsPassValidateWithoutThrowing() + { + var options = new VisionServerOptions(); + + Assert.That(options.Validate, Throws.Nothing); + } + + [Test] + public void DefaultsExposeTheDocumentedConstants() + { + var options = new VisionServerOptions(); + + Assert.Multiple(() => + { + Assert.That(options.InstanceNamespaceUri, + Is.EqualTo(VisionServerOptions.DefaultInstanceNamespaceUri)); + Assert.That(options.SpecificationVersion, + Is.EqualTo(VisionServerOptions.DefaultSpecificationVersion)); + Assert.That(options.AdditionalFacets.Count, Is.EqualTo(0)); + }); + } + + [Test] + public void ValidateThrowsArgumentExceptionWhenInstanceNamespaceUriIsEmpty() + { + var options = new VisionServerOptions { InstanceNamespaceUri = string.Empty }; + + Assert.That(options.Validate, + Throws.TypeOf() + .With.Property("ParamName").EqualTo("InstanceNamespaceUri")); + } + + [Test] + public void ValidateThrowsArgumentExceptionWhenInstanceNamespaceUriIsWhitespace() + { + var options = new VisionServerOptions { InstanceNamespaceUri = " " }; + + Assert.That(options.Validate, + Throws.TypeOf() + .With.Property("ParamName").EqualTo("InstanceNamespaceUri")); + } + + [Test] + public void ValidateThrowsArgumentExceptionWhenInstanceNamespaceUriIsRelative() + { + var options = new VisionServerOptions { InstanceNamespaceUri = "not/absolute" }; + + Assert.That(options.Validate, + Throws.TypeOf() + .With.Property("ParamName").EqualTo("InstanceNamespaceUri")); + } + + [Test] + public void ValidateThrowsArgumentExceptionWhenSpecificationVersionIsEmpty() + { + var options = new VisionServerOptions + { + InstanceNamespaceUri = "urn:custom:instances", + SpecificationVersion = string.Empty + }; + + Assert.That(options.Validate, + Throws.TypeOf() + .With.Property("ParamName").EqualTo("SpecificationVersion")); + } + + [Test] + public void ValidateThrowsBadConfigurationErrorWhenNamespaceEqualsOpcUaBase() + { + var options = new VisionServerOptions + { + InstanceNamespaceUri = Namespaces.OpcUa + }; + + ServiceResultException ex = Assert.Throws(options.Validate)!; + Assert.That(ex.StatusCode, Is.EqualTo(StatusCodes.BadConfigurationError)); + } + + [Test] + public void ValidateThrowsBadConfigurationErrorWhenNamespaceEqualsVisionModel() + { + var options = new VisionServerOptions + { + InstanceNamespaceUri = global::Opc.Ua.Vision.Namespaces.Vision + }; + + ServiceResultException ex = Assert.Throws(options.Validate)!; + Assert.That(ex.StatusCode, Is.EqualTo(StatusCodes.BadConfigurationError)); + } + + [Test] + public void ValidateAcceptsCustomAdditionalFacets() + { + var options = new VisionServerOptions + { + InstanceNamespaceUri = "urn:custom:vision:instances", + AdditionalFacets = new[] { "VIS-Custom" }.ToArrayOf() + }; + + Assert.That(options.Validate, Throws.Nothing); + Assert.That(options.AdditionalFacets.Count, Is.EqualTo(1)); + } + } +} diff --git a/tests/Opc.Ua.Vision.Tests/VisionSessionHarness.cs b/tests/Opc.Ua.Vision.Tests/VisionSessionHarness.cs new file mode 100644 index 0000000000..7ae6a05c9d --- /dev/null +++ b/tests/Opc.Ua.Vision.Tests/VisionSessionHarness.cs @@ -0,0 +1,410 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Moq; +using Opc.Ua.Client; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Vision.Tests +{ + /// + /// Reusable mock session harness for exercising the Vision client + /// facades. Wires namespace tables, message context, browse-path + /// translation, browse, read and call handlers over dictionary-backed + /// stores so a caller can populate the address space by BrowseName + /// and read raw values without booting a real + /// server. + /// + internal sealed class VisionSessionHarness + { + private readonly Dictionary<(NodeId Parent, string BrowseName), NodeId> m_children = []; + private readonly Dictionary> m_browse = []; + private readonly Dictionary m_values = []; + private readonly Dictionary m_valueStatus = []; + private StatusCode m_callStatus = StatusCodes.Good; + private ArrayOf m_callOutput = ArrayOf.Empty; + + public VisionSessionHarness() + { + Telemetry = new Mock().Object; + NamespaceUris.GetIndexOrAppend(Opc.Ua.Namespaces.OpcUa); + NamespaceUris.GetIndexOrAppend(global::Opc.Ua.Vision.Namespaces.Vision); + MessageContext = ServiceMessageContext.Create(Telemetry); + MessageContext.NamespaceUris.GetIndexOrAppend(Opc.Ua.Namespaces.OpcUa); + MessageContext.NamespaceUris.GetIndexOrAppend(global::Opc.Ua.Vision.Namespaces.Vision); + Session.SetupGet(s => s.NamespaceUris).Returns(NamespaceUris); + Session.SetupGet(s => s.MessageContext).Returns(MessageContext); + Session.SetupGet(s => s.Factory).Returns(MessageContext.Factory); + Session.SetupGet(s => s.OperationLimits).Returns(new OperationLimits()); + Session.SetupGet(s => s.ServerCapabilities).Returns(new ServerCapabilities()); + Session.SetupGet(s => s.ContinuationPointPolicy).Returns(ContinuationPointPolicy.Default); + Session.SetupGet(s => s.NodeCache).Returns(NodeCache.Object); + NodeCache + .Setup(c => c.IsTypeOfAsync( + It.IsAny(), It.IsAny(), It.IsAny())) + .Returns(new ValueTask(true)); + SetupTranslate(); + SetupBrowse(); + SetupRead(); + SetupCall(); + Client = new VisionClient(Session.Object, Telemetry); + } + + public Mock Session { get; } = new(MockBehavior.Loose); + + public Mock NodeCache { get; } = new(MockBehavior.Loose); + + public ITelemetryContext Telemetry { get; } + + public NamespaceTable NamespaceUris { get; } = new(); + + public ServiceMessageContext MessageContext { get; } + + public VisionClient Client { get; } + + public ushort VisionNamespaceIndex => + (ushort)NamespaceUris.GetIndex(global::Opc.Ua.Vision.Namespaces.Vision); + + public NodeId VisionRootId => NodeId.Create( + Objects.Vision, global::Opc.Ua.Vision.Namespaces.Vision, NamespaceUris); + + public NodeId SensorsFolderId => NodeId.Create( + Objects.Vision_Sensors, global::Opc.Ua.Vision.Namespaces.Vision, NamespaceUris); + + public NodeId PipelinesFolderId { get; } = new(1001u, 3); + + public NodeId FramesFolderId { get; } = new(1002u, 3); + + public NodeId SensorNodeId { get; } = new(2000u, 3); + + public NodeId PipelineNodeId { get; } = new(3000u, 3); + + public NodeId FeedbackNodeId { get; } = new(3100u, 3); + + public NodeId ResultNodeId { get; } = new(3200u, 3); + + public NodeId FrameNodeId { get; } = new(4000u, 3); + + public NodeId OpticsNodeId { get; } = new(2100u, 3); + + public NodeId IlluminationNodeId { get; } = new(2101u, 3); + + public NodeId CalibrationsFolderId { get; } = new(2200u, 3); + + public NodeId IntrinsicCalibrationNodeId { get; } = new(2201u, 3); + + public NodeId ExtrinsicCalibrationNodeId { get; } = new(2202u, 3); + + public NodeId MediaNodeId { get; } = new(2300u, 3); + + public NodeId StreamEndpointsFolderId { get; } = new(2310u, 3); + + public NodeId ClipEndpointsFolderId { get; } = new(2320u, 3); + + public NodeId StreamEndpointNodeId { get; } = new(2311u, 3); + + public NodeId ClipEndpointNodeId { get; } = new(2321u, 3); + + public NodeId ResultsFolderId { get; } = new(3300u, 3); + + public NodeId InferenceResultNodeId { get; } = new(3301u, 3); + + /// + /// Populates the Vision root's Pipelines and Frames folders so + /// GetPipelinesFolderIdAsync and GetFramesFolderIdAsync return + /// non-null NodeIds when consumers call them. + /// + public void ConfigureVisionFolders() + { + AddChild(VisionRootId, BrowseNames.Pipelines, PipelinesFolderId); + AddChild(VisionRootId, BrowseNames.Frames, FramesFolderId); + } + + /// + /// Populates a single sensor with the given type definition under + /// the Vision/Sensors folder, browse-reachable both by browse-path + /// and by hierarchical browse. Returns the sensor NodeId. + /// + public NodeId AddSensor(uint typeDefinition, string browseName = "Sensor1") + { + AddBrowse(SensorsFolderId, + [Ref(SensorNodeId, browseName, typeDefinition)]); + return SensorNodeId; + } + + /// + /// Populates a pipeline browse-reachable under Vision/Pipelines. + /// + public NodeId AddPipeline(string browseName = "Pipeline1") + { + AddBrowse(PipelinesFolderId, + [Ref(PipelineNodeId, browseName, ObjectTypes.InferencePipelineType)]); + return PipelineNodeId; + } + + /// + /// Populates a coordinate frame browse-reachable under Vision/Frames. + /// + public NodeId AddFrame(string browseName = "Frame1") + { + AddBrowse(FramesFolderId, + [Ref(FrameNodeId, browseName, ObjectTypes.CoordinateFrameType)]); + return FrameNodeId; + } + + public void ConfigureCall(StatusCode status, params Variant[] outputs) + { + m_callStatus = status; + m_callOutput = outputs?.ToArrayOf() ?? ArrayOf.Empty; + } + + public ReferenceDescription Ref(NodeId nodeId, string browseName, uint typeId) + { + return new ReferenceDescription + { + NodeId = new ExpandedNodeId(nodeId), + BrowseName = new QualifiedName(browseName, VisionNamespaceIndex), + DisplayName = new LocalizedText(browseName), + NodeClass = NodeClass.Object, + TypeDefinition = new ExpandedNodeId( + new NodeId(typeId, VisionNamespaceIndex)), + ReferenceTypeId = Opc.Ua.ReferenceTypeIds.HierarchicalReferences, + IsForward = true + }; + } + + public void AddBrowse(NodeId folder, IReadOnlyList references) + { + m_browse[folder] = [.. references]; + } + + public void AppendBrowse(NodeId folder, ReferenceDescription reference) + { + if (!m_browse.TryGetValue(folder, out List? list)) + { + list = []; + m_browse[folder] = list; + } + list.Add(reference); + } + + public void AddChild(NodeId parent, string browseName, NodeId child) + { + m_children[(parent, browseName)] = child; + } + + public void AddValue(NodeId nodeId, Variant value) + { + m_values[nodeId] = value; + } + + public void AddValueStatus(NodeId nodeId, StatusCode statusCode) + { + m_valueStatus[nodeId] = statusCode; + } + + public void AddValueChild(NodeId parent, string browseName, NodeId nodeId, Variant value) + { + AddChild(parent, browseName, nodeId); + AddValue(nodeId, value); + } + + private void SetupTranslate() + { + Session.Setup(s => s.TranslateBrowsePathsToNodeIdsAsync( + It.IsAny(), + It.IsAny>(), + It.IsAny())) + .Returns, CancellationToken>((_, paths, _) => + { + var results = new List(); + for (int ii = 0; ii < paths.Count; ii++) + { + BrowsePath path = paths[ii]; + NodeId current = path.StartingNode; + bool found = true; + for (int jj = 0; jj < path.RelativePath.Elements.Count; jj++) + { + string name = path.RelativePath.Elements[jj].TargetName.Name ?? string.Empty; + if (!m_children.TryGetValue((current, name), out NodeId next)) + { + found = false; + break; + } + current = next; + } + results.Add(found ? GoodPath(current) : BadPath()); + } + return new ValueTask( + new TranslateBrowsePathsToNodeIdsResponse + { + ResponseHeader = new ResponseHeader(), + Results = results.ToArrayOf(), + DiagnosticInfos = default + }); + }); + } + + private void SetupBrowse() + { + Session.Setup(s => s.BrowseAsync( + It.IsAny(), + It.IsAny(), + It.IsAny(), + It.IsAny>(), + It.IsAny())) + .Returns, CancellationToken>( + (_, _, _, descriptions, _) => + { + var results = new List(descriptions.Count); + for (int ii = 0; ii < descriptions.Count; ii++) + { + BrowseDescription description = descriptions[ii]; + List refs = m_browse.TryGetValue( + description.NodeId, out List? value) + ? value + : []; + results.Add(new BrowseResult + { + StatusCode = StatusCodes.Good, + References = refs.ToArrayOf(), + ContinuationPoint = default + }); + } + return new ValueTask(new BrowseResponse + { + ResponseHeader = new ResponseHeader(), + Results = results.ToArrayOf(), + DiagnosticInfos = default + }); + }); + Session.Setup(s => s.BrowseNextAsync( + It.IsAny(), + It.IsAny(), + It.IsAny>(), + It.IsAny())) + .Returns(new ValueTask(new BrowseNextResponse + { + ResponseHeader = new ResponseHeader(), + Results = [new BrowseResult { StatusCode = StatusCodes.Good, References = [] }], + DiagnosticInfos = default + })); + } + + private void SetupRead() + { + Session.Setup(s => s.ReadAsync( + It.IsAny(), + It.IsAny(), + It.IsAny(), + It.IsAny>(), + It.IsAny())) + .Returns, CancellationToken>( + (_, _, _, nodes, _) => + { + var values = new List(); + for (int ii = 0; ii < nodes.Count; ii++) + { + ReadValueId node = nodes[ii]; + if (node.AttributeId == Attributes.Value) + { + if (m_valueStatus.TryGetValue(node.NodeId, out StatusCode status) && + StatusCode.IsBad(status)) + { + values.Add(new DataValue(Variant.Null, status)); + } + else if (m_values.TryGetValue(node.NodeId, out Variant variant)) + { + values.Add(new DataValue(variant, StatusCodes.Good, + DateTime.UtcNow, DateTime.UtcNow)); + } + else + { + values.Add(new DataValue(Variant.Null, StatusCodes.BadNodeIdUnknown)); + } + } + else + { + values.Add(new DataValue(Variant.Null, StatusCodes.BadNotSupported)); + } + } + return new ValueTask(new ReadResponse + { + ResponseHeader = new ResponseHeader(), + Results = values.ToArrayOf(), + DiagnosticInfos = default + }); + }); + } + + private void SetupCall() + { + Session.Setup(s => s.CallAsync( + It.IsAny(), + It.IsAny>(), + It.IsAny())) + .Returns, CancellationToken>((_, _, _) => + new ValueTask(new CallResponse + { + ResponseHeader = new ResponseHeader(), + Results = + [ + new CallMethodResult + { + StatusCode = m_callStatus, + OutputArguments = m_callOutput + } + ], + DiagnosticInfos = default + })); + } + + private static BrowsePathResult GoodPath(NodeId nodeId) + { + return new BrowsePathResult + { + StatusCode = StatusCodes.Good, + Targets = [new BrowsePathTarget { TargetId = new ExpandedNodeId(nodeId) }] + }; + } + + private static BrowsePathResult BadPath() + { + return new BrowsePathResult { StatusCode = StatusCodes.BadNoMatch, Targets = [] }; + } + } +} diff --git a/tests/Opc.Ua.VisualInspection.Tests/Opc.Ua.VisualInspection.Tests.csproj b/tests/Opc.Ua.VisualInspection.Tests/Opc.Ua.VisualInspection.Tests.csproj new file mode 100644 index 0000000000..a0a0f8985b --- /dev/null +++ b/tests/Opc.Ua.VisualInspection.Tests/Opc.Ua.VisualInspection.Tests.csproj @@ -0,0 +1,32 @@ + + + $(TestsTargetFrameworks) + Opc.Ua.VisualInspection.Tests + Opc.Ua.VisualInspection.Tests + enable + false + $(NoWarn);CS1591;CA2007;CA2000;CA1014;CA1861;CA1859;NUnit2026;NUnit4002;NUnit2046;RCS1174 + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + all + runtime; build; native; contentfiles; analyzers + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + diff --git a/tests/Opc.Ua.VisualInspection.Tests/VisualInspectionFixtureGeometryTests.cs b/tests/Opc.Ua.VisualInspection.Tests/VisualInspectionFixtureGeometryTests.cs new file mode 100644 index 0000000000..7264ff72d6 --- /dev/null +++ b/tests/Opc.Ua.VisualInspection.Tests/VisualInspectionFixtureGeometryTests.cs @@ -0,0 +1,547 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.IO.Compression; +using System.Linq; +using NUnit.Framework; + +namespace Opc.Ua.VisualInspection.Tests +{ + [TestFixture] + [Category("Vision")] + public sealed class VisualInspectionFixtureGeometryTests + { + [TestCaseSource(nameof(FixtureGeometryCases))] + public void FixtureImagesMeasureTheDesignedBoreAndSlotGeometry( + string imageName, + double expectedBoreMm, + double expectedSlotMm) + { + PngImage image = PngImage.Load(GetFixturePath(imageName)); + FeatureMeasurements measurements = MeasureFeatures(image); + + // These assertions prevent regenerated fixtures from silently changing the sample's verdicts. + Assert.Multiple(() => + { + Assert.That(measurements.BoreDiameterMm, Is.EqualTo(expectedBoreMm).Within(0.000_001), + $"the decoded bore diameter must remain {FormatMm(expectedBoreMm)} mm"); + Assert.That(measurements.SlotWidthMm, Is.EqualTo(expectedSlotMm).Within(0.000_001), + $"the decoded slot width must remain {FormatMm(expectedSlotMm)} mm"); + }); + } + + [TestCase("bracket-ok.png", VisualInspectionVerdict.Ok)] + [TestCase("bracket-not-ok.png", VisualInspectionVerdict.NotOk)] + [TestCase("bracket-ambiguous.png", VisualInspectionVerdict.NotDecidable)] + public void FixtureMeasurementsProduceTheDocumentedPartVerdicts( + string imageName, + VisualInspectionVerdict expectedVerdict) + { + PngImage image = PngImage.Load(GetFixturePath(imageName)); + FeatureMeasurements measurements = MeasureFeatures(image); + + IReadOnlyList actual = new[] + { + new CharacteristicMeasurement(BoreDiameter, measurements.BoreDiameterMm, PixelPitchMm), + new CharacteristicMeasurement(SlotWidth, measurements.SlotWidthMm, PixelPitchMm), + new CharacteristicMeasurement(EdgeOffset, 20.00, PixelPitchMm) + }; + + // The fixture pixels are the real camera measurements that drive the sample's intended outcome. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(expectedVerdict)); + } + + [Test] + public void IntervalTouchingLimitFromInsideIsOk() + { + IReadOnlyList actual = ReplaceOkMeasurement( + new CharacteristicMeasurement(BoreDiameter, 12.10, 0.10)); + + // Inclusive tolerance limits make an exactly in-band measurement a pass, not an ambiguity. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(VisualInspectionVerdict.Ok)); + } + + [Test] + public void IntervalTouchingLimitFromOutsideIsNotDecidable() + { + IReadOnlyList actual = ReplaceOkMeasurement( + new CharacteristicMeasurement(BoreDiameter, 12.30, 0.10)); + + // The interval still contains the upper limit, so it is not wholly outside and must escalate. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(VisualInspectionVerdict.NotDecidable)); + } + + [Test] + public void NegativeUncertaintyCannotNarrowIntervalIntoPass() + { + IReadOnlyList actual = ReplaceOkMeasurement( + new CharacteristicMeasurement(SlotWidth, 8.20, -0.05)); + + // A caller controls uncertainty; treating it as signed could report this out-of-tolerance slot as good. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(VisualInspectionVerdict.NotDecidable)); + } + + [Test] + public void EnormousUncertaintyMakesVerdictNotDecidable() + { + IReadOnlyList actual = ReplaceOkMeasurement( + new CharacteristicMeasurement(BoreDiameter, 12.00, 100.00)); + + // Large uncertainty must widen the interval and force escalation rather than being trusted as a pass. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(VisualInspectionVerdict.NotDecidable)); + } + + [Test] + public void MissingRequiredCharacteristicMakesVerdictNotDecidable() + { + IReadOnlyList actual = new[] + { + new CharacteristicMeasurement(BoreDiameter, 12.00, PixelPitchMm), + new CharacteristicMeasurement(SlotWidth, 8.00, PixelPitchMm) + }; + + // The recipe requires EdgeOffset; no measurement is an escalation, not a silent pass. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(VisualInspectionVerdict.NotDecidable)); + } + + [Test] + public void WorstOfOrderingNotOkAmongOkCharacteristicsWins() + { + IReadOnlyList actual = ReplaceOkMeasurement( + new CharacteristicMeasurement(BoreDiameter, 12.60, PixelPitchMm)); + + // Any confirmed failing characteristic makes the whole part fail. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(VisualInspectionVerdict.NotOk)); + } + + [Test] + public void WorstOfOrderingNotDecidableAmongOkCharacteristicsWins() + { + IReadOnlyList actual = ReplaceOkMeasurement( + new CharacteristicMeasurement(SlotWidth, 8.10, PixelPitchMm)); + + // One ambiguous characteristic among otherwise good measurements must still escalate the whole part. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(VisualInspectionVerdict.NotDecidable)); + } + + [Test] + public void WorstOfOrderingNotOkBeatsNotDecidable() + { + IReadOnlyList actual = new[] + { + new CharacteristicMeasurement(BoreDiameter, 12.60, PixelPitchMm), + new CharacteristicMeasurement(SlotWidth, 8.10, PixelPitchMm), + new CharacteristicMeasurement(EdgeOffset, 20.00, PixelPitchMm) + }; + + // Reading of "worst": a confirmed failure is more severe than an escalation request. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(VisualInspectionVerdict.NotOk)); + } + + [Test] + public void FloatingPointStressAtLowerToleranceLimitIsNotDecidable() + { + IReadOnlyList actual = ReplaceOkMeasurement( + new CharacteristicMeasurement(BoreDiameter, 11.70, 0.10)); + + // This reachable 11.7 +/- 0.1 case touches 11.8; integer micrometres avoid double drift to NotOk. + Assert.That(LocalVerdictRule.EvaluatePart(actual), Is.EqualTo(VisualInspectionVerdict.NotDecidable)); + } + + private static IEnumerable FixtureGeometryCases() + { + yield return new TestCaseData("bracket-ok.png", 12.00, 8.00); + yield return new TestCaseData("bracket-not-ok.png", 12.60, 8.00); + yield return new TestCaseData("bracket-ambiguous.png", 12.00, 8.10); + } + + private static FeatureMeasurements MeasureFeatures(PngImage image) + { + IReadOnlyList boreRuns = FindDarkRuns(image, BoreCenterY, BoreSearchStartX, BoreSearchEndX); + IReadOnlyList slotRuns = FindDarkRuns(image, SlotCenterY, SlotSearchStartX, SlotSearchEndX); + + PixelRun boreRun = boreRuns.OrderByDescending(run => run.Length).First(); + PixelRun slotRun = slotRuns.OrderByDescending(run => run.Length).First(); + + return new FeatureMeasurements(boreRun.Length / PixelsPerMillimetre, slotRun.Length / PixelsPerMillimetre); + } + + private static IReadOnlyList FindDarkRuns(PngImage image, int y, int startX, int endX) + { + var runs = new List(); + int? runStart = null; + for (int x = startX; x <= endX; x++) + { + bool isDark = image.IsDark(x, y); + if (isDark && runStart is null) + { + runStart = x; + } + else if (!isDark && runStart is not null) + { + runs.Add(new PixelRun(runStart.Value, x - 1)); + runStart = null; + } + } + + if (runStart is not null) + { + runs.Add(new PixelRun(runStart.Value, endX)); + } + + Assert.That(runs, Is.Not.Empty, "each fixture must contain the dark feature being measured"); + return runs; + } + + private static IReadOnlyList ReplaceOkMeasurement( + CharacteristicMeasurement replacement) + { + List measurements = CreateOkMeasurements().ToList(); + int index = measurements.FindIndex(measurement => measurement.Name == replacement.Name); + Assert.That(index, Is.GreaterThanOrEqualTo(0), "test setup must replace a recipe characteristic"); + measurements[index] = replacement; + return measurements; + } + + private static IReadOnlyList CreateOkMeasurements() + { + return new[] + { + new CharacteristicMeasurement(BoreDiameter, 12.00, PixelPitchMm), + new CharacteristicMeasurement(SlotWidth, 8.00, PixelPitchMm), + new CharacteristicMeasurement(EdgeOffset, 20.00, PixelPitchMm) + }; + } + + private static string GetFixturePath(string imageName) + { + return Path.Combine(TestContext.CurrentContext.TestDirectory, "Fixtures", imageName); + } + + private static string FormatMm(double value) + { + return value.ToString("0.00", CultureInfo.InvariantCulture); + } + + private const double PixelsPerMillimetre = 10.00; + private const double PixelPitchMm = 0.10; + private const int BoreCenterY = 300; + private const int SlotCenterY = 300; + private const int BoreSearchStartX = 200; + private const int BoreSearchEndX = 380; + private const int SlotSearchStartX = 400; + private const int SlotSearchEndX = 560; + private const string BoreDiameter = "BoreDiameter"; + private const string SlotWidth = "SlotWidth"; + private const string EdgeOffset = "EdgeOffset"; + } + + public enum VisualInspectionVerdict + { + Ok, + NotDecidable, + NotOk + } + + internal static class LocalVerdictRule + { + public static VisualInspectionVerdict EvaluatePart(IReadOnlyList measurements) + { + var lookup = measurements.ToDictionary(measurement => measurement.Name, StringComparer.Ordinal); + VisualInspectionVerdict worst = VisualInspectionVerdict.Ok; + for (int ii = 0; ii < s_recipe.Length; ii++) + { + VisualInspectionVerdict verdict = lookup.TryGetValue( + s_recipe[ii].Name, + out CharacteristicMeasurement measurement) + ? EvaluateCharacteristic(s_recipe[ii], measurement) + : VisualInspectionVerdict.NotDecidable; + + if (verdict > worst) + { + worst = verdict; + } + } + + return worst; + } + + private static VisualInspectionVerdict EvaluateCharacteristic( + CharacteristicRecipe recipe, + CharacteristicMeasurement measurement) + { + int actual = ToMicrometres(measurement.ActualMm); + int uncertainty = Math.Abs(ToMicrometres(measurement.UncertaintyMm)); + int actualLower = actual - uncertainty; + int actualUpper = actual + uncertainty; + int toleranceLower = ToMicrometres(recipe.NominalMm - recipe.LowerToleranceMm); + int toleranceUpper = ToMicrometres(recipe.NominalMm + recipe.UpperToleranceMm); + + if (actualLower >= toleranceLower && actualUpper <= toleranceUpper) + { + return VisualInspectionVerdict.Ok; + } + + if (actualUpper < toleranceLower || actualLower > toleranceUpper) + { + return VisualInspectionVerdict.NotOk; + } + + return VisualInspectionVerdict.NotDecidable; + } + + private static int ToMicrometres(double value) + { + return checked((int)Math.Round(value * 1000.00, MidpointRounding.AwayFromZero)); + } + + private static readonly CharacteristicRecipe[] s_recipe = + { + new CharacteristicRecipe("BoreDiameter", 12.00, 0.20, 0.20), + new CharacteristicRecipe("SlotWidth", 8.00, 0.15, 0.15), + new CharacteristicRecipe("EdgeOffset", 20.00, 0.25, 0.25) + }; + } + + internal sealed class PngImage + { + private PngImage(int width, int height, byte[] pixels) + { + Width = width; + Height = height; + m_pixels = pixels; + } + + public int Width { get; } + + public int Height { get; } + + public static PngImage Load(string path) + { + byte[] bytes = File.ReadAllBytes(path); + int position = PngSignature.Length; + int width = 0; + int height = 0; + byte[] idat = Array.Empty(); + + while (position < bytes.Length) + { + int length = ReadBigEndianInt32(bytes, position); + string chunkType = System.Text.Encoding.ASCII.GetString(bytes, position + 4, 4); + int dataOffset = position + 8; + + if (chunkType == "IHDR") + { + width = ReadBigEndianInt32(bytes, dataOffset); + height = ReadBigEndianInt32(bytes, dataOffset + 4); + Assert.That(bytes[dataOffset + 8], Is.EqualTo(8), "fixtures must be 8-bit PNGs"); + Assert.That(bytes[dataOffset + 9], Is.EqualTo(2), "fixtures must be true-colour RGB PNGs"); + Assert.That(bytes[dataOffset + 12], Is.EqualTo(0), "fixtures must not be interlaced"); + } + else if (chunkType == "IDAT") + { + byte[] next = new byte[idat.Length + length]; + Buffer.BlockCopy(idat, 0, next, 0, idat.Length); + Buffer.BlockCopy(bytes, dataOffset, next, idat.Length, length); + idat = next; + } + else if (chunkType == "IEND") + { + break; + } + + position = dataOffset + length + PngCrcLength; + } + + return new PngImage(width, height, DecodePixels(idat, width, height)); + } + + public bool IsDark(int x, int y) + { + int offset = ((y * Width) + x) * BytesPerPixel; + return m_pixels[offset] < DarkThreshold && + m_pixels[offset + 1] < DarkThreshold && + m_pixels[offset + 2] < DarkThreshold; + } + + private static byte[] DecodePixels(byte[] compressed, int width, int height) + { + byte[] raw = InflateZlib(compressed); + int stride = width * BytesPerPixel; + var pixels = new byte[height * stride]; + var previous = new byte[stride]; + int sourceOffset = 0; + + for (int y = 0; y < height; y++) + { + byte filter = raw[sourceOffset++]; + var scanline = new byte[stride]; + Buffer.BlockCopy(raw, sourceOffset, scanline, 0, stride); + sourceOffset += stride; + Unfilter(scanline, previous, filter); + Buffer.BlockCopy(scanline, 0, pixels, y * stride, stride); + previous = scanline; + } + + return pixels; + } + + private static byte[] InflateZlib(byte[] compressed) + { + using var source = new MemoryStream(compressed, ZlibHeaderLength, compressed.Length - ZlibWrapperLength); + using var inflater = new DeflateStream(source, CompressionMode.Decompress); + using var output = new MemoryStream(); + inflater.CopyTo(output); + return output.ToArray(); + } + + private static void Unfilter(byte[] scanline, byte[] previous, byte filter) + { + for (int x = 0; x < scanline.Length; x++) + { + int left = x >= BytesPerPixel ? scanline[x - BytesPerPixel] : 0; + int up = previous[x]; + int upLeft = x >= BytesPerPixel ? previous[x - BytesPerPixel] : 0; + int predictor = filter switch + { + 0 => 0, + 1 => left, + 2 => up, + 3 => (left + up) / 2, + 4 => PaethPredictor(left, up, upLeft), + _ => throw new InvalidDataException("Unsupported PNG filter type.") + }; + + scanline[x] = unchecked((byte)(scanline[x] + predictor)); + } + } + + private static int PaethPredictor(int left, int up, int upLeft) + { + int estimate = left + up - upLeft; + int leftDistance = Math.Abs(estimate - left); + int upDistance = Math.Abs(estimate - up); + int upLeftDistance = Math.Abs(estimate - upLeft); + + if (leftDistance <= upDistance && leftDistance <= upLeftDistance) + { + return left; + } + + return upDistance <= upLeftDistance ? up : upLeft; + } + + private static int ReadBigEndianInt32(byte[] bytes, int offset) + { + return (bytes[offset] << 24) | + (bytes[offset + 1] << 16) | + (bytes[offset + 2] << 8) | + bytes[offset + 3]; + } + + private readonly byte[] m_pixels; + private static readonly byte[] PngSignature = { 137, 80, 78, 71, 13, 10, 26, 10 }; + private const int BytesPerPixel = 3; + private const int DarkThreshold = 64; + private const int PngCrcLength = 4; + private const int ZlibHeaderLength = 2; + private const int ZlibWrapperLength = 6; + } + + internal readonly struct CharacteristicRecipe + { + public CharacteristicRecipe( + string name, + double nominalMm, + double lowerToleranceMm, + double upperToleranceMm) + { + Name = name; + NominalMm = nominalMm; + LowerToleranceMm = lowerToleranceMm; + UpperToleranceMm = upperToleranceMm; + } + + public string Name { get; } + + public double NominalMm { get; } + + public double LowerToleranceMm { get; } + + public double UpperToleranceMm { get; } + } + + internal readonly struct CharacteristicMeasurement + { + public CharacteristicMeasurement(string name, double actualMm, double uncertaintyMm) + { + Name = name; + ActualMm = actualMm; + UncertaintyMm = uncertaintyMm; + } + + public string Name { get; } + + public double ActualMm { get; } + + public double UncertaintyMm { get; } + } + + internal readonly struct FeatureMeasurements + { + public FeatureMeasurements(double boreDiameterMm, double slotWidthMm) + { + BoreDiameterMm = boreDiameterMm; + SlotWidthMm = slotWidthMm; + } + + public double BoreDiameterMm { get; } + + public double SlotWidthMm { get; } + } + + internal readonly struct PixelRun + { + public PixelRun(int startX, int endX) + { + StartX = startX; + EndX = endX; + } + + public int StartX { get; } + + public int EndX { get; } + + public int Length => EndX - StartX + 1; + } +} diff --git a/tools/Opc.Ua.Mcp.Core/McpToolProfile.cs b/tools/Opc.Ua.Mcp.Core/McpToolProfile.cs index c3eb20ca63..d19a1d3a68 100644 --- a/tools/Opc.Ua.Mcp.Core/McpToolProfile.cs +++ b/tools/Opc.Ua.Mcp.Core/McpToolProfile.cs @@ -64,6 +64,12 @@ public enum McpToolProfile /// Robotics, + /// + /// Vision discovery, monitoring, seeing (image content), inference, + /// feedback, and coordinate-frame tools. + /// + Vision, + /// /// Every available tool, preserving the current-major default catalog. /// diff --git a/tools/Opc.Ua.Mcp.Core/McpToolProfileSet.cs b/tools/Opc.Ua.Mcp.Core/McpToolProfileSet.cs new file mode 100644 index 0000000000..e35e424021 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Core/McpToolProfileSet.cs @@ -0,0 +1,349 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Diagnostics.CodeAnalysis; +using System.Globalization; +using System.Numerics; +using System.Text; + +namespace Opc.Ua.Mcp +{ + /// + /// A set of values a host wants to expose, + /// so a single MCP server can carry the tools of several bounded profiles + /// at once. + /// + /// + /// This is the composition primitive an application uses when it needs + /// tools from more than one profile - a vision-guided pick-and-place agent, + /// for example, that has to both look at a camera through the + /// tools and command a robot through + /// the tools. The set is passed to the + /// McpToolProfileSet overloads of each package's + /// With…Tools extension so the packages together register the tools + /// of every selected profile exactly once - notably including the + /// ConnectionTools that every session-scoped profile needs but that + /// would otherwise be registered several times. + /// + public readonly struct McpToolProfileSet : IEquatable + { + private const char kListSeparatorComma = ','; + private const char kListSeparatorPlus = '+'; + private const char kListSeparatorSemicolon = ';'; + private const char kListSeparatorPipe = '|'; + + /// + /// Creates a set from a single profile. + /// + /// The profile to include. + /// + /// is not a defined profile. + /// + public McpToolProfileSet(McpToolProfile profile) + { + ValidateProfile(profile); + Bits = ToBit(profile); + } + + /// + /// Creates a set from a sequence of profiles. Duplicates are collapsed. + /// + /// The profiles to include. + /// + /// is null. + /// + /// + /// contains a value that is not a defined profile. + /// + public McpToolProfileSet(IEnumerable profiles) + { + ArgumentNullException.ThrowIfNull(profiles); + uint bits = 0; + foreach (McpToolProfile profile in profiles) + { + ValidateProfile(profile); + bits |= ToBit(profile); + } + Bits = bits; + } + + private McpToolProfileSet(uint bits) + { + Bits = bits; + } + + /// + /// The empty set - no profiles selected. + /// + public static McpToolProfileSet Empty => default; + + /// + /// Whether the set is empty. + /// + public bool IsEmpty => Bits == 0; + + /// + /// The number of distinct profiles in the set. + /// + public int Count => BitOperations.PopCount(Bits); + + /// + /// Whether is a member of the set. + /// + /// The profile to test for membership. + /// + /// true when the set contains . + /// + public bool Contains(McpToolProfile profile) + { + return Enum.IsDefined(profile) && (Bits & ToBit(profile)) != 0; + } + + /// + /// Returns the same set with added. + /// + /// The profile to add. + /// A set containing the union. + /// + /// is not a defined profile. + /// + public McpToolProfileSet With(McpToolProfile profile) + { + ValidateProfile(profile); + return new McpToolProfileSet(Bits | ToBit(profile)); + } + + /// + /// Enumerates the profiles in the set in + /// declaration order. + /// + /// The profiles in the set. + public IEnumerable Enumerate() + { + uint bits = Bits; + foreach (McpToolProfile profile in Enum.GetValues()) + { + if ((bits & ToBit(profile)) != 0) + { + yield return profile; + } + } + } + + /// + /// Parses a comma or plus separated list of profile names + /// (case-insensitive). Duplicates are collapsed. + /// + /// The text to parse. + /// The parsed set. + /// + /// is null. + /// + /// + /// is empty or contains a token that is not + /// a profile name. + /// + public static McpToolProfileSet Parse(string value) + { + ArgumentNullException.ThrowIfNull(value); + if (!TryParse(value, out McpToolProfileSet set, out string? error)) + { + throw new FormatException(error); + } + return set; + } + + /// + /// Tries to parse a comma or plus separated list of profile names + /// (case-insensitive). Duplicates are collapsed. + /// + /// The text to parse. + /// The parsed set on success. + /// + /// true when the text is a well-formed list of known profile + /// names; false when the text is null, empty, or names + /// an unknown profile. + /// + public static bool TryParse( + [NotNullWhen(true)] string? value, + out McpToolProfileSet set) + { + return TryParse(value, out set, out _); + } + + /// + /// Tries to parse a comma or plus separated list of profile names + /// (case-insensitive) and reports the parse error on failure. + /// + /// The text to parse. + /// The parsed set on success. + /// The parse error on failure. + /// + /// true when the text is a well-formed list of known profile + /// names. + /// + public static bool TryParse( + [NotNullWhen(true)] string? value, + out McpToolProfileSet set, + [NotNullWhen(false)] out string? error) + { + set = default; + if (string.IsNullOrWhiteSpace(value)) + { + error = "The tool profile list is empty."; + return false; + } + + uint bits = 0; + string[] tokens = value.Split( + [ + kListSeparatorComma, + kListSeparatorPlus, + kListSeparatorSemicolon, + kListSeparatorPipe, + ' ', + '\t' + ], + StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries); + if (tokens.Length == 0) + { + error = "The tool profile list is empty."; + return false; + } + + foreach (string token in tokens) + { + if (!Enum.TryParse(token, ignoreCase: true, out McpToolProfile profile) || + !Enum.IsDefined(profile)) + { + error = string.Format( + CultureInfo.InvariantCulture, + "Unknown MCP tool profile '{0}'. Valid profiles: {1}.", + token, + string.Join(", ", Enum.GetNames())); + return false; + } + bits |= ToBit(profile); + } + + set = new McpToolProfileSet(bits); + error = null; + return true; + } + + /// + /// Renders the set as a comma-separated list of profile names in + /// declaration order. + /// + /// The text form of the set. + public override string ToString() + { + if (Bits == 0) + { + return string.Empty; + } + + var builder = new StringBuilder(); + foreach (McpToolProfile profile in Enumerate()) + { + if (builder.Length > 0) + { + builder.Append(kListSeparatorComma); + } + builder.Append(profile.ToString()); + } + return builder.ToString(); + } + + /// + public bool Equals(McpToolProfileSet other) + { + return Bits == other.Bits; + } + + /// + public override bool Equals(object? obj) + { + return obj is McpToolProfileSet other && Equals(other); + } + + /// + public override int GetHashCode() + { + return unchecked((int)Bits); + } + + /// + /// Whether two sets contain the same profiles. + /// + public static bool operator ==(McpToolProfileSet left, McpToolProfileSet right) + { + return left.Equals(right); + } + + /// + /// Whether two sets contain different profiles. + /// + public static bool operator !=(McpToolProfileSet left, McpToolProfileSet right) + { + return !left.Equals(right); + } + + /// + /// Implicit conversion from a single profile so an existing + /// single-profile call site can be re-typed to + /// without changing its intent. + /// + /// The profile to wrap in a set. + public static implicit operator McpToolProfileSet(McpToolProfile profile) + { + return new McpToolProfileSet(profile); + } + + internal uint Bits { get; } + + private static uint ToBit(McpToolProfile profile) + { + return 1u << (int)profile; + } + + private static void ValidateProfile(McpToolProfile profile) + { + if (!Enum.IsDefined(profile)) + { + throw new ArgumentOutOfRangeException( + nameof(profile), + profile, + "Unknown MCP tool profile."); + } + } + } +} diff --git a/tools/Opc.Ua.Mcp.Core/OpcUaMcpCoreExtensions.cs b/tools/Opc.Ua.Mcp.Core/OpcUaMcpCoreExtensions.cs index 9964be45cb..fd1923cb12 100644 --- a/tools/Opc.Ua.Mcp.Core/OpcUaMcpCoreExtensions.cs +++ b/tools/Opc.Ua.Mcp.Core/OpcUaMcpCoreExtensions.cs @@ -28,6 +28,7 @@ * ======================================================================*/ using System; +using System.Collections.Generic; using Microsoft.Extensions.DependencyInjection; using ModelContextProtocol.Server; using Opc.Ua.Mcp.Tools; @@ -190,6 +191,7 @@ public static IMcpServerBuilder WithOpcUaCoreTools( case McpToolProfile.PubSub: case McpToolProfile.Diagnostics: case McpToolProfile.Robotics: + case McpToolProfile.Vision: break; case McpToolProfile.Full: AddFullTools(mcpServerBuilder); @@ -205,6 +207,226 @@ public static IMcpServerBuilder WithOpcUaCoreTools( return mcpServerBuilder; } + /// + /// Registers the Part 4 tools that satisfy the composed set of + /// profiles in , together with the + /// session resources. + /// + /// + /// This overload is the composition entry point a host uses when it + /// wants tools from more than one bounded profile - a vision-guided + /// pick-and-place agent that has to both see through the + /// Vision tools and command a robot through the Robotics + /// tools, for example. Every profile in the set that owns a Part 4 + /// tool set contributes it, deduplicated by tool type. Profiles that + /// only rely on Part 4 tools indirectly - , + /// and + /// - cause to be registered exactly once + /// through , so + /// composing Vision and Robotics on the same builder does not register + /// the connection tools twice. + /// + /// The MCP server builder. + /// The composed set of profiles. + /// The builder, for chaining. + /// + /// is null. + /// + public static IMcpServerBuilder WithOpcUaCoreTools( + this IMcpServerBuilder mcpServerBuilder, + McpToolProfileSet toolProfiles) + { + ArgumentNullException.ThrowIfNull(mcpServerBuilder); + + if (toolProfiles.Contains(McpToolProfile.Full)) + { + AddFullTools(mcpServerBuilder); + } + else + { + var registered = new HashSet(); + if (toolProfiles.Contains(McpToolProfile.Services)) + { + AddServiceToolTypes(registered); + } + if (toolProfiles.Contains(McpToolProfile.Administration)) + { + AddAdministrationToolTypes(registered); + } + if (toolProfiles.Contains(McpToolProfile.Core)) + { + AddCoreToolTypes(registered); + } + // ConnectionTools is registered once through the shared helper + // below, so remove it from the set of tools to register here + // even when a Part 4 profile listed it as owned. + registered.Remove(typeof(ConnectionTools)); + RegisterCoreToolTypes(mcpServerBuilder, registered); + } + + if (NeedsConnectionTools(toolProfiles)) + { + mcpServerBuilder.WithOpcUaConnectionTools(); + } + + mcpServerBuilder.WithResources(); + return mcpServerBuilder; + } + + /// + /// Registers at most once on the + /// supplied builder, so several tool packages composed on the same MCP + /// server share one set of connection tools instead of registering + /// their own. + /// + /// + /// Every session-scoped OPC UA tool - Part 4 services, Vision, + /// Robotics and Diagnostics - resolves a named session that only the + /// connection tools can open, so a composed host has to expose them. + /// This method uses a marker service on + /// to detect a prior registration on the same builder and skip a + /// second one. Each package's McpToolProfileSet overload calls + /// it, so composing Vision and Robotics yields one Connect + /// entry rather than two. + /// + /// The MCP server builder. + /// The builder, for chaining. + /// + /// is null. + /// + public static IMcpServerBuilder WithOpcUaConnectionTools( + this IMcpServerBuilder mcpServerBuilder) + { + ArgumentNullException.ThrowIfNull(mcpServerBuilder); + + IServiceCollection services = mcpServerBuilder.Services; + for (int i = 0; i < services.Count; i++) + { + if (services[i].ServiceType == typeof(ConnectionToolsMarker)) + { + return mcpServerBuilder; + } + } + + services.AddSingleton(); + return mcpServerBuilder.WithTools(); + } + + // Keep the helper as a regular static method; it is not part of the fluent API. + // TODO: Remove when RCS1224 supports intentionally non-extension helpers in extension classes. +#pragma warning disable RCS1224 + internal static bool NeedsConnectionTools(McpToolProfileSet toolProfiles) + { + return toolProfiles.Contains(McpToolProfile.Core) || + toolProfiles.Contains(McpToolProfile.Services) || + toolProfiles.Contains(McpToolProfile.Administration) || + toolProfiles.Contains(McpToolProfile.Diagnostics) || + toolProfiles.Contains(McpToolProfile.Robotics) || + toolProfiles.Contains(McpToolProfile.Vision) || + toolProfiles.Contains(McpToolProfile.Full); + } +#pragma warning restore RCS1224 + + private static void AddCoreToolTypes(HashSet registered) + { + registered.Add(typeof(ConfigurationReadTools)); + registered.Add(typeof(ConfigurationUpdateTools)); + registered.Add(typeof(ConnectionTools)); + registered.Add(typeof(ConvenienceTools)); + } + + private static void AddServiceToolTypes(HashSet registered) + { + registered.Add(typeof(AttributeServiceTools)); + registered.Add(typeof(ConfigurationReadTools)); + registered.Add(typeof(ConfigurationUpdateTools)); + registered.Add(typeof(ConnectionTools)); + registered.Add(typeof(ConvenienceTools)); + registered.Add(typeof(DiscoveryServiceTools)); + registered.Add(typeof(MethodServiceTools)); + registered.Add(typeof(MonitoredItemServiceTools)); + registered.Add(typeof(NodeManagementServiceTools)); + registered.Add(typeof(SubscriptionServiceTools)); + registered.Add(typeof(ViewServiceTools)); + } + + private static void AddAdministrationToolTypes(HashSet registered) + { + registered.Add(typeof(ConfigurationReadTools)); + registered.Add(typeof(ConfigurationUpdateTools)); + registered.Add(typeof(ConnectionTools)); + registered.Add(typeof(NodeSetExportTools)); + registered.Add(typeof(PkiTools)); + } + + private sealed class ConnectionToolsMarker; + + /// + /// Registers each tool class the composed profiles asked for, exactly + /// once. + /// + /// + /// The set decides *what* to register and this decides *how*. It has to + /// be a fixed list of generic calls rather than a loop over the set: the + /// non-generic WithTools(IEnumerable<Type>) looks method + /// metadata up dynamically, which is annotated + /// RequiresUnreferencedCode and breaks the Native AOT guarantee + /// this repository builds under. + /// + private static void RegisterCoreToolTypes( + IMcpServerBuilder mcpServerBuilder, + HashSet registered) + { + if (registered.Contains(typeof(AttributeServiceTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(ConfigurationReadTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(ConfigurationUpdateTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(ConvenienceTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(DiscoveryServiceTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(MethodServiceTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(MonitoredItemServiceTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(NodeManagementServiceTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(NodeSetExportTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(PkiTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(SubscriptionServiceTools))) + { + mcpServerBuilder.WithTools(); + } + if (registered.Contains(typeof(ViewServiceTools))) + { + mcpServerBuilder.WithTools(); + } + } + private static void AddCoreTools(IMcpServerBuilder mcpServerBuilder) { mcpServerBuilder diff --git a/tools/Opc.Ua.Mcp.Core/OpcUaMcpOptions.cs b/tools/Opc.Ua.Mcp.Core/OpcUaMcpOptions.cs index f3a741145b..78a3b5c58f 100644 --- a/tools/Opc.Ua.Mcp.Core/OpcUaMcpOptions.cs +++ b/tools/Opc.Ua.Mcp.Core/OpcUaMcpOptions.cs @@ -43,8 +43,38 @@ public sealed class OpcUaMcpOptions /// /// Gets or sets the tool catalog exposed by the MCP server. /// + /// + /// When is empty this single value drives + /// the tool set, so a host that only wants one profile does not have to + /// think about composition. Setting takes + /// precedence over this value, so an agent that needs several profiles + /// - vision plus robotics, for instance - configures the set and leaves + /// this property alone. + /// public McpToolProfile ToolProfile { get; set; } = McpToolProfile.Full; + /// + /// Gets or sets the composed tool catalog exposed by the MCP server. + /// + /// + /// When non-empty this set takes precedence over + /// and lists every profile the server should carry. This is how a host + /// composes several bounded profiles into one meaningful catalog - for + /// example, Vision plus Robotics for a vision-guided + /// pick-and-place agent - without pulling in every other profile + /// through . + /// + public McpToolProfileSet ToolProfiles { get; set; } + + /// + /// The effective set of tool profiles the host should register, so a + /// caller does not have to know whether the single-profile or composed + /// selection is in force. + /// + public McpToolProfileSet EffectiveToolProfiles => ToolProfiles.IsEmpty + ? new McpToolProfileSet(ToolProfile) + : ToolProfiles; + /// /// Base directory under which the /// is diff --git a/tools/Opc.Ua.Mcp.Diagnostics/OpcUaMcpDiagnosticsExtensions.cs b/tools/Opc.Ua.Mcp.Diagnostics/OpcUaMcpDiagnosticsExtensions.cs index 08cb6818c3..1884250dbb 100644 --- a/tools/Opc.Ua.Mcp.Diagnostics/OpcUaMcpDiagnosticsExtensions.cs +++ b/tools/Opc.Ua.Mcp.Diagnostics/OpcUaMcpDiagnosticsExtensions.cs @@ -140,6 +140,7 @@ public static IMcpServerBuilder WithOpcUaDiagnosticsTools( case McpToolProfile.Administration: case McpToolProfile.PubSub: case McpToolProfile.Robotics: + case McpToolProfile.Vision: break; default: throw new ArgumentOutOfRangeException( @@ -151,6 +152,53 @@ public static IMcpServerBuilder WithOpcUaDiagnosticsTools( return mcpServerBuilder; } + /// + /// Registers the diagnostics tools when the composed + /// includes + /// . + /// + /// + /// This overload is the composition entry point a host uses when it + /// wants the packet capture tools alongside other bounded profiles. + /// It never registers directly; + /// the McpToolProfileSet overload of WithOpcUaCoreTools + /// owns that registration and deduplicates it across every package + /// that contributes to the same MCP server. The key-disclosing decode + /// and replay tools are still gated by + /// . + /// + /// The MCP server builder. + /// The composed set of profiles. + /// + /// Whether the key-disclosing tools are opted in. + /// + /// The builder, for chaining. + /// + /// is null. + /// + public static IMcpServerBuilder WithOpcUaDiagnosticsTools( + this IMcpServerBuilder mcpServerBuilder, + McpToolProfileSet toolProfiles, + bool diagnosticsToolsEnabled) + { + ArgumentNullException.ThrowIfNull(mcpServerBuilder); + + if (!toolProfiles.Contains(McpToolProfile.Diagnostics) && + !toolProfiles.Contains(McpToolProfile.Full)) + { + return mcpServerBuilder; + } + + mcpServerBuilder.WithTools(); + if (diagnosticsToolsEnabled) + { + mcpServerBuilder + .WithTools() + .WithTools(); + } + return mcpServerBuilder; + } + /// /// Reads the Pcap:EnableDiagnosticsTools configuration value /// into a new instance. Non-boolean or @@ -161,6 +209,9 @@ public static IMcpServerBuilder WithOpcUaDiagnosticsTools( /// /// is null. /// + // Preserve the established static API rather than exposing configuration values as extensions. + // TODO: Remove when RCS1224 supports intentionally static APIs in extension classes. +#pragma warning disable RCS1224 public static PcapOptions CreatePcapOptions(IConfiguration configuration) { ArgumentNullException.ThrowIfNull(configuration); @@ -202,5 +253,6 @@ public static bool AreDiagnosticsToolsEnabled(PcapOptions pcapOptions) "true", StringComparison.OrdinalIgnoreCase); } +#pragma warning restore RCS1224 } } diff --git a/tools/Opc.Ua.Mcp.PubSub.Diagnostics/OpcUaMcpPubSubDiagnosticsExtensions.cs b/tools/Opc.Ua.Mcp.PubSub.Diagnostics/OpcUaMcpPubSubDiagnosticsExtensions.cs index f3df4195ce..f3e265bf07 100644 --- a/tools/Opc.Ua.Mcp.PubSub.Diagnostics/OpcUaMcpPubSubDiagnosticsExtensions.cs +++ b/tools/Opc.Ua.Mcp.PubSub.Diagnostics/OpcUaMcpPubSubDiagnosticsExtensions.cs @@ -102,10 +102,7 @@ public static IMcpServerBuilder WithOpcUaPubSubDiagnosticsTools( { case McpToolProfile.PubSub: case McpToolProfile.Full: - mcpServerBuilder.WithRequestFilters(filters => - { - filters.AddCallToolFilter(PubSubPcapMcpFilters.SurfaceDiagnosticsErrors); - }); + mcpServerBuilder.WithRequestFilters(filters => filters.AddCallToolFilter(PubSubPcapMcpFilters.SurfaceDiagnosticsErrors)); mcpServerBuilder.WithTools(); if (diagnosticsToolsEnabled) @@ -119,6 +116,7 @@ public static IMcpServerBuilder WithOpcUaPubSubDiagnosticsTools( case McpToolProfile.Administration: case McpToolProfile.Diagnostics: case McpToolProfile.Robotics: + case McpToolProfile.Vision: break; default: throw new ArgumentOutOfRangeException( @@ -129,5 +127,46 @@ public static IMcpServerBuilder WithOpcUaPubSubDiagnosticsTools( return mcpServerBuilder; } + + /// + /// Registers the PubSub diagnostics tools when the composed + /// includes + /// . + /// + /// + /// This overload is the composition entry point a host uses when it + /// wants the PubSub capture tools alongside other bounded profiles. The + /// key-loading decode tool is still gated by + /// . + /// + /// The MCP server builder. + /// The composed set of profiles. + /// + /// Whether the key-loading decode tool is opted in. + /// + /// The builder, for chaining. + /// + /// is null. + /// + public static IMcpServerBuilder WithOpcUaPubSubDiagnosticsTools( + this IMcpServerBuilder mcpServerBuilder, + McpToolProfileSet toolProfiles, + bool diagnosticsToolsEnabled) + { + ArgumentNullException.ThrowIfNull(mcpServerBuilder); + + if (!toolProfiles.Contains(McpToolProfile.PubSub) && + !toolProfiles.Contains(McpToolProfile.Full)) + { + return mcpServerBuilder; + } + + mcpServerBuilder.WithTools(); + if (diagnosticsToolsEnabled) + { + mcpServerBuilder.WithTools(); + } + return mcpServerBuilder; + } } } diff --git a/tools/Opc.Ua.Mcp.PubSub/OpcUaMcpPubSubExtensions.cs b/tools/Opc.Ua.Mcp.PubSub/OpcUaMcpPubSubExtensions.cs index fad88daac9..516b30dd81 100644 --- a/tools/Opc.Ua.Mcp.PubSub/OpcUaMcpPubSubExtensions.cs +++ b/tools/Opc.Ua.Mcp.PubSub/OpcUaMcpPubSubExtensions.cs @@ -101,6 +101,7 @@ public static IMcpServerBuilder WithOpcUaPubSubTools( case McpToolProfile.Administration: case McpToolProfile.Diagnostics: case McpToolProfile.Robotics: + case McpToolProfile.Vision: break; default: throw new ArgumentOutOfRangeException( @@ -111,5 +112,40 @@ public static IMcpServerBuilder WithOpcUaPubSubTools( return mcpServerBuilder; } + + /// + /// Registers the PubSub tools when the composed + /// includes + /// . + /// + /// + /// This overload is the composition entry point a host uses when it + /// wants the PubSub runtime tools alongside other bounded profiles. + /// The PubSub tools do not resolve a named OPC UA session, so this + /// method never touches . + /// + /// The MCP server builder. + /// The composed set of profiles. + /// The builder, for chaining. + /// + /// is null. + /// + public static IMcpServerBuilder WithOpcUaPubSubTools( + this IMcpServerBuilder mcpServerBuilder, + McpToolProfileSet toolProfiles) + { + ArgumentNullException.ThrowIfNull(mcpServerBuilder); + + if (!toolProfiles.Contains(McpToolProfile.PubSub) && + !toolProfiles.Contains(McpToolProfile.Full)) + { + return mcpServerBuilder; + } + + return mcpServerBuilder + .WithTools() + .WithTools() + .WithTools(); + } } } diff --git a/tools/Opc.Ua.Mcp.Robotics/NugetREADME.md b/tools/Opc.Ua.Mcp.Robotics/NugetREADME.md index d98930c36d..a19dd1e74c 100644 --- a/tools/Opc.Ua.Mcp.Robotics/NugetREADME.md +++ b/tools/Opc.Ua.Mcp.Robotics/NugetREADME.md @@ -12,7 +12,42 @@ manage missions through the OPC UA Robotics Client API. Discovery, live state and outstanding-work monitoring, direct-control tools for authority/cancel/pause/resume/retry and one submit tool per intent kind, plus -mission submit/update/cancel. +mission submit/update/cancel, bounded operation/mission waits, and +`robotics_vision_pick`. + +Every tool takes a controller selector (unique display name, BrowseName, or +NodeId). The selector is resolved once per call, and the controller's published +lookup tables are then used to resolve the frame, tool, location, output, +and program names named inside the request. Full NodeIds are always +accepted and validated by the server. Resolution is read-only: it never submits +work and never requests command authority. + +Intent and mission inputs are typed MCP objects and arrays. There is no JSON +document encoded inside an `intentJson` or `stepsJson` string. Mission intent +`kind` is a closed discriminator, and exactly one matching payload is required. +Operation and mission list tools return filtered, cursor-paged summaries by +default; use Full detail only when the complete snapshots are needed. + +`robotics_vision_pick` resolves a Vision pipeline on the same OPC UA session, +runs one detection inference, applies exact ID/class and confidence filters, +then submits either one Pick or a two-step Pick/Place mission. It returns the +selected detection provenance and the authoritative operation or mission +handles: + +```json +{ + "request": { + "controller": "BinPickingController", + "pipeline": "BinPickingPipeline", + "source": "Bin", + "tool": "ParallelGripper", + "destination": "Fixture", + "classLabel": "RedCube", + "minimumConfidence": 0.9, + "missionId": "place-red-cube" + } +} +``` Refusals are returned with the server's exact `IntentFailureEnum` and message; the MCP layer does not retry, request authority implicitly, or reinterpret @@ -42,6 +77,7 @@ host references. |---|---| | `OPCFoundation.NetStandard.Opc.Ua.Mcp.Core` | Part 4 service tools, session management, filters (required) | | `OPCFoundation.NetStandard.Opc.Ua.Robotics.Client` | Robot Intent discovery, state, authority, operation and mission client API | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Client` | same-session inference used by `robotics_vision_pick` | | `OPCFoundation.NetStandard.Opc.Ua.Mcp` | the ready-to-run `opcua-mcp` server composing all OPC UA MCP tool packages | ## License diff --git a/tools/Opc.Ua.Mcp.Robotics/Opc.Ua.Mcp.Robotics.csproj b/tools/Opc.Ua.Mcp.Robotics/Opc.Ua.Mcp.Robotics.csproj index cae7a3d061..3b185e916b 100644 --- a/tools/Opc.Ua.Mcp.Robotics/Opc.Ua.Mcp.Robotics.csproj +++ b/tools/Opc.Ua.Mcp.Robotics/Opc.Ua.Mcp.Robotics.csproj @@ -29,6 +29,9 @@ + + diff --git a/tools/Opc.Ua.Mcp.Robotics/OpcUaMcpRoboticsExtensions.cs b/tools/Opc.Ua.Mcp.Robotics/OpcUaMcpRoboticsExtensions.cs index 67cd00a89c..5f61ab89fd 100644 --- a/tools/Opc.Ua.Mcp.Robotics/OpcUaMcpRoboticsExtensions.cs +++ b/tools/Opc.Ua.Mcp.Robotics/OpcUaMcpRoboticsExtensions.cs @@ -40,13 +40,15 @@ namespace Opc.Ua.Mcp public static class OpcUaMcpRoboticsExtensions { /// - /// Registers the Robot Intent controller manager the Robotics tools resolve. + /// Registers the Robot Intent controller manager the Robotics tools resolve + /// and the vision-guided helper that composes it with the Vision client. /// public static IServiceCollection AddOpcUaMcpRobotics(this IServiceCollection services) { ArgumentNullException.ThrowIfNull(services); services.AddSingleton(); + services.AddSingleton(); return services; } @@ -59,6 +61,7 @@ public static IServiceCollection AddOpcUaMcpRobotics(this IServiceCollection ser /// tools can open one. already carries them through the /// core package, so they are not added twice. /// + /// public static IMcpServerBuilder WithOpcUaRoboticsTools( this IMcpServerBuilder mcpServerBuilder, McpToolProfile toolProfile = McpToolProfile.Full) @@ -73,7 +76,8 @@ public static IMcpServerBuilder WithOpcUaRoboticsTools( .WithTools() .WithTools() .WithTools() - .WithTools(); + .WithTools() + .WithTools(); if (toolProfile == McpToolProfile.Robotics) { @@ -90,6 +94,7 @@ public static IMcpServerBuilder WithOpcUaRoboticsTools( case McpToolProfile.Administration: case McpToolProfile.PubSub: case McpToolProfile.Diagnostics: + case McpToolProfile.Vision: break; default: throw new ArgumentOutOfRangeException( @@ -100,5 +105,45 @@ public static IMcpServerBuilder WithOpcUaRoboticsTools( return mcpServerBuilder; } + + /// + /// Registers the Robot Intent tools when the composed + /// includes . + /// + /// + /// This overload is the composition entry point a host uses when it + /// wants the Robot Intent tools alongside the tools of another bounded + /// profile - , for a vision-guided + /// pick-and-place agent, for example. It never registers + /// directly; the + /// McpToolProfileSet overload of WithOpcUaCoreTools owns + /// that registration and deduplicates it across every package that + /// contributes to the same MCP server. + /// + /// The MCP server builder. + /// The composed set of profiles. + /// The builder, for chaining. + /// + /// is null. + /// + public static IMcpServerBuilder WithOpcUaRoboticsTools( + this IMcpServerBuilder mcpServerBuilder, + McpToolProfileSet toolProfiles) + { + ArgumentNullException.ThrowIfNull(mcpServerBuilder); + + if (!toolProfiles.Contains(McpToolProfile.Robotics) && + !toolProfiles.Contains(McpToolProfile.Full)) + { + return mcpServerBuilder; + } + + return mcpServerBuilder + .WithTools() + .WithTools() + .WithTools() + .WithTools() + .WithTools(); + } } } diff --git a/tools/Opc.Ua.Mcp.Robotics/RoboticsIntentManager.cs b/tools/Opc.Ua.Mcp.Robotics/RoboticsIntentManager.cs index d9222635f6..801873eb8a 100644 --- a/tools/Opc.Ua.Mcp.Robotics/RoboticsIntentManager.cs +++ b/tools/Opc.Ua.Mcp.Robotics/RoboticsIntentManager.cs @@ -28,7 +28,10 @@ * ======================================================================*/ using System; +using System.Threading; +using System.Threading.Tasks; using Opc.Ua.Client; +using Opc.Ua.Mcp.Tools; using Opc.Ua.Robotics.Client.Intent; namespace Opc.Ua.Mcp @@ -57,6 +60,7 @@ public RobotIntentClient CreateClient(string? sessionName = null) /// /// Creates a controller client over the named or sole active session. + /// Accepts a NodeId string directly; does not resolve names. /// public RobotIntentControllerClient OpenController(string controllerId, string? sessionName = null) { @@ -65,6 +69,19 @@ public RobotIntentControllerClient OpenController(string controllerId, string? s return CreateClient(sessionName).Controller(Serialization.OpcUaJsonHelper.ParseNodeId(controllerId)); } + /// + /// Resolves a controller selector (unique name, BrowseName, or NodeId string) to a + /// controller client. The selector is trimmed and matched with exact ordinal comparison. + /// Exactly one discovery client is created per call. + /// + public ValueTask ResolveControllerAsync( + string controller, + string? sessionName = null, + CancellationToken ct = default) + { + return RoboticsControllerResolver.ResolveAsync(CreateClient(sessionName), controller, ct); + } + private readonly OpcUaSessionManager m_sessionManager; } } diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsControlTools.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsControlTools.cs index 70ea677c34..139b87ec2a 100644 --- a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsControlTools.cs +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsControlTools.cs @@ -27,6 +27,7 @@ * http://opcfoundation.org/License/MIT/1.00/ * ======================================================================*/ +using System; using System.ComponentModel; using System.Threading; using System.Threading.Tasks; @@ -50,11 +51,13 @@ public sealed class RoboticsControlTools "authority, the current owner is returned; this tool never synthesizes authority as a side effect.")] public static async Task RequestControlAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(ControllerDescription)] string controller, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).Transport.RequestControlAsync(ct) + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await resolved.Transport.RequestControlAsync(ct) .ConfigureAwait(false); } @@ -66,11 +69,13 @@ public static async Task RequestControlAsync( "ownership; server-side errors are returned as OPC UA call errors.")] public static async Task ReleaseControlAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(ControllerDescription)] string controller, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - await manager.OpenController(controllerId, sessionName).ReleaseControlAsync(ct).ConfigureAwait(false); + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + await resolved.ReleaseControlAsync(ct).ConfigureAwait(false); } /// @@ -81,13 +86,15 @@ public static async Task ReleaseControlAsync( "ControlNotOwned is returned by the client API; this tool never retries or turns it into a resubmit.")] public static async Task CancelIntentAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(ControllerDescription)] string controller, [Description("IntentId to cancel.")] string intentId, [Description("Stop mode requested from the server.")] StopModeEnum stopMode = StopModeEnum.QuickStop, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).CancelIntentAsync(intentId, stopMode, ct) + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await resolved.CancelIntentAsync(intentId, stopMode, ct) .ConfigureAwait(false); } @@ -99,12 +106,14 @@ public static async Task CancelIntentAsync( "the MCP layer does not maintain its own outstanding-work list or retry refusals.")] public static async Task CancelAllAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(ControllerDescription)] string controller, [Description("Stop mode requested from the server.")] StopModeEnum stopMode = StopModeEnum.QuickStop, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).CancelAllAsync(stopMode, ct) + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await resolved.CancelAllAsync(stopMode, ct) .ConfigureAwait(false); } @@ -119,11 +128,13 @@ public static async Task CancelAllAsync( "as a side effect. Returns IntentCommandOutcome.")] public static async Task PauseAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(ControllerDescription)] string controller, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).PauseAsync(ct).ConfigureAwait(false); + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await resolved.PauseAsync(ct).ConfigureAwait(false); } /// @@ -136,11 +147,13 @@ public static async Task PauseAsync( "are never retried. Command authority is never acquired as a side effect. Returns IntentCommandOutcome.")] public static async Task ResumeAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(ControllerDescription)] string controller, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).ResumeAsync(ct).ConfigureAwait(false); + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await resolved.ResumeAsync(ct).ConfigureAwait(false); } /// @@ -151,17 +164,19 @@ public static async Task ResumeAsync( "IntentFailureEnum and message when refused; this tool performs no client-side retry loop.")] public static async Task RetryAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(ControllerDescription)] string controller, [Description("IntentId to retry.")] string intentId, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).RetryAsync(intentId, ct) + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await resolved.RetryAsync(intentId, ct) .ConfigureAwait(false); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a JointMove motion intent. /// [McpServerTool(Name = "robotics_submit_joint_move")] [Description("Submits a JointMove motion intent: point-to-point joint motion to jointTargets in radians " + @@ -171,24 +186,24 @@ public static async Task RetryAsync( "are never retried. Command authority is never acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitJointMoveAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description("Required JSON object for JointMove: either jointTargets as an array of joint positions " + - "in radians, or targetPose with position metres and quaternion orientation [x,y,z,w]. Optional " + - "common fields include intentId, label, bufferMode, blockingMode, toolFrame, constraints, and blend. " + - "Malformed JSON, wrong array lengths, or NodeIds outside the controller fail before or during " + - "submission as ParameterInvalid or an argument error.")] - string intentJson, - [Description("Axis count used only for local JointMove builder validation. Use the controller's declared " + - "AxisCount; the default 0 disables local count validation and leaves validation to the server.")] + [Description(ControllerDescription)] string controller, + [Description("JointMove input: set jointTargets (radians) or targetPose with position/orientation.")] + JointMoveIntentInput input, + [Description("Axis count for local validation. Default 0 disables and leaves validation to the server.")] uint axisCount = 0, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "jointMove", intentJson, axisCount, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertJointMove(input, axisCount, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a LinearMove motion intent. /// [McpServerTool(Name = "robotics_submit_linear_move")] [Description("Submits a LinearMove motion intent: a straight Cartesian segment to target. " + @@ -198,20 +213,22 @@ public static Task SubmitJointMoveAsync( "effect. Returns IntentSubmissionResult.")] public static Task SubmitLinearMoveAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description("Required JSON object for LinearMove with target pose: position is [x,y,z] in metres and " + - "orientation is quaternion [x,y,z,w], plus optional speedFraction, constraints, blend, toolFrame, " + - "intentId, label, bufferMode, and blockingMode. Missing target, malformed pose arrays, or invalid " + - "NodeIds fail before or during submission as ParameterInvalid or an argument error.")] - string intentJson, + [Description(ControllerDescription)] string controller, + [Description("LinearMove input with target pose: position in metres, quaternion orientation.")] + LinearMoveIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "linearMove", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertLinearMove(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a CircularMove motion intent. /// [McpServerTool(Name = "robotics_submit_circular_move")] [Description("Submits a CircularMove motion intent: a Cartesian arc through viaPoint to " + @@ -221,20 +238,22 @@ public static Task SubmitLinearMoveAsync( "effect. Returns IntentSubmissionResult.")] public static Task SubmitCircularMoveAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description("Required JSON object for CircularMove with viaPoint and target poses. Each pose uses " + - "position metres and quaternion orientation [x,y,z,w]; optional common motion fields are " + - "constraints, blend, toolFrame, intentId, label, bufferMode, and blockingMode. Missing poses or " + - "malformed arrays fail before or during submission as ParameterInvalid or an argument error.")] - string intentJson, + [Description(ControllerDescription)] string controller, + [Description("CircularMove input with viaPoint and target poses.")] + CircularMoveIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "circularMove", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertCircularMove(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Trajectory motion intent. /// [McpServerTool(Name = "robotics_submit_trajectory")] [Description("Submits a Trajectory motion intent: a complete time-parameterised joint path made of points " + @@ -245,21 +264,23 @@ public static Task SubmitCircularMoveAsync( "Returns IntentSubmissionResult.")] public static Task SubmitTrajectoryAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description("Required JSON object for Trajectory with points array. Each point has timeFromStart in " + - "seconds, positions as joint values in radians, and optional velocities and accelerations arrays. " + - "Optional common fields include intentId, label, bufferMode, blockingMode, constraints, blend, and " + - "toolFrame. Missing points or inconsistent arrays fail before or during submission as " + - "ParameterInvalid or an argument error.")] - string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Trajectory input with points array (timeFromStart, positions, optional " + + "velocities/accelerations).")] + TrajectoryIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "trajectory", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertTrajectory(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a CartesianPath motion intent. /// [McpServerTool(Name = "robotics_submit_cartesian_path")] [Description("Submits a CartesianPath motion intent: taught Cartesian waypoints with optional " + @@ -269,20 +290,22 @@ public static Task SubmitTrajectoryAsync( "as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitCartesianPathAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description("Required JSON object for CartesianPath with waypoints array. Each waypoint has pose with " + - "position metres and quaternion orientation [x,y,z,w], plus optional blend; the intent also accepts " + - "constraints, toolFrame, intentId, label, bufferMode, and blockingMode. Missing waypoints or invalid " + - "poses fail before or during submission as ParameterInvalid or an argument error.")] - string intentJson, + [Description(ControllerDescription)] string controller, + [Description("CartesianPath input with waypoints array, each with pose and optional blend.")] + CartesianPathIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "cartesianPath", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertCartesianPath(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Force motion intent. /// [McpServerTool(Name = "robotics_submit_force")] [Description("Submits a Force motion intent: move along direction until contactForce is reached, optionally " + @@ -292,20 +315,23 @@ public static Task SubmitCartesianPathAsync( "IntentSubmissionResult.")] public static Task SubmitForceAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description("Required JSON object for Force with direction array, contactForce in newtons, and optional " + - "frameId, maxDistance in metres, holdForce, constraints, blend, toolFrame, intentId, label, " + - "bufferMode, and blockingMode. Wrong units, malformed arrays, or frame NodeIds outside the " + - "controller are refused as ParameterInvalid or fail as argument errors.")] - string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Force input with direction (3-element unit vector), contactForce, optional frameId, " + + "maxDistance, holdForce.")] + ForceIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "force", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertForce(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits an ArcWeld process intent. /// [McpServerTool(Name = "robotics_submit_arc_weld")] [Description("Submits an ArcWeld process intent for continuous welding along the motion path, using optional " + @@ -315,16 +341,23 @@ public static Task SubmitForceAsync( "acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitArcWeldAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(ArcWeldIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("ArcWeld input with optional processProgram, voltage, wireFeedSpeed, travelSpeed, " + + "seamTrackingEnabled, weldProcedureRef.")] + ArcWeldIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "arcWeld", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertArcWeld(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a SpotWeld process intent. /// [McpServerTool(Name = "robotics_submit_spot_weld")] [Description("Submits a SpotWeld process intent for discrete resistance weld points, with processProgram " + @@ -333,16 +366,22 @@ public static Task SubmitArcWeldAsync( "Command authority is never acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitSpotWeldAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(SpotWeldIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("SpotWeld input with optional processProgram, weldSchedule, gunForce.")] + SpotWeldIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "spotWeld", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertSpotWeld(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Dispense process intent. /// [McpServerTool(Name = "robotics_submit_dispense")] [Description("Submits a Dispense process intent for applying material along a " + @@ -353,16 +392,22 @@ public static Task SubmitSpotWeldAsync( "effect. Returns IntentSubmissionResult.")] public static Task SubmitDispenseAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(DispenseIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Dispense input with optional processProgram, flowRate, beadWidth, purgeCycles.")] + DispenseIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "dispense", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertDispense(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Fasten process intent. /// [McpServerTool(Name = "robotics_submit_fasten")] [Description("Submits a Fasten process intent for tightening or joining at a fastener/joint, using optional " + @@ -372,16 +417,22 @@ public static Task SubmitDispenseAsync( "authority is never acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitFastenAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(FastenIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Fasten input with optional joint NodeId, programNumber, targetTorque, processProgram.")] + FastenIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "fasten", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertFasten(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Palletise process intent. /// [McpServerTool(Name = "robotics_submit_palletise")] [Description("Submits a Palletise process intent that places workpieces using a controller-defined pattern " + @@ -391,16 +442,22 @@ public static Task SubmitFastenAsync( "authority is never acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitPalletiseAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(PalletiseIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Palletise input with optional pattern NodeId, layer, row, column.")] + PalletiseIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "palletise", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertPalletise(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a SurfaceFinish process intent. /// [McpServerTool(Name = "robotics_submit_surface_finish")] [Description("Submits a SurfaceFinish process intent for sanding, polishing, deburring, or finishing, " + @@ -411,16 +468,23 @@ public static Task SubmitPalletiseAsync( "effect. Returns IntentSubmissionResult.")] public static Task SubmitSurfaceFinishAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(SurfaceFinishIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("SurfaceFinish input with optional processProgram, contactForce, feedRate, " + + "toolSpeed, stepOver.")] + SurfaceFinishIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "surfaceFinish", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertSurfaceFinish(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Grasp intent. /// [McpServerTool(Name = "robotics_submit_grasp")] [Description("Submits a Grasp intent to close or activate a tool on an object, with tool NodeId and " + @@ -430,16 +494,22 @@ public static Task SubmitSurfaceFinishAsync( "never retried. Command authority is never acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitGraspAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(GraspIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Grasp input with tool NodeId and force in newtons.")] + GraspIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "grasp", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertGrasp(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Release intent. /// [McpServerTool(Name = "robotics_submit_release")] [Description("Submits a Release intent to open or deactivate a gripper/tool by tool NodeId. Use this to " + @@ -449,16 +519,22 @@ public static Task SubmitGraspAsync( "never acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitReleaseAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(ReleaseIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Release input with tool NodeId.")] + ReleaseIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "release", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertRelease(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Pick intent. /// [McpServerTool(Name = "robotics_submit_pick")] [Description("Submits a Pick intent for inbound material handling: approach the source LocationType " + @@ -469,16 +545,22 @@ public static Task SubmitReleaseAsync( "a side effect. Returns IntentSubmissionResult.")] public static Task SubmitPickAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(PickIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Pick input with source NodeId, tool NodeId, and optional objectClass label.")] + PickIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "pick", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertPick(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Place intent. /// [McpServerTool(Name = "robotics_submit_place")] [Description("Submits a Place intent for outbound material handling: carry the held workpiece to the " + @@ -488,16 +570,22 @@ public static Task SubmitPickAsync( "without retrying or silently taking command authority. Returns IntentSubmissionResult.")] public static Task SubmitPlaceAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(PlaceIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Place input with destination NodeId and tool NodeId.")] + PlaceIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "place", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertPlace(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a ToolChange intent. /// [McpServerTool(Name = "robotics_submit_tool_change")] [Description("Submits a ToolChange intent to fit a docked tool or release the current tool when tool is " + @@ -507,16 +595,22 @@ public static Task SubmitPlaceAsync( "Returns IntentSubmissionResult.")] public static Task SubmitToolChangeAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(ToolChangeIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("ToolChange input with optional tool NodeId and dockStation NodeId.")] + ToolChangeIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "toolChange", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertToolChange(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a SetOutput intent. /// [McpServerTool(Name = "robotics_submit_set_output")] [Description("Submits a SetOutput intent that writes a controller OutputSignalType to a supplied value. " + @@ -525,16 +619,22 @@ public static Task SubmitToolChangeAsync( "never retried. Command authority is never acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitSetOutputAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(SetOutputIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("SetOutput input with output NodeId and typed value.")] + SetOutputIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "setOutput", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertSetOutput(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a CallProgram intent. /// [McpServerTool(Name = "robotics_submit_call_program")] [Description("Submits a CallProgram intent that starts a server ProgramType by program NodeId with optional " + @@ -544,16 +644,22 @@ public static Task SubmitSetOutputAsync( "Command authority is never acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitCallProgramAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(CallProgramIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("CallProgram input with program NodeId and optional arguments.")] + CallProgramIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "callProgram", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertCallProgram(input, scope), + ct); } /// - /// Executes a Robot Intent direct-control MCP tool. + /// Submits a Wait intent. /// [McpServerTool(Name = "robotics_submit_wait")] [Description("Submits a Wait intent that delays for a duration in seconds or until a signal/Boolean node " + @@ -563,127 +669,44 @@ public static Task SubmitCallProgramAsync( "retried. Command authority is never acquired as a side effect. Returns IntentSubmissionResult.")] public static Task SubmitWaitAsync( RoboticsIntentManager manager, - [Description(ControllerIdDescription)] string controllerId, - [Description(WaitIntentJsonDescription)] string intentJson, + [Description(ControllerDescription)] string controller, + [Description("Wait input with duration in seconds and optional signal NodeId.")] + WaitIntentInput input, [Description(SessionNameDescription)] string? sessionName = null, CancellationToken ct = default) { - return SubmitIntentAsync(manager, controllerId, "wait", intentJson, 0, sessionName, ct); + return SubmitAsync( + manager, + controller, + sessionName, + scope => RoboticsIntentDtoConverter.ConvertWait(input, scope), + ct); } - private const string ControllerIdDescription = - "Required OPC UA NodeId string for the Robot Intent controller to command, for example " + - "ns=2;s=RobotIntent/Controllers/Controller1. It must identify a controller already discovered in the " + - "selected session; malformed NodeIds or nodes that are not Robot Intent controllers fail before any " + - "robot command is submitted."; + internal const string ControllerDescription = + "Controller selector: unique display name or BrowseName (e.g. 'Controller1') or OPC UA " + + "NodeId string (e.g. ns=2;s=RobotIntent/Controllers/Controller1). Matched with exact " + + "ordinal comparison after trimming. Fails with available names and NodeIds when zero " + + "or multiple controllers match. Names of frames, tools, locations, outputs, programs, " + + "and axes inside the request are resolved against this controller's published lookup " + + "tables; full NodeIds are always accepted and validated by the server."; private const string SessionNameDescription = "Optional MCP OPC UA session name. Omit it only when exactly one OPC UA session is active; provide the " + "name returned by the session-management tools when multiple sessions are connected. If the name is " + "missing, ambiguous, or unknown, the tool fails before sending any robot command."; - private const string ArcWeldIntentJsonDescription = - "Required JSON object for ArcWeld. Optional fields are processProgram as a ProgramType NodeId under the " + - "controller, voltage, wireFeedSpeed, travelSpeed, seamTrackingEnabled, weldProcedureRef, attributes, and " + - "common motion fields such as intentId, label, bufferMode, and blockingMode. Wrong NodeIds or malformed " + - "numbers are refused as ParameterInvalid or fail as argument errors."; - - private const string SpotWeldIntentJsonDescription = - "Required JSON object for SpotWeld. Optional fields are processProgram as a ProgramType NodeId under the " + - "controller, weldSchedule number, gunForce in newtons, attributes, intentId, label, bufferMode, and " + - "blockingMode. Wrong NodeIds or malformed numeric values are refused as ParameterInvalid or fail as " + - "argument errors."; - - private const string DispenseIntentJsonDescription = - "Required JSON object for Dispense. Optional fields are processProgram ProgramType NodeId, flowRate, " + - "beadWidth in metres, purgeCycles, attributes, intentId, label, bufferMode, and blockingMode. Use only " + - "controller-published process programs; malformed values or foreign NodeIds are refused as " + - "ParameterInvalid or fail as argument errors."; - - private const string FastenIntentJsonDescription = - "Required JSON object for Fasten. Optional fields are joint as a controller joining-model NodeId, " + - "programNumber, targetTorque in newton-metres, processProgram, attributes, intentId, label, bufferMode, " + - "and blockingMode. A joint outside the controller or unsupported joining model is refused as " + - "ParameterInvalid or CapabilityNotSupported."; - - private const string PalletiseIntentJsonDescription = - "Required JSON object for Palletise. Optional fields are pattern as a LocationType NodeId under the " + - "controller, layer, row, column indexes, processProgram, attributes, intentId, label, bufferMode, and " + - "blockingMode. Pattern NodeIds outside the controller or malformed indexes are refused as " + - "ParameterInvalid or fail as argument errors."; - - private const string SurfaceFinishIntentJsonDescription = - "Required JSON object for SurfaceFinish. Optional fields are processProgram, contactForce in newtons, " + - "feedRate, toolSpeed, stepOver in metres, attributes, intentId, label, bufferMode, and blockingMode. Use " + - "only when the controller declares the SurfaceFinish/Force capability; unsupported or malformed values " + - "are returned as CapabilityNotSupported, ParameterInvalid, or argument errors."; - - private const string GraspIntentJsonDescription = - "Required JSON object for Grasp with controller ToolType NodeId and force in newtons. " + - "Optional common fields are intentId, label, bufferMode, and blockingMode. A missing/foreign tool NodeId " + - "or malformed force is refused as ParameterInvalid or fails as an argument error."; - - private const string ReleaseIntentJsonDescription = - "Required JSON object for Release with tool as ToolType NodeId to open/deactivate. Optional common " + - "fields are intentId, label, bufferMode, and blockingMode. A missing, malformed, or foreign tool NodeId " + - "is refused as ParameterInvalid or fails as an argument error."; - - private const string PickIntentJsonDescription = - "Required JSON object for Pick with source as a LocationType NodeId under the controller and tool as the " + - "ToolType NodeId used to acquire the object. Optional common fields are intentId, label, bufferMode, and " + - "blockingMode. Foreign or wrong-type NodeIds are refused as ParameterInvalid."; - - private const string PlaceIntentJsonDescription = - "Required JSON object for Place with destination LocationType NodeId and tool as " + - "the ToolType NodeId used to release the object. Optional common fields are intentId, label, bufferMode, " + - "and blockingMode. Foreign or wrong-type NodeIds are refused as ParameterInvalid."; - - private const string ToolChangeIntentJsonDescription = - "Required JSON object for ToolChange with tool as the ToolType NodeId to fit, or null/omitted to release " + - "the current tool; dockStation identifies the dock LocationType or station NodeId when required by the " + - "server. Optional common fields are intentId, label, bufferMode, and blockingMode. Invalid NodeIds are " + - "refused as ParameterInvalid."; - - private const string SetOutputIntentJsonDescription = - "Required JSON object for SetOutput with controller OutputSignalType NodeId, value " + - "as the JSON value to write, and optional dataType to guide Variant conversion. Optional common fields " + - "are intentId, label, bufferMode, and blockingMode. Wrong output NodeIds or values that do not match the " + - "signal DataType are refused as ParameterInvalid."; - - private const string CallProgramIntentJsonDescription = - "Required JSON object for CallProgram with program as controller ProgramType NodeId " + - "and optional arguments object of name/value pairs. Optional common fields are intentId, label, " + - "bufferMode, and blockingMode. A NodeId that names anything other than a controller ProgramType is " + - "refused as ParameterInvalid."; - - private const string WaitIntentJsonDescription = - "Required JSON object for Wait with duration in seconds and/or signal as an OutputSignalType or Boolean " + - "Variable NodeId under the controller. Optional common fields are intentId, label, bufferMode, and " + - "blockingMode. Invalid signal NodeIds or unsupported wait semantics are refused as ParameterInvalid or " + - "CapabilityNotSupported."; - - internal static async Task SubmitIntentAsync( - RobotIntentControllerClient controller, - string intentKind, - string? intentJson, - uint axisCount, - CancellationToken ct) - { - IntentDataType intent = RoboticsIntentJson.BuildIntent(intentKind, intentJson, axisCount); - return await controller.TrySubmitIntentAsync(intent, ct).ConfigureAwait(false); - } - - private static Task SubmitIntentAsync( + private static async Task SubmitAsync( RoboticsIntentManager manager, - string controllerId, - string intentKind, - string? intentJson, - uint axisCount, + string controller, string? sessionName, + Func convert, CancellationToken ct) { - RobotIntentControllerClient controller = manager.OpenController(controllerId, sessionName); - return SubmitIntentAsync(controller, intentKind, intentJson, axisCount, ct); + RoboticsResolutionContext context = await RoboticsResolutionContext.CreateAsync( + manager, controller, sessionName, ct).ConfigureAwait(false); + IntentDataType intent = convert(context.Scope); + return await context.Client.TrySubmitIntentAsync(intent, ct).ConfigureAwait(false); } } } diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsControllerResolver.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsControllerResolver.cs new file mode 100644 index 0000000000..753bc56f81 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsControllerResolver.cs @@ -0,0 +1,190 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Robotics.Client.Intent; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// Resolves a controller selector (unique display name, BrowseName, or NodeId string) to a + /// , and resolves scoped resource selectors against + /// a controller's published lookup tables. Name resolution is unambiguous: when two entries + /// share a name the caller must use the full NodeId to disambiguate. + /// + internal static class RoboticsControllerResolver + { + /// + /// Resolves to a unique controller on + /// . Accepts either an OPC UA NodeId string (for example + /// ns=2;s=RobotIntent/Controllers/C1) or a controller display name or BrowseName + /// (for example Controller1). Names are trimmed and compared with exact ordinal + /// comparison; the discovery browse happens only when the selector is not a NodeId. + /// + /// + /// Thrown when is empty or whitespace, names zero + /// controllers, or names more than one controller. + /// + public static async ValueTask ResolveAsync( + RobotIntentClient client, + string controllerIdOrName, + CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(client); + ArgumentException.ThrowIfNullOrWhiteSpace(controllerIdOrName); + + string trimmed = controllerIdOrName.Trim(); + if (NodeId.TryParse(trimmed, out NodeId nodeId) && !nodeId.IsNull) + { + return client.Controller(nodeId); + } + + ArrayOf controllers = + await client.DiscoverControllersAsync(ct).ConfigureAwait(false); + + List matches = MatchByName(controllers, trimmed); + if (matches.Count == 0) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"No controller named '{trimmed}' was found. " + + $"Available: [{FormatNamesAndNodeIds(controllers)}]."), + nameof(controllerIdOrName)); + } + + if (matches.Count > 1) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Ambiguous controller name '{trimmed}' matches {matches.Count} controllers. " + + $"Use the full NodeId to disambiguate: [{FormatNodeIds(matches)}]."), + nameof(controllerIdOrName)); + } + + return client.Controller(matches[0].NodeId); + } + + /// + /// Resolves a scoped resource (tool, frame, location, output, program, axis) within the + /// controller's published lookup tables. Accepts a NodeId string, a unique display name, + /// or a unique BrowseName from the corresponding lookup list. + /// + /// Display name, BrowseName, or NodeId string. + /// The lookup entries for the resource category. + /// Human-readable category name used in error messages. + /// The resolved NodeId, or a null NodeId when the selector is empty. + /// + public static NodeId ResolveScopedResource( + string? nameOrNodeId, + ArrayOf entries, + string category) + { + if (string.IsNullOrWhiteSpace(nameOrNodeId)) + { + return NodeId.Null; + } + + string trimmed = nameOrNodeId.Trim(); + if (NodeId.TryParse(trimmed, out NodeId nodeId) && !nodeId.IsNull) + { + return nodeId; + } + + List matches = MatchByName(entries, trimmed); + if (matches.Count == 0) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"No {category} named '{trimmed}' found. " + + $"Available: [{FormatNamesAndNodeIds(entries)}]."), + nameof(nameOrNodeId)); + } + + if (matches.Count > 1) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Ambiguous {category} name '{trimmed}' matches {matches.Count} entries. " + + $"Use the full NodeId to disambiguate: [{FormatNodeIds(matches)}]."), + nameof(nameOrNodeId)); + } + + return matches[0].NodeId; + } + + private static List MatchByName( + ArrayOf entries, + string trimmed) + { + var matches = new List(); + for (int i = 0; i < entries.Count; i++) + { + RobotIntentNodeLookupEntry entry = entries[i]; + if (string.Equals(entry.Name, trimmed, StringComparison.Ordinal) || + string.Equals(entry.BrowseName.Name, trimmed, StringComparison.Ordinal)) + { + matches.Add(entry); + } + } + + return matches; + } + + private static string FormatNamesAndNodeIds(ArrayOf entries) + { + if (entries.Count == 0) + { + return "(none)"; + } + + var items = new List(entries.Count); + for (int i = 0; i < entries.Count; i++) + { + items.Add(string.Create(CultureInfo.InvariantCulture, + $"{entries[i].Name} ({entries[i].NodeId})")); + } + return string.Join(", ", items); + } + + private static string FormatNodeIds(List entries) + { + var ids = new List(entries.Count); + for (int i = 0; i < entries.Count; i++) + { + ids.Add(string.Create(CultureInfo.InvariantCulture, + $"{entries[i].Name} ({entries[i].NodeId})")); + } + return string.Join(", ", ids); + } + } +} diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsDiscoveryTools.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsDiscoveryTools.cs index bc09b859f6..9e1d984f77 100644 --- a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsDiscoveryTools.cs +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsDiscoveryTools.cs @@ -47,12 +47,21 @@ public sealed class RoboticsDiscoveryTools [McpServerTool(Name = "robotics_list_controllers")] [Description("Lists Robot Intent controllers visible in the active OPC UA session. This performs discovery " + "only; it does not request command authority and therefore cannot be refused for ControlNotOwned.")] - public static async Task> ListControllersAsync( + public static async Task ListControllersAsync( RoboticsIntentManager manager, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.CreateClient(sessionName).DiscoverControllersAsync(ct).ConfigureAwait(false); + ArrayOf controllers = await manager + .CreateClient(sessionName) + .DiscoverControllersAsync(ct) + .ConfigureAwait(false); + var result = new RobotIntentNodeLookupEntry[controllers.Count]; + for (int i = 0; i < controllers.Count; i++) + { + result[i] = controllers[i]; + } + return result; } /// @@ -64,11 +73,13 @@ public static async Task> ListControllersAsy "does not infer missing capabilities or command authority.")] public static async Task ReadControllerAsync( RoboticsIntentManager manager, - [Description("Controller NodeId, for example ns=2;s=RobotIntent/Controllers/Controller1.")] string controllerId, + [Description(RoboticsControlTools.ControllerDescription)] string controller, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).ReadAsync(ct).ConfigureAwait(false); + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await resolved.ReadAsync(ct).ConfigureAwait(false); } } } diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsIntentDtoConverter.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsIntentDtoConverter.cs new file mode 100644 index 0000000000..b849271158 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsIntentDtoConverter.cs @@ -0,0 +1,1123 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Text.Json; +using Opc.Ua.Mcp.Serialization; +using Opc.Ua.Robotics.Client.Intent; +using Opc.Ua.RobotIntent; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// Converts strongly-typed MCP DTOs into OPC UA Robot Intent data types. + /// Every scoped name reference is resolved through the per-call + /// before conversion; a null resolver + /// means the caller has no controller scope and every reference must then + /// be a full NodeId. + /// + internal static class RoboticsIntentDtoConverter + { + public static IntentDataType ConvertIntent( + MissionIntentInput input, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(input); + RejectConflictingPayloads(input); + + return input.Kind switch + { + IntentKind.JointMove => ConvertJointMove( + GetPayload(input.Kind, input.JointMove), 0, scope), + IntentKind.LinearMove => ConvertLinearMove( + GetPayload(input.Kind, input.LinearMove), scope), + IntentKind.CircularMove => ConvertCircularMove( + GetPayload(input.Kind, input.CircularMove), scope), + IntentKind.Trajectory => ConvertTrajectory( + GetPayload(input.Kind, input.Trajectory), scope), + IntentKind.CartesianPath => ConvertCartesianPath( + GetPayload(input.Kind, input.CartesianPath), scope), + IntentKind.Force => ConvertForce( + GetPayload(input.Kind, input.Force), scope), + IntentKind.ArcWeld => ConvertArcWeld( + GetPayload(input.Kind, input.ArcWeld), scope), + IntentKind.SpotWeld => ConvertSpotWeld( + GetPayload(input.Kind, input.SpotWeld), scope), + IntentKind.Dispense => ConvertDispense( + GetPayload(input.Kind, input.Dispense), scope), + IntentKind.Fasten => ConvertFasten( + GetPayload(input.Kind, input.Fasten), scope), + IntentKind.Palletise => ConvertPalletise( + GetPayload(input.Kind, input.Palletise), scope), + IntentKind.SurfaceFinish => ConvertSurfaceFinish( + GetPayload(input.Kind, input.SurfaceFinish), scope), + IntentKind.Grasp => ConvertGrasp( + GetPayload(input.Kind, input.Grasp), scope), + IntentKind.Release => ConvertRelease( + GetPayload(input.Kind, input.Release), scope), + IntentKind.Pick => ConvertPick( + GetPayload(input.Kind, input.Pick), scope), + IntentKind.Place => ConvertPlace( + GetPayload(input.Kind, input.Place), scope), + IntentKind.ToolChange => ConvertToolChange( + GetPayload(input.Kind, input.ToolChange), scope), + IntentKind.SetOutput => ConvertSetOutput( + GetPayload(input.Kind, input.SetOutput), scope), + IntentKind.CallProgram => ConvertCallProgram( + GetPayload(input.Kind, input.CallProgram), scope), + IntentKind.Wait => ConvertWait( + GetPayload(input.Kind, input.Wait), scope), + _ => throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Unknown intent kind '{input.Kind}'."), + nameof(input)) + }; + } + + public static IntentDataType ConvertJointMove( + JointMoveIntentInput dto, + uint axisCount, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + bool hasJointTargets = dto.JointTargets is { Length: > 0 }; + if (hasJointTargets && dto.TargetPose != null) + { + throw new ArgumentException( + "JointMove accepts either jointTargets or targetPose, not both.", + nameof(dto)); + } + + JointMoveIntentBuilder builder = RobotIntentBuilder.JointMove(axisCount); + IntentDataType intent; + if (hasJointTargets) + { + ValidateFiniteValues(dto.JointTargets!, "jointTargets"); + intent = builder.ToJoints(dto.JointTargets!).Build(); + } + else if (dto.TargetPose != null) + { + intent = builder.ToPose(ConvertPose(dto.TargetPose, "targetPose", scope)).Build(); + } + else + { + throw new ArgumentException( + "JointMove requires either jointTargets or targetPose.", + nameof(dto)); + } + + ApplyMotionCommon(dto, intent, scope); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertLinearMove( + LinearMoveIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + ValidateSpeedFraction(dto.SpeedFraction, "speedFraction"); + IntentDataType intent = RobotIntentBuilder.LinearMove( + ConvertPose(dto.Target, "target", scope), + dto.SpeedFraction).Build(); + ApplyMotionCommon(dto, intent, scope); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertCircularMove( + CircularMoveIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + IntentDataType intent = RobotIntentBuilder.CircularMove( + ConvertPose(dto.ViaPoint, "viaPoint", scope), + ConvertPose(dto.Target, "target", scope)).Build(); + ApplyMotionCommon(dto, intent, scope); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertTrajectory( + TrajectoryIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + if (dto.Points is null || dto.Points.Length == 0) + { + throw new ArgumentException("Trajectory requires at least one point.", nameof(dto)); + } + + var points = new List(dto.Points.Length); + int jointCount = -1; + double previousTime = double.NegativeInfinity; + for (int i = 0; i < dto.Points.Length; i++) + { + TrajectoryPointDto point = dto.Points[i]; + if (point is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"Trajectory point [{i}] is required."), + nameof(dto)); + } + + string prefix = string.Create(CultureInfo.InvariantCulture, $"points[{i}]"); + if (point.Positions is null || point.Positions.Length == 0) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{prefix}.positions' must have at least one value."), + nameof(dto)); + } + + if (jointCount < 0) + { + jointCount = point.Positions.Length; + } + else if (point.Positions.Length != jointCount) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{prefix}.positions' has {point.Positions.Length} values but the " + + $"trajectory uses {jointCount}."), + nameof(dto)); + } + + ValidateFiniteValues(point.Positions, prefix + ".positions"); + ValidateOptionalTrajectoryComponent(point.Velocities, jointCount, prefix + ".velocities"); + ValidateOptionalTrajectoryComponent(point.Accelerations, jointCount, prefix + ".accelerations"); + + ValidateFinite(point.TimeFromStart, prefix + ".timeFromStart"); + if (point.TimeFromStart < 0) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{prefix}.timeFromStart' must not be negative but was {point.TimeFromStart}."), + nameof(dto)); + } + + if (i > 0 && point.TimeFromStart <= previousTime) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{prefix}.timeFromStart' must be strictly greater than the previous " + + $"point ({previousTime})."), + nameof(dto)); + } + + previousTime = point.TimeFromStart; + points.Add(new TrajectoryPointDataType + { + TimeFromStart = point.TimeFromStart, + Positions = point.Positions, + Velocities = point.Velocities ?? [], + Accelerations = point.Accelerations ?? [] + }); + } + + IntentDataType intent = RobotIntentBuilder.Trajectory().WithPoints([.. points]).Build(); + ApplyMotionCommon(dto, intent, scope); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertCartesianPath( + CartesianPathIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + if (dto.Waypoints is null || dto.Waypoints.Length == 0) + { + throw new ArgumentException("CartesianPath requires at least one waypoint.", nameof(dto)); + } + + var waypoints = new List(dto.Waypoints.Length); + for (int i = 0; i < dto.Waypoints.Length; i++) + { + CartesianWaypointDto waypoint = dto.Waypoints[i]; + if (waypoint is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"Waypoint [{i}] is required."), + nameof(dto)); + } + + string prefix = string.Create(CultureInfo.InvariantCulture, $"waypoints[{i}]"); + waypoints.Add(new PathWaypointDataType + { + Pose = ConvertPose(waypoint.Pose, prefix + ".pose", scope), + Blend = waypoint.Blend != null + ? ConvertBlend(waypoint.Blend, prefix + ".blend") + : new BlendDataType() + }); + } + + IntentDataType intent = RobotIntentBuilder.CartesianPath().WithWaypoints([.. waypoints]).Build(); + ApplyMotionCommon(dto, intent, scope); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertForce( + ForceIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + ValidateDirection(dto.Direction, "direction"); + ValidateFinite(dto.ContactForce, "contactForce"); + if (dto.ContactForce <= 0) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'contactForce' must be greater than zero but was {dto.ContactForce}."), + nameof(dto)); + } + + ValidateNonNegative(dto.MaxDistance, "maxDistance"); + + ForceIntentDataType intent = RobotIntentBuilder.Force(dto.Direction, dto.ContactForce).Build(); + intent.FrameId = ResolveFrameId(dto.FrameId, scope); + intent.MaxDistance = dto.MaxDistance; + intent.HoldForce = dto.HoldForce; + ApplyMotionCommon(dto, intent, scope); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertArcWeld( + ArcWeldIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + ValidateNonNegative(dto.Voltage, "voltage"); + ValidateNonNegative(dto.WireFeedSpeed, "wireFeedSpeed"); + ValidateNonNegative(dto.TravelSpeed, "travelSpeed"); + + ProcessIntentBuilder builder = RobotIntentBuilder.ArcWeld(); + ApplyProcessBuilder(builder, dto, scope); + ArcWeldIntentDataType intent = builder.Build(); + intent.Voltage = dto.Voltage; + intent.WireFeedSpeed = dto.WireFeedSpeed; + intent.TravelSpeed = dto.TravelSpeed; + intent.SeamTrackingEnabled = dto.SeamTrackingEnabled; + intent.WeldProcedureRef = dto.WeldProcedureRef ?? string.Empty; + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertSpotWeld( + SpotWeldIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + ValidateNonNegative(dto.GunForce, "gunForce"); + + ProcessIntentBuilder builder = RobotIntentBuilder.SpotWeld(); + ApplyProcessBuilder(builder, dto, scope); + SpotWeldIntentDataType intent = builder.Build(); + intent.WeldSchedule = dto.WeldSchedule; + intent.GunForce = dto.GunForce; + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertDispense( + DispenseIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + ValidateNonNegative(dto.FlowRate, "flowRate"); + ValidateNonNegative(dto.BeadWidth, "beadWidth"); + + ProcessIntentBuilder builder = RobotIntentBuilder.Dispense(); + ApplyProcessBuilder(builder, dto, scope); + DispenseIntentDataType intent = builder.Build(); + intent.FlowRate = dto.FlowRate; + intent.BeadWidth = dto.BeadWidth; + intent.PurgeCycles = dto.PurgeCycles; + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertFasten( + FastenIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + ValidateNonNegative(dto.TargetTorque, "targetTorque"); + + ProcessIntentBuilder builder = RobotIntentBuilder.Fasten(); + ApplyProcessBuilder(builder, dto, scope); + FastenIntentDataType intent = builder.Build(); + intent.Joint = ResolveNodeId(dto.Joint); + intent.ProgramNumber = dto.ProgramNumber; + intent.TargetTorque = dto.TargetTorque; + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertPalletise( + PalletiseIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + ProcessIntentBuilder builder = RobotIntentBuilder.Palletise(); + ApplyProcessBuilder(builder, dto, scope); + PalletiseIntentDataType intent = builder.Build(); + intent.Pattern = scope != null + ? scope.ResolveLocation(dto.Pattern) + : ResolveNodeId(dto.Pattern); + intent.Layer = dto.Layer; + intent.Row = dto.Row; + intent.Column = dto.Column; + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertSurfaceFinish( + SurfaceFinishIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + ValidateNonNegative(dto.ContactForce, "contactForce"); + ValidateNonNegative(dto.FeedRate, "feedRate"); + ValidateNonNegative(dto.ToolSpeed, "toolSpeed"); + ValidateNonNegative(dto.StepOver, "stepOver"); + + ProcessIntentBuilder builder = RobotIntentBuilder.SurfaceFinish(); + ApplyProcessBuilder(builder, dto, scope); + SurfaceFinishIntentDataType intent = builder.Build(); + intent.ContactForce = dto.ContactForce; + intent.FeedRate = dto.FeedRate; + intent.ToolSpeed = dto.ToolSpeed; + intent.StepOver = dto.StepOver; + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertGrasp( + GraspIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + ValidateNonNegative(dto.Force, "force"); + + NodeId tool = scope != null + ? scope.ResolveRequiredTool(dto.Tool, "tool") + : ResolveRequiredNodeId(dto.Tool, "tool"); + IntentDataType intent = RobotIntentBuilder.Grasp(tool, dto.Force).Build(); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertRelease( + ReleaseIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + NodeId tool = scope != null + ? scope.ResolveRequiredTool(dto.Tool, "tool") + : ResolveRequiredNodeId(dto.Tool, "tool"); + IntentDataType intent = RobotIntentBuilder.Release(tool).Build(); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertPick( + PickIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + NodeId source = scope != null + ? scope.ResolveRequiredLocation(dto.Source, "source") + : ResolveRequiredNodeId(dto.Source, "source"); + NodeId tool = scope != null + ? scope.ResolveRequiredTool(dto.Tool, "tool") + : ResolveRequiredNodeId(dto.Tool, "tool"); + + IntentDataType intent = RobotIntentBuilder.Pick( + source, tool, dto.ObjectClass ?? string.Empty).Build(); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertPlace( + PlaceIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + NodeId destination = scope != null + ? scope.ResolveRequiredLocation(dto.Destination, "destination") + : ResolveRequiredNodeId(dto.Destination, "destination"); + NodeId tool = scope != null + ? scope.ResolveRequiredTool(dto.Tool, "tool") + : ResolveRequiredNodeId(dto.Tool, "tool"); + + IntentDataType intent = RobotIntentBuilder.Place(destination, tool).Build(); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertToolChange( + ToolChangeIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + NodeId tool = scope != null ? scope.ResolveTool(dto.Tool) : ResolveNodeId(dto.Tool); + NodeId dock = scope != null + ? scope.ResolveLocation(dto.DockStation) + : ResolveNodeId(dto.DockStation); + + IntentDataType intent = RobotIntentBuilder.ToolChange(tool, dock).Build(); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertSetOutput( + SetOutputIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + NodeId output = scope != null + ? scope.ResolveRequiredOutput(dto.Output, "output") + : ResolveRequiredNodeId(dto.Output, "output"); + Variant value = ConvertTypedValue(dto.Value, "value"); + + IntentDataType intent = RobotIntentBuilder.SetOutput(output, value).Build(); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertCallProgram( + CallProgramIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + + NodeId program = scope != null + ? scope.ResolveRequiredProgram(dto.Program, "program") + : ResolveRequiredNodeId(dto.Program, "program"); + + CallProgramIntentDataType intent = RobotIntentBuilder.CallProgram(program).Build(); + intent.Arguments = ConvertNamedTypedValues(dto.Arguments, "arguments"); + ApplyCommon(dto, intent); + return intent; + } + + public static IntentDataType ConvertWait( + WaitIntentInput dto, + RoboticsScopeResolver? scope) + { + ArgumentNullException.ThrowIfNull(dto); + ValidateNonNegative(dto.Duration, "duration"); + + NodeId signal = scope != null + ? scope.ResolveOutput(dto.Signal) + : ResolveNodeId(dto.Signal); + if (signal.IsNull && dto.Duration <= 0) + { + throw new ArgumentException( + "Wait requires a positive duration or a signal.", + nameof(dto)); + } + + WaitIntentDataType intent = RobotIntentBuilder.Wait(dto.Duration).Build(); + intent.Signal = signal; + ApplyCommon(dto, intent); + return intent; + } + + public static ArrayOf ConvertMissionSteps( + MissionStepInput[]? steps, + RoboticsScopeResolver? scope) + { + if (steps is null || steps.Length == 0) + { + return []; + } + + var result = new List(steps.Length); + var seenStepIds = new HashSet(StringComparer.Ordinal); + uint sequence = 1; + for (int i = 0; i < steps.Length; i++) + { + MissionStepInput step = steps[i]; + if (step is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"Step [{i}] is required."), + nameof(steps)); + } + + if (string.IsNullOrWhiteSpace(step.StepId)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"Step [{i}] is missing stepId."), + nameof(steps)); + } + + if (!seenStepIds.Add(step.StepId)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Step [{i}] repeats stepId '{step.StepId}'."), + nameof(steps)); + } + + if (step.Intent is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Step '{step.StepId}' is missing its intent."), + nameof(steps)); + } + + IntentDataType intent = ConvertIntent(step.Intent, scope); + result.Add(new MissionStepDataType + { + StepId = step.StepId, + SequenceId = step.SequenceId ?? sequence, + Released = step.Released, + Intent = intent, + ErrorPolicy = step.ErrorPolicy ?? ErrorPolicyEnum.Abort, + FallbackStepId = step.FallbackStepId ?? string.Empty + }); + sequence++; + } + + return [.. result]; + } + + public static ArrayOf ConvertMissionTransitions( + MissionTransitionInput[]? transitions) + { + if (transitions is null || transitions.Length == 0) + { + return []; + } + + var result = new List(transitions.Length); + for (int i = 0; i < transitions.Length; i++) + { + MissionTransitionInput transition = transitions[i]; + if (transition is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"Transition [{i}] is required."), + nameof(transitions)); + } + + if (string.IsNullOrWhiteSpace(transition.FromStepId)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"Transition [{i}] is missing fromStepId."), + nameof(transitions)); + } + + if (string.IsNullOrWhiteSpace(transition.ToStepId)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"Transition [{i}] is missing toStepId."), + nameof(transitions)); + } + + result.Add(new MissionTransitionDataType + { + FromStepId = transition.FromStepId, + ToStepId = transition.ToStepId, + DivergenceKind = transition.DivergenceKind ?? DivergenceKindEnum.Alternative, + Condition = MissionCondition.Always() + }); + } + + return [.. result]; + } + + internal static NodeId ResolveNodeId(string? nameOrNodeId) + { + if (string.IsNullOrWhiteSpace(nameOrNodeId)) + { + return NodeId.Null; + } + + if (!NodeId.TryParse(nameOrNodeId, out NodeId nodeId)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{nameOrNodeId}' is not a valid NodeId."), + nameof(nameOrNodeId)); + } + + return nodeId; + } + + private static T GetPayload(IntentKind kind, T? payload) + where T : class + { + return payload ?? + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Intent kind '{kind}' requires the matching payload.")); + } + + private static void RejectConflictingPayloads(MissionIntentInput input) + { + var present = new List(); + AddIfSet(present, nameof(input.JointMove), input.JointMove); + AddIfSet(present, nameof(input.LinearMove), input.LinearMove); + AddIfSet(present, nameof(input.CircularMove), input.CircularMove); + AddIfSet(present, nameof(input.Trajectory), input.Trajectory); + AddIfSet(present, nameof(input.CartesianPath), input.CartesianPath); + AddIfSet(present, nameof(input.Force), input.Force); + AddIfSet(present, nameof(input.ArcWeld), input.ArcWeld); + AddIfSet(present, nameof(input.SpotWeld), input.SpotWeld); + AddIfSet(present, nameof(input.Dispense), input.Dispense); + AddIfSet(present, nameof(input.Fasten), input.Fasten); + AddIfSet(present, nameof(input.Palletise), input.Palletise); + AddIfSet(present, nameof(input.SurfaceFinish), input.SurfaceFinish); + AddIfSet(present, nameof(input.Grasp), input.Grasp); + AddIfSet(present, nameof(input.Release), input.Release); + AddIfSet(present, nameof(input.Pick), input.Pick); + AddIfSet(present, nameof(input.Place), input.Place); + AddIfSet(present, nameof(input.ToolChange), input.ToolChange); + AddIfSet(present, nameof(input.SetOutput), input.SetOutput); + AddIfSet(present, nameof(input.CallProgram), input.CallProgram); + AddIfSet(present, nameof(input.Wait), input.Wait); + + if (present.Count > 1) + { + // All concatenated operands must remain interpolated for string.Create handler binding. + // TODO: Remove when RCS1214 preserves interpolated-string-handler overload binding. +#pragma warning disable RCS1214 + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Intent kind '{input.Kind}' has {present.Count} payloads set " + + $"([{string.Join(", ", present)}]); only the payload matching the kind " + + $"is allowed."), + nameof(input)); +#pragma warning restore RCS1214 + } + + if (present.Count == 1 && + !string.Equals(present[0], input.Kind.ToString(), StringComparison.Ordinal)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Intent kind '{input.Kind}' does not match the '{present[0]}' payload."), + nameof(input)); + } + } + + private static void AddIfSet(List present, string name, object? payload) + { + if (payload != null) + { + present.Add(name); + } + } + + private static Pose3DDataType ConvertPose( + PoseDto? dto, + string name, + RoboticsScopeResolver? scope) + { + if (dto is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{name}' is required."), name); + } + + if (dto.Position is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{name}.position' is required."), name); + } + + if (dto.Orientation is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{name}.orientation' is required."), name); + } + + ValidateFinite(dto.Position.X, name + ".position.x"); + ValidateFinite(dto.Position.Y, name + ".position.y"); + ValidateFinite(dto.Position.Z, name + ".position.z"); + ValidateFinite(dto.Orientation.X, name + ".orientation.x"); + ValidateFinite(dto.Orientation.Y, name + ".orientation.y"); + ValidateFinite(dto.Orientation.Z, name + ".orientation.z"); + ValidateFinite(dto.Orientation.W, name + ".orientation.w"); + + double norm = Math.Sqrt( + (dto.Orientation.X * dto.Orientation.X) + + (dto.Orientation.Y * dto.Orientation.Y) + + (dto.Orientation.Z * dto.Orientation.Z) + + (dto.Orientation.W * dto.Orientation.W)); + if (Math.Abs(norm - 1.0) > 1e-3) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}.orientation' must be a unit quaternion but its norm was {norm}."), + name); + } + + return RobotIntentBuilder.Pose( + dto.Position.X, + dto.Position.Y, + dto.Position.Z, + dto.Orientation.X, + dto.Orientation.Y, + dto.Orientation.Z, + dto.Orientation.W, + ResolveFrameId(dto.FrameId, scope)); + } + + private static string ResolveFrameId(string? frameId, RoboticsScopeResolver? scope) + { + if (scope != null) + { + return scope.ResolveFrameId(frameId); + } + + return frameId?.Trim() ?? string.Empty; + } + + private static BlendDataType ConvertBlend(BlendDto dto, string name) + { + ValidateNonNegative(dto.Radius, name + ".radius"); + if (dto.Termination == TerminationModeEnum.Blend && dto.Radius <= 0) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}.radius' must be greater than zero when termination is Blend."), + name); + } + + return new BlendDataType + { + Termination = dto.Termination, + Radius = dto.Radius + }; + } + + private static MotionConstraintsDataType ConvertConstraints(MotionConstraintsDto dto, string name) + { + ValidateSpeedFraction(dto.SpeedFraction, name + ".speedFraction"); + ValidateNonNegative(dto.CartesianSpeed, name + ".cartesianSpeed"); + ValidateNonNegative(dto.CartesianAcceleration, name + ".cartesianAcceleration"); + ValidateNonNegative(dto.Jerk, name + ".jerk"); + + return new MotionConstraintsDataType + { + SpeedFraction = dto.SpeedFraction, + CartesianSpeed = dto.CartesianSpeed, + CartesianAcceleration = dto.CartesianAcceleration, + Jerk = dto.Jerk + }; + } + + private static void ApplyCommon(IntentCommonDto dto, IntentDataType intent) + { + if (!string.IsNullOrEmpty(dto.IntentId)) + { + intent.IntentId = dto.IntentId; + } + if (!string.IsNullOrEmpty(dto.Label)) + { + intent.Label = new LocalizedText(dto.Label); + } + if (dto.BufferMode.HasValue) + { + intent.BufferMode = dto.BufferMode.Value; + } + if (dto.BlockingMode.HasValue) + { + intent.BlockingMode = dto.BlockingMode.Value; + } + } + + private static void ApplyMotionCommon( + MotionIntentDto dto, + IntentDataType intent, + RoboticsScopeResolver? scope) + { + if (intent is not MotionIntentDataType motion) + { + return; + } + + motion.ToolFrame = scope != null + ? scope.ResolveFrame(dto.ToolFrame) + : ResolveNodeId(dto.ToolFrame); + + if (dto.Constraints != null) + { + motion.Constraints = ConvertConstraints(dto.Constraints, "constraints"); + } + else + { + ValidateSpeedFraction(dto.SpeedFraction, "speedFraction"); + ValidateNonNegative(dto.CartesianSpeed, "cartesianSpeed"); + motion.Constraints = new MotionConstraintsDataType + { + SpeedFraction = dto.SpeedFraction, + CartesianSpeed = dto.CartesianSpeed + }; + } + + if (dto.Blend != null) + { + motion.Blend = ConvertBlend(dto.Blend, "blend"); + } + } + + private static void ApplyProcessBuilder( + ProcessIntentBuilder builder, + ProcessIntentDto dto, + RoboticsScopeResolver? scope) + where T : ProcessIntentDataType + { + NodeId processProgram = scope != null + ? scope.ResolveProgram(dto.ProcessProgram) + : ResolveNodeId(dto.ProcessProgram); + if (!processProgram.IsNull) + { + builder.WithProcessProgram(processProgram); + } + builder.WithAttributes(ConvertNamedTypedValues(dto.Attributes, "attributes")); + } + + private static Variant ConvertTypedValue(TypedValueDto? dto, string name) + { + if (dto is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{name}' is required."), name); + } + + if (string.IsNullOrWhiteSpace(dto.DataType)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}.dataType' is required so the value is written with an explicit type."), + name); + } + + if (dto.Value.ValueKind is JsonValueKind.Undefined or JsonValueKind.Null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{name}.value' is required."), name); + } + + return OpcUaJsonHelper.JsonElementToVariant(dto.Value, dto.DataType); + } + + private static ArrayOf ConvertNamedTypedValues( + NamedTypedValueDto[]? values, + string name) + { + if (values is null || values.Length == 0) + { + return []; + } + + var result = new List(values.Length); + var seen = new HashSet(StringComparer.Ordinal); + for (int i = 0; i < values.Length; i++) + { + NamedTypedValueDto value = values[i]; + string prefix = string.Create(CultureInfo.InvariantCulture, $"{name}[{i}]"); + if (value is null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{prefix}' is required."), name); + } + + if (string.IsNullOrWhiteSpace(value.Name)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{prefix}.name' is required."), name); + } + + if (!seen.Add(value.Name)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{prefix}.name' repeats '{value.Name}'."), + name); + } + + if (string.IsNullOrWhiteSpace(value.DataType)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{prefix}.dataType' is required so the value is sent with an explicit type."), + name); + } + + if (value.Value.ValueKind is JsonValueKind.Undefined or JsonValueKind.Null) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{prefix}.value' is required."), name); + } + + result.Add(new KeyValuePair + { + Key = new QualifiedName(value.Name), + Value = OpcUaJsonHelper.JsonElementToVariant(value.Value, value.DataType) + }); + } + + return [.. result]; + } + + private static NodeId ResolveRequiredNodeId(string? nameOrNodeId, string parameterName) + { + if (string.IsNullOrWhiteSpace(nameOrNodeId)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{parameterName}' is required."), + parameterName); + } + + if (!NodeId.TryParse(nameOrNodeId, out NodeId nodeId) || nodeId.IsNull) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{parameterName}' value '{nameOrNodeId}' is not a valid NodeId."), + parameterName); + } + + return nodeId; + } + + private static void ValidateDirection(double[] vector, string name) + { + if (vector is null || vector.Length != 3) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}' must be exactly 3 elements but had {(vector?.Length) ?? 0}."), + name); + } + + ValidateFiniteValues(vector, name); + + double magnitude = Math.Sqrt( + (vector[0] * vector[0]) + (vector[1] * vector[1]) + (vector[2] * vector[2])); + if (magnitude <= 1e-9) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}' must not be the zero vector."), + name); + } + } + + private static void ValidateOptionalTrajectoryComponent( + double[]? values, + int expectedCount, + string name) + { + if (values is null || values.Length == 0) + { + return; + } + + if (values.Length != expectedCount) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}' has {values.Length} values but the trajectory uses {expectedCount}."), + name); + } + + ValidateFiniteValues(values, name); + } + + private static void ValidateFiniteValues(double[] values, string name) + { + for (int i = 0; i < values.Length; i++) + { + if (!double.IsFinite(values[i])) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}[{i}]' must be a finite number but was {values[i]}."), + name); + } + } + } + + private static void ValidateFinite(double value, string name) + { + if (!double.IsFinite(value)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}' must be a finite number but was {value}."), + name); + } + } + + private static void ValidateNonNegative(double value, string name) + { + ValidateFinite(value, name); + if (value < 0) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}' must not be negative but was {value}."), + name); + } + } + + private static void ValidateSpeedFraction(double value, string name) + { + ValidateFinite(value, name); + if (value is < 0 or > 1) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{name}' must be within [0, 1] but was {value}."), + name); + } + } + } +} diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsIntentDtos.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsIntentDtos.cs new file mode 100644 index 0000000000..5710487304 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsIntentDtos.cs @@ -0,0 +1,1407 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.ComponentModel; +using System.Text.Json; +using System.Text.Json.Serialization; +using Opc.Ua.Robotics.Client.Intent; +using Opc.Ua.RobotIntent; + +namespace Opc.Ua.Mcp.Tools +{ + // CLR arrays are intentional at this JSON boundary: the MCP schema generator exposes + // ArrayOf as its backing memory object and cannot bind an incoming JSON array to it. + // The converter immediately projects these values into ArrayOf for OPC UA APIs. + + /// + /// Detail level for list operations. + /// + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum DetailLevel + { + /// + /// Return concise summaries (default). + /// + Summary, + + /// + /// Return full snapshots. + /// + Full + } + + /// + /// Work selector for filtering operations or missions. + /// + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum WorkSelector + { + /// + /// Return all operations/missions regardless of state. + /// + All, + + /// + /// Return only active (non-terminal) operations/missions. + /// + Active, + + /// + /// Return only terminal (succeeded/cancelled/failed) operations/missions. + /// + Terminal + } + + /// + /// Discriminator for intent kinds in missions and submit tools. + /// + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum IntentKind + { + /// + /// Point-to-point joint motion. + /// + JointMove, + + /// + /// Straight Cartesian segment. + /// + LinearMove, + + /// + /// Circular arc through a via point. + /// + CircularMove, + + /// + /// Time-parameterised joint trajectory. + /// + Trajectory, + + /// + /// Multi-waypoint Cartesian path. + /// + CartesianPath, + + /// + /// Force-controlled contact motion. + /// + Force, + + /// + /// Continuous arc welding. + /// + ArcWeld, + + /// + /// Discrete spot welding. + /// + SpotWeld, + + /// + /// Material dispensing. + /// + Dispense, + + /// + /// Fastening/joining. + /// + Fasten, + + /// + /// Pattern-based palletising. + /// + Palletise, + + /// + /// Surface finishing (sanding, polishing). + /// + SurfaceFinish, + + /// + /// Close or activate a gripper/tool. + /// + Grasp, + + /// + /// Open or deactivate a gripper/tool. + /// + Release, + + /// + /// Inbound pick from a source location. + /// + Pick, + + /// + /// Outbound place at a destination location. + /// + Place, + + /// + /// Fit or release a docked tool. + /// + ToolChange, + + /// + /// Write a controller output signal. + /// + SetOutput, + + /// + /// Call a server-side program. + /// + CallProgram, + + /// + /// Time or signal wait. + /// + Wait + } + + /// + /// A typed named value: name + dataType tag + JSON payload. + /// + public sealed class NamedTypedValueDto + { + /// + /// Gets or sets the attribute or argument name. + /// + [Description("Attribute or argument name.")] + public string Name { get; set; } = string.Empty; + + /// + /// Gets or sets the OPC UA data type name. + /// + [Description("OPC UA data type: Boolean, Int32, UInt32, Double, String, Float, " + + "Int16, UInt16, Int64, UInt64.")] + public string? DataType { get; set; } + + /// + /// Gets or sets the value as a JSON element. + /// + [Description("The value.")] + public JsonElement Value { get; set; } + } + + /// + /// A 3-D position as [x, y, z] in metres. + /// + public sealed class PosePositionDto + { + /// + /// Gets or sets the X coordinate in metres. + /// + [Description("X coordinate in metres.")] + public double X { get; set; } + + /// + /// Gets or sets the Y coordinate in metres. + /// + [Description("Y coordinate in metres.")] + public double Y { get; set; } + + /// + /// Gets or sets the Z coordinate in metres. + /// + [Description("Z coordinate in metres.")] + public double Z { get; set; } + } + + /// + /// A quaternion orientation [x, y, z, w]. + /// + public sealed class QuaternionDto + { + /// + /// Gets or sets the X quaternion component. + /// + [Description("Quaternion X component.")] + public double X { get; set; } + + /// + /// Gets or sets the Y quaternion component. + /// + [Description("Quaternion Y component.")] + public double Y { get; set; } + + /// + /// Gets or sets the Z quaternion component. + /// + [Description("Quaternion Z component.")] + public double Z { get; set; } + + /// + /// Gets or sets the W quaternion component. + /// + [Description("Quaternion W component.")] + public double W { get; set; } + } + + /// + /// A 3-D pose: position in metres and quaternion orientation, with optional frame reference. + /// + public sealed class PoseDto + { + /// + /// Gets or sets the position in metres. + /// + [Description("Position in metres. Required.")] + public PosePositionDto? Position { get; set; } + + /// + /// Gets or sets the quaternion orientation. + /// + [Description("Unit quaternion orientation [x, y, z, w]. Required.")] + public QuaternionDto? Orientation { get; set; } + + /// + /// Gets or sets the optional frame name or NodeId the pose is expressed in. + /// + [Description("Optional frame selector: published FrameId, frame name/BrowseName, or NodeId.")] + public string? FrameId { get; set; } + } + + /// + /// Motion constraints shared by all motion intents. + /// + public sealed class MotionConstraintsDto + { + /// + /// Gets or sets the speed fraction (0..1]. + /// + [Description("Speed fraction within [0, 1]; 0 leaves the controller default.")] + public double SpeedFraction { get; set; } + + /// + /// Gets or sets the Cartesian speed limit in m/s. + /// + [Description("Cartesian speed limit in m/s.")] + public double CartesianSpeed { get; set; } + + /// + /// Gets or sets the Cartesian acceleration limit in m/s². + /// + [Description("Cartesian acceleration limit in m/s².")] + public double CartesianAcceleration { get; set; } + + /// + /// Gets or sets the jerk limit. + /// + [Description("Jerk limit.")] + public double Jerk { get; set; } + } + + /// + /// Blend/termination mode for motion transitions. + /// + public sealed class BlendDto + { + /// + /// Gets or sets the termination mode: Exact or Blend. + /// + [Description("Termination mode: Exact or Blend.")] + public TerminationModeEnum Termination { get; set; } = TerminationModeEnum.Exact; + + /// + /// Gets or sets the blend radius in metres. + /// + [Description("Blend radius in metres.")] + public double Radius { get; set; } + } + + /// + /// Common fields shared by all intent inputs. + /// + public abstract class IntentCommonDto + { + /// + /// Gets or sets the optional intent identifier. + /// + [Description("Optional intent identifier.")] + public string? IntentId { get; set; } + + /// + /// Gets or sets the optional human-readable label. + /// + [Description("Optional human-readable label.")] + public string? Label { get; set; } + + /// + /// Gets or sets the buffer mode: Immediate, Buffered, or Aborting. + /// + [Description("Buffer mode: Immediate, Buffered, or Aborting.")] + public BufferModeEnum? BufferMode { get; set; } + + /// + /// Gets or sets the blocking mode: NonBlocking or Single. + /// + [Description("Blocking mode: NonBlocking or Single.")] + public BlockingModeEnum? BlockingMode { get; set; } + } + + /// + /// Common fields for all motion intents. + /// + public abstract class MotionIntentDto : IntentCommonDto + { + /// + /// Gets or sets the tool frame name or NodeId. + /// + [Description("Tool frame selector: frame name/BrowseName or NodeId.")] + public string? ToolFrame { get; set; } + + /// + /// Gets or sets the motion constraints. + /// + [Description("Motion constraints.")] + public MotionConstraintsDto? Constraints { get; set; } + + /// + /// Gets or sets the blend/termination for this motion. + /// + [Description("Blend/termination for this motion.")] + public BlendDto? Blend { get; set; } + + /// + /// Gets or sets the speed fraction as a shorthand for constraints.speedFraction. + /// + [Description("Shorthand for constraints.speedFraction, within [0, 1].")] + public double SpeedFraction { get; set; } + + /// + /// Gets or sets the Cartesian speed as a shorthand for constraints.cartesianSpeed. + /// + [Description("Shorthand for constraints.cartesianSpeed.")] + public double CartesianSpeed { get; set; } + } + + /// + /// Typed input for a JointMove intent. + /// + public sealed class JointMoveIntentInput : MotionIntentDto + { + /// + /// Gets or sets the joint target positions in radians. + /// + [Description("Joint target positions in radians. Provide this or targetPose.")] + public double[]? JointTargets { get; set; } + + /// + /// Gets or sets the target pose when joint targets are not supplied. + /// + [Description("Target pose (IK solved by controller). Provide this or jointTargets.")] + public PoseDto? TargetPose { get; set; } + } + + /// + /// Typed input for a LinearMove intent. + /// + public sealed class LinearMoveIntentInput : MotionIntentDto + { + /// + /// Gets or sets the target pose. + /// + [Description("Target pose: position in metres, unit quaternion orientation. Required.")] + public PoseDto? Target { get; set; } + } + + /// + /// Typed input for a CircularMove intent. + /// + public sealed class CircularMoveIntentInput : MotionIntentDto + { + /// + /// Gets or sets the intermediate via point on the arc. + /// + [Description("Intermediate via point on the arc. Required.")] + public PoseDto? ViaPoint { get; set; } + + /// + /// Gets or sets the end target of the arc. + /// + [Description("End target of the arc. Required.")] + public PoseDto? Target { get; set; } + } + + /// + /// A single trajectory point. + /// + public sealed class TrajectoryPointDto + { + /// + /// Gets or sets the time from start in seconds. + /// + [Description("Time from start in seconds.")] + public double TimeFromStart { get; set; } + + /// + /// Gets or sets the joint positions in radians. + /// + [Description("Joint positions in radians.")] + public double[] Positions { get; set; } = []; + + /// + /// Gets or sets optional joint velocities. + /// + [Description("Optional joint velocities.")] + public double[]? Velocities { get; set; } + + /// + /// Gets or sets optional joint accelerations. + /// + [Description("Optional joint accelerations.")] + public double[]? Accelerations { get; set; } + } + + /// + /// Typed input for a Trajectory intent. + /// + public sealed class TrajectoryIntentInput : MotionIntentDto + { + /// + /// Gets or sets the trajectory points. + /// + [Description("Trajectory points with timeFromStart and positions.")] + public TrajectoryPointDto[] Points { get; set; } = []; + } + + /// + /// A single Cartesian path waypoint. + /// + public sealed class CartesianWaypointDto + { + /// + /// Gets or sets the waypoint pose. + /// + [Description("Waypoint pose. Required.")] + public PoseDto? Pose { get; set; } + + /// + /// Gets or sets optional per-waypoint blend. + /// + [Description("Optional per-waypoint blend.")] + public BlendDto? Blend { get; set; } + } + + /// + /// Typed input for a CartesianPath intent. + /// + public sealed class CartesianPathIntentInput : MotionIntentDto + { + /// + /// Gets or sets the waypoints. + /// + [Description("Cartesian waypoints.")] + public CartesianWaypointDto[] Waypoints { get; set; } = []; + } + + /// + /// Typed input for a Force intent. + /// + public sealed class ForceIntentInput : MotionIntentDto + { + /// + /// Gets or sets the force direction as a 3-element unit vector. + /// + [Description("Force direction as 3-element unit vector.")] + public double[] Direction { get; set; } = []; + + /// + /// Gets or sets the contact force threshold in newtons. + /// + [Description("Contact force threshold in newtons.")] + public double ContactForce { get; set; } + + /// + /// Gets or sets the optional reference frame name or NodeId. + /// + [Description("Optional frame selector: published FrameId, frame name/BrowseName, or NodeId.")] + public string? FrameId { get; set; } + + /// + /// Gets or sets the maximum contact-search distance in metres. + /// + [Description("Maximum contact-search distance in metres.")] + public double MaxDistance { get; set; } + + /// + /// Gets or sets a value indicating whether to hold force after contact. + /// + [Description("Whether to hold force after contact.")] + public bool HoldForce { get; set; } + } + + /// + /// A typed variant value: dataType tag plus JSON payload. + /// + public sealed class TypedValueDto + { + /// + /// Gets or sets the OPC UA data type name. + /// + [Description("OPC UA data type: Boolean, Int32, UInt32, Double, String, Float, " + + "Int16, UInt16, Int64, UInt64.")] + public string? DataType { get; set; } + + /// + /// Gets or sets the value as a JSON element. + /// + [Description("The value to write.")] + public JsonElement Value { get; set; } + } + + /// + /// Common fields for all process intents. + /// + public abstract class ProcessIntentDto : IntentCommonDto + { + /// + /// Gets or sets the optional process program name or NodeId. + /// + [Description("Optional process program name or NodeId.")] + public string? ProcessProgram { get; set; } + + /// + /// Gets or sets optional key-value attributes as a JSON object. + /// + [Description("Optional typed key-value attributes.")] + public NamedTypedValueDto[]? Attributes { get; set; } + } + + /// + /// Typed input for an ArcWeld process intent. + /// + public sealed class ArcWeldIntentInput : ProcessIntentDto + { + /// + /// Gets or sets the welding voltage. + /// + [Description("Welding voltage.")] + public double Voltage { get; set; } + + /// + /// Gets or sets the wire feed speed. + /// + [Description("Wire feed speed.")] + public double WireFeedSpeed { get; set; } + + /// + /// Gets or sets the travel speed. + /// + [Description("Travel speed.")] + public double TravelSpeed { get; set; } + + /// + /// Gets or sets a value indicating whether seam tracking is enabled. + /// + [Description("Whether seam tracking is enabled.")] + public bool SeamTrackingEnabled { get; set; } + + /// + /// Gets or sets the weld procedure reference. + /// + [Description("Weld procedure reference.")] + public string? WeldProcedureRef { get; set; } + } + + /// + /// Typed input for a SpotWeld process intent. + /// + public sealed class SpotWeldIntentInput : ProcessIntentDto + { + /// + /// Gets or sets the weld schedule number. + /// + [Description("Weld schedule number.")] + public uint WeldSchedule { get; set; } + + /// + /// Gets or sets the gun force in newtons. + /// + [Description("Gun force in newtons.")] + public double GunForce { get; set; } + } + + /// + /// Typed input for a Dispense process intent. + /// + public sealed class DispenseIntentInput : ProcessIntentDto + { + /// + /// Gets or sets the flow rate. + /// + [Description("Flow rate.")] + public double FlowRate { get; set; } + + /// + /// Gets or sets the bead width in metres. + /// + [Description("Bead width in metres.")] + public double BeadWidth { get; set; } + + /// + /// Gets or sets the purge cycles count. + /// + [Description("Purge cycles count.")] + public uint PurgeCycles { get; set; } + } + + /// + /// Typed input for a Fasten process intent. + /// + public sealed class FastenIntentInput : ProcessIntentDto + { + /// + /// Gets or sets the joining-model node. + /// + [Description("Joining-model NodeId. Robot axes are not fastening joints.")] + public string? Joint { get; set; } + + /// + /// Gets or sets the program number. + /// + [Description("Program number.")] + public uint ProgramNumber { get; set; } + + /// + /// Gets or sets the target torque in newton-metres. + /// + [Description("Target torque in Nm.")] + public double TargetTorque { get; set; } + } + + /// + /// Typed input for a Palletise process intent. + /// + public sealed class PalletiseIntentInput : ProcessIntentDto + { + /// + /// Gets or sets the pattern name or NodeId. + /// + [Description("Pattern location name or NodeId.")] + public string? Pattern { get; set; } + + /// + /// Gets or sets the layer index. + /// + [Description("Layer index.")] + public uint Layer { get; set; } + + /// + /// Gets or sets the row index. + /// + [Description("Row index.")] + public uint Row { get; set; } + + /// + /// Gets or sets the column index. + /// + [Description("Column index.")] + public uint Column { get; set; } + } + + /// + /// Typed input for a SurfaceFinish process intent. + /// + public sealed class SurfaceFinishIntentInput : ProcessIntentDto + { + /// + /// Gets or sets the contact force in newtons. + /// + [Description("Contact force in newtons.")] + public double ContactForce { get; set; } + + /// + /// Gets or sets the feed rate. + /// + [Description("Feed rate.")] + public double FeedRate { get; set; } + + /// + /// Gets or sets the tool speed. + /// + [Description("Tool speed.")] + public double ToolSpeed { get; set; } + + /// + /// Gets or sets the step-over distance in metres. + /// + [Description("Step-over distance in metres.")] + public double StepOver { get; set; } + } + + /// + /// Typed input for a Grasp intent. + /// + public sealed class GraspIntentInput : IntentCommonDto + { + /// + /// Gets or sets the tool name or NodeId. + /// + [Description("Tool name or NodeId to activate.")] + public string Tool { get; set; } = string.Empty; + + /// + /// Gets or sets the grasp force in newtons. + /// + [Description("Grasp force in newtons.")] + public double Force { get; set; } + } + + /// + /// Typed input for a Release intent. + /// + public sealed class ReleaseIntentInput : IntentCommonDto + { + /// + /// Gets or sets the tool name or NodeId to deactivate. + /// + [Description("Tool name or NodeId to deactivate.")] + public string Tool { get; set; } = string.Empty; + } + + /// + /// Typed input for a Pick intent. + /// + public sealed class PickIntentInput : IntentCommonDto + { + /// + /// Gets or sets the source location name or NodeId. + /// + [Description("Source location name or NodeId.")] + public string Source { get; set; } = string.Empty; + + /// + /// Gets or sets the tool name or NodeId. + /// + [Description("Tool name or NodeId.")] + public string Tool { get; set; } = string.Empty; + + /// + /// Gets or sets the object class label of the workpiece being taken. + /// + [Description("Class label of the workpiece being taken.")] + public string? ObjectClass { get; set; } + } + + /// + /// Typed input for a Place intent. + /// + public sealed class PlaceIntentInput : IntentCommonDto + { + /// + /// Gets or sets the destination location name or NodeId. + /// + [Description("Destination location name or NodeId.")] + public string Destination { get; set; } = string.Empty; + + /// + /// Gets or sets the tool name or NodeId. + /// + [Description("Tool name or NodeId.")] + public string Tool { get; set; } = string.Empty; + } + + /// + /// Typed input for a ToolChange intent. + /// + public sealed class ToolChangeIntentInput : IntentCommonDto + { + /// + /// Gets or sets the tool name or NodeId to fit, or null to release. + /// + [Description("Tool name or NodeId to fit, or null to release.")] + public string? Tool { get; set; } + + /// + /// Gets or sets the dock station name or NodeId. + /// + [Description("Dock station name or NodeId.")] + public string? DockStation { get; set; } + } + + /// + /// Typed input for a SetOutput intent. + /// + public sealed class SetOutputIntentInput : IntentCommonDto + { + /// + /// Gets or sets the output signal name or NodeId. + /// + [Description("Output signal name or NodeId.")] + public string Output { get; set; } = string.Empty; + + /// + /// Gets or sets the value to write with data type guidance. + /// + [Description("Value to write. Required, with an explicit dataType.")] + public TypedValueDto? Value { get; set; } + } + + /// + /// Typed input for a CallProgram intent. + /// + public sealed class CallProgramIntentInput : IntentCommonDto + { + /// + /// Gets or sets the program name or NodeId. + /// + [Description("Program name or NodeId.")] + public string Program { get; set; } = string.Empty; + + /// + /// Gets or sets optional arguments as a JSON object of name/value pairs. + /// + [Description("Optional typed arguments.")] + public NamedTypedValueDto[]? Arguments { get; set; } + } + + /// + /// Typed input for a Wait intent. + /// + public sealed class WaitIntentInput : IntentCommonDto + { + /// + /// Gets or sets the wait duration in seconds. + /// + [Description("Wait duration in seconds.")] + public double Duration { get; set; } + + /// + /// Gets or sets the signal name or NodeId to wait for. + /// + [Description("Signal name or NodeId to wait for.")] + public string? Signal { get; set; } + } + + /// + /// Typed input for a mission step, carrying a discriminated intent. + /// + public sealed class MissionStepInput + { + /// + /// Gets or sets the step identifier. + /// + [Description("Step identifier.")] + public string StepId { get; set; } = string.Empty; + + /// + /// Gets or sets the optional sequence identifier. + /// + [Description("Optional sequence identifier.")] + public uint? SequenceId { get; set; } + + /// + /// Gets or sets a value indicating whether the step is released for execution. + /// + [Description("Whether the step is released for execution.")] + public bool Released { get; set; } + + /// + /// Gets or sets the error policy: Abort or Skip. + /// + [Description("Error policy: Abort or Skip.")] + public ErrorPolicyEnum? ErrorPolicy { get; set; } + + /// + /// Gets or sets the fallback step identifier on error. + /// + [Description("Fallback step identifier on error.")] + public string? FallbackStepId { get; set; } + + /// + /// Gets or sets the intent for this step. + /// + [Description("The intent for this step. Required.")] + public MissionIntentInput? Intent { get; set; } + } + + /// + /// A discriminated union for mission step intents. Set kind and the matching typed payload. + /// + public sealed class MissionIntentInput + { + /// + /// Gets or sets the intent kind discriminator. + /// + [Description("Intent kind discriminator.")] + public IntentKind Kind { get; set; } + + /// + /// Gets or sets the JointMove payload. + /// + [Description("JointMove payload. Set when kind is 'JointMove'.")] + public JointMoveIntentInput? JointMove { get; set; } + + /// + /// Gets or sets the LinearMove payload. + /// + [Description("LinearMove payload. Set when kind is 'LinearMove'.")] + public LinearMoveIntentInput? LinearMove { get; set; } + + /// + /// Gets or sets the CircularMove payload. + /// + [Description("CircularMove payload. Set when kind is 'CircularMove'.")] + public CircularMoveIntentInput? CircularMove { get; set; } + + /// + /// Gets or sets the Trajectory payload. + /// + [Description("Trajectory payload. Set when kind is 'Trajectory'.")] + public TrajectoryIntentInput? Trajectory { get; set; } + + /// + /// Gets or sets the CartesianPath payload. + /// + [Description("CartesianPath payload. Set when kind is 'CartesianPath'.")] + public CartesianPathIntentInput? CartesianPath { get; set; } + + /// + /// Gets or sets the Force payload. + /// + [Description("Force payload. Set when kind is 'Force'.")] + public ForceIntentInput? Force { get; set; } + + /// + /// Gets or sets the ArcWeld payload. + /// + [Description("ArcWeld payload. Set when kind is 'ArcWeld'.")] + public ArcWeldIntentInput? ArcWeld { get; set; } + + /// + /// Gets or sets the SpotWeld payload. + /// + [Description("SpotWeld payload. Set when kind is 'SpotWeld'.")] + public SpotWeldIntentInput? SpotWeld { get; set; } + + /// + /// Gets or sets the Dispense payload. + /// + [Description("Dispense payload. Set when kind is 'Dispense'.")] + public DispenseIntentInput? Dispense { get; set; } + + /// + /// Gets or sets the Fasten payload. + /// + [Description("Fasten payload. Set when kind is 'Fasten'.")] + public FastenIntentInput? Fasten { get; set; } + + /// + /// Gets or sets the Palletise payload. + /// + [Description("Palletise payload. Set when kind is 'Palletise'.")] + public PalletiseIntentInput? Palletise { get; set; } + + /// + /// Gets or sets the SurfaceFinish payload. + /// + [Description("SurfaceFinish payload. Set when kind is 'SurfaceFinish'.")] + public SurfaceFinishIntentInput? SurfaceFinish { get; set; } + + /// + /// Gets or sets the Grasp payload. + /// + [Description("Grasp payload. Set when kind is 'Grasp'.")] + public GraspIntentInput? Grasp { get; set; } + + /// + /// Gets or sets the Release payload. + /// + [Description("Release payload. Set when kind is 'Release'.")] + public ReleaseIntentInput? Release { get; set; } + + /// + /// Gets or sets the Pick payload. + /// + [Description("Pick payload. Set when kind is 'Pick'.")] + public PickIntentInput? Pick { get; set; } + + /// + /// Gets or sets the Place payload. + /// + [Description("Place payload. Set when kind is 'Place'.")] + public PlaceIntentInput? Place { get; set; } + + /// + /// Gets or sets the ToolChange payload. + /// + [Description("ToolChange payload. Set when kind is 'ToolChange'.")] + public ToolChangeIntentInput? ToolChange { get; set; } + + /// + /// Gets or sets the SetOutput payload. + /// + [Description("SetOutput payload. Set when kind is 'SetOutput'.")] + public SetOutputIntentInput? SetOutput { get; set; } + + /// + /// Gets or sets the CallProgram payload. + /// + [Description("CallProgram payload. Set when kind is 'CallProgram'.")] + public CallProgramIntentInput? CallProgram { get; set; } + + /// + /// Gets or sets the Wait payload. + /// + [Description("Wait payload. Set when kind is 'Wait'.")] + public WaitIntentInput? Wait { get; set; } + } + + /// + /// A mission transition between steps. + /// + public sealed class MissionTransitionInput + { + /// + /// Gets or sets the source step identifier. + /// + [Description("Source step identifier.")] + public string FromStepId { get; set; } = string.Empty; + + /// + /// Gets or sets the target step identifier. + /// + [Description("Target step identifier.")] + public string ToStepId { get; set; } = string.Empty; + + /// + /// Gets or sets the divergence kind: Alternative or Parallel. + /// + [Description("Divergence kind: Alternative or Parallel.")] + public DivergenceKindEnum? DivergenceKind { get; set; } + } + + /// + /// Query parameters for listing operations with paging. + /// + public sealed class OperationListQuery + { + /// + /// Gets or sets an optional intent identifier to filter by. + /// + [Description("Optional intent identifier to filter by.")] + public string? IntentId { get; set; } + + /// + /// Gets or sets an optional mission identifier to filter by. + /// + [Description("Optional mission identifier to filter by.")] + public string? MissionId { get; set; } + + /// + /// Gets or sets an optional execution state filter. + /// + [Description("Execution state filter.")] + public ExecutionStateEnum? ExecutionState { get; set; } + + /// + /// Gets or sets the work selector: All, Active, or Terminal. + /// + [Description("Work selector: All (default), Active, or Terminal.")] + public WorkSelector Work { get; set; } = WorkSelector.All; + + /// + /// Gets or sets the detail level: Summary or Full. + /// + [Description("Detail level: Summary (default) or Full.")] + public DetailLevel Detail { get; set; } = DetailLevel.Summary; + + /// + /// Gets or sets the maximum page size (default 20, max 100). + /// + [Description("Maximum page size (default 20, max 100).")] + public int? PageSize { get; set; } + + /// + /// Gets or sets the opaque cursor for the next page. + /// + [Description("Opaque cursor for continuation.")] + public string? Cursor { get; set; } + } + + /// + /// A summary operation snapshot, omitting pose and full output. + /// + public sealed class OperationSummary + { + /// + /// Gets or sets the operation NodeId string. + /// + [Description("Operation NodeId.")] + public string Operation { get; set; } = string.Empty; + + /// + /// Gets or sets the intent identifier. + /// + [Description("Intent identifier.")] + public string IntentId { get; set; } = string.Empty; + + /// + /// Gets or sets the execution state name. + /// + [Description("Execution state.")] + public ExecutionStateEnum ExecutionState { get; set; } + + /// + /// Gets or sets the progress fraction, or -1 when unknown. + /// + [Description("Progress fraction, or -1 when unknown.")] + public double Progress { get; set; } = -1; + + /// + /// Gets or sets the queue position. + /// + [Description("Queue position (1=next, 0=not queued).")] + public uint QueuePosition { get; set; } + + /// + /// Gets or sets the failure reason. + /// + [Description("Failure classification when in a terminal state.")] + public IntentFailureEnum? Failure { get; set; } + + /// + /// Gets or sets a human-readable message. + /// + [Description("Human-readable message.")] + public string? Message { get; set; } + + /// + /// Gets or sets the mission identifier. + /// + [Description("Mission identifier, if applicable.")] + public string? MissionId { get; set; } + } + + /// + /// Paged result for operation listing. + /// + public sealed class OperationListResult + { + /// + /// Gets or sets the total matching operations. + /// + [Description("Total number of matching operations.")] + public int Total { get; set; } + + /// + /// Gets or sets the returned count. + /// + [Description("Number of items on this page.")] + public int Returned { get; set; } + + /// + /// Gets or sets the next cursor, or null if no more pages. + /// + [Description("Opaque cursor for the next page, or null.")] + public string? NextCursor { get; set; } + + /// + /// Gets or sets summaries when detail is 'summary'. + /// + [Description("Operation summaries.")] + public OperationSummary[]? Summaries { get; set; } + + /// + /// Gets or sets full snapshots when detail is 'full'. + /// + [Description("Full operation snapshots.")] + public IntentOperationSnapshot[]? Operations { get; set; } + } + + /// + /// Query parameters for listing missions with paging. + /// + public sealed class MissionListQuery + { + /// + /// Gets or sets an optional mission identifier to filter by. + /// + [Description("Optional mission identifier to filter by.")] + public string? MissionId { get; set; } + + /// + /// Gets or sets an optional execution state filter. + /// + [Description("Execution state filter.")] + public ExecutionStateEnum? ExecutionState { get; set; } + + /// + /// Gets or sets the work selector: All, Active, or Terminal. + /// + [Description("Work selector: All (default), Active, or Terminal.")] + public WorkSelector Work { get; set; } = WorkSelector.All; + + /// + /// Gets or sets the detail level: Summary or Full. + /// + [Description("Detail level: Summary (default) or Full.")] + public DetailLevel Detail { get; set; } = DetailLevel.Summary; + + /// + /// Gets or sets the maximum page size (default 20, max 100). + /// + [Description("Maximum page size (default 20, max 100).")] + public int? PageSize { get; set; } + + /// + /// Gets or sets the opaque cursor for the next page. + /// + [Description("Opaque cursor for continuation.")] + public string? Cursor { get; set; } + } + + /// + /// A concise mission summary. + /// + public sealed class MissionSummary + { + /// + /// Gets or sets the mission NodeId string. + /// + [Description("Mission NodeId.")] + public string MissionNode { get; set; } = string.Empty; + + /// + /// Gets or sets the mission identifier. + /// + [Description("Mission identifier.")] + public string MissionId { get; set; } = string.Empty; + + /// + /// Gets or sets the mission update identifier. + /// + [Description("Mission update identifier.")] + public uint MissionUpdateId { get; set; } + + /// + /// Gets or sets the execution state name. + /// + [Description("Execution state.")] + public ExecutionStateEnum ExecutionState { get; set; } + + /// + /// Gets or sets the currently executing step identifier. + /// + [Description("Currently executing step.")] + public string? CurrentStepId { get; set; } + + /// + /// Gets or sets the failure reason. + /// + [Description("Failure classification when in a terminal state.")] + public IntentFailureEnum? Failure { get; set; } + + /// + /// Gets or sets a human-readable message. + /// + [Description("Human-readable message.")] + public string? Message { get; set; } + + /// + /// Gets or sets the number of released steps. + /// + [Description("Number of released steps.")] + public uint ReleasedStepCount { get; set; } + + /// + /// Gets or sets the per-step operation summaries. + /// + [Description("Per-step operation summaries: stepId, intentId, operation NodeId, state.")] + public MissionStepOperationSummary[] Steps { get; set; } = []; + } + + /// + /// A concise per-step summary within a mission. + /// + public sealed class MissionStepOperationSummary + { + /// + /// Gets or sets the step identifier. + /// + [Description("Step identifier.")] + public string StepId { get; set; } = string.Empty; + + /// + /// Gets or sets the intent identifier. + /// + [Description("Intent identifier.")] + public string IntentId { get; set; } = string.Empty; + + /// + /// Gets or sets the operation NodeId string, or null if not yet executing. + /// + [Description("Operation NodeId, or null if not yet executing.")] + public string? Operation { get; set; } + + /// + /// Gets or sets the step execution state. + /// + [Description("Step execution state.")] + public ExecutionStateEnum State { get; set; } + } + + /// + /// Paged result for mission listing. + /// + public sealed class MissionListResult + { + /// + /// Gets or sets the total matching missions. + /// + [Description("Total number of matching missions.")] + public int Total { get; set; } + + /// + /// Gets or sets the returned count. + /// + [Description("Number of items on this page.")] + public int Returned { get; set; } + + /// + /// Gets or sets the next cursor, or null if no more pages. + /// + [Description("Opaque cursor for the next page, or null.")] + public string? NextCursor { get; set; } + + /// + /// Gets or sets summaries when detail is 'summary'. + /// + [Description("Mission summaries.")] + public MissionSummary[]? Summaries { get; set; } + + /// + /// Gets or sets full snapshots when detail is 'full'. + /// + [Description("Full mission snapshots.")] + public MissionSnapshot[]? Missions { get; set; } + } +} diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsIntentJson.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsIntentJson.cs deleted file mode 100644 index b8d1496d25..0000000000 --- a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsIntentJson.cs +++ /dev/null @@ -1,598 +0,0 @@ -/* ======================================================================== - * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. - * - * OPC Foundation MIT License 1.00 - * - * Permission is hereby granted, free of charge, to any person - * obtaining a copy of this software and associated documentation - * files (the "Software"), to deal in the Software without - * restriction, including without limitation the rights to use, - * copy, modify, merge, publish, distribute, sublicense, and/or sell - * copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following - * conditions: - * - * The above copyright notice and this permission notice shall be - * included in all copies or substantial portions of the Software. - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, - * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES - * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND - * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT - * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, - * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING - * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - * - * The complete license agreement can be found here: - * http://opcfoundation.org/License/MIT/1.00/ - * ======================================================================*/ - -using System; -using System.Collections.Generic; -using System.Globalization; -using System.Text.Json; -using Opc.Ua.Mcp.Serialization; -using Opc.Ua.Robotics.Client.Intent; -using Opc.Ua.RobotIntent; - -namespace Opc.Ua.Mcp.Tools -{ - internal static class RoboticsIntentJson - { - public static IntentDataType BuildIntent(string intentKind, string? intentJson, uint axisCount = 0) - { - ArgumentException.ThrowIfNullOrWhiteSpace(intentKind); - - using JsonDocument document = ParseObject(intentJson); - JsonElement root = document.RootElement; - IntentDataType intent = intentKind.Trim().ToLowerInvariant() switch - { - "jointmove" or "joint_move" => BuildJointMove(root, axisCount), - "linearmove" or "linear_move" => BuildLinearMove(root), - "circularmove" or "circular_move" => BuildCircularMove(root), - "trajectory" => BuildTrajectory(root), - "cartesianpath" or "cartesian_path" => BuildCartesianPath(root), - "force" => BuildForce(root), - "arcweld" or "arc_weld" => BuildProcess(RobotIntentBuilder.ArcWeld(), root), - "spotweld" or "spot_weld" => BuildProcess(RobotIntentBuilder.SpotWeld(), root), - "dispense" => BuildProcess(RobotIntentBuilder.Dispense(), root), - "fasten" => BuildProcess(RobotIntentBuilder.Fasten(), root), - "palletise" or "palletize" => BuildProcess(RobotIntentBuilder.Palletise(), root), - "surfacefinish" or "surface_finish" => BuildProcess(RobotIntentBuilder.SurfaceFinish(), root), - "grasp" => RobotIntentBuilder.Grasp(GetNode(root, "tool"), GetDouble(root, "force")).Build(), - "release" => RobotIntentBuilder.Release(GetNode(root, "tool")).Build(), - "pick" => RobotIntentBuilder.Pick(GetNode(root, "source"), GetNode(root, "tool")).Build(), - "place" => RobotIntentBuilder.Place(GetNode(root, "destination"), GetNode(root, "tool")).Build(), - "toolchange" or "tool_change" => RobotIntentBuilder.ToolChange( - GetNode(root, "tool"), - GetNode(root, "dockStation")).Build(), - "setoutput" or "set_output" => RobotIntentBuilder.SetOutput( - GetNode(root, "output"), - GetVariant(root, "value", GetString(root, "dataType"))).Build(), - "callprogram" or "call_program" => BuildCallProgram(root), - "wait" => BuildWait(root), - _ => throw new ArgumentException( - string.Create(CultureInfo.InvariantCulture, $"Unknown Robot Intent kind '{intentKind}'."), - nameof(intentKind)) - }; - ApplyCommon(root, intent); - return intent; - } - - public static ArrayOf BuildMissionSteps(string? stepsJson) - { - if (string.IsNullOrWhiteSpace(stepsJson)) - { - return []; - } - - using JsonDocument document = ParseDocument(stepsJson, nameof(stepsJson), "Mission steps JSON"); - if (document.RootElement.ValueKind != JsonValueKind.Array) - { - throw new ArgumentException("Mission steps JSON must be an array.", nameof(stepsJson)); - } - - var steps = new List(); - uint sequence = 1; - foreach (JsonElement element in document.RootElement.EnumerateArray()) - { - JsonElement intentElement = GetRequiredProperty(element, "intent"); - string kind = GetRequiredString(intentElement, "kind"); - string payload = intentElement.GetRawText(); - var step = new MissionStepDataType - { - StepId = GetRequiredString(element, "stepId"), - SequenceId = GetUInt(element, "sequenceId", sequence), - Released = GetBool(element, "released", false), - Intent = BuildIntent(kind, payload), - ErrorPolicy = GetEnum(element, "errorPolicy", ErrorPolicyEnum.Abort), - FallbackStepId = GetString(element, "fallbackStepId") ?? string.Empty - }; - steps.Add(step); - sequence++; - } - - return [.. steps]; - } - - public static ArrayOf BuildMissionTransitions(string? transitionsJson) - { - if (string.IsNullOrWhiteSpace(transitionsJson)) - { - return []; - } - - using JsonDocument document = ParseDocument( - transitionsJson, - nameof(transitionsJson), - "Mission transitions JSON"); - if (document.RootElement.ValueKind != JsonValueKind.Array) - { - throw new ArgumentException("Mission transitions JSON must be an array.", nameof(transitionsJson)); - } - - var transitions = new List(); - foreach (JsonElement element in document.RootElement.EnumerateArray()) - { - transitions.Add(new MissionTransitionDataType - { - FromStepId = GetRequiredString(element, "fromStepId"), - ToStepId = GetRequiredString(element, "toStepId"), - DivergenceKind = GetEnum(element, "divergenceKind", DivergenceKindEnum.Alternative), - Condition = MissionCondition.Always() - }); - } - - return [.. transitions]; - } - - private static JointMoveIntentDataType BuildJointMove(JsonElement root, uint axisCount) - { - JointMoveIntentBuilder builder = RobotIntentBuilder.JointMove(axisCount); - if (root.TryGetProperty("jointTargets", out JsonElement jointTargets)) - { - return builder.ToJoints(GetDoubleArray(jointTargets, "jointTargets")).Build(); - } - - return builder.ToPose(GetPose(root, "targetPose")).Build(); - } - - private static LinearMoveIntentDataType BuildLinearMove(JsonElement root) - { - return RobotIntentBuilder.LinearMove(GetPose(root, "target"), GetDouble(root, "speedFraction", 0)).Build(); - } - - private static CircularMoveIntentDataType BuildCircularMove(JsonElement root) - { - return RobotIntentBuilder.CircularMove(GetPose(root, "viaPoint"), GetPose(root, "target")).Build(); - } - - private static TrajectoryIntentDataType BuildTrajectory(JsonElement root) - { - var points = new List(); - foreach (JsonElement point in GetRequiredArray(root, "points").EnumerateArray()) - { - points.Add(new TrajectoryPointDataType - { - TimeFromStart = GetDouble(point, "timeFromStart"), - Positions = GetDoubleArray(point, "positions", "points"), - Velocities = TryGetDoubleArray(point, "velocities"), - Accelerations = TryGetDoubleArray(point, "accelerations") - }); - } - - return RobotIntentBuilder.Trajectory().WithPoints([.. points]).Build(); - } - - private static CartesianPathIntentDataType BuildCartesianPath(JsonElement root) - { - var waypoints = new List(); - foreach (JsonElement waypoint in GetRequiredArray(root, "waypoints").EnumerateArray()) - { - waypoints.Add(new PathWaypointDataType - { - Pose = GetPose(waypoint, "pose"), - Blend = GetBlend(waypoint) - }); - } - - return RobotIntentBuilder.CartesianPath().WithWaypoints([.. waypoints]).Build(); - } - - private static ForceIntentDataType BuildForce(JsonElement root) - { - ForceIntentDataType intent = RobotIntentBuilder.Force( - GetDoubleArray(root, "direction", "force"), - GetDouble(root, "contactForce")).Build(); - intent.FrameId = GetString(root, "frameId") ?? string.Empty; - intent.MaxDistance = GetDouble(root, "maxDistance", 0); - intent.HoldForce = GetBool(root, "holdForce", false); - return intent; - } - - private static IntentDataType BuildProcess(ProcessIntentBuilder builder, JsonElement root) - where TIntent : ProcessIntentDataType - { - NodeId processProgram = GetNode(root, "processProgram"); - if (!processProgram.IsNull) - { - builder.WithProcessProgram(processProgram); - } - builder.WithAttributes(GetAttributes(root, "attributes")); - TIntent intent = builder.Build(); - ApplyProcessFields(root, intent); - return intent; - } - - private static CallProgramIntentDataType BuildCallProgram(JsonElement root) - { - CallProgramIntentDataType intent = RobotIntentBuilder.CallProgram(GetNode(root, "program")).Build(); - intent.Arguments = GetAttributes(root, "arguments"); - return intent; - } - - private static WaitIntentDataType BuildWait(JsonElement root) - { - WaitIntentDataType intent = RobotIntentBuilder.Wait(GetDouble(root, "duration")).Build(); - intent.Signal = GetNode(root, "signal"); - return intent; - } - - private static void ApplyCommon(JsonElement root, IntentDataType intent) - { - intent.IntentId = GetString(root, "intentId") ?? intent.IntentId; - string? label = GetString(root, "label"); - if (!string.IsNullOrEmpty(label)) - { - intent.Label = new LocalizedText(label); - } - intent.BufferMode = GetEnum(root, "bufferMode", intent.BufferMode); - intent.BlockingMode = GetEnum(root, "blockingMode", intent.BlockingMode); - if (intent is MotionIntentDataType motion) - { - motion.ToolFrame = GetNode(root, "toolFrame"); - motion.Constraints = GetConstraints(root); - motion.Blend = GetBlend(root); - } - } - - private static void ApplyProcessFields(JsonElement root, TIntent intent) - where TIntent : ProcessIntentDataType - { - switch (intent) - { - case ArcWeldIntentDataType arcWeld: - arcWeld.Voltage = GetDouble(root, "voltage", arcWeld.Voltage); - arcWeld.WireFeedSpeed = GetDouble(root, "wireFeedSpeed", arcWeld.WireFeedSpeed); - arcWeld.TravelSpeed = GetDouble(root, "travelSpeed", arcWeld.TravelSpeed); - arcWeld.SeamTrackingEnabled = GetBool(root, "seamTrackingEnabled", arcWeld.SeamTrackingEnabled); - arcWeld.WeldProcedureRef = GetString(root, "weldProcedureRef") ?? arcWeld.WeldProcedureRef; - break; - case SpotWeldIntentDataType spotWeld: - spotWeld.WeldSchedule = GetUInt(root, "weldSchedule", spotWeld.WeldSchedule); - spotWeld.GunForce = GetDouble(root, "gunForce", spotWeld.GunForce); - break; - case DispenseIntentDataType dispense: - dispense.FlowRate = GetDouble(root, "flowRate", dispense.FlowRate); - dispense.BeadWidth = GetDouble(root, "beadWidth", dispense.BeadWidth); - dispense.PurgeCycles = GetUInt(root, "purgeCycles", dispense.PurgeCycles); - break; - case FastenIntentDataType fasten: - fasten.Joint = GetNode(root, "joint"); - fasten.ProgramNumber = GetUInt(root, "programNumber", fasten.ProgramNumber); - fasten.TargetTorque = GetDouble(root, "targetTorque", fasten.TargetTorque); - break; - case PalletiseIntentDataType palletise: - palletise.Pattern = GetNode(root, "pattern"); - palletise.Layer = GetUInt(root, "layer", palletise.Layer); - palletise.Row = GetUInt(root, "row", palletise.Row); - palletise.Column = GetUInt(root, "column", palletise.Column); - break; - case SurfaceFinishIntentDataType surfaceFinish: - surfaceFinish.ContactForce = GetDouble(root, "contactForce", surfaceFinish.ContactForce); - surfaceFinish.FeedRate = GetDouble(root, "feedRate", surfaceFinish.FeedRate); - surfaceFinish.ToolSpeed = GetDouble(root, "toolSpeed", surfaceFinish.ToolSpeed); - surfaceFinish.StepOver = GetDouble(root, "stepOver", surfaceFinish.StepOver); - break; - } - } - - private static JsonDocument ParseObject(string? json) - { - JsonDocument document = ParseDocument( - string.IsNullOrWhiteSpace(json) ? "{}" : json, - nameof(json), - "Intent JSON"); - if (document.RootElement.ValueKind != JsonValueKind.Object) - { - document.Dispose(); - throw new ArgumentException("Intent JSON must be an object.", nameof(json)); - } - return document; - } - - /// - /// Parses agent-supplied JSON, reporting a syntax error as an argument - /// error. - /// - /// - /// The text comes from a language model, so malformed input is expected - /// rather than exceptional. - /// signals it with a , which does not match - /// what the tool descriptions promise the agent, and which a caller - /// distinguishing bad input from a genuine fault would have to know to - /// catch separately. Every rejection from this class is an - /// ; the original error is kept as the - /// inner exception so the position information is not lost. - /// - private static JsonDocument ParseDocument(string json, string paramName, string description) - { - try - { - return JsonDocument.Parse(json); - } - catch (JsonException ex) - { - throw new ArgumentException( - string.Create( - CultureInfo.InvariantCulture, - $"{description} is not valid JSON: {ex.Message}"), - paramName, - ex); - } - } - - private static Pose3DDataType GetPose(JsonElement root, string name) - { - JsonElement pose = GetRequiredObject(root, name); - ArrayOf position = GetDoubleArray(pose, "position", name); - ArrayOf orientation = GetDoubleArray(pose, "orientation", name); - if (position.Count != 3) - { - throw new ArgumentException( - string.Create( - CultureInfo.InvariantCulture, - $"'{name}.position' must hold exactly 3 numbers (x, y, z) but held {position.Count}.")); - } - if (orientation.Count != 4) - { - throw new ArgumentException( - string.Create( - CultureInfo.InvariantCulture, - $"'{name}.orientation' must hold exactly 4 numbers (quaternion x, y, z, w) " + - $"but held {orientation.Count}.")); - } - - return RobotIntentBuilder.Pose( - position[0], - position[1], - position[2], - orientation[0], - orientation[1], - orientation[2], - orientation[3], - GetString(pose, "frameId") ?? string.Empty); - } - - private static MotionConstraintsDataType GetConstraints(JsonElement root) - { - if (!root.TryGetProperty("constraints", out JsonElement constraints)) - { - return new MotionConstraintsDataType - { - SpeedFraction = GetDouble(root, "speedFraction", 0), - CartesianSpeed = GetDouble(root, "cartesianSpeed", 0) - }; - } - - return new MotionConstraintsDataType - { - SpeedFraction = GetDouble(constraints, "speedFraction", 0), - CartesianSpeed = GetDouble(constraints, "cartesianSpeed", 0), - CartesianAcceleration = GetDouble(constraints, "cartesianAcceleration", 0), - Jerk = GetDouble(constraints, "jerk", 0) - }; - } - - private static BlendDataType GetBlend(JsonElement root) - { - JsonElement blend = root.TryGetProperty("blend", out JsonElement nested) ? nested : root; - return new BlendDataType - { - Termination = GetEnum(blend, "termination", TerminationModeEnum.Exact), - Radius = GetDouble(blend, "radius", 0) - }; - } - - private static ArrayOf GetAttributes(JsonElement root, string propertyName) - { - if (!root.TryGetProperty(propertyName, out JsonElement attributes) || - attributes.ValueKind == JsonValueKind.Null) - { - return []; - } - - if (attributes.ValueKind != JsonValueKind.Object) - { - throw new ArgumentException( - string.Create(CultureInfo.InvariantCulture, $"{propertyName} must be a JSON object.")); - } - - var values = new List(); - foreach (JsonProperty attribute in attributes.EnumerateObject()) - { - values.Add(new KeyValuePair - { - Key = new QualifiedName(attribute.Name), - Value = OpcUaJsonHelper.JsonElementToVariant(attribute.Value) - }); - } - - return [.. values]; - } - - private static ArrayOf TryGetDoubleArray(JsonElement root, string propertyName) - { - return root.TryGetProperty(propertyName, out JsonElement value) - ? GetDoubleArray(value, propertyName) - : []; - } - - private static ArrayOf GetDoubleArray(JsonElement root, string propertyName, string owner) - { - _ = GetRequiredProperty(root, propertyName); - return GetDoubleArray( - root.GetProperty(propertyName), - string.Create(CultureInfo.InvariantCulture, $"{owner}.{propertyName}")); - } - - private static ArrayOf GetDoubleArray(JsonElement element, string description) - { - if (element.ValueKind != JsonValueKind.Array) - { - throw new ArgumentException( - string.Create( - CultureInfo.InvariantCulture, - $"'{description}' must be a JSON array of numbers but was {element.ValueKind}.")); - } - - var values = new List(); - foreach (JsonElement item in element.EnumerateArray()) - { - if (item.ValueKind != JsonValueKind.Number || !item.TryGetDouble(out double value)) - { - throw new ArgumentException( - string.Create( - CultureInfo.InvariantCulture, - $"'{description}' must hold only numbers a Double can represent.")); - } - values.Add(value); - } - - return [.. values]; - } - - private static JsonElement GetRequiredArray(JsonElement root, string propertyName) - { - JsonElement value = GetRequiredProperty(root, propertyName); - if (value.ValueKind != JsonValueKind.Array) - { - throw new ArgumentException( - string.Create( - CultureInfo.InvariantCulture, - $"Property '{propertyName}' must be a JSON array but was {value.ValueKind}.")); - } - return value; - } - - private static JsonElement GetRequiredObject(JsonElement root, string propertyName) - { - JsonElement value = GetRequiredProperty(root, propertyName); - if (value.ValueKind != JsonValueKind.Object) - { - throw new ArgumentException( - string.Create( - CultureInfo.InvariantCulture, - $"Property '{propertyName}' must be a JSON object but was {value.ValueKind}.")); - } - return value; - } - - private static NodeId GetNode(JsonElement root, string propertyName) - { - string? value = GetString(root, propertyName); - if (string.IsNullOrWhiteSpace(value)) - { - return NodeId.Null; - } - - // NodeId.Parse signals a bad identifier with either ArgumentException - // or ServiceResultException depending on the error; TryParse collapses - // both into the argument error the tool descriptions promise. - if (!NodeId.TryParse(value, out NodeId nodeId)) - { - throw new ArgumentException( - string.Create( - CultureInfo.InvariantCulture, - $"Property '{propertyName}' is not a valid NodeId: '{value}'."), - propertyName); - } - - return nodeId; - } - - private static Variant GetVariant(JsonElement root, string propertyName, string? dataType) - { - return root.TryGetProperty(propertyName, out JsonElement value) - ? OpcUaJsonHelper.JsonElementToVariant(value, dataType) - : Variant.Null; - } - - private static JsonElement GetRequiredProperty(JsonElement root, string propertyName) - { - if (!root.TryGetProperty(propertyName, out JsonElement value)) - { - throw new ArgumentException( - string.Create(CultureInfo.InvariantCulture, $"Missing required property '{propertyName}'.")); - } - return value; - } - - private static string GetRequiredString(JsonElement root, string propertyName) - { - string? value = GetString(root, propertyName); - return string.IsNullOrWhiteSpace(value) - ? throw new ArgumentException( - string.Create(CultureInfo.InvariantCulture, $"Missing required property '{propertyName}'.")) - : value; - } - - private static string? GetString(JsonElement root, string propertyName) - { - return root.TryGetProperty(propertyName, out JsonElement value) && value.ValueKind == JsonValueKind.String - ? value.GetString() - : null; - } - - private static double GetDouble(JsonElement root, string propertyName, double defaultValue = 0) - { - return root.TryGetProperty(propertyName, out JsonElement value) && value.ValueKind == JsonValueKind.Number - ? value.GetDouble() - : defaultValue; - } - - private static uint GetUInt(JsonElement root, string propertyName, uint defaultValue) - { - if (!root.TryGetProperty(propertyName, out JsonElement value) || - value.ValueKind != JsonValueKind.Number) - { - return defaultValue; - } - if (!value.TryGetUInt32(out uint parsed)) - { - throw new ArgumentException( - string.Create( - CultureInfo.InvariantCulture, - $"Property '{propertyName}' must be a whole number a UInt32 can represent.")); - } - return parsed; - } - - private static bool GetBool(JsonElement root, string propertyName, bool defaultValue) - { - return root.TryGetProperty(propertyName, out JsonElement value) && - (value.ValueKind == JsonValueKind.True || value.ValueKind == JsonValueKind.False) - ? value.GetBoolean() - : defaultValue; - } - - private static TEnum GetEnum(JsonElement root, string propertyName, TEnum defaultValue) - where TEnum : struct - { - string? value = GetString(root, propertyName); - return value != null && Enum.TryParse(value, ignoreCase: true, out TEnum parsed) - ? parsed - : defaultValue; - } - } -} diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsListPaging.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsListPaging.cs new file mode 100644 index 0000000000..4d7431df15 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsListPaging.cs @@ -0,0 +1,402 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Buffers.Binary; +using System.Collections.Generic; +using System.Globalization; +using Opc.Ua.Robotics.Client.Intent; +using Opc.Ua.RobotIntent; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// Applies deterministic filtering, sorting, and cursor-based paging to + /// operation and mission snapshot collections. + /// + internal static class RoboticsListPaging + { + private const int DefaultPageSize = 20; + private const int MaxPageSize = 100; + private const byte CursorVersion = 1; + private const int CursorLength = 5; + + public static OperationListResult PageOperations( + ArrayOf allOps, + OperationListQuery? query) + { + query ??= new OperationListQuery(); + int pageSize = ResolvePageSize(query.PageSize); + int startIndex = DecodeCursor(query.Cursor); + + ArrayOf filtered = FilterOperations(allOps, query); + ArrayOf sorted = SortOperationsDeterministically(filtered); + + int total = sorted.Count; + if (startIndex > total) + { + startIndex = total; + } + int end = Math.Min(startIndex + pageSize, total); + int returned = Math.Max(0, end - startIndex); + + string? nextCursor = end < total + ? EncodeCursor(end) + : null; + + if (query.Detail == DetailLevel.Full) + { + var items = new List(returned); + for (int i = startIndex; i < end; i++) + { + items.Add(sorted[i]); + } + return new OperationListResult + { + Total = total, + Returned = returned, + NextCursor = nextCursor, + Operations = [.. items] + }; + } + + var summaries = new List(returned); + for (int i = startIndex; i < end; i++) + { + summaries.Add(ToSummary(sorted[i])); + } + return new OperationListResult + { + Total = total, + Returned = returned, + NextCursor = nextCursor, + Summaries = [.. summaries] + }; + } + + public static MissionListResult PageMissions( + ArrayOf allMissions, + MissionListQuery? query) + { + query ??= new MissionListQuery(); + int pageSize = ResolvePageSize(query.PageSize); + int startIndex = DecodeCursor(query.Cursor); + + ArrayOf filtered = FilterMissions(allMissions, query); + ArrayOf sorted = SortMissionsDeterministically(filtered); + + int total = sorted.Count; + if (startIndex > total) + { + startIndex = total; + } + int end = Math.Min(startIndex + pageSize, total); + int returned = Math.Max(0, end - startIndex); + + string? nextCursor = end < total + ? EncodeCursor(end) + : null; + + if (query.Detail == DetailLevel.Full) + { + var items = new List(returned); + for (int i = startIndex; i < end; i++) + { + items.Add(sorted[i]); + } + return new MissionListResult + { + Total = total, + Returned = returned, + NextCursor = nextCursor, + Missions = [.. items] + }; + } + + var summaries = new List(returned); + for (int i = startIndex; i < end; i++) + { + summaries.Add(ToMissionSummary(sorted[i])); + } + return new MissionListResult + { + Total = total, + Returned = returned, + NextCursor = nextCursor, + Summaries = [.. summaries] + }; + } + + private static ArrayOf FilterOperations( + ArrayOf ops, + OperationListQuery query) + { + var result = new List(ops.Count); + for (int i = 0; i < ops.Count; i++) + { + IntentOperationSnapshot op = ops[i]; + + if (!string.IsNullOrEmpty(query.IntentId) && + !string.Equals(op.IntentId, query.IntentId, StringComparison.Ordinal)) + { + continue; + } + + if (!string.IsNullOrEmpty(query.MissionId) && + !string.Equals(op.MissionId, query.MissionId, StringComparison.Ordinal)) + { + continue; + } + + if (query.ExecutionState.HasValue && op.ExecutionState != query.ExecutionState.Value) + { + continue; + } + + if (query.Work == WorkSelector.Active && IsTerminal(op.ExecutionState)) + { + continue; + } + + if (query.Work == WorkSelector.Terminal && !IsTerminal(op.ExecutionState)) + { + continue; + } + + result.Add(op); + } + return [.. result]; + } + + private static ArrayOf FilterMissions( + ArrayOf missions, + MissionListQuery query) + { + var result = new List(missions.Count); + for (int i = 0; i < missions.Count; i++) + { + MissionSnapshot m = missions[i]; + + if (!string.IsNullOrEmpty(query.MissionId) && + !string.Equals(m.MissionId, query.MissionId, StringComparison.Ordinal)) + { + continue; + } + + if (query.ExecutionState.HasValue && m.ExecutionState != query.ExecutionState.Value) + { + continue; + } + + if (query.Work == WorkSelector.Active && IsTerminal(m.ExecutionState)) + { + continue; + } + + if (query.Work == WorkSelector.Terminal && !IsTerminal(m.ExecutionState)) + { + continue; + } + + result.Add(m); + } + return [.. result]; + } + + private static ArrayOf SortOperationsDeterministically( + ArrayOf ops) + { + var list = new List(ops.Count); + for (int i = 0; i < ops.Count; i++) + { + list.Add(ops[i]); + } + list.Sort((a, b) => + { + int cmp = string.CompareOrdinal(a.IntentId, b.IntentId); + if (cmp != 0) + { + return cmp; + } + return string.CompareOrdinal( + a.Operation.ToString(), b.Operation.ToString()); + }); + return [.. list]; + } + + private static ArrayOf SortMissionsDeterministically( + ArrayOf missions) + { + var list = new List(missions.Count); + for (int i = 0; i < missions.Count; i++) + { + list.Add(missions[i]); + } + list.Sort((a, b) => string.CompareOrdinal(a.MissionId, b.MissionId)); + return [.. list]; + } + + private static OperationSummary ToSummary(IntentOperationSnapshot op) + { + return new OperationSummary + { + Operation = op.Operation.ToString(), + IntentId = op.IntentId, + ExecutionState = op.ExecutionState, + Progress = op.Progress, + QueuePosition = op.QueuePosition, + Failure = op.Result.Failure != IntentFailureEnum.None + ? op.Result.Failure + : null, + Message = op.Result.Message.Text is { Length: > 0 } text + ? text + : null, + MissionId = op.MissionId + }; + } + + private static MissionSummary ToMissionSummary(MissionSnapshot m) + { + var stepSummaries = new List(); + if (!m.Steps.IsNull && m.Steps.Count > 0) + { + for (int i = 0; i < m.Steps.Count; i++) + { + MissionStepOperation step = m.Steps[i]; + stepSummaries.Add(new MissionStepOperationSummary + { + StepId = step.StepId, + IntentId = step.IntentId, + Operation = step.OperationNodeId.IsNull + ? null + : step.OperationNodeId.ToString(), + State = step.State + }); + } + } + + return new MissionSummary + { + MissionNode = m.MissionNode.ToString(), + MissionId = m.MissionId, + MissionUpdateId = m.MissionUpdateId, + ExecutionState = m.ExecutionState, + CurrentStepId = m.CurrentStepId, + Failure = m.Failure != IntentFailureEnum.None + ? m.Failure + : null, + Message = m.FailureMessage.Text is { Length: > 0 } text + ? text + : null, + ReleasedStepCount = m.ReleasedStepCount, + Steps = [.. stepSummaries] + }; + } + + private static bool IsTerminal(ExecutionStateEnum state) + { + return state is ExecutionStateEnum.Succeeded + or ExecutionStateEnum.Cancelled + or ExecutionStateEnum.Failed + or ExecutionStateEnum.Retriable; + } + + internal static int ResolvePageSize(int? requested) + { + if (requested is null) + { + return DefaultPageSize; + } + + int value = requested.Value; + if (value <= 0) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'pageSize' must be between 1 and {MaxPageSize} but was {value}. " + + $"Omit it to use the default of {DefaultPageSize}."), + nameof(requested)); + } + + if (value > MaxPageSize) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'pageSize' must be between 1 and {MaxPageSize} but was {value}."), + nameof(requested)); + } + + return value; + } + + internal static int DecodeCursor(string? cursor) + { + if (string.IsNullOrWhiteSpace(cursor)) + { + return 0; + } + + Span bytes = stackalloc byte[CursorLength]; + if (!Convert.TryFromBase64String(cursor, bytes, out int written) || written != CursorLength) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Invalid cursor '{cursor}': a cursor is exactly {CursorLength} base64-encoded bytes."), + nameof(cursor)); + } + + if (bytes[0] != CursorVersion) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Invalid cursor '{cursor}': unsupported cursor version {bytes[0]}."), + nameof(cursor)); + } + + int index = BinaryPrimitives.ReadInt32LittleEndian(bytes[1..]); + if (index < 0) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"Invalid cursor '{cursor}': the offset {index} is negative."), + nameof(cursor)); + } + + return index; + } + + internal static string EncodeCursor(int index) + { + Span bytes = stackalloc byte[CursorLength]; + bytes[0] = CursorVersion; + BinaryPrimitives.WriteInt32LittleEndian(bytes[1..], index); + return Convert.ToBase64String(bytes); + } + } +} diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsMissionTools.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsMissionTools.cs index 3d723af533..a7d85fe2e5 100644 --- a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsMissionTools.cs +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsMissionTools.cs @@ -43,50 +43,56 @@ namespace Opc.Ua.Mcp.Tools public sealed class RoboticsMissionTools { /// - /// Builds and submits a mission. + /// Builds and submits a mission from typed step/transition DTOs. /// [McpServerTool(Name = "robotics_submit_mission")] - [Description("Compiles a mission from JSON steps/transitions using the Robot Intent MissionBuilder and " + + [Description("Compiles a mission from typed steps and transitions using the Robot Intent MissionBuilder and " + "submits it. Server refusals such as NotPermittedInMode, SafetyLimitExceeded, ControlNotOwned, " + "CapabilityNotSupported, ParameterInvalid, or QueueFull are returned verbatim with message; this tool " + "does not retry or request command authority implicitly.")] public static async Task SubmitMissionAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(RoboticsControlTools.ControllerDescription)] string controller, [Description("MissionId to submit.")] string missionId, [Description("MissionUpdateId for this submission.")] uint missionUpdateId, - [Description("JSON array of mission steps. Each step has stepId, released, and intent { kind, ... }.")] - string stepsJson, - [Description("Optional JSON array of transitions; an omitted/empty array is a flat ordered mission.")] - string? transitionsJson = null, + [Description("Array of mission steps. Each step has stepId, released, and intent with kind discriminator.")] + MissionStepInput[] steps, + [Description("Optional array of transitions; an omitted/empty array is a flat ordered mission.")] + MissionTransitionInput[]? transitions = null, [Description("Optional localized label text.")] string? label = null, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - RobotIntentControllerClient controller = manager.OpenController(controllerId, sessionName); - MissionDataType mission = BuildMission(missionId, missionUpdateId, stepsJson, transitionsJson, label); - return await controller.SubmitMissionAsync(mission, ct).ConfigureAwait(false); + RoboticsResolutionContext context = await RoboticsResolutionContext.CreateAsync( + manager, controller, sessionName, ct).ConfigureAwait(false); + MissionDataType mission = BuildMission( + missionId, missionUpdateId, steps, transitions, label, context.Scope); + return await context.Client.SubmitMissionAsync(mission, ct).ConfigureAwait(false); } /// - /// Updates a mission horizon. + /// Updates a mission horizon from typed step DTOs. /// [McpServerTool(Name = "robotics_update_mission")] - [Description("Updates a mission horizon from a JSON step list. The client API performs stale-update checks " + + [Description("Updates a mission horizon from a typed step list. The client API performs stale-update checks " + "and the server returns the authoritative MissionUpdateResult and message; the MCP layer does not " + "invent mission state or retry refused updates.")] public static async Task UpdateMissionAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(RoboticsControlTools.ControllerDescription)] string controller, [Description("MissionId to update.")] string missionId, [Description("Strictly increasing MissionUpdateId.")] uint missionUpdateId, - [Description("JSON array of replacement horizon steps.")] string horizonStepsJson, + [Description("Array of replacement horizon steps.")] + MissionStepInput[] horizonSteps, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - RobotIntentControllerClient controller = manager.OpenController(controllerId, sessionName); - ArrayOf steps = RoboticsIntentJson.BuildMissionSteps(horizonStepsJson); - return await controller.UpdateMissionAsync(missionId, missionUpdateId, steps, ct).ConfigureAwait(false); + RoboticsResolutionContext context = await RoboticsResolutionContext.CreateAsync( + manager, controller, sessionName, ct).ConfigureAwait(false); + ArrayOf steps = RoboticsIntentDtoConverter.ConvertMissionSteps( + horizonSteps, context.Scope); + return await context.Client.UpdateMissionAsync(missionId, missionUpdateId, steps, ct) + .ConfigureAwait(false); } /// @@ -97,22 +103,24 @@ public static async Task UpdateMissionAsync( "ControlNotOwned is returned by the client API; this tool never retries or submits compensating work.")] public static async Task CancelMissionAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(RoboticsControlTools.ControllerDescription)] string controller, [Description("MissionId to cancel.")] string missionId, [Description("Stop mode requested from the server.")] StopModeEnum stopMode = StopModeEnum.QuickStop, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).CancelMissionAsync(missionId, stopMode, ct) - .ConfigureAwait(false); + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await resolved.CancelMissionAsync(missionId, stopMode, ct).ConfigureAwait(false); } internal static MissionDataType BuildMission( string missionId, uint missionUpdateId, - string stepsJson, - string? transitionsJson, - string? label) + MissionStepInput[] steps, + MissionTransitionInput[]? transitions, + string? label, + RoboticsScopeResolver? scope) { MissionBuilder builder = RobotIntentBuilder.Mission(missionId).WithMissionUpdateId(missionUpdateId); if (!string.IsNullOrEmpty(label)) @@ -120,10 +128,12 @@ internal static MissionDataType BuildMission( builder.WithLabel(new LocalizedText(label)); } - ArrayOf steps = RoboticsIntentJson.BuildMissionSteps(stepsJson); - ArrayOf transitions = RoboticsIntentJson.BuildMissionTransitions(transitionsJson); - builder.WithSteps(steps); - builder.WithTransitions(transitions); + ArrayOf convertedSteps = + RoboticsIntentDtoConverter.ConvertMissionSteps(steps, scope); + ArrayOf convertedTransitions = + RoboticsIntentDtoConverter.ConvertMissionTransitions(transitions); + builder.WithSteps(convertedSteps) + .WithTransitions(convertedTransitions); return builder.Build(); } } diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsMonitoringTools.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsMonitoringTools.cs index 8b56f30cd4..1abb12caa8 100644 --- a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsMonitoringTools.cs +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsMonitoringTools.cs @@ -29,6 +29,7 @@ using System; using System.ComponentModel; +using System.Globalization; using System.Threading; using System.Threading.Tasks; using ModelContextProtocol.Server; @@ -52,15 +53,17 @@ public sealed class RoboticsMonitoringTools "state from capabilities and does not request command authority.")] public static async Task ReadStateAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(RoboticsControlTools.ControllerDescription)] string controller, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).ReadStateAsync(ct).ConfigureAwait(false); + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await resolved.ReadStateAsync(ct).ConfigureAwait(false); } /// - /// Lists outstanding intent operations. + /// Lists outstanding intent operations with filtering and paging. /// [McpServerTool(Name = "robotics_list_operations")] [Description("Lists Robot Intent operation snapshots published by the controller: active, queued, " + @@ -68,33 +71,45 @@ public static async Task ReadStateAsync( "this after submitting or retrying one intent, or before robotics_wait_operation. Use " + "robotics_list_missions instead when you need mission containers and MissionIds rather than per-intent " + "operations. It reports server state only, never invents work, and never requests command authority. " + - "Returns an array of IntentOperationSnapshot.")] - public static async Task> ListOperationsAsync( + "Returns a paged OperationListResult with summaries or full snapshots.")] + public static async Task ListOperationsAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(RoboticsControlTools.ControllerDescription)] string controller, + [Description("Optional query: filter by intentId, missionId, executionState, work selector; choose " + + "summary or full detail; page with pageSize (default 20, max 100) and cursor.")] + OperationListQuery? query = null, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).ListOperationsAsync(ct) - .ConfigureAwait(false); + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + ArrayOf all = await resolved.ListOperationsAsync(ct).ConfigureAwait(false); + return RoboticsListPaging.PageOperations(all, query); } /// - /// Lists outstanding missions. + /// Lists outstanding missions with filtering and paging. /// [McpServerTool(Name = "robotics_list_missions")] [Description("Lists outstanding Robot Intent mission containers from the controller, including MissionId, " + "update state, and mission-level progress when the server exposes it. Use this after submitting, " + "updating, or cancelling a mission. Use robotics_list_operations instead to inspect the individual " + "intent operations spawned by a mission or single-intent submission. It reports server state only, " + - "never invents mission state, and never requests command authority. Returns an array of MissionSnapshot.")] - public static async Task> ListMissionsAsync( + "never invents mission state, and never requests command authority. Returns a paged MissionListResult " + + "with summaries or full snapshots.")] + public static async Task ListMissionsAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(RoboticsControlTools.ControllerDescription)] string controller, + [Description("Optional query: filter by missionId, executionState, work selector; choose summary or " + + "full detail; page with pageSize (default 20, max 100) and cursor.")] + MissionListQuery? query = null, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - return await manager.OpenController(controllerId, sessionName).ListMissionsAsync(ct).ConfigureAwait(false); + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + ArrayOf all = await resolved.ListMissionsAsync(ct).ConfigureAwait(false); + return RoboticsListPaging.PageMissions(all, query); } /// @@ -104,20 +119,22 @@ public static async Task> ListMissionsAsync( [Description("Waits for an intent operation to complete up to timeoutMs. Timeout is not an error: the " + "result has completed=false and includes the current operation snapshot refreshed from the server. This " + "tool does not retry refusals or resubmit work.")] - public static Task WaitOperationAsync( + public static async Task WaitOperationAsync( RoboticsIntentManager manager, - [Description("Controller NodeId.")] string controllerId, + [Description(RoboticsControlTools.ControllerDescription)] string controller, [Description("IntentId associated with the operation.")] string intentId, [Description("IntentOperation NodeId to observe.")] string operationNodeId, [Description("Maximum wait in milliseconds; <=0 performs a poll/refresh.")] int timeoutMs = 2000, [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, CancellationToken ct = default) { - RobotIntentControllerClient controller = manager.OpenController(controllerId, sessionName); - return WaitOperationAsync(controller, intentId, operationNodeId, timeoutMs, ct); + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await WaitOperationCoreAsync(resolved, intentId, operationNodeId, timeoutMs, ct) + .ConfigureAwait(false); } - internal static async Task WaitOperationAsync( + internal static async Task WaitOperationCoreAsync( RobotIntentControllerClient controller, string intentId, string operationNodeId, @@ -139,5 +156,69 @@ internal static async Task WaitOperationAsync( await handle.DisposeAsync().ConfigureAwait(false); } } + + /// + /// Waits for an existing mission with a bounded timeout. + /// + [McpServerTool(Name = "robotics_wait_mission")] + [Description("Waits for a Robot Intent mission to reach a terminal state, up to timeoutMs. The wait is " + + "always bounded and observes the mission node the server published; it never polls " + + "robotics_list_missions, never retries, and never resubmits or updates the mission. Timeout is not an " + + "error: the result has completed=false with the mission snapshot refreshed from the server. Command " + + "authority is never acquired as a side effect. Returns MissionWaitResult.")] + public static async Task WaitMissionAsync( + RoboticsIntentManager manager, + [Description(RoboticsControlTools.ControllerDescription)] string controller, + [Description("MissionId associated with the mission operation.")] string missionId, + [Description("Mission operation NodeId to observe, as returned by robotics_submit_mission or " + + "robotics_list_missions.")] + string missionNodeId, + [Description("Maximum wait in milliseconds; <=0 performs a poll/refresh. Bounded by " + + "MaxWaitMilliseconds.")] + int timeoutMs = 2000, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + RobotIntentControllerClient resolved = await manager.ResolveControllerAsync( + controller, sessionName, ct).ConfigureAwait(false); + return await WaitMissionCoreAsync(resolved, missionId, missionNodeId, timeoutMs, ct) + .ConfigureAwait(false); + } + + internal static async Task WaitMissionCoreAsync( + RobotIntentControllerClient controller, + string missionId, + string missionNodeId, + int timeoutMs, + CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(controller); + ArgumentException.ThrowIfNullOrWhiteSpace(missionId); + + if (timeoutMs > MaxWaitMilliseconds) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'timeoutMs' must not exceed {MaxWaitMilliseconds} but was {timeoutMs}."), + nameof(timeoutMs)); + } + + NodeId missionNode = OpcUaJsonHelper.ParseNodeId(missionNodeId); + MissionHandle handle = await controller.TrackMissionAsync( + missionId, + missionNode, + ct).ConfigureAwait(false); + try + { + var timeout = TimeSpan.FromMilliseconds(timeoutMs <= 0 ? 0 : timeoutMs); + return await handle.WaitForCompletionAsync(timeout, ct).ConfigureAwait(false); + } + finally + { + await handle.DisposeAsync().ConfigureAwait(false); + } + } + + private const int MaxWaitMilliseconds = 600000; } } diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsResolutionContext.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsResolutionContext.cs new file mode 100644 index 0000000000..aaca5e66d3 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsResolutionContext.cs @@ -0,0 +1,100 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Robotics.Client.Intent; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// A per-call resolution context. It resolves the controller selector once + /// and reads the controller info exactly once, then resolves every scoped + /// name reference in the request against that single snapshot. Resolution + /// is read-only: it never submits work and never requests or releases + /// command authority. + /// + internal sealed class RoboticsResolutionContext + { + private RoboticsResolutionContext( + RobotIntentControllerClient client, + RobotIntentControllerInfo info) + { + Client = client; + Info = info; + Scope = new RoboticsScopeResolver(info.Lookups); + } + + /// + /// Gets the resolved controller client. + /// + public RobotIntentControllerClient Client { get; } + + /// + /// Gets the controller info snapshot, read exactly once per call. + /// + public RobotIntentControllerInfo Info { get; } + + /// + /// Gets the scoped name resolver over the snapshot lookups. + /// + public RoboticsScopeResolver Scope { get; } + + /// + /// Resolves the controller selector and reads its info snapshot once. + /// + public static async ValueTask CreateAsync( + RoboticsIntentManager manager, + string controller, + string? sessionName, + CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(manager); + + RobotIntentControllerClient client = await manager + .ResolveControllerAsync(controller, sessionName, ct) + .ConfigureAwait(false); + return await CreateAsync(client, ct).ConfigureAwait(false); + } + + /// + /// Reads the info snapshot of an already resolved controller once. + /// + public static async ValueTask CreateAsync( + RobotIntentControllerClient client, + CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(client); + + RobotIntentControllerInfo info = await client.ReadAsync(ct).ConfigureAwait(false); + return new RoboticsResolutionContext(client, info); + } + } +} diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsScopeResolver.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsScopeResolver.cs new file mode 100644 index 0000000000..84d91f2ae8 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsScopeResolver.cs @@ -0,0 +1,195 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Globalization; +using Opc.Ua.Robotics.Client.Intent; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// Resolves scoped name references (frames, tools, locations, outputs, + /// programs) against the lookup tables one controller publishes. + /// The resolver is a pure projection over a single lookup snapshot: it + /// never reads from the server, never submits work, and never requests + /// command authority. + /// + internal sealed class RoboticsScopeResolver + { + /// + /// Initializes the resolver over one lookup snapshot. + /// + public RoboticsScopeResolver(RobotIntentLookups lookups) + { + Lookups = lookups ?? RobotIntentLookups.Empty; + } + + /// + /// Gets the lookup snapshot the resolver projects. + /// + public RobotIntentLookups Lookups { get; } + + /// + /// Resolves a frame name or NodeId through the Frames lookup. + /// + public NodeId ResolveFrame(string? nameOrNodeId) + { + return RoboticsControllerResolver.ResolveScopedResource( + nameOrNodeId, Lookups.Frames, "frame"); + } + + /// + /// Resolves a pose or force frame selector to the FrameId string the + /// controller publishes. + /// + /// + /// A value that already matches a published FrameId is returned as-is. + /// A frame Name or NodeId is resolved through the Frames lookup and + /// mapped back to the published FrameId. Anything else is rejected so + /// that a mistyped selector never reaches the server as a silently + /// unscoped string. + /// + public string ResolveFrameId(string? frameId) + { + if (string.IsNullOrWhiteSpace(frameId)) + { + return string.Empty; + } + + string trimmed = frameId.Trim(); + for (int i = 0; i < Lookups.FramesByFrameId.Count; i++) + { + if (string.Equals(Lookups.FramesByFrameId[i].Name, trimmed, StringComparison.Ordinal)) + { + return Lookups.FramesByFrameId[i].Name; + } + } + + if (Lookups.Frames.Count == 0 && Lookups.FramesByFrameId.Count == 0) + { + return trimmed; + } + + NodeId resolved = RoboticsControllerResolver.ResolveScopedResource( + trimmed, Lookups.Frames, "frame"); + for (int i = 0; i < Lookups.FramesByFrameId.Count; i++) + { + if (Lookups.FramesByFrameId[i].NodeId == resolved) + { + return Lookups.FramesByFrameId[i].Name; + } + } + + return trimmed; + } + + /// + /// Resolves a tool name or NodeId through the Tools lookup. + /// + public NodeId ResolveTool(string? nameOrNodeId) + { + return RoboticsControllerResolver.ResolveScopedResource( + nameOrNodeId, Lookups.Tools, "tool"); + } + + /// + /// Resolves a location name or NodeId through the Locations lookup. + /// + public NodeId ResolveLocation(string? nameOrNodeId) + { + return RoboticsControllerResolver.ResolveScopedResource( + nameOrNodeId, Lookups.Locations, "location"); + } + + /// + /// Resolves an output or Boolean signal name or NodeId through the + /// Outputs lookup. A full NodeId is always accepted so that a signal + /// the controller does not publish as an output can still be named; + /// the server validates the scope. + /// + public NodeId ResolveOutput(string? nameOrNodeId) + { + return RoboticsControllerResolver.ResolveScopedResource( + nameOrNodeId, Lookups.Outputs, "output"); + } + + /// + /// Resolves a program name or NodeId through the Programs lookup. + /// + public NodeId ResolveProgram(string? nameOrNodeId) + { + return RoboticsControllerResolver.ResolveScopedResource( + nameOrNodeId, Lookups.Programs, "program"); + } + + /// + /// Resolves a required tool selector. + /// + public NodeId ResolveRequiredTool(string? nameOrNodeId, string parameterName) + { + return Require(ResolveTool(nameOrNodeId), parameterName); + } + + /// + /// Resolves a required location selector. + /// + public NodeId ResolveRequiredLocation(string? nameOrNodeId, string parameterName) + { + return Require(ResolveLocation(nameOrNodeId), parameterName); + } + + /// + /// Resolves a required output selector. + /// + public NodeId ResolveRequiredOutput(string? nameOrNodeId, string parameterName) + { + return Require(ResolveOutput(nameOrNodeId), parameterName); + } + + /// + /// Resolves a required program selector. + /// + public NodeId ResolveRequiredProgram(string? nameOrNodeId, string parameterName) + { + return Require(ResolveProgram(nameOrNodeId), parameterName); + } + + private static NodeId Require(NodeId resolved, string parameterName) + { + if (resolved.IsNull) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{parameterName}' is required."), + parameterName); + } + + return resolved; + } + } +} diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsVisionPickDtos.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsVisionPickDtos.cs new file mode 100644 index 0000000000..eadd39972f --- /dev/null +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsVisionPickDtos.cs @@ -0,0 +1,495 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.ComponentModel; +using System.ComponentModel.DataAnnotations; +using System.Text.Json.Serialization; +using Opc.Ua.RobotIntent; + +namespace Opc.Ua.Mcp.Tools +{ + // CLR arrays are intentional at this JSON boundary: the MCP schema generator exposes + // ArrayOf as its backing memory object and cannot bind an incoming JSON array to it. + // The same projection applies on the way out, so the result DTOs below publish plain + // JSON arrays and the manager projects the OPC UA ArrayOf values into them. + + /// + /// Deterministic policy used to select one detection from the filtered set. + /// + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum VisionPickSelectionPolicy + { + /// + /// Select the detection with the highest confidence. Ties are broken by + /// ordinal DetectionId order and then by the original result order, so the + /// same detection set always selects the same detection. + /// + HighestConfidence + } + + /// + /// Discriminator for the single piece of work a vision-guided pick submitted. + /// + [JsonConverter(typeof(JsonStringEnumConverter))] + public enum VisionPickSubmissionKind + { + /// + /// A single Pick intent was submitted because no destination was requested. + /// + Pick, + + /// + /// A two-step Pick/Place mission was submitted because a destination was requested. + /// + Mission + } + + /// + /// Structured input for robotics_vision_pick. + /// + public sealed class VisionGuidedPickRequest + { + /// + /// Gets or sets the controller selector. + /// + [Description("Controller selector: unique display name or BrowseName (e.g. 'Controller1') or " + + "OPC UA NodeId string. Matched with exact ordinal comparison after trimming.")] + public required string Controller { get; set; } + + /// + /// Gets or sets the Vision pipeline selector. + /// + [Description("Vision pipeline selector: an exact BrowseName, DisplayName, or NodeId string " + + "(e.g. 'BinPickingPipeline'). Resolved on the same OPC UA session as the controller.")] + public required string Pipeline { get; set; } + + /// + /// Gets or sets the source location selector for the Pick intent. + /// + [Description("Source location name or NodeId the Pick intent takes the workpiece from.")] + public required string Source { get; set; } + + /// + /// Gets or sets the tool selector for the submitted intents. + /// + [Description("Tool name or NodeId used for the Pick and, when requested, the Place intent.")] + public required string Tool { get; set; } + + /// + /// Gets or sets the optional destination location selector. + /// + [Description("Optional destination location name or NodeId. When set, a two-step Pick/Place " + + "mission is submitted instead of a single Pick intent.")] + public string? Destination { get; set; } + + /// + /// Gets or sets the exact DetectionId filter. + /// + [Description("Optional exact DetectionId filter, compared with ordinal equality.")] + public string? DetectionId { get; set; } + + /// + /// Gets or sets the exact class label filter. + /// + [Description("Optional exact detection ClassLabel filter, compared with ordinal equality.")] + public string? ClassLabel { get; set; } + + /// + /// Gets or sets the inclusive minimum confidence filter. + /// + [Range(0.0, 1.0)] + [Description("Optional inclusive minimum detection confidence within [0, 1].")] + public double? MinimumConfidence { get; set; } + + /// + /// Gets or sets the deterministic selection policy. + /// + [DefaultValue(VisionPickSelectionPolicy.HighestConfidence)] + [Description("Deterministic selection policy applied to the filtered detections. " + + "HighestConfidence breaks ties by ordinal DetectionId and then original order.")] + public VisionPickSelectionPolicy Selection { get; set; } = VisionPickSelectionPolicy.HighestConfidence; + + /// + /// Gets or sets the object class override for the Pick intent. + /// + [Description("Optional ObjectClass override for the Pick intent. Defaults to the selected " + + "detection's ClassLabel.")] + public string? ObjectClass { get; set; } + + /// + /// Gets or sets the IntentId of the Pick intent. + /// + [Description("Optional IntentId for the Pick intent.")] + public string? PickIntentId { get; set; } + + /// + /// Gets or sets the IntentId of the Place intent. + /// + [Description("Optional IntentId for the Place intent. Requires destination.")] + public string? PlaceIntentId { get; set; } + + /// + /// Gets or sets the localized label applied to both intents. + /// + [Description("Optional localized label applied to the submitted intents and mission.")] + public string? Label { get; set; } + + /// + /// Gets or sets the buffer mode applied to both intents. + /// + [Description("Optional buffer mode applied to the submitted intents.")] + public BufferModeEnum? BufferMode { get; set; } + + /// + /// Gets or sets the blocking mode applied to both intents. + /// + [Description("Optional blocking mode applied to the submitted intents.")] + public BlockingModeEnum? BlockingMode { get; set; } + + /// + /// Gets or sets the MissionId of the submitted mission. + /// + [Description("Optional MissionId for the Pick/Place mission. Requires destination.")] + public string? MissionId { get; set; } + + /// + /// Gets or sets the MissionUpdateId of the submitted mission. + /// + [Description("Optional MissionUpdateId for the Pick/Place mission. Requires destination. Default 1.")] + public uint? MissionUpdateId { get; set; } + + /// + /// Gets or sets the session name. + /// + [Description("Session name to use; defaults to the only active session. The Vision pipeline and " + + "the Robot Intent controller are always resolved on the same session.")] + public string? SessionName { get; set; } + } + + /// + /// The closed result of one vision-guided pick. Exactly one of + /// and is populated, + /// selected by . + /// + public sealed class VisionGuidedPickResult + { + /// + /// Gets or sets the kind of work that was submitted. + /// + [Description("Which submission was made: Pick or Mission.")] + public VisionPickSubmissionKind Kind { get; set; } + + /// + /// Gets or sets the perception provenance of the selected detection. + /// + [Description("Perception provenance: the Vision result, pipeline, sensor, model, frame, and the " + + "selected detection the submitted work was derived from.")] + public VisionPickProvenance Provenance { get; set; } = new(); + + /// + /// Gets or sets the authoritative single-intent submission outcome. + /// + [Description("Authoritative IntentSubmissionResult when kind is 'Pick'; null otherwise.")] + public VisionPickIntentSubmission? PickSubmission { get; set; } + + /// + /// Gets or sets the authoritative mission submission outcome. + /// + [Description("Authoritative MissionSubmissionResult when kind is 'Mission'; null otherwise.")] + public VisionPickMissionSubmission? MissionSubmission { get; set; } + + /// + /// Gets or sets the immediate mission step to operation mapping. + /// + [Description("Immediate mission step-to-operation mapping read once after an accepted mission " + + "submission. Empty for a Pick submission or a refused mission.")] + public VisionPickMissionStep[] Steps { get; set; } = []; + } + + /// + /// Refusal-shaped outcome of the single Pick intent submission. + /// + public sealed class VisionPickIntentSubmission + { + /// + /// Gets or sets a value indicating whether the server accepted the intent. + /// + [Description("Whether the server accepted the Pick intent.")] + public bool Accepted { get; set; } + + /// + /// Gets or sets the IntentId the server acknowledged. + /// + [Description("IntentId the server acknowledged.")] + public string IntentId { get; set; } = string.Empty; + + /// + /// Gets or sets the operation NodeId, or null when none was returned. + /// + [Description("Operation NodeId, or null when the server returned none.")] + public string? Operation { get; set; } + + /// + /// Gets or sets the authoritative failure reason. + /// + [Description("Authoritative IntentFailureEnum reported by the server.")] + public IntentFailureEnum Failure { get; set; } + + /// + /// Gets or sets the authoritative refusal message. + /// + [Description("Authoritative refusal message reported by the server, or null.")] + public string? Message { get; set; } + } + + /// + /// Refusal-shaped outcome of the Pick/Place mission submission. + /// + public sealed class VisionPickMissionSubmission + { + /// + /// Gets or sets a value indicating whether the server accepted the mission. + /// + [Description("Whether the server accepted the mission.")] + public bool Accepted { get; set; } + + /// + /// Gets or sets the MissionId the server acknowledged. + /// + [Description("MissionId the server acknowledged.")] + public string MissionId { get; set; } = string.Empty; + + /// + /// Gets or sets the submitted MissionUpdateId. + /// + [Description("MissionUpdateId that was submitted.")] + public uint MissionUpdateId { get; set; } + + /// + /// Gets or sets the mission NodeId, or null when none was returned. + /// + [Description("Mission NodeId, or null when the server returned none.")] + public string? Operation { get; set; } + + /// + /// Gets or sets the authoritative failure reason. + /// + [Description("Authoritative IntentFailureEnum reported by the server.")] + public IntentFailureEnum Failure { get; set; } + + /// + /// Gets or sets the authoritative refusal message. + /// + [Description("Authoritative refusal message reported by the server, or null.")] + public string? Message { get; set; } + + /// + /// Gets or sets the StepId of the generated Pick step. + /// + [Description("StepId of the generated Pick step.")] + public string PickStepId { get; set; } = string.Empty; + + /// + /// Gets or sets the StepId of the generated Place step. + /// + [Description("StepId of the generated Place step.")] + public string PlaceStepId { get; set; } = string.Empty; + } + + /// + /// One mission step mapped to the operation the server created for it. + /// + public sealed class VisionPickMissionStep + { + /// + /// Gets or sets the step identifier. + /// + [Description("Step identifier.")] + public string StepId { get; set; } = string.Empty; + + /// + /// Gets or sets the intent identifier. + /// + [Description("Intent identifier.")] + public string IntentId { get; set; } = string.Empty; + + /// + /// Gets or sets the operation NodeId, or null if not yet executing. + /// + [Description("Operation NodeId, or null if the server has not created one yet.")] + public string? Operation { get; set; } + + /// + /// Gets or sets the step execution state. + /// + [Description("Step execution state.")] + public ExecutionStateEnum State { get; set; } + } + + /// + /// The perception provenance of a vision-guided pick: which Vision result the + /// submitted work was derived from and which detection was selected. + /// + public sealed class VisionPickProvenance + { + /// + /// Gets or sets the ResultId the Vision server assigned to the run. + /// + [Description("ResultId the Vision server assigned to the inference run.")] + public string ResultId { get; set; } = string.Empty; + + /// + /// Gets or sets the NodeId of the published result. + /// + [Description("NodeId of the published Vision result.")] + public string ResultNodeId { get; set; } = string.Empty; + + /// + /// Gets or sets the NodeId of the pipeline that was asked to run. + /// + [Description("NodeId of the Vision pipeline the inference was requested on.")] + public string RequestedPipelineNodeId { get; set; } = string.Empty; + + /// + /// Gets or sets the published name of the requested pipeline. + /// + [Description("Published name of the requested Vision pipeline, when available.")] + public string? RequestedPipelineName { get; set; } + + /// + /// Gets or sets the PipelineId the result published. + /// + [Description("Pipeline NodeId published by the result, when available.")] + public string? PipelineId { get; set; } + + /// + /// Gets or sets the sensor NodeId the result published. + /// + [Description("Sensor NodeId that produced the frame, when available.")] + public string? SensorId { get; set; } + + /// + /// Gets or sets the model version the server reported. + /// + [Description("Model version used by the pipeline, when reported.")] + public string? ModelVersionUsed { get; set; } + + /// + /// Gets or sets the result creation time. + /// + [Description("Result creation time in ISO 8601, when available.")] + public string? CreationTime { get; set; } + + /// + /// Gets or sets the frame identifier the pose is expressed in. + /// + [Description("Frame identifier the selected pose is expressed in, when available.")] + public string? FrameId { get; set; } + + /// + /// Gets or sets the total number of detections the result published. + /// + [Description("Total number of detections in the published result.")] + public int TotalDetections { get; set; } + + /// + /// Gets or sets the number of detections that survived the filters. + /// + [Description("Number of detections that matched the requested filters.")] + public int MatchedDetections { get; set; } + + /// + /// Gets or sets a value indicating whether the full detection snapshot was read. + /// + [Description("Whether the full detection snapshot was read because the bounded summary " + + "did not carry every detection.")] + public bool FullResultRead { get; set; } + + /// + /// Gets or sets the selected detection. + /// + [Description("The detection the submitted work was derived from.")] + public VisionPickDetection SelectedDetection { get; set; } = new(); + } + + /// + /// The detection a vision-guided pick selected, with its pose. + /// + public sealed class VisionPickDetection + { + /// + /// Gets or sets the detection identifier. + /// + [Description("Detection identifier.")] + public string DetectionId { get; set; } = string.Empty; + + /// + /// Gets or sets the detection class label. + /// + [Description("Detection class label.")] + public string ClassLabel { get; set; } = string.Empty; + + /// + /// Gets or sets the detection class identifier. + /// + [Description("Detection class identifier.")] + public uint ClassId { get; set; } + + /// + /// Gets or sets the detection confidence. + /// + [Description("Detection confidence within [0, 1].")] + public double Confidence { get; set; } + + /// + /// Gets or sets a value indicating whether the detection carries a pose. + /// + [Description("Whether the detection carries a pose.")] + public bool HasPose { get; set; } + + /// + /// Gets or sets the frame identifier of the pose. + /// + [Description("Frame identifier the pose is expressed in, or null when the detection has no pose.")] + public string? PoseFrameId { get; set; } + + /// + /// Gets or sets the pose position as [x, y, z]. + /// + [Description("Pose position as [x, y, z] in metres, or null when the detection has no pose. " + + "The pose covariance is deliberately not published here.")] + public double[]? PosePosition { get; set; } + + /// + /// Gets or sets the pose orientation quaternion as [x, y, z, w]. + /// + [Description("Pose orientation quaternion as [x, y, z, w], or null when the detection has no pose.")] + public double[]? PoseOrientation { get; set; } + } +} diff --git a/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsVisionTools.cs b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsVisionTools.cs new file mode 100644 index 0000000000..4546e63196 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Robotics/Tools/RoboticsVisionTools.cs @@ -0,0 +1,71 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.ComponentModel; +using System.Threading; +using System.Threading.Tasks; +using ModelContextProtocol.Server; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// MCP tools that combine Vision perception with Robot Intent actuation on + /// one OPC UA session. + /// + [McpServerToolType] + public sealed class RoboticsVisionTools + { + /// + /// Runs one Vision inference and submits the resulting pick. + /// + [McpServerTool(Name = "robotics_vision_pick")] + [Description("Runs a single Vision inference on a pipeline, selects one detection " + + "deterministically, and submits exactly one piece of Robot Intent work on the same OPC UA " + + "session: a Pick intent when no destination is given, or a two-step released Pick/Place " + + "mission when it is. Use it to close the perception-to-motion loop in one call instead of " + + "chaining vision_run_inference with robotics_submit_pick. Filters detections by exact " + + "detectionId, exact classLabel, and minimumConfidence, then selects the highest confidence " + + "detection with ordinal detectionId tie-breaking. Reports the full perception provenance - " + + "result, pipeline, sensor, model version, frame, and the selected detection with its pose - " + + "alongside the authoritative submission outcome. Refusals such as ParameterInvalid, " + + "CapabilityNotSupported, SafetyLimitExceeded, ControlNotOwned, or QueueFull are returned " + + "verbatim. Command authority is never acquired as a side effect, and the tool never waits " + + "for completion, retries, or cancels.")] + public static Task VisionPickAsync( + VisionGuidedRoboticsManager manager, + [Description("The vision-guided pick request.")] VisionGuidedPickRequest request, + CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(manager); + + return manager.PickAsync(request, ct); + } + } +} diff --git a/tools/Opc.Ua.Mcp.Robotics/VisionGuidedRoboticsManager.cs b/tools/Opc.Ua.Mcp.Robotics/VisionGuidedRoboticsManager.cs new file mode 100644 index 0000000000..ff5264753f --- /dev/null +++ b/tools/Opc.Ua.Mcp.Robotics/VisionGuidedRoboticsManager.cs @@ -0,0 +1,614 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client; +using Opc.Ua.Mcp.Tools; +using Opc.Ua.Robotics.Client.Intent; +using Opc.Ua.RobotIntent; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Mcp +{ + /// + /// Runs one Vision inference and turns the detection it selects into exactly + /// one piece of Robot Intent work on the same OPC UA session. + /// + /// + /// The manager is the cross-companion helper the robotics_vision_pick + /// tool delegates to. It composes the Vision and Robot Intent client SDKs + /// directly - it never calls another MCP tool - so the same orchestration is + /// available to application code that does not host an MCP server. It is + /// read-only towards command authority: it never requests or releases + /// control, never waits for completion, never retries, and never cancels. + /// Every refusal the server reports is returned verbatim. + /// + public sealed class VisionGuidedRoboticsManager + { + /// + /// Initializes the manager over the MCP session manager and the Robot + /// Intent manager. This is the constructor the container resolves, and + /// it is equally usable as a direct-construction fallback. + /// + /// + /// The session manager that owns the named OPC UA sessions. + /// + /// + /// The Robot Intent manager used to resolve the controller. + /// + /// + /// Any argument is null. + /// + public VisionGuidedRoboticsManager( + OpcUaSessionManager sessionManager, + RoboticsIntentManager roboticsManager) + { + m_sessionManager = sessionManager ?? throw new ArgumentNullException(nameof(sessionManager)); + Robotics = roboticsManager ?? throw new ArgumentNullException(nameof(roboticsManager)); + } + + /// + /// Gets the Robot Intent manager the controller is resolved through. + /// + public RoboticsIntentManager Robotics { get; } + + /// + /// Creates a Vision client over the very same named session the Robot + /// Intent clients are created on, so perception and actuation always + /// observe one server through one session. + /// + /// + /// Session name to use; defaults to the only active session. + /// + public VisionClient CreateVisionClient(string? sessionName = null) + { + ISession session = m_sessionManager.GetSessionOrThrow(sessionName); + return new VisionClient(session, m_sessionManager.Telemetry); + } + + /// + /// Runs one Vision inference, selects one detection deterministically, + /// and submits either a single Pick intent or a two-step Pick/Place + /// mission exactly once. + /// + /// + /// The vision-guided pick request. + /// + /// + /// Cancels the operation. + /// + /// + /// is null. + /// + /// + /// A request field is empty, out of range, non-finite, or conflicts + /// with another field. + /// + /// + /// The Vision result could not be resolved, or no detection matched. + /// + public async Task PickAsync( + VisionGuidedPickRequest request, + CancellationToken ct = default) + { + ValidateRequest(request); + + RoboticsResolutionContext context = await RoboticsResolutionContext.CreateAsync( + Robotics, request.Controller, request.SessionName, ct).ConfigureAwait(false); + VisionPickObservation observation = await ObserveAsync(request, ct).ConfigureAwait(false); + return await SubmitAsync(request, context, observation, ct).ConfigureAwait(false); + } + + /// + /// Runs the Vision half of a vision-guided pick on the same session: + /// resolves the pipeline, runs one bounded one-shot inference, and reads + /// the full detection snapshot when the bounded summary is incomplete. + /// + internal async Task ObserveAsync( + VisionGuidedPickRequest request, + CancellationToken ct) + { + VisionClient client = CreateVisionClient(request.SessionName); + VisionNodeEntry entry = await client.ResolvePipelineAsync(request.Pipeline, ct) + .ConfigureAwait(false); + VisionPipelineClient pipeline = client.Pipeline(entry.NodeId); + VisionInferenceService inference = client.Inference(); + + VisionInferenceResult result = await inference.RunOneShotAsync( + pipeline, + entry.BrowseName.Name, + VisionResultDetail.Summary, + VisionExpectedResultKind.Detection, + kMaxSummaryItems, + ct).ConfigureAwait(false); + + return await ResolveObservationAsync( + result, + token => client.Result(result.ResultNodeId).ReadDetectionAsync(token), + ct).ConfigureAwait(false); + } + + /// + /// Projects one inference result into the detection set selection runs + /// over, reading the full snapshot when the bounded summary truncated it. + /// + /// + internal static async Task ResolveObservationAsync( + VisionInferenceResult result, + Func> readFullSnapshot, + CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(result); + ArgumentNullException.ThrowIfNull(readFullSnapshot); + + if (!result.Resolved || result.ResultNodeId.IsNull) + { + // All concatenated operands must remain interpolated for string.Create handler binding. + // TODO: Remove when RCS1214 preserves interpolated-string-handler overload binding. +#pragma warning disable RCS1214 + throw new InvalidOperationException(string.Create(CultureInfo.InvariantCulture, + $"The Vision pipeline published result '{result.ResultId}' but the result NodeId " + + $"could not be resolved, so no detection can be selected.")); +#pragma warning restore RCS1214 + } + + VisionDetectionSummary? summary = result.DetectionSummary; + if (summary is null) + { + throw new InvalidOperationException(string.Create(CultureInfo.InvariantCulture, + $"The Vision result '{result.ResultId}' carries no detection summary.")); + } + + if (summary.TotalDetections <= summary.Items.Count) + { + return new VisionPickObservation(result, summary.Items, summary.TotalDetections, false); + } + + VisionDetectionResultSnapshot snapshot = await readFullSnapshot(ct).ConfigureAwait(false); + var items = new List(snapshot.Detections.Count); + for (int i = 0; i < snapshot.Detections.Count; i++) + { + VisionDetectionDataType detection = snapshot.Detections[i]; + items.Add(new VisionDetectionItem + { + DetectionId = detection.DetectionId ?? string.Empty, + ClassLabel = detection.ClassLabel ?? string.Empty, + ClassId = detection.ClassId, + Confidence = detection.Confidence, + HasPose = detection.HasPose, + Pose = detection.HasPose ? detection.Pose : null + }); + } + + return new VisionPickObservation(result, items.ToArrayOf(), items.Count, true); + } + + /// + /// Turns one observation into exactly one submission against an already + /// resolved controller. This is the seam the tests drive with a fake + /// controller client and a supplied observation. + /// + internal static async Task SubmitAsync( + VisionGuidedPickRequest request, + RoboticsResolutionContext context, + VisionPickObservation observation, + CancellationToken ct) + { + ValidateRequest(request); + ArgumentNullException.ThrowIfNull(context); + ArgumentNullException.ThrowIfNull(observation); + + (VisionDetectionItem selected, int matched) = SelectDetection(request, observation); + VisionPickProvenance provenance = BuildProvenance(observation, selected, matched); + string objectClass = string.IsNullOrEmpty(request.ObjectClass) + ? selected.ClassLabel + : request.ObjectClass; + + var pick = new PickIntentInput + { + Source = request.Source, + Tool = request.Tool, + ObjectClass = objectClass, + IntentId = request.PickIntentId, + Label = request.Label, + BufferMode = request.BufferMode, + BlockingMode = request.BlockingMode + }; + + if (string.IsNullOrEmpty(request.Destination)) + { + IntentDataType intent = RoboticsIntentDtoConverter.ConvertPick(pick, context.Scope); + IntentSubmissionResult submission = await context.Client + .TrySubmitIntentAsync(intent, ct).ConfigureAwait(false); + return new VisionGuidedPickResult + { + Kind = VisionPickSubmissionKind.Pick, + Provenance = provenance, + PickSubmission = new VisionPickIntentSubmission + { + Accepted = submission.Accepted, + IntentId = submission.IntentId, + Operation = submission.Operation.IsNull ? null : submission.Operation.ToString(), + Failure = submission.Failure, + Message = submission.Message.Text + } + }; + } + + var place = new PlaceIntentInput + { + Destination = request.Destination, + Tool = request.Tool, + IntentId = request.PlaceIntentId, + Label = request.Label, + BufferMode = request.BufferMode, + BlockingMode = request.BlockingMode + }; + + MissionStepInput[] steps = + [ + new MissionStepInput + { + StepId = kPickStepId, + Released = true, + Intent = new MissionIntentInput { Kind = IntentKind.Pick, Pick = pick } + }, + new MissionStepInput + { + StepId = kPlaceStepId, + Released = true, + Intent = new MissionIntentInput { Kind = IntentKind.Place, Place = place } + } + ]; + + string missionId = string.IsNullOrEmpty(request.MissionId) + ? kMissionIdPrefix + Guid.NewGuid().ToString("N", CultureInfo.InvariantCulture) + : request.MissionId; + uint missionUpdateId = request.MissionUpdateId ?? 1; + MissionDataType mission = RoboticsMissionTools.BuildMission( + missionId, missionUpdateId, steps, null, request.Label, context.Scope); + + MissionSubmissionResult missionResult = await context.Client + .SubmitMissionAsync(mission, ct).ConfigureAwait(false); + + var result = new VisionGuidedPickResult + { + Kind = VisionPickSubmissionKind.Mission, + Provenance = provenance, + MissionSubmission = new VisionPickMissionSubmission + { + Accepted = missionResult.Accepted, + // The server echoes the MissionId it accepted; an empty echo falls back to the + // submitted id exactly as the Robot Intent client SDK does, so the caller can + // always address the mission it just created. + MissionId = missionResult.MissionId.Length == 0 ? missionId : missionResult.MissionId, + MissionUpdateId = missionUpdateId, + Operation = missionResult.Operation.IsNull ? null : missionResult.Operation.ToString(), + Failure = missionResult.Failure, + Message = missionResult.Message.Text, + PickStepId = kPickStepId, + PlaceStepId = kPlaceStepId + } + }; + + if (!missionResult.Accepted || missionResult.Operation.IsNull) + { + return result; + } + + MissionSnapshot snapshot = await context.Client.Transport + .ReadMissionSnapshotAsync(missionResult.Operation, ct).ConfigureAwait(false); + result.Steps = MapSteps(snapshot); + return result; + } + + /// + /// Filters the observed detections and selects one deterministically. + /// + /// + internal static (VisionDetectionItem Selected, int Matched) SelectDetection( + VisionGuidedPickRequest request, + VisionPickObservation observation) + { + ArgumentNullException.ThrowIfNull(request); + ArgumentNullException.ThrowIfNull(observation); + + ArrayOf detections = observation.Detections; + VisionDetectionItem? best = null; + int matched = 0; + + for (int i = 0; i < detections.Count; i++) + { + VisionDetectionItem candidate = detections[i]; + if (!Matches(request, candidate)) + { + continue; + } + + matched++; + if (best is null || IsBetter(candidate, best)) + { + best = candidate; + } + } + + if (best is null) + { + throw new InvalidOperationException(string.Create(CultureInfo.InvariantCulture, + $"No detection in Vision result '{observation.Result.ResultId}' matched the requested " + + $"filters (detectionId='{request.DetectionId ?? ""}', " + + $"classLabel='{request.ClassLabel ?? ""}', " + + $"minimumConfidence={FormatConfidence(request.MinimumConfidence)}). " + + $"The result carries {observation.TotalDetections} detection(s), " + + $"{detections.Count} of which were considered.")); + } + + return (best, matched); + } + + /// + /// Validates every request field explicitly before any server call. + /// + /// + /// + internal static void ValidateRequest(VisionGuidedPickRequest request) + { + ArgumentNullException.ThrowIfNull(request); + + RequireText(request.Controller, "controller"); + RequireText(request.Pipeline, "pipeline"); + RequireText(request.Source, "source"); + RequireText(request.Tool, "tool"); + RejectBlank(request.Destination, "destination"); + RejectBlank(request.DetectionId, "detectionId"); + RejectBlank(request.ClassLabel, "classLabel"); + RejectBlank(request.ObjectClass, "objectClass"); + RejectBlank(request.PickIntentId, "pickIntentId"); + RejectBlank(request.PlaceIntentId, "placeIntentId"); + RejectBlank(request.Label, "label"); + RejectBlank(request.MissionId, "missionId"); + + if (request.MinimumConfidence.HasValue) + { + double confidence = request.MinimumConfidence.Value; + if (!double.IsFinite(confidence)) + { + throw new ArgumentException( + "'minimumConfidence' must be a finite number.", nameof(request)); + } + + if (confidence is < 0.0 or > 1.0) + { + throw new ArgumentOutOfRangeException( + nameof(request), + confidence, + "'minimumConfidence' must be between 0 and 1 inclusive."); + } + } + + if (!Enum.IsDefined(request.Selection)) + { + throw new ArgumentOutOfRangeException( + nameof(request), request.Selection, "Invalid 'selection' value."); + } + + if (request.BufferMode.HasValue && !Enum.IsDefined(request.BufferMode.Value)) + { + throw new ArgumentOutOfRangeException( + nameof(request), request.BufferMode.Value, "Invalid 'bufferMode' value."); + } + + if (request.BlockingMode.HasValue && !Enum.IsDefined(request.BlockingMode.Value)) + { + throw new ArgumentOutOfRangeException( + nameof(request), request.BlockingMode.Value, "Invalid 'blockingMode' value."); + } + + if (!string.IsNullOrEmpty(request.Destination)) + { + return; + } + + RejectWithoutDestination(request.PlaceIntentId, "placeIntentId"); + RejectWithoutDestination(request.MissionId, "missionId"); + if (request.MissionUpdateId.HasValue) + { + throw new ArgumentException( + "'missionUpdateId' requires 'destination'; without it a single Pick intent is submitted.", + nameof(request)); + } + } + + private static bool Matches(VisionGuidedPickRequest request, VisionDetectionItem candidate) + { + if (!string.IsNullOrEmpty(request.DetectionId) && + !string.Equals(request.DetectionId, candidate.DetectionId, StringComparison.Ordinal)) + { + return false; + } + + if (!string.IsNullOrEmpty(request.ClassLabel) && + !string.Equals(request.ClassLabel, candidate.ClassLabel, StringComparison.Ordinal)) + { + return false; + } + + return !request.MinimumConfidence.HasValue || + candidate.Confidence >= request.MinimumConfidence.Value; + } + + private static bool IsBetter(VisionDetectionItem candidate, VisionDetectionItem best) + { + // double.CompareTo orders NaN below every number, so a detection whose + // confidence the server did not report can never displace a real one. + int comparison = candidate.Confidence.CompareTo(best.Confidence); + if (comparison != 0) + { + return comparison > 0; + } + + // Equal confidence: ordinal DetectionId decides, and an identical id keeps + // the earlier item, which is the original result order. + return string.CompareOrdinal(candidate.DetectionId, best.DetectionId) < 0; + } + + private static VisionPickProvenance BuildProvenance( + VisionPickObservation observation, + VisionDetectionItem selected, + int matched) + { + VisionInferenceResult result = observation.Result; + return new VisionPickProvenance + { + ResultId = result.ResultId, + ResultNodeId = result.ResultNodeId.IsNull ? string.Empty : result.ResultNodeId.ToString(), + RequestedPipelineNodeId = result.RequestedPipelineNodeId.IsNull + ? string.Empty + : result.RequestedPipelineNodeId.ToString(), + RequestedPipelineName = result.RequestedPipelineName, + PipelineId = result.PipelineId.IsNull ? null : result.PipelineId.ToString(), + SensorId = result.SensorId.IsNull ? null : result.SensorId.ToString(), + ModelVersionUsed = result.ModelVersionUsed, + CreationTime = result.CreationTime == default + ? null + : result.CreationTime.ToString("o", CultureInfo.InvariantCulture), + FrameId = result.FrameId, + TotalDetections = observation.TotalDetections, + MatchedDetections = matched, + FullResultRead = observation.FullResultRead, + SelectedDetection = BuildDetection(selected) + }; + } + + private static VisionPickDetection BuildDetection(VisionDetectionItem selected) + { + VisionPose3DDataType? pose = selected.HasPose ? selected.Pose : null; + return new VisionPickDetection + { + DetectionId = selected.DetectionId, + ClassLabel = selected.ClassLabel, + ClassId = selected.ClassId, + Confidence = selected.Confidence, + HasPose = selected.HasPose, + PoseFrameId = pose?.FrameId, + PosePosition = (pose?.Position.ToArray()), + PoseOrientation = (pose?.Orientation.ToArray()) + }; + } + + private static VisionPickMissionStep[] MapSteps(MissionSnapshot snapshot) + { + if (snapshot.Steps.Count == 0) + { + return []; + } + + var steps = new VisionPickMissionStep[snapshot.Steps.Count]; + for (int i = 0; i < snapshot.Steps.Count; i++) + { + MissionStepOperation step = snapshot.Steps[i]; + steps[i] = new VisionPickMissionStep + { + StepId = step.StepId, + IntentId = step.IntentId, + Operation = step.OperationNodeId.IsNull ? null : step.OperationNodeId.ToString(), + State = step.State + }; + } + + return steps; + } + + private static string FormatConfidence(double? confidence) + { + return confidence.HasValue + ? confidence.Value.ToString("R", CultureInfo.InvariantCulture) + : ""; + } + + private static void RequireText(string? value, string parameterName) + { + if (string.IsNullOrWhiteSpace(value)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, $"'{parameterName}' is required."), + parameterName); + } + } + + private static void RejectBlank(string? value, string parameterName) + { + if (value != null && string.IsNullOrWhiteSpace(value)) + { + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{parameterName}' must be omitted or non-empty."), + parameterName); + } + } + + private static void RejectWithoutDestination(string? value, string parameterName) + { + if (!string.IsNullOrEmpty(value)) + { + // All concatenated operands must remain interpolated for string.Create handler binding. + // TODO: Remove when RCS1214 preserves interpolated-string-handler overload binding. +#pragma warning disable RCS1214 + throw new ArgumentException( + string.Create(CultureInfo.InvariantCulture, + $"'{parameterName}' requires 'destination'; without it a single Pick intent " + + $"is submitted."), + parameterName); +#pragma warning restore RCS1214 + } + } + + private const int kMaxSummaryItems = 100; + private const string kPickStepId = "pick"; + private const string kPlaceStepId = "place"; + private const string kMissionIdPrefix = "vision-pick-"; + + private readonly OpcUaSessionManager m_sessionManager; + } + + /// + /// One bounded perception observation: the inference result, the detections + /// selection considers, and whether the full snapshot had to be read. + /// + internal sealed record VisionPickObservation( + VisionInferenceResult Result, + ArrayOf Detections, + int TotalDetections, + bool FullResultRead); +} diff --git a/tools/Opc.Ua.Mcp.Vision/EventIds.cs b/tools/Opc.Ua.Mcp.Vision/EventIds.cs new file mode 100644 index 0000000000..d8eb31b2c3 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/EventIds.cs @@ -0,0 +1,39 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +namespace Opc.Ua.Mcp +{ + /// + /// Centrally managed event id offsets for source-generated log messages in this assembly. + /// + internal static class McpVisionEventIds + { + public const int VisionClientAccessor = 0; + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/NugetREADME.md b/tools/Opc.Ua.Mcp.Vision/NugetREADME.md new file mode 100644 index 0000000000..42e05f9e5f --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/NugetREADME.md @@ -0,0 +1,128 @@ +# OPCFoundation.NetStandard.Opc.Ua.Mcp.Vision + +OPC UA Model Context Protocol (MCP) tools that let a language model **see** +through an OPC UA Vision server and act on what it sees, packaged so they can +be embedded in any MCP server host rather than only run as the shipped +`opcua-mcp` tool. + +Use this package when an application wants an LLM agent to enumerate Vision +sensors and pipelines, capture the current camera frame as MCP image content, +run inference or submit off-server perception feedback, and compose poses +between named coordinate frames. + +## The headline capability + +`vision_get_frame` returns the encoded still image as an MCP `ImageContentBlock` +with the correct MIME type, so the model actually sees pixels rather than +reading a description of them. When the server cannot render — for example on +CI without a graphics device — the tool returns an actionable text explanation +instead of a broken image. + +`vision_run_inference` accepts one structured request and returns the ResultId, +result NodeId, authoritative result kind and provenance in the same call. Its +default `Summary` detail includes a bounded detection, inspection or +segmentation summary; use the corresponding `vision_read_*_result` tool for the +complete result. Pipelines can be selected by NodeId or by an exact, +unambiguous BrowseName/DisplayName: + +```json +{ + "request": { + "pipeline": "BinPickingPipeline", + "expectedKind": "Detection", + "detail": "Summary", + "maxItems": 20 + } +} +``` + +## Tools (22) + +- **Discovery (4)** — `vision_list_sensors`, `vision_list_pipelines`, + `vision_list_frames`, `vision_list_calibrations`. +- **Monitoring (6)** — `vision_read_sensor`, + `vision_read_extrinsic_calibration`, `vision_read_pipeline`, + `vision_read_detection_result`, `vision_read_inspection_result`, + `vision_read_segmentation_result`. +- **Seeing (2)** — `vision_get_frame` (`ImageContentBlock`), + `vision_get_frame_metadata`. +- **Inference (3)** — `vision_run_inference`, + `vision_start_continuous_inference`, `vision_stop_inference`. +- **Feedback (4)** — `vision_submit_detections`, + `vision_submit_inspection_result`, `vision_submit_correction`, + `vision_submit_image_reference`. +- **Geometry (3)** — `vision_read_frame`, `vision_compose_pose`, + `vision_compose_transform`. + +Server refusals are returned honestly with the exact `StatusCode` and message; +the MCP layer does not retry and never acquires command authority as a side +effect. + +## Usage + +### Standalone `vision` profile + +Every Vision tool resolves a named OPC UA session, and only the connection +tools can open one. The single-profile overload therefore carries +`ConnectionTools` itself, so the bounded `vision` profile is usable +end-to-end without composing with any other package: + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Opc.Ua.Mcp; + +builder.Services.AddOpcUaMcpCore(); +builder.Services.AddOpcUaMcpVision(); + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithOpcUaMcpFilters() + .WithOpcUaCoreTools(McpToolProfile.Vision) + .WithOpcUaVisionTools(McpToolProfile.Vision); +``` + +### Composed with `robotics` + +Use the `McpToolProfileSet` overloads to combine Vision with Robotics — the +composition the [BinPickingClient sample](https://github.com/OPCFoundation/UA-.NETStandard/tree/main/samples/Robotics/BinPickingClient) +runs. The core-tools overload owns and deduplicates `ConnectionTools` +across every package that references the same MCP server: + +```csharp +using Opc.Ua.Mcp; + +McpToolProfileSet profiles = new McpToolProfileSet( + new[] { McpToolProfile.Vision, McpToolProfile.Robotics }); + +builder.Services.AddOpcUaMcpCore(); +builder.Services.AddOpcUaMcpVision(); +builder.Services.AddOpcUaMcpRobotics(); + +builder.Services.AddMcpServer() + .WithStdioServerTransport() + .WithOpcUaMcpFilters() + .WithOpcUaCoreTools(profiles) + .WithOpcUaVisionTools(profiles) + .WithOpcUaRoboticsTools(profiles); +``` + +A profile that does not select Vision contributes no tools rather than +failing, so the same profile value can be passed to every OPC UA tool package a +host references. + +## Related packages + +| Package | Adds | +|---|---| +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Core` | Part 4 service tools, session management, filters (required) | +| `OPCFoundation.NetStandard.Opc.Ua.Vision.Client` | Vision discovery, sensors, frames, media, inference, feedback client API | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Robotics` | Robot Intent tools that pair with Vision for pick-and-pack scenarios | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp` | the ready-to-run `opcua-mcp` server composing all OPC UA MCP tool packages | + +See the [Vision developer guide](https://github.com/OPCFoundation/UA-.NETStandard/blob/main/docs/Vision.md#mcp-tools) +and the [MCP Server guide](https://github.com/OPCFoundation/UA-.NETStandard/blob/main/docs/McpServer.md) +for the profile table, composition rules and the bin-picking sample. + +## License + +OPC Foundation MIT License 1.00 — diff --git a/tools/Opc.Ua.Mcp.Vision/Opc.Ua.Mcp.Vision.csproj b/tools/Opc.Ua.Mcp.Vision/Opc.Ua.Mcp.Vision.csproj new file mode 100644 index 0000000000..d4b8d40446 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/Opc.Ua.Mcp.Vision.csproj @@ -0,0 +1,36 @@ + + + + net8.0;net9.0;net10.0 + $(CustomTestTarget) + net10.0 + $(CustomTestTarget) + true + $(AssemblyPrefix).Mcp.Vision + $(PackagePrefix).Opc.Ua.Mcp.Vision + Opc.Ua.Mcp + OPC UA MCP tools for Vision: let a language model see through an OPC UA Vision server and act on what it sees. Discovery, image-content capture, inference, off-server perception feedback, and frame-graph composition. Embed in any MCP server host. + OPC Foundation + Copyright © 2004-2026 OPC Foundation, Inc + OPCFoundation OPC UA MCP ModelContextProtocol Vision AI LLM VLM + true + NugetREADME.md + true + enable + disable + + + + + + + + + + + + + + + + diff --git a/tools/Opc.Ua.Mcp.Vision/OpcUaMcpVisionExtensions.cs b/tools/Opc.Ua.Mcp.Vision/OpcUaMcpVisionExtensions.cs new file mode 100644 index 0000000000..738453574b --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/OpcUaMcpVisionExtensions.cs @@ -0,0 +1,153 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using Microsoft.Extensions.DependencyInjection; +using ModelContextProtocol.Server; +using Opc.Ua.Mcp.Tools; + +namespace Opc.Ua.Mcp +{ + /// + /// Registers the OPC UA Vision MCP tools and the runtime they need on a host. + /// + public static class OpcUaMcpVisionExtensions + { + /// + /// Registers the Vision client accessor the Vision tools resolve. + /// + public static IServiceCollection AddOpcUaMcpVision(this IServiceCollection services) + { + ArgumentNullException.ThrowIfNull(services); + + services.AddSingleton(); + return services; + } + + /// + /// Registers the Vision tools when selects them. + /// + /// + /// The bounded catalogue also carries the connection + /// tools, because every Vision tool resolves a named OPC UA session and only those + /// tools can open one. already carries them through the + /// core package, so they are not added twice. + /// + /// + public static IMcpServerBuilder WithOpcUaVisionTools( + this IMcpServerBuilder mcpServerBuilder, + McpToolProfile toolProfile = McpToolProfile.Full) + { + ArgumentNullException.ThrowIfNull(mcpServerBuilder); + + switch (toolProfile) + { + case McpToolProfile.Vision: + case McpToolProfile.Full: + mcpServerBuilder.WithRequestFilters(filters => + filters.AddListToolsFilter(VisionMcpFilters.AddInferenceRequestSchema)); + mcpServerBuilder + .WithTools() + .WithTools() + .WithTools() + .WithTools() + .WithTools() + .WithTools(); + + if (toolProfile == McpToolProfile.Vision) + { + // Every Vision tool resolves a named OPC UA session and only the connection + // tools can open one, so the bounded vision catalogue has to carry them to + // be usable at all. Full already gets them from the core package, so adding + // them there would register the same tools twice. + mcpServerBuilder.WithTools(); + } + + break; + case McpToolProfile.Core: + case McpToolProfile.Services: + case McpToolProfile.Administration: + case McpToolProfile.PubSub: + case McpToolProfile.Diagnostics: + case McpToolProfile.Robotics: + break; + default: + throw new ArgumentOutOfRangeException( + nameof(toolProfile), + toolProfile, + "Unknown MCP tool profile."); + } + + return mcpServerBuilder; + } + + /// + /// Registers the Vision tools when the composed + /// includes . + /// + /// + /// This overload is the composition entry point a host uses when it + /// wants the Vision tools alongside the tools of another bounded + /// profile - , for a vision-guided + /// pick-and-place agent, for example. It never registers + /// directly; the + /// McpToolProfileSet overload of WithOpcUaCoreTools owns + /// that registration and deduplicates it across every package that + /// contributes to the same MCP server. + /// + /// The MCP server builder. + /// The composed set of profiles. + /// The builder, for chaining. + /// + /// is null. + /// + public static IMcpServerBuilder WithOpcUaVisionTools( + this IMcpServerBuilder mcpServerBuilder, + McpToolProfileSet toolProfiles) + { + ArgumentNullException.ThrowIfNull(mcpServerBuilder); + + if (!toolProfiles.Contains(McpToolProfile.Vision) && + !toolProfiles.Contains(McpToolProfile.Full)) + { + return mcpServerBuilder; + } + + return mcpServerBuilder + .WithRequestFilters(filters => + filters.AddListToolsFilter(VisionMcpFilters.AddInferenceRequestSchema)) + .WithTools() + .WithTools() + .WithTools() + .WithTools() + .WithTools() + .WithTools(); + } + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/Properties/AssemblyInfo.cs b/tools/Opc.Ua.Mcp.Vision/Properties/AssemblyInfo.cs new file mode 100644 index 0000000000..7798c9bd57 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/Properties/AssemblyInfo.cs @@ -0,0 +1,32 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; + +[assembly: CLSCompliant(false)] diff --git a/tools/Opc.Ua.Mcp.Vision/Tools/VisionDiscoveryTools.cs b/tools/Opc.Ua.Mcp.Vision/Tools/VisionDiscoveryTools.cs new file mode 100644 index 0000000000..4260bdbbf8 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/Tools/VisionDiscoveryTools.cs @@ -0,0 +1,146 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using System.ComponentModel; +using System.Threading; +using System.Threading.Tasks; +using ModelContextProtocol.Server; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// MCP tools for discovering Vision sensors, inference pipelines, and + /// coordinate frames on a connected OPC UA server. + /// + [McpServerToolType] + public sealed class VisionDiscoveryTools + { + /// + /// Lists sensors exposed under the Vision root. + /// + [McpServerTool(Name = "vision_list_sensors")] + [Description("Lists the cameras and 3D sensors this server exposes, under Server/Vision/Sensors. " + + "Start here: a sensor is what produces imagery, so its NodeId is the input to " + + "vision_read_sensor, vision_get_frame and vision_read_extrinsic_calibration. Prefer " + + "vision_list_pipelines when you want the inference bindings that interpret imagery, and " + + "vision_list_frames when you want coordinate frames. Discovery only: never requests command " + + "authority, and yields nothing on a server without the Vision namespace. Returns one entry " + + "per sensor.")] + public static async Task> ListSensorsAsync( + VisionClientAccessor accessor, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionClient client = accessor.CreateClient(sessionName); + var entries = new List(); + await foreach (VisionNodeEntry entry in client.EnumerateSensorsAsync(ct).ConfigureAwait(false)) + { + entries.Add(entry); + } + return [.. entries]; + } + + /// + /// Lists inference pipelines exposed under the Vision root. + /// + [McpServerTool(Name = "vision_list_pipelines")] + [Description("Lists the inference pipelines under Server/Vision/Pipelines. A pipeline is where " + + "perception actually runs: it binds a sensor to a deployment and publishes results, and its " + + "NodeId is what vision_run_inference, vision_start_continuous_inference, vision_read_pipeline " + + "and the vision_submit_* feedback tools all take. Read its InferenceLocation to learn whether " + + "the work happens on the server or off it. Prefer vision_list_sensors when you want the imaging " + + "hardware rather than the perception bound to it. Discovery only: never requests command " + + "authority, and yields nothing on a server without the Vision namespace. Returns one entry per " + + "pipeline.")] + public static async Task> ListPipelinesAsync( + VisionClientAccessor accessor, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionClient client = accessor.CreateClient(sessionName); + var entries = new List(); + await foreach (VisionNodeEntry entry in client.EnumeratePipelinesAsync(ct).ConfigureAwait(false)) + { + entries.Add(entry); + } + return [.. entries]; + } + + /// + /// Lists coordinate frames exposed under the Vision root. + /// + [McpServerTool(Name = "vision_list_frames")] + [Description("Lists the coordinate frames under Server/Vision/Frames — the named right-handed " + + "systems that give a pose meaning, such as the robot base, the flange, the tool centre point " + + "and the camera. Their names are what vision_compose_pose takes to re-express a detection from " + + "camera coordinates into something a robot can act on. Prefer vision_list_calibrations when you " + + "want the transforms measured between two frames rather than the frames themselves. Discovery " + + "only: never requests command authority, and yields nothing on a server without the Vision " + + "namespace. Returns one entry per frame.")] + public static async Task> ListFramesAsync( + VisionClientAccessor accessor, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionClient client = accessor.CreateClient(sessionName); + var entries = new List(); + await foreach (VisionNodeEntry entry in client.EnumerateFramesAsync(ct).ConfigureAwait(false)) + { + entries.Add(entry); + } + return [.. entries]; + } + + /// + /// Lists the calibrations attached to a sensor. + /// + [McpServerTool(Name = "vision_list_calibrations")] + [Description("Lists the calibrations attached to a Vision sensor either directly via HasCalibration " + + "or nested under the sensor's Calibrations folder. Use this to find the extrinsic calibration " + + "NodeId needed by vision_read_extrinsic_calibration for a hand-eye lookup. Use " + + "vision_read_sensor instead when you only need identity, imaging members and the mount frame. " + + "Discovery only; never requests authority. Returns an array of VisionNodeEntry.")] + public static async Task> ListCalibrationsAsync( + VisionClientAccessor accessor, + [Description("Sensor NodeId, for example ns=2;s=Vision/Sensors/Camera1.")] string sensorNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionSensorClient sensor = accessor.OpenSensor(sensorNodeId, sessionName); + var entries = new List(); + await foreach (VisionNodeEntry entry in sensor.EnumerateCalibrationsAsync(ct).ConfigureAwait(false)) + { + entries.Add(entry); + } + return [.. entries]; + } + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/Tools/VisionFeedbackTools.cs b/tools/Opc.Ua.Mcp.Vision/Tools/VisionFeedbackTools.cs new file mode 100644 index 0000000000..4f693c99f1 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/Tools/VisionFeedbackTools.cs @@ -0,0 +1,286 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.ComponentModel; +using System.Threading; +using System.Threading.Tasks; +using ModelContextProtocol.Server; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// MCP tools for publishing off-server detections, inspections, corrections + /// and image references back to a Vision server via its Feedback object. + /// + [McpServerToolType] + public sealed class VisionFeedbackTools + { + /// + /// Submits detections to a Vision Feedback object. + /// + [McpServerTool(Name = "vision_submit_detections")] + [Description("Publishes a batch of off-server detections to a Vision Feedback object. Use this " + + "when the language model itself produced the detections and wants the server to persist them " + + "as first-class results. Use vision_submit_correction instead when correcting an existing " + + "result, and vision_submit_image_reference to publish only the frame the detections apply to. " + + "Detections are JSON: an array of objects with detectionId, classLabel, classId, confidence, " + + "optional boundingBox2D { centerX, centerY, width, height, rotation }, optional boundingBox3D " + + "{ center pose, size[3] }, optional pose { frameId, position[3], orientation[4], covariance[36] } " + + "and optional trackId. To report that a frame was examined and contains nothing — the " + + "terminating condition of a pick-and-place task, and a valid negative training label — pass " + + "an empty array and set sceneIsEmpty. Reports the server's refusal honestly; never retries " + + "silently and never acquires authority as a side effect.")] + public static Task SubmitDetectionsAsync( + VisionClientAccessor accessor, + [Description("Pipeline selector: an exact BrowseName, DisplayName, or NodeId string " + + "whose Feedback object should receive the detections.")] string pipeline, + [Description("Purpose the detections are submitted for: Overlay, Reconciliation, GroundTruthLabel " + + "or Trigger.")] + VisionFeedbackPurposeEnum purpose, + [Description("JSON array of detections as documented on this tool.")] string detectionsJson, + [Description("Set when the frame was examined and found to contain nothing. Required for an " + + "empty detections array, and rejected when detections are present.")] + bool sceneIsEmpty = false, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + ArrayOf detections = VisionJson.BuildDetections(detectionsJson); + return SubmitDetectionsCoreAsync( + accessor, pipeline, purpose, detections, sceneIsEmpty, sessionName, ct); + } + + /// + /// Submits an inspection result to a Vision Feedback object. + /// + [McpServerTool(Name = "vision_submit_inspection_result")] + [Description("Publishes an off-server inspection verdict to a Vision Feedback object. Use this " + + "when the language model evaluated the sensor's part against a recipe and wants the server to " + + "persist the verdict and its measured characteristics. Use vision_submit_detections instead " + + "for detection payloads and vision_submit_correction to correct an existing published result. " + + "Characteristics are JSON: an array of objects with characteristicId, name, nominal, actual, " + + "deviation, lowerTolerance, upperTolerance, uncertainty, optional unit NodeId string and status " + + "(Ok, NotOk, NotDecidable, Undefined). Reports the server's refusal honestly; never retries " + + "silently.")] + public static Task SubmitInspectionResultAsync( + VisionClientAccessor accessor, + [Description("Pipeline selector: an exact BrowseName, DisplayName, or NodeId string " + + "whose Feedback object should receive the inspection result.")] string pipeline, + [Description("Stable identifier of the inspection result.")] string resultId, + [Description("Overall evaluation: Ok, NotOk, NotDecidable or Undefined.")] + VisionResultEvaluationEnum evaluation, + [Description("JSON array of measured characteristics as documented on this tool.")] + string characteristicsJson, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + ArrayOf characteristics = VisionJson.BuildCharacteristics( + characteristicsJson); + return SubmitInspectionCoreAsync( + accessor, pipeline, resultId, evaluation, characteristics, sessionName, ct); + } + + /// + /// Submits a correction against an existing result. + /// + /// + [McpServerTool(Name = "vision_submit_correction")] + [Description("Publishes a correction to an existing Vision result identified by its ResultId. Use " + + "this when the language model disagrees with a server-published detection or inspection and " + + "wants to attach a corrected version. Use vision_submit_detections instead to publish fresh " + + "detections without a target result, and vision_submit_inspection_result for a fresh verdict. " + + "At most one of correctedDetectionsJson or correctedCharacteristicsJson may be provided. To " + + "retract a false positive — asserting the result should contain nothing at all, which is the " + + "error class an operator can label most confidently — omit both and set retractAll. Reports " + + "the server's refusal honestly; never retries silently.")] + public static Task SubmitCorrectionAsync( + VisionClientAccessor accessor, + [Description("Pipeline selector: an exact BrowseName, DisplayName, or NodeId string " + + "whose Feedback object should receive the correction.")] string pipeline, + [Description("Stable identifier of the result being corrected.")] string resultId, + [Description("Purpose the correction is submitted for: Overlay, Reconciliation, GroundTruthLabel " + + "or Trigger.")] + VisionFeedbackPurposeEnum purpose, + [Description("Optional JSON array of corrected detections; mutually exclusive with " + + "correctedCharacteristicsJson.")] + string? correctedDetectionsJson = null, + [Description("Optional JSON array of corrected characteristics; mutually exclusive with " + + "correctedDetectionsJson.")] + string? correctedCharacteristicsJson = null, + [Description("Human-readable reason attached to the correction.")] + string? reason = null, + [Description("Set to retract the referenced result entirely, asserting it should contain " + + "nothing. Requires both corrected arrays to be omitted.")] + bool retractAll = false, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + bool hasDetections = !string.IsNullOrWhiteSpace(correctedDetectionsJson); + bool hasCharacteristics = !string.IsNullOrWhiteSpace(correctedCharacteristicsJson); + if (retractAll) + { + if (hasDetections || hasCharacteristics) + { + throw new ArgumentException( + "Both corrected arrays must be omitted when retractAll is set.", + nameof(correctedDetectionsJson)); + } + } + else if (hasDetections == hasCharacteristics) + { + throw new ArgumentException( + "Provide exactly one of correctedDetectionsJson and correctedCharacteristicsJson, " + + "or set retractAll to retract the result entirely.", + nameof(correctedDetectionsJson)); + } + ArrayOf detections = hasDetections + ? VisionJson.BuildDetections(correctedDetectionsJson!) + : ArrayOf.Empty; + ArrayOf characteristics = hasCharacteristics + ? VisionJson.BuildCharacteristics(correctedCharacteristicsJson!) + : ArrayOf.Empty; + LocalizedText localizedReason = string.IsNullOrEmpty(reason) + ? LocalizedText.Null + : new LocalizedText(reason); + return SubmitCorrectionCoreAsync( + accessor, + pipeline, + resultId, + purpose, + detections, + characteristics, + localizedReason, + retractAll, + sessionName, + ct); + } + + /// + /// Submits an image reference to a Vision Feedback object. + /// + [McpServerTool(Name = "vision_submit_image_reference")] + [Description("Publishes an image reference to a Vision Feedback object. Use this when the model " + + "wants the server to persist the frame that a detection or correction was reasoned about, or " + + "to attach a ground-truth image without any detections. Use vision_submit_detections when you " + + "also want to publish detections against the frame. The image JSON is a single object with " + + "uri, format (Jpeg, Png, ...), pixelFormat, width, height, sizeBytes, timestamp (ISO 8601), " + + "digest (base64) and digestAlgorithm. Reports the server's refusal honestly; never retries " + + "silently.")] + public static Task SubmitImageReferenceAsync( + VisionClientAccessor accessor, + [Description("Pipeline selector: an exact BrowseName, DisplayName, or NodeId string " + + "whose Feedback object should receive the image reference.")] string pipeline, + [Description("Purpose the image is submitted for: Overlay, Reconciliation, GroundTruthLabel " + + "or Trigger.")] + VisionFeedbackPurposeEnum purpose, + [Description("JSON object describing the image reference as documented on this tool.")] string imageJson, + [Description("Stable identifier of the target result, or an empty string.")] string resultId = "", + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionImageReferenceDataType image = VisionJson.BuildImageReference(imageJson, nameof(imageJson)); + return SubmitImageReferenceCoreAsync( + accessor, pipeline, purpose, image, resultId, sessionName, ct); + } + + private static async Task SubmitDetectionsCoreAsync( + VisionClientAccessor accessor, + string pipeline, + VisionFeedbackPurposeEnum purpose, + ArrayOf detections, + bool sceneIsEmpty, + string? sessionName, + CancellationToken ct) + { + VisionFeedbackClient feedback = await accessor.OpenPipelineFeedbackAsync( + pipeline, sessionName, ct).ConfigureAwait(false); + await feedback.SubmitDetectionsAsync( + purpose, detections, frameReference: null, inlineImage: ByteString.Empty, + sceneIsEmpty, ct) + .ConfigureAwait(false); + } + + private static async Task SubmitInspectionCoreAsync( + VisionClientAccessor accessor, + string pipeline, + string resultId, + VisionResultEvaluationEnum evaluation, + ArrayOf characteristics, + string? sessionName, + CancellationToken ct) + { + VisionFeedbackClient feedback = await accessor.OpenPipelineFeedbackAsync( + pipeline, sessionName, ct).ConfigureAwait(false); + await feedback.SubmitInspectionResultAsync(resultId, evaluation, characteristics, ct) + .ConfigureAwait(false); + } + + private static async Task SubmitCorrectionCoreAsync( + VisionClientAccessor accessor, + string pipeline, + string resultId, + VisionFeedbackPurposeEnum purpose, + ArrayOf detections, + ArrayOf characteristics, + LocalizedText reason, + bool retractAll, + string? sessionName, + CancellationToken ct) + { + VisionFeedbackClient feedback = await accessor.OpenPipelineFeedbackAsync( + pipeline, sessionName, ct).ConfigureAwait(false); + await feedback.SubmitCorrectionAsync( + resultId, + purpose, + detections, + characteristics, + reason, + inlineImage: ByteString.Empty, + retractAll, + ct).ConfigureAwait(false); + } + + private static async Task SubmitImageReferenceCoreAsync( + VisionClientAccessor accessor, + string pipeline, + VisionFeedbackPurposeEnum purpose, + VisionImageReferenceDataType image, + string resultId, + string? sessionName, + CancellationToken ct) + { + VisionFeedbackClient feedback = await accessor.OpenPipelineFeedbackAsync( + pipeline, sessionName, ct).ConfigureAwait(false); + await feedback.SubmitImageReferenceAsync(purpose, image, resultId, ct) + .ConfigureAwait(false); + } + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/Tools/VisionGeometryTools.cs b/tools/Opc.Ua.Mcp.Vision/Tools/VisionGeometryTools.cs new file mode 100644 index 0000000000..a51d4ddf22 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/Tools/VisionGeometryTools.cs @@ -0,0 +1,119 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.ComponentModel; +using System.Threading; +using System.Threading.Tasks; +using ModelContextProtocol.Server; +using Opc.Ua.Mcp.Serialization; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// MCP tools for composing and inspecting Vision coordinate frames. + /// + [McpServerToolType] + public sealed class VisionGeometryTools + { + /// + /// Reads a single frame node from the Vision frame graph. + /// + [McpServerTool(Name = "vision_read_frame")] + [Description("Reads a coordinate frame from the Vision frame graph: FrameId, ParentFrameId, and " + + "the six-degree-of-freedom Transform relative to the parent. Use this to inspect the tree " + + "without composing anything. Use vision_compose_pose instead when translating a pose from one " + + "named frame into another, and vision_compose_transform to obtain only the transform between " + + "two frames. Reports what the server declares; a null-NodeId parent means the frame is a root. " + + "Returns a VisionFrameSnapshot.")] + public static Task ReadFrameAsync( + VisionClientAccessor accessor, + [Description("Frame NodeId, for example ns=2;s=Vision/Frames/RobotBase.")] string frameNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionFrameGraph graph = accessor.OpenFrames(sessionName); + return graph.ReadAsync(OpcUaJsonHelper.ParseNodeId(frameNodeId), ct); + } + + /// + /// Composes a pose from one Vision frame into another. + /// + [McpServerTool(Name = "vision_compose_pose")] + [Description("Composes a pose expressed in one Vision frame into a target frame by walking the " + + "frame graph. Use this to convert a detection expressed in camera coordinates into robot base " + + "or tool-centre-point coordinates so a controller can act on it. Use vision_compose_transform " + + "instead when you only need the transform between the two frames. Pose JSON is a single " + + "object: { frameId, position:[x,y,z], orientation:[x,y,z,w], covariance:[36 doubles or omit] }. " + + "Reports the server's refusal honestly if the frames are not connected; never invents an " + + "identity transform. Returns the transformed VisionPose3DDataType.")] + public static Task ComposePoseAsync( + VisionClientAccessor accessor, + [Description("JSON pose in the source frame; the frameId field is optional and, when set, " + + "must match fromFrameNodeId's FrameId.")] + string poseJson, + [Description("Source frame NodeId (the one the input pose is expressed in).")] string fromFrameNodeId, + [Description("Target frame NodeId (the one the returned pose is expressed in).")] string toFrameNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionPose3DDataType pose = VisionJson.BuildPose(poseJson, nameof(poseJson)); + VisionFrameGraph graph = accessor.OpenFrames(sessionName); + return graph.ComposeAsync( + pose, + OpcUaJsonHelper.ParseNodeId(fromFrameNodeId), + OpcUaJsonHelper.ParseNodeId(toFrameNodeId), + ct); + } + + /// + /// Composes the transform between two Vision frames. + /// + [McpServerTool(Name = "vision_compose_transform")] + [Description("Composes the six-degree-of-freedom transform between two Vision frames by walking " + + "the frame graph and returns it as a VisionPose3DDataType with FrameId set to the target " + + "frame. Use vision_compose_pose instead when you also have a pose expressed in the source " + + "frame that you want transformed. Reports the server's refusal honestly if the frames are not " + + "connected; never invents an identity transform. Returns the composed transform.")] + public static Task ComposeTransformAsync( + VisionClientAccessor accessor, + [Description("Source frame NodeId.")] string fromFrameNodeId, + [Description("Target frame NodeId.")] string toFrameNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionFrameGraph graph = accessor.OpenFrames(sessionName); + return graph.ComposeTransformAsync( + OpcUaJsonHelper.ParseNodeId(fromFrameNodeId), + OpcUaJsonHelper.ParseNodeId(toFrameNodeId), + ct); + } + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/Tools/VisionInferenceTools.cs b/tools/Opc.Ua.Mcp.Vision/Tools/VisionInferenceTools.cs new file mode 100644 index 0000000000..f82424be03 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/Tools/VisionInferenceTools.cs @@ -0,0 +1,514 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.Collections.Generic; +using System.ComponentModel; +using System.ComponentModel.DataAnnotations; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using ModelContextProtocol.Server; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Mcp.Tools +{ + // CLR arrays are intentional on these MCP result DTOs: ArrayOf serializes + // its backing memory object instead of the JSON array agents need. The Vision + // client service keeps ArrayOf; this boundary projects it for MCP only. + + /// + /// MCP tools for driving Vision inference pipelines. + /// + [McpServerToolType] + public sealed class VisionInferenceTools + { + /// + /// Runs a single one-shot inference on a Vision pipeline with structured + /// result. Accepts a pipeline selector (name or NodeId), expected result + /// kind, detail level, and bounded items. + /// + [McpServerTool(Name = "vision_run_inference")] + [Description("Invokes RunInference on a Vision pipeline for a single acquisition. Accepts a " + + "pipeline selector (exact BrowseName, DisplayName, or NodeId string) so you do not need to " + + "discover a NodeId before calling. Resolves the result, determines its kind (detection, " + + "inspection, segmentation), and optionally reads a bounded concise summary. Set detail to " + + "HandleOnly to return a result handle without reading the result payload. Set expectedKind " + + "to enforce the produced result kind. Reports the server's refusal honestly; never retries " + + "silently and never adjusts the pipeline configuration.")] + public static async Task RunInferenceAsync( + VisionClientAccessor accessor, + [Description("The one-shot inference request.")] VisionInferenceRequest request, + CancellationToken ct = default) + { + System.ArgumentNullException.ThrowIfNull(request); + ValidateRequest(request); + System.ArgumentNullException.ThrowIfNull(accessor); + + (VisionNodeEntry entry, VisionPipelineClient pipelineClient) = + await accessor.ResolvePipelineAsync( + request.Pipeline, request.SessionName, ct) + .ConfigureAwait(false); + + VisionInferenceService service = accessor.CreateInferenceService(request.SessionName); + VisionInferenceResult result = await service.RunOneShotAsync( + pipelineClient, + entry.BrowseName.Name, + request.Detail, + request.ExpectedKind, + request.MaxItems, + ct).ConfigureAwait(false); + + return VisionInferenceRunResult.FromServiceResult(result); + } + + /// + /// Starts continuous inference on a Vision pipeline. + /// + [McpServerTool(Name = "vision_start_continuous_inference")] + [Description("Invokes StartContinuous on a Vision pipeline so the server acquires and publishes " + + "results repeatedly. Use vision_run_inference for a single one-shot invocation. Use " + + "vision_stop_inference to halt the loop. Reports the server's refusal honestly if the " + + "pipeline is not runnable; never retries silently and never asks the server to change mode. " + + "Returns no value on success.")] + public static async Task StartContinuousInferenceAsync( + VisionClientAccessor accessor, + [Description("Pipeline selector: an exact BrowseName, DisplayName, or NodeId string.")] string pipeline, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + (_, VisionPipelineClient pipelineClient) = + await accessor.ResolvePipelineAsync(pipeline, sessionName, ct) + .ConfigureAwait(false); + await pipelineClient.StartContinuousAsync(ct).ConfigureAwait(false); + } + + /// + /// Stops continuous or in-progress inference on a Vision pipeline. + /// + [McpServerTool(Name = "vision_stop_inference")] + [Description("Invokes Stop on a Vision pipeline to halt continuous or in-progress inference. Use " + + "vision_start_continuous_inference to start it again. Use vision_run_inference to run a " + + "single pass instead. Reports the server's refusal honestly if the pipeline cannot be " + + "stopped in its current state; never retries silently. Returns no value on success.")] + public static async Task StopInferenceAsync( + VisionClientAccessor accessor, + [Description("Pipeline selector: an exact BrowseName, DisplayName, or NodeId string.")] string pipeline, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + (_, VisionPipelineClient pipelineClient) = + await accessor.ResolvePipelineAsync(pipeline, sessionName, ct) + .ConfigureAwait(false); + await pipelineClient.StopAsync(ct).ConfigureAwait(false); + } + + private static void ValidateRequest(VisionInferenceRequest request) + { + System.ArgumentException.ThrowIfNullOrWhiteSpace(request.Pipeline); + + if (!System.Enum.IsDefined(request.ExpectedKind)) + { + throw new System.ArgumentOutOfRangeException( + nameof(request), + request.ExpectedKind, + "Invalid expectedKind value."); + } + + if (!System.Enum.IsDefined(request.Detail)) + { + throw new System.ArgumentOutOfRangeException( + nameof(request), + request.Detail, + "Invalid detail value."); + } + + if (request.MaxItems < 0 || request.MaxItems > 100) + { + throw new System.ArgumentOutOfRangeException( + nameof(request), + request.MaxItems, + "maxItems must be between 0 and 100 inclusive."); + } + } + } + + /// + /// Structured input for vision_run_inference. + /// + public sealed record VisionInferenceRequest + { + /// + /// Exact pipeline BrowseName, DisplayName, or NodeId selector. + /// + [Description("Pipeline selector: an exact BrowseName, DisplayName, or NodeId string " + + "(e.g. 'BinPickingPipeline' or 'ns=2;s=Vision/Pipelines/BinPickingPipeline').")] + public required string Pipeline { get; init; } + + /// + /// Expected result kind. Auto accepts any recognized result kind. + /// + [Description("Expected result kind: Auto (default) accepts any kind; Detection, Inspection, " + + "or Segmentation requires a matching result.")] + [DefaultValue(VisionExpectedResultKind.Auto)] + public VisionExpectedResultKind ExpectedKind { get; init; } = VisionExpectedResultKind.Auto; + + /// + /// Requested payload detail. Summary reads a concise bounded response. + /// + [Description("Detail level: Summary (default) reads a concise bounded summary; HandleOnly " + + "returns only the result handle.")] + [DefaultValue(VisionResultDetail.Summary)] + public VisionResultDetail Detail { get; init; } = VisionResultDetail.Summary; + + /// + /// Maximum detection or inspection items in a concise summary. + /// + [Range(0, 100)] + [Description("Maximum number of detections or characteristics in the concise summary. " + + "Must be between 0 and 100 inclusive. Default 20.")] + [DefaultValue(20)] + public int MaxItems { get; init; } = 20; + + /// + /// Session name to use; defaults to the only active session. + /// + [Description("Session name to use; defaults to the only active session.")] + public string? SessionName { get; init; } + } + + /// + /// Structured result of a single inference run, including handle information and + /// an optional concise summary. Returned by the vision_run_inference tool. + /// + public sealed record VisionInferenceRunResult + { + /// + /// The ResultId the Server assigned to this run. + /// + public required string ResultId { get; init; } + + /// + /// The NodeId the Server published the result at, which is what the + /// vision_read_*_result tools take. Empty when it could not be resolved. + /// + public required string ResultNodeId { get; init; } + + /// + /// True when is usable. + /// + public bool Resolved { get; init; } + + /// + /// The detected result kind. + /// + public VisionResultKind ResultKind { get; init; } + + /// + /// The requested pipeline's published name, when available. + /// + public string? RequestedPipelineName { get; init; } + + /// + /// The Pipeline NodeId requested to run this inference. + /// + public string? RequestedPipelineNodeId { get; init; } + + /// + /// The Pipeline NodeId published by the result, when available. + /// + public string? PipelineId { get; init; } + + /// + /// The sensor NodeId that produced the frame, when available. + /// + public string? SensorId { get; init; } + + /// + /// The model version used, when reported. + /// + public string? ModelVersionUsed { get; init; } + + /// + /// Result creation time (ISO 8601), when available. + /// + public string? CreationTime { get; init; } + + /// + /// Frame identifier the poses are expressed in, when available. + /// + public string? FrameId { get; init; } + + /// + /// Detection summary, populated when resultKind is Detection and detail + /// was Summary. + /// + public VisionInferenceDetectionSummary? Detection { get; init; } + + /// + /// Inspection summary, populated when resultKind is Inspection and detail + /// was Summary. + /// + public VisionInferenceInspectionSummary? Inspection { get; init; } + + /// + /// Segmentation summary, populated when resultKind is Segmentation and + /// detail was Summary. + /// + public VisionInferenceSegmentationSummary? Segmentation { get; init; } + + /// + /// Creates a run result from the service-level result. + /// + internal static VisionInferenceRunResult FromServiceResult(VisionInferenceResult r) + { + return new VisionInferenceRunResult + { + ResultId = r.ResultId, + ResultNodeId = r.ResultNodeId.IsNull ? string.Empty : r.ResultNodeId.ToString(), + Resolved = r.Resolved, + ResultKind = r.ResultKind, + RequestedPipelineName = r.RequestedPipelineName, + RequestedPipelineNodeId = r.RequestedPipelineNodeId.IsNull + ? null + : r.RequestedPipelineNodeId.ToString(), + PipelineId = r.PipelineId.IsNull ? null : r.PipelineId.ToString(), + SensorId = r.SensorId.IsNull ? null : r.SensorId.ToString(), + ModelVersionUsed = r.ModelVersionUsed, + CreationTime = r.CreationTime == default + ? null + : r.CreationTime.ToString("o", CultureInfo.InvariantCulture), + FrameId = r.FrameId, + Detection = ToDetectionSummary(r.DetectionSummary), + Inspection = ToInspectionSummary(r.InspectionSummary), + Segmentation = ToSegmentationSummary(r.SegmentationSummary) + }; + } + + private static VisionInferenceDetectionSummary? ToDetectionSummary( + VisionDetectionSummary? summary) + { + if (summary is null) + { + return null; + } + + var items = new List(summary.Items.Count); + for (int i = 0; i < summary.Items.Count; i++) + { + VisionDetectionItem item = summary.Items[i]; + items.Add(new VisionInferenceDetectionItem + { + DetectionId = item.DetectionId, + ClassLabel = item.ClassLabel, + ClassId = item.ClassId, + Confidence = item.Confidence, + HasPose = item.HasPose + }); + } + + return new VisionInferenceDetectionSummary + { + TotalDetections = summary.TotalDetections, + Items = items.ToArray() + }; + } + + private static VisionInferenceInspectionSummary? ToInspectionSummary( + VisionInspectionSummary? summary) + { + if (summary is null) + { + return null; + } + + var items = new List(summary.Items.Count); + for (int i = 0; i < summary.Items.Count; i++) + { + VisionCharacteristicItem item = summary.Items[i]; + items.Add(new VisionInferenceCharacteristicSummary + { + Name = item.Name, + Status = item.Status, + Deviation = item.Deviation + }); + } + + return new VisionInferenceInspectionSummary + { + Evaluation = summary.Evaluation, + PartId = summary.PartId, + RecipeId = summary.RecipeId, + TotalCharacteristics = summary.TotalCharacteristics, + Items = items.ToArray() + }; + } + + private static VisionInferenceSegmentationSummary? ToSegmentationSummary( + VisionSegmentationSummary? summary) + { + if (summary is null) + { + return null; + } + + return new VisionInferenceSegmentationSummary + { + LabelClasses = summary.LabelClasses.ToArray() ?? [], + MaskWidth = summary.MaskWidth, + MaskHeight = summary.MaskHeight, + MaskFormat = summary.MaskFormat + }; + } + } + + /// + /// Lean detection summary returned by vision_run_inference. + /// + public sealed record VisionInferenceDetectionSummary + { + /// + /// Total number of detections in the published result. + /// + public int TotalDetections { get; init; } + + /// + /// Bounded detection items without geometry payloads. + /// + public VisionInferenceDetectionItem[] Items { get; init; } = []; + } + + /// + /// Lean detection item returned by vision_run_inference. + /// + public sealed record VisionInferenceDetectionItem + { + /// + /// Detection identifier. + /// + public string DetectionId { get; init; } = string.Empty; + + /// + /// Detection class label. + /// + public string ClassLabel { get; init; } = string.Empty; + + /// + /// Detection class identifier. + /// + public uint ClassId { get; init; } + + /// + /// Detection confidence. + /// + public double Confidence { get; init; } + + /// + /// Whether the source detection has a pose. + /// + public bool HasPose { get; init; } + } + + /// + /// Lean inspection summary returned by vision_run_inference. + /// + public sealed record VisionInferenceInspectionSummary + { + /// + /// Overall inspection evaluation. + /// + public Vision.VisionResultEvaluationEnum Evaluation { get; init; } + + /// + /// Inspected part identifier, when reported. + /// + public string? PartId { get; init; } + + /// + /// Inspection recipe identifier, when reported. + /// + public string? RecipeId { get; init; } + + /// + /// Total number of characteristics in the result. + /// + public int TotalCharacteristics { get; init; } + + /// + /// Bounded characteristic summaries. + /// + public VisionInferenceCharacteristicSummary[] Items { get; init; } = []; + } + + /// + /// Lean characteristic summary returned by vision_run_inference. + /// + public sealed record VisionInferenceCharacteristicSummary + { + /// + /// Characteristic name. + /// + public string Name { get; init; } = string.Empty; + + /// + /// Tolerance status. + /// + public Vision.VisionToleranceStatusEnum Status { get; init; } + + /// + /// Deviation from the nominal value. + /// + public double Deviation { get; init; } + } + + /// + /// Lean segmentation summary returned by vision_run_inference. + /// + public sealed record VisionInferenceSegmentationSummary + { + /// + /// Label classes associated with the mask. + /// + public string[] LabelClasses { get; init; } = []; + + /// + /// Mask width, when reported. + /// + public uint MaskWidth { get; init; } + + /// + /// Mask height, when reported. + /// + public uint MaskHeight { get; init; } + + /// + /// Mask format, when reported. + /// + public string? MaskFormat { get; init; } + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/Tools/VisionJson.cs b/tools/Opc.Ua.Mcp.Vision/Tools/VisionJson.cs new file mode 100644 index 0000000000..c737048e0d --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/Tools/VisionJson.cs @@ -0,0 +1,389 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Text.Json; +using Opc.Ua.Mcp.Serialization; +using Opc.Ua.Vision; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// Parses Vision detection and characteristic payloads submitted through MCP + /// tools as JSON, mapping the ROS 2 vision_msgs conventions the Vision + /// NodeSet documents onto the generated Vision data types. + /// + internal static class VisionJson + { + public static ArrayOf BuildDetections(string detectionsJson) + { + if (string.IsNullOrWhiteSpace(detectionsJson)) + { + throw new ArgumentException( + "Detections JSON must be a non-empty array.", nameof(detectionsJson)); + } + + using JsonDocument document = JsonDocument.Parse(detectionsJson); + if (document.RootElement.ValueKind != JsonValueKind.Array) + { + throw new ArgumentException( + "Detections JSON must be an array.", nameof(detectionsJson)); + } + + var detections = new List(); + foreach (JsonElement element in document.RootElement.EnumerateArray()) + { + if (element.ValueKind != JsonValueKind.Object) + { + throw new ArgumentException( + "Each detection must be a JSON object.", nameof(detectionsJson)); + } + detections.Add(BuildDetection(element)); + } + + return [.. detections]; + } + + public static ArrayOf BuildCharacteristics(string characteristicsJson) + { + if (string.IsNullOrWhiteSpace(characteristicsJson)) + { + throw new ArgumentException( + "Characteristics JSON must be a non-empty array.", nameof(characteristicsJson)); + } + + using JsonDocument document = JsonDocument.Parse(characteristicsJson); + if (document.RootElement.ValueKind != JsonValueKind.Array) + { + throw new ArgumentException( + "Characteristics JSON must be an array.", nameof(characteristicsJson)); + } + + var characteristics = new List(); + foreach (JsonElement element in document.RootElement.EnumerateArray()) + { + if (element.ValueKind != JsonValueKind.Object) + { + throw new ArgumentException( + "Each characteristic must be a JSON object.", nameof(characteristicsJson)); + } + characteristics.Add(BuildCharacteristic(element)); + } + + return [.. characteristics]; + } + + public static VisionPose3DDataType BuildPose(string poseJson, string parameterName) + { + if (string.IsNullOrWhiteSpace(poseJson)) + { + throw new ArgumentException("Pose JSON must be a non-empty object.", parameterName); + } + + using JsonDocument document = JsonDocument.Parse(poseJson); + if (document.RootElement.ValueKind != JsonValueKind.Object) + { + throw new ArgumentException("Pose JSON must be an object.", parameterName); + } + + return BuildPose(document.RootElement, parameterName); + } + + public static VisionImageReferenceDataType BuildImageReference(string imageJson, string parameterName) + { + if (string.IsNullOrWhiteSpace(imageJson)) + { + throw new ArgumentException( + "Image reference JSON must be a non-empty object.", parameterName); + } + + using JsonDocument document = JsonDocument.Parse(imageJson); + if (document.RootElement.ValueKind != JsonValueKind.Object) + { + throw new ArgumentException( + "Image reference JSON must be an object.", parameterName); + } + + JsonElement root = document.RootElement; + return new VisionImageReferenceDataType + { + Uri = GetString(root, "uri") ?? string.Empty, + Digest = GetByteString(root, "digest"), + DigestAlgorithm = GetString(root, "digestAlgorithm") ?? "SHA-256", + Format = GetEnum(root, "format", VisionClipFormatEnum.Jpeg), + PixelFormat = GetString(root, "pixelFormat") ?? string.Empty, + Width = GetUInt32(root, "width", 0), + Height = GetUInt32(root, "height", 0), + SizeBytes = GetUInt32(root, "sizeBytes", 0), + Timestamp = GetDateTimeUtc(root, "timestamp") + }; + } + + private static VisionDetectionDataType BuildDetection(JsonElement element) + { + var detection = new VisionDetectionDataType + { + DetectionId = GetString(element, "detectionId") ?? string.Empty, + ClassLabel = GetString(element, "classLabel") ?? string.Empty, + ClassId = GetUInt32(element, "classId", 0), + Confidence = GetDouble(element, "confidence", 0.0), + TrackId = GetString(element, "trackId") ?? string.Empty + }; + + if (element.TryGetProperty("boundingBox2D", out JsonElement box2D) && + box2D.ValueKind == JsonValueKind.Object) + { + detection.HasBoundingBox2D = true; + detection.BoundingBox2D = new VisionBoundingBox2DDataType + { + CenterX = GetDouble(box2D, "centerX", 0.0), + CenterY = GetDouble(box2D, "centerY", 0.0), + Width = GetDouble(box2D, "width", 0.0), + Height = GetDouble(box2D, "height", 0.0), + Rotation = GetDouble(box2D, "rotation", 0.0) + }; + } + + if (element.TryGetProperty("boundingBox3D", out JsonElement box3D) && + box3D.ValueKind == JsonValueKind.Object) + { + detection.HasBoundingBox3D = true; + detection.BoundingBox3D = new VisionBoundingBox3DDataType + { + Center = BuildPose(GetRequiredProperty(box3D, "center"), "center"), + Size = ReadDoubleArray(box3D, "size", 3, "boundingBox3D.size") + }; + } + + if (element.TryGetProperty("pose", out JsonElement pose) && + pose.ValueKind == JsonValueKind.Object) + { + detection.HasPose = true; + detection.Pose = BuildPose(pose, "pose"); + } + + return detection; + } + + private static VisionCharacteristicDataType BuildCharacteristic(JsonElement element) + { + var unit = new EUInformation(); + if (element.TryGetProperty("unit", out JsonElement unitElement) && + unitElement.ValueKind == JsonValueKind.Object) + { + string? namespaceUri = GetString(unitElement, "namespaceUri"); + string? shortName = GetString(unitElement, "shortName"); + string? longName = GetString(unitElement, "longName") ?? shortName; + if (!string.IsNullOrEmpty(shortName) && !string.IsNullOrEmpty(namespaceUri)) + { + unit = new EUInformation(shortName, longName ?? shortName, namespaceUri); + } + } + return new VisionCharacteristicDataType + { + CharacteristicId = GetString(element, "characteristicId") ?? string.Empty, + Name = GetString(element, "name") ?? string.Empty, + Nominal = GetDouble(element, "nominal", 0.0), + Actual = GetDouble(element, "actual", 0.0), + Deviation = GetDouble(element, "deviation", 0.0), + LowerTolerance = GetDouble(element, "lowerTolerance", 0.0), + UpperTolerance = GetDouble(element, "upperTolerance", 0.0), + Uncertainty = GetDouble(element, "uncertainty", 0.0), + Unit = unit, + Status = GetEnum(element, "status", VisionToleranceStatusEnum.InTolerance) + }; + } + + private static VisionPose3DDataType BuildPose(JsonElement element, string parameterName) + { + ArrayOf position = ReadDoubleArray(element, "position", 3, $"{parameterName}.position"); + ArrayOf orientation = ReadDoubleArray( + element, "orientation", 4, $"{parameterName}.orientation"); + ArrayOf covariance = ArrayOf.Empty; + if (element.TryGetProperty("covariance", out JsonElement covarianceElement) && + covarianceElement.ValueKind == JsonValueKind.Array) + { + covariance = ReadDoubleArray( + element, "covariance", 36, $"{parameterName}.covariance"); + } + return new VisionPose3DDataType + { + FrameId = GetString(element, "frameId") ?? string.Empty, + Position = position, + Orientation = orientation, + Covariance = covariance + }; + } + + private static ArrayOf ReadDoubleArray( + JsonElement element, + string propertyName, + int expectedLength, + string context) + { + if (!element.TryGetProperty(propertyName, out JsonElement value) || + value.ValueKind != JsonValueKind.Array) + { + throw new ArgumentException( + string.Create( + CultureInfo.InvariantCulture, + $"Missing or malformed array '{propertyName}' in {context}."), + context); + } + var buffer = new List(expectedLength); + foreach (JsonElement item in value.EnumerateArray()) + { + if (item.ValueKind != JsonValueKind.Number) + { + throw new ArgumentException( + string.Create( + CultureInfo.InvariantCulture, + $"All entries in '{propertyName}' must be numbers in {context}."), + context); + } + buffer.Add(item.GetDouble()); + } + if (buffer.Count != expectedLength) + { + throw new ArgumentException( + string.Create( + CultureInfo.InvariantCulture, + $"Array '{propertyName}' must have exactly {expectedLength} entries in {context}."), + context); + } + return [.. buffer]; + } + + private static JsonElement GetRequiredProperty(JsonElement element, string propertyName) + { + if (!element.TryGetProperty(propertyName, out JsonElement value)) + { + throw new ArgumentException( + string.Create( + CultureInfo.InvariantCulture, + $"Required property '{propertyName}' is missing."), + propertyName); + } + return value; + } + + private static string? GetString(JsonElement element, string propertyName) + { + if (!element.TryGetProperty(propertyName, out JsonElement value)) + { + return null; + } + return value.ValueKind == JsonValueKind.String ? value.GetString() : null; + } + + private static uint GetUInt32(JsonElement element, string propertyName, uint fallback) + { + if (!element.TryGetProperty(propertyName, out JsonElement value) || + value.ValueKind != JsonValueKind.Number) + { + return fallback; + } + return value.TryGetUInt32(out uint parsed) ? parsed : fallback; + } + + private static double GetDouble(JsonElement element, string propertyName, double fallback) + { + if (!element.TryGetProperty(propertyName, out JsonElement value) || + value.ValueKind != JsonValueKind.Number) + { + return fallback; + } + return value.GetDouble(); + } + + private static ByteString GetByteString(JsonElement element, string propertyName) + { + if (!element.TryGetProperty(propertyName, out JsonElement value) || + value.ValueKind != JsonValueKind.String) + { + return ByteString.Empty; + } + string? base64 = value.GetString(); + if (string.IsNullOrEmpty(base64)) + { + return ByteString.Empty; + } + return new ByteString(Convert.FromBase64String(base64)); + } + + private static TEnum GetEnum(JsonElement element, string propertyName, TEnum fallback) + where TEnum : struct, Enum + { + if (!element.TryGetProperty(propertyName, out JsonElement value)) + { + return fallback; + } + if (value.ValueKind == JsonValueKind.String) + { + string? name = value.GetString(); + if (!string.IsNullOrEmpty(name) && + Enum.TryParse(name, ignoreCase: true, out TEnum parsed)) + { + return parsed; + } + return fallback; + } + if (value.ValueKind == JsonValueKind.Number && value.TryGetInt32(out int number)) + { + return (TEnum)Enum.ToObject(typeof(TEnum), number); + } + return fallback; + } + + private static DateTimeUtc GetDateTimeUtc(JsonElement element, string propertyName) + { + if (!element.TryGetProperty(propertyName, out JsonElement value) || + value.ValueKind != JsonValueKind.String) + { + return default; + } + string? text = value.GetString(); + if (string.IsNullOrEmpty(text)) + { + return default; + } + if (DateTime.TryParse( + text, + CultureInfo.InvariantCulture, + DateTimeStyles.RoundtripKind | DateTimeStyles.AssumeUniversal, + out DateTime parsed)) + { + return new DateTimeUtc(parsed.ToUniversalTime()); + } + return default; + } + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/Tools/VisionMonitoringTools.cs b/tools/Opc.Ua.Mcp.Vision/Tools/VisionMonitoringTools.cs new file mode 100644 index 0000000000..29d1164b3e --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/Tools/VisionMonitoringTools.cs @@ -0,0 +1,230 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System.ComponentModel; +using System.Threading; +using System.Threading.Tasks; +using ModelContextProtocol.Server; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// MCP tools for reading Vision sensor and pipeline state. + /// + [McpServerToolType] + public sealed class VisionMonitoringTools + { + /// + /// Reads sensor identity, imaging members, optics and mount frame. + /// + [McpServerTool(Name = "vision_read_sensor")] + [Description("Reads a Vision sensor's identity (SensorId, RealityKind, Modality, manufacturer, " + + "model, serial number, device URI, frame id), image members (Width, Height, PixelFormat, " + + "ExposureTime, Gain, AcquisitionFrameRate, intrinsics) when it is an ImageSensorType, and the " + + "mounted frame NodeId when declared via MountedOn. Use this after vision_list_sensors to obtain " + + "the details needed for image-space work; use vision_read_extrinsic_calibration when you need " + + "the camera-to-robot transform for a hand-eye lookup. Reports only what the server declares; " + + "members the sensor does not carry come back as null. Returns a VisionSensorSnapshot.")] + public static async Task ReadSensorAsync( + VisionClientAccessor accessor, + [Description("Sensor NodeId, for example ns=2;s=Vision/Sensors/Camera1.")] string sensorNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionSensorClient sensor = accessor.OpenSensor(sensorNodeId, sessionName); + VisionSensorIdentity identity = await sensor.ReadIdentityAsync(ct).ConfigureAwait(false); + VisionImageSensorSnapshot? image = await sensor.ReadImageMembersAsync(ct).ConfigureAwait(false); + VisionDepth3DSensorSnapshot? depth = await sensor.ReadDepthMembersAsync(ct).ConfigureAwait(false); + VisionOpticsSnapshot? optics = await sensor.ReadOpticsAsync(ct).ConfigureAwait(false); + VisionIlluminationSnapshot? illumination = await sensor.ReadIlluminationAsync(ct) + .ConfigureAwait(false); + NodeId mounted = await sensor.GetMountedFrameIdAsync(ct).ConfigureAwait(false); + return new VisionSensorSnapshot + { + Identity = identity, + Image = image, + Depth = depth, + Optics = optics, + Illumination = illumination, + MountedFrameId = mounted + }; + } + + /// + /// Reads an extrinsic calibration. + /// + [McpServerTool(Name = "vision_read_extrinsic_calibration")] + [Description("Reads a Vision extrinsic-calibration snapshot: the camera-to-target 6-DoF transform, " + + "mount arrangement (EyeInHand, EyeToHand, Fixed), source and target frame NodeIds, residual " + + "error, method, and validity. Use this to obtain the hand-eye transform needed to convert " + + "detection poses into robot base or tool-centre-point coordinates. Use vision_list_calibrations " + + "first to enumerate the available calibration NodeIds. Reports only what the server declares; " + + "an invalid calibration should be treated as unusable rather than substituting a default. " + + "Returns a VisionExtrinsicCalibrationSnapshot.")] + public static Task ReadExtrinsicCalibrationAsync( + VisionClientAccessor accessor, + [Description("Sensor NodeId the calibration is attached to.")] string sensorNodeId, + [Description("Extrinsic calibration NodeId.")] string calibrationNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionSensorClient sensor = accessor.OpenSensor(sensorNodeId, sessionName); + NodeId parsed = Serialization.OpcUaJsonHelper.ParseNodeId(calibrationNodeId); + return sensor.ReadExtrinsicCalibrationAsync(parsed, ct); + } + + /// + /// Reads an inference pipeline's current state. + /// + [McpServerTool(Name = "vision_read_pipeline")] + [Description("Reads a Vision inference pipeline's live state: PipelineId, current EndpointState, " + + "Continuous flag, bound Sensor NodeId, Deployment NodeId, and any LearningJob NodeId. Use this " + + "before calling vision_run_inference or vision_start_continuous_inference to check the " + + "pipeline is ready and to record its Deployment NodeId. When the Server also implements OPC UA " + + "AI Model Management, the Deployment NodeId points at the AI Model Management deployment whose " + + "InferenceLocation says where inference physically runs. Use vision_read_result instead when " + + "you want a specific published result. Reports server state only, never infers pipeline state, " + + "and never requests authority. Returns a VisionPipelineSnapshot.")] + public static async Task ReadPipelineAsync( + VisionClientAccessor accessor, + [Description("Pipeline selector: an exact BrowseName, DisplayName, or NodeId string " + + "(e.g. 'BinPickingPipeline' or 'ns=2;s=Vision/Pipelines/BinPickingPipeline').")] string pipeline, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + (_, VisionPipelineClient pipelineClient) = + await accessor.ResolvePipelineAsync(pipeline, sessionName, ct) + .ConfigureAwait(false); + return await pipelineClient.ReadAsync(ct).ConfigureAwait(false); + } + + /// + /// Reads a Vision detection result. + /// + [McpServerTool(Name = "vision_read_detection_result")] + [Description("Reads a Vision detection result (DetectionResultType) into its snapshot: ResultId, " + + "CreationTime, sensor and pipeline NodeIds, ModelVersionUsed, the acquisition Frame image " + + "reference, and the array of detections with class label, confidence, and 2D or 3D geometry. " + + "Use this after vision_run_inference or when observing the pipeline's Results folder. Use " + + "vision_read_inspection_result instead when the pipeline publishes an InspectionResultType, " + + "and vision_read_segmentation_result for a SegmentationResultType. Reports server state only, " + + "never fabricates detections. Returns a VisionDetectionResultSnapshot.")] + public static Task ReadDetectionResultAsync( + VisionClientAccessor accessor, + [Description("Result NodeId of a DetectionResultType instance.")] string resultNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionResultReader reader = accessor.OpenResult(resultNodeId, sessionName); + return reader.ReadDetectionAsync(ct); + } + + /// + /// Reads a Vision inspection result. + /// + [McpServerTool(Name = "vision_read_inspection_result")] + [Description("Reads a Vision inspection result (InspectionResultType) into its snapshot: ResultId, " + + "CreationTime, sensor and pipeline NodeIds, ModelVersionUsed, the acquisition Frame image " + + "reference, overall Evaluation (Ok, NotOk, NotDecidable, Undefined), PartId, RecipeId, and the " + + "measured characteristics. Use this when the pipeline publishes verdicts against tolerances. " + + "Use vision_read_detection_result instead for DetectionResultType and " + + "vision_read_segmentation_result for SegmentationResultType. Reports server state only. " + + "Returns a VisionInspectionResultSnapshot.")] + public static Task ReadInspectionResultAsync( + VisionClientAccessor accessor, + [Description("Result NodeId of an InspectionResultType instance.")] string resultNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionResultReader reader = accessor.OpenResult(resultNodeId, sessionName); + return reader.ReadInspectionAsync(ct); + } + + /// + /// Reads a Vision segmentation result. + /// + [McpServerTool(Name = "vision_read_segmentation_result")] + [Description("Reads a Vision segmentation result (SegmentationResultType) into its snapshot: " + + "ResultId, CreationTime, sensor and pipeline NodeIds, the acquisition Frame image reference, " + + "the label class names, and the mask image reference. Use this when the pipeline publishes " + + "per-pixel segmentation labels. Use vision_read_detection_result for detections and " + + "vision_read_inspection_result for verdicts. Reports server state only. Returns a " + + "VisionSegmentationResultSnapshot.")] + public static Task ReadSegmentationResultAsync( + VisionClientAccessor accessor, + [Description("Result NodeId of a SegmentationResultType instance.")] string resultNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionResultReader reader = accessor.OpenResult(resultNodeId, sessionName); + return reader.ReadSegmentationAsync(ct); + } + } + + /// + /// Combined sensor snapshot returned by the vision_read_sensor MCP tool. It + /// bundles identity, imaging members, depth members (when the sensor is a + /// Depth3D sensor), optics, illumination, and the mounted frame NodeId so an + /// agent gets everything a single round-trip in the tool call. + /// + public sealed record VisionSensorSnapshot + { + /// + /// Sensor identity nameplate. + /// + public required VisionSensorIdentity Identity { get; init; } + + /// + /// Imaging members when the sensor is an ImageSensorType, or null. + /// + public VisionImageSensorSnapshot? Image { get; init; } + + /// + /// Depth members when the sensor is a Depth3DSensorType, or null. + /// + public VisionDepth3DSensorSnapshot? Depth { get; init; } + + /// + /// Optics description when the sensor declares one, or null. + /// + public VisionOpticsSnapshot? Optics { get; init; } + + /// + /// Illumination description when the sensor declares one, or null. + /// + public VisionIlluminationSnapshot? Illumination { get; init; } + + /// + /// The NodeId of the frame the sensor is mounted on, or a null NodeId + /// when the sensor does not declare a MountedOn frame. + /// + public NodeId MountedFrameId { get; init; } = NodeId.Null; + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/Tools/VisionSeeingTools.cs b/tools/Opc.Ua.Mcp.Vision/Tools/VisionSeeingTools.cs new file mode 100644 index 0000000000..8d5b636fd3 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/Tools/VisionSeeingTools.cs @@ -0,0 +1,376 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.ComponentModel; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using ModelContextProtocol.Protocol; +using ModelContextProtocol.Server; +using Opc.Ua.Vision; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Mcp.Tools +{ + /// + /// MCP tools that surface the latest Vision-sensor frame to the language + /// model as image content it can actually see. + /// + [McpServerToolType] + public sealed class VisionSeeingTools + { + /// + /// Returns the latest still frame from a sensor as MCP image content. + /// + [McpServerTool(Name = "vision_get_frame")] + [Description("Returns the latest still frame from a Vision sensor as an MCP ImageContentBlock the " + + "model can inspect directly. The image is delivered at the sensor's own resolution and encoding " + + "and is not resampled by this tool; if the encoded bytes exceed the model's context, request a " + + "smaller PNG/JPEG format on the sensor. Use vision_get_frame_metadata when you only need the " + + "frame descriptor (URI, dimensions, timestamp, digest) without the pixels. When the server has " + + "no rendering backend, has inline delivery disabled, or has not yet published a frame, returns " + + "a TextContentBlock explaining the reason rather than an empty image. Never retries silently " + + "and never asks the server to switch mode as a side-effect.")] + public static async Task GetFrameAsync( + VisionClientAccessor accessor, + [Description("Sensor NodeId, for example ns=2;s=Vision/Sensors/Camera1.")] string sensorNodeId, + [Description("Preferred encoded image format when the sensor supports selection: Jpeg, Png, " + + "Tiff, Bmp, WebP, GenDc or Other. Defaults to Jpeg.")] + VisionClipFormatEnum format = VisionClipFormatEnum.Jpeg, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + (ByteString bytes, VisionClipFormatEnum actualFormat, string? reason) = await AcquireFrameAsync( + accessor, sensorNodeId, format, ct).ConfigureAwait(false); + + if (reason is not null) + { + return new CallToolResult + { + IsError = true, + Content = [new TextContentBlock { Text = reason }] + }; + } + + return new CallToolResult + { + Content = + [ + // FromBytes base64-encodes for the wire. Assigning the encoded frame + // straight to Data put raw bytes where the protocol requires base64: + // every byte that is not valid UTF-8 was serialised as U+FFFD, so the + // image reached the model corrupted beyond recovery - and six times + // larger, because each byte became a \uXXXX escape. + ImageContentBlock.FromBytes(bytes.Memory, MimeTypeFor(actualFormat)) + ] + }; + } + + /// + /// Returns the metadata descriptor for the latest still frame from a + /// sensor, without the pixel bytes. + /// + [McpServerTool(Name = "vision_get_frame_metadata")] + [Description("Returns the latest still-frame descriptor from a Vision sensor without transferring " + + "pixels: URI, MIME type, width, height, size, timestamp and digest. Use this to check that a " + + "frame is available and inspect its dimensions before calling vision_get_frame, or to log the " + + "URI a downstream tool should fetch out of band. This tool never returns the encoded image; " + + "use vision_get_frame for that. Returns a VisionFrameMetadata record; when the sensor has no " + + "clip metadata, StatusMessage explains why and Available is false.")] + public static async Task GetFrameMetadataAsync( + VisionClientAccessor accessor, + [Description("Sensor NodeId, for example ns=2;s=Vision/Sensors/Camera1.")] string sensorNodeId, + [Description("Session name to use; defaults to the only active session.")] string? sessionName = null, + CancellationToken ct = default) + { + VisionSensorClient sensor = accessor.OpenSensor(sensorNodeId, sessionName); + VisionMediaClient? media = await sensor.OpenMediaAsync(ct).ConfigureAwait(false); + if (media is null) + { + return VisionFrameMetadata.Unavailable( + "Sensor exposes no Media object; there are no clip endpoints to read a frame from."); + } + + NodeId clipEndpoint = await FirstClipEndpointAsync(media, ct).ConfigureAwait(false); + if (clipEndpoint.IsNull) + { + return VisionFrameMetadata.Unavailable( + "Sensor's Media object exposes no ClipEndpoint; no still frame descriptors are published."); + } + + try + { + VisionImageReferenceDataType? metadata = await media + .ReadLatestClipMetadataAsync(clipEndpoint, ct).ConfigureAwait(false); + if (metadata is null) + { + return VisionFrameMetadata.Unavailable( + "Clip endpoint has no LatestClipMetadata; the server has not published a frame yet."); + } + return new VisionFrameMetadata + { + Available = true, + Uri = metadata.Uri, + MimeType = MimeTypeFor(metadata.Format), + Format = metadata.Format, + Width = metadata.Width, + Height = metadata.Height, + SizeBytes = metadata.SizeBytes, + PixelFormat = metadata.PixelFormat, + Timestamp = metadata.Timestamp, + Digest = metadata.Digest, + DigestAlgorithm = metadata.DigestAlgorithm, + StatusMessage = null + }; + } + catch (ServiceResultException ex) + { + return VisionFrameMetadata.Unavailable( + string.Create( + CultureInfo.InvariantCulture, + $"Server refused reading the latest clip metadata: {ex.Result}")); + } + } + + private static async Task<(ByteString Bytes, VisionClipFormatEnum Format, string? Reason)> + AcquireFrameAsync( + VisionClientAccessor accessor, + string sensorNodeId, + VisionClipFormatEnum format, + CancellationToken ct) + { + VisionSensorClient sensor = accessor.OpenSensor(sensorNodeId); + VisionMediaClient? media = await sensor.OpenMediaAsync(ct).ConfigureAwait(false); + if (media is null) + { + return ( + ByteString.Empty, + format, + "Sensor exposes no Media object; there are no clip endpoints to acquire a frame from."); + } + + NodeId clipEndpoint = await FirstClipEndpointAsync(media, ct).ConfigureAwait(false); + if (clipEndpoint.IsNull) + { + return ( + ByteString.Empty, + format, + "Sensor's Media object exposes no ClipEndpoint; no still frame can be acquired."); + } + + try + { + VisionClipResult clip = await media.GetClipAsync( + clipEndpoint, + resultId: null, + timestamp: default, + format: format, + requestInline: true, + cancellationToken: ct).ConfigureAwait(false); + if (clip.HasInlineImage) + { + return (clip.InlineImage, clip.Image.Format, null); + } + return ( + ByteString.Empty, + clip.Image.Format, + string.Create( + CultureInfo.InvariantCulture, + $"Server returned only an out-of-band image reference at '{clip.Image.Uri}'. Fetch it out of band or ask the server to enable inline delivery.")); + } + catch (ServiceResultException ex) + { + VisionInlineClipReading fallback; + try + { + fallback = await media.ReadLatestClipAsync(clipEndpoint, ct).ConfigureAwait(false); + } + catch (ServiceResultException inner) + { + return ( + ByteString.Empty, + format, + string.Create( + CultureInfo.InvariantCulture, + $"Server refused GetClip ({ex.Result}) and refused LatestClip ({inner.Result}). Likely no rendering backend is attached to the server.")); + } + return ClassifyFallback(fallback, format, ex.Result); + } + } + + private static (ByteString Bytes, VisionClipFormatEnum Format, string? Reason) ClassifyFallback( + VisionInlineClipReading fallback, + VisionClipFormatEnum format, + ServiceResult getClipError) + { + switch (fallback.State) + { + case VisionInlineClipState.Available: + VisionClipFormatEnum actualFormat = fallback.Metadata?.Format ?? format; + return (fallback.Bytes, actualFormat, null); + case VisionInlineClipState.NotYetAvailable: + return ( + ByteString.Empty, + format, + string.Create( + CultureInfo.InvariantCulture, + $"Server has not published a frame yet (LatestClip = Bad_NoDataAvailable). GetClip returned {getClipError}. Wait or trigger acquisition.")); + case VisionInlineClipState.InlineDisabled: + return ( + ByteString.Empty, + format, + string.Create( + CultureInfo.InvariantCulture, + $"Inline image delivery is disabled on this clip endpoint (LatestClip = Bad_NotSupported). Fall back to the out-of-band URI or enable inline delivery. GetClip returned {getClipError}.")); + case VisionInlineClipState.Overflow: + return ( + ByteString.Empty, + format, + string.Create( + CultureInfo.InvariantCulture, + $"Last frame exceeded the inline delivery limit (LatestClip = Bad_EncodingLimitsExceeded). Request a smaller PNG/JPEG format or fetch out of band. GetClip returned {getClipError}.")); + default: + return ( + ByteString.Empty, + format, + string.Create( + CultureInfo.InvariantCulture, + $"Server cannot render a frame right now. GetClip returned {getClipError}; LatestClip returned {fallback.StatusCode}. This is the pattern reported when a scene camera has no rendering backend attached.")); + } + } + + private static async Task FirstClipEndpointAsync( + VisionMediaClient media, + CancellationToken ct) + { + await foreach (VisionNodeEntry entry in media.EnumerateClipEndpointsAsync(ct) + .ConfigureAwait(false)) + { + if (!entry.NodeId.IsNull) + { + return entry.NodeId; + } + } + return NodeId.Null; + } + + private static string MimeTypeFor(VisionClipFormatEnum format) + { + return format switch + { + VisionClipFormatEnum.Jpeg => "image/jpeg", + VisionClipFormatEnum.Png => "image/png", + VisionClipFormatEnum.Tiff => "image/tiff", + VisionClipFormatEnum.Bmp => "image/bmp", + VisionClipFormatEnum.WebP => "image/webp", + _ => "application/octet-stream" + }; + } + } + + /// + /// Descriptor of the latest still frame available on a Vision sensor. Returned + /// by the vision_get_frame_metadata tool without transferring the encoded + /// pixels. + /// + public sealed record VisionFrameMetadata + { + /// + /// True when the server published a frame descriptor for the sensor. + /// + public required bool Available { get; init; } + + /// + /// The out-of-band URI a downstream tool can fetch to obtain the encoded + /// image bytes, or null when no metadata is available. + /// + public string? Uri { get; init; } + + /// + /// The MIME type derived from the encoded image format, for example + /// image/jpeg. + /// + public string? MimeType { get; init; } + + /// + /// The encoded image format the server used. + /// + public VisionClipFormatEnum Format { get; init; } + + /// + /// Image width in pixels, or zero when unknown. + /// + public uint Width { get; init; } + + /// + /// Image height in pixels, or zero when unknown. + /// + public uint Height { get; init; } + + /// + /// Encoded image size in bytes, or zero when unknown. + /// + public uint SizeBytes { get; init; } + + /// + /// Pixel format string as declared by the server, or null when not set. + /// + public string? PixelFormat { get; init; } + + /// + /// Timestamp associated with the frame. + /// + public DateTimeUtc Timestamp { get; init; } + + /// + /// Content digest of the encoded image, or empty when not set. + /// + public ByteString Digest { get; init; } = ByteString.Empty; + + /// + /// The digest algorithm the server used, defaulting to SHA-256. + /// + public string? DigestAlgorithm { get; init; } + + /// + /// Human-readable status message when Available is false; otherwise null. + /// + public string? StatusMessage { get; init; } + + internal static VisionFrameMetadata Unavailable(string message) + { + return new VisionFrameMetadata + { + Available = false, + StatusMessage = message + }; + } + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/VisionClientAccessor.cs b/tools/Opc.Ua.Mcp.Vision/VisionClientAccessor.cs new file mode 100644 index 0000000000..2e87ae7784 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/VisionClientAccessor.cs @@ -0,0 +1,178 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Threading; +using System.Threading.Tasks; +using Opc.Ua.Client; +using Opc.Ua.Vision.Client; + +namespace Opc.Ua.Mcp +{ + /// + /// Creates Vision clients from active MCP OPC UA sessions and hands out + /// focused sub-clients for a named sensor, pipeline, media manager, + /// feedback object, result, or the frame graph. + /// + public sealed class VisionClientAccessor + { + /// + /// Initializes the accessor. + /// + public VisionClientAccessor(OpcUaSessionManager sessionManager) + { + m_sessionManager = sessionManager ?? throw new ArgumentNullException(nameof(sessionManager)); + } + + /// + /// Creates the top-level Vision client over the named or sole active session. + /// + public VisionClient CreateClient(string? sessionName = null) + { + ISession session = m_sessionManager.GetSessionOrThrow(sessionName); + return new VisionClient(session, m_sessionManager.Telemetry); + } + + /// + /// Opens a focused sensor client over the named or sole active session. + /// + public VisionSensorClient OpenSensor(string sensorNodeId, string? sessionName = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(sensorNodeId); + + return CreateClient(sessionName).Sensor(Serialization.OpcUaJsonHelper.ParseNodeId(sensorNodeId)); + } + + /// + /// Opens a focused pipeline client over the named or sole active session. + /// + public VisionPipelineClient OpenPipeline(string pipelineNodeId, string? sessionName = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(pipelineNodeId); + + return CreateClient(sessionName).Pipeline( + Serialization.OpcUaJsonHelper.ParseNodeId(pipelineNodeId)); + } + + /// + /// Opens a focused media-management client over the named or sole active session. + /// + public VisionMediaClient OpenMedia(string mediaNodeId, string? sessionName = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(mediaNodeId); + + return CreateClient(sessionName).Media( + Serialization.OpcUaJsonHelper.ParseNodeId(mediaNodeId)); + } + + /// + /// Opens a focused feedback client over the named or sole active session. + /// + public VisionFeedbackClient OpenFeedback(string feedbackNodeId, string? sessionName = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(feedbackNodeId); + + return CreateClient(sessionName).Feedback( + Serialization.OpcUaJsonHelper.ParseNodeId(feedbackNodeId)); + } + + /// + /// Opens the feedback client attached to a selected pipeline over the named or sole active session. + /// + /// + public async Task OpenPipelineFeedbackAsync( + string pipelineSelector, + string? sessionName = null, + CancellationToken ct = default) + { + (_, VisionPipelineClient pipeline) = await ResolvePipelineAsync( + pipelineSelector, sessionName, ct).ConfigureAwait(false); + VisionFeedbackClient? feedback = await pipeline.OpenFeedbackAsync(ct).ConfigureAwait(false); + return feedback ?? + throw new InvalidOperationException( + "Pipeline does not expose a Feedback object."); + } + + /// + /// Opens a focused result reader over the named or sole active session. + /// + public VisionResultReader OpenResult(string resultNodeId, string? sessionName = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(resultNodeId); + + return CreateClient(sessionName).Result( + Serialization.OpcUaJsonHelper.ParseNodeId(resultNodeId)); + } + + /// + /// Opens the frame graph over the named or sole active session. + /// + public VisionFrameGraph OpenFrames(string? sessionName = null) + { + return CreateClient(sessionName).Frames(); + } + + /// + /// Resolves a pipeline by exact unique name (BrowseName.Name or + /// DisplayName.Text) or by NodeId string, returning both the resolved + /// entry and a ready-to-use pipeline client. + /// + /// + /// A NodeId string or an exact pipeline name. + /// + /// + /// Session name to use; defaults to the only active session. + /// + /// + /// Cancels the operation. + /// + public async Task<(VisionNodeEntry Entry, VisionPipelineClient Pipeline)> + ResolvePipelineAsync( + string pipelineSelector, + string? sessionName = null, + CancellationToken ct = default) + { + VisionClient client = CreateClient(sessionName); + VisionNodeEntry entry = await client.ResolvePipelineAsync( + pipelineSelector, ct).ConfigureAwait(false); + VisionPipelineClient pipeline = client.Pipeline(entry.NodeId); + return (entry, pipeline); + } + + /// + /// Creates the one-shot inference service from the named or sole active session. + /// + public VisionInferenceService CreateInferenceService(string? sessionName = null) + { + return CreateClient(sessionName).Inference(); + } + + private readonly OpcUaSessionManager m_sessionManager; + } +} diff --git a/tools/Opc.Ua.Mcp.Vision/VisionMcpFilters.cs b/tools/Opc.Ua.Mcp.Vision/VisionMcpFilters.cs new file mode 100644 index 0000000000..34f0fda7e5 --- /dev/null +++ b/tools/Opc.Ua.Mcp.Vision/VisionMcpFilters.cs @@ -0,0 +1,170 @@ +/* ======================================================================== + * Copyright (c) 2005-2026 The OPC Foundation, Inc. All rights reserved. + * + * OPC Foundation MIT License 1.00 + * + * Permission is hereby granted, free of charge, to any person + * obtaining a copy of this software and associated documentation + * files (the "Software"), to deal in the Software without + * restriction, including without limitation the rights to use, + * copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES + * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT + * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, + * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * The complete license agreement can be found here: + * http://opcfoundation.org/License/MIT/1.00/ + * ======================================================================*/ + +using System; +using System.Text.Json; +using System.Text.Json.Nodes; +using System.Threading.Tasks; +using ModelContextProtocol.Protocol; +using ModelContextProtocol.Server; + +namespace Opc.Ua.Mcp +{ + /// + /// Normalizes the generated Vision inference request schema at the MCP boundary. + /// + internal static class VisionMcpFilters + { + /// + /// Adds the defaults and bounds required by the structured + /// vision_run_inference request contract. + /// + public static McpRequestHandler + AddInferenceRequestSchema( + McpRequestHandler next) + { + ArgumentNullException.ThrowIfNull(next); + + return async (request, ct) => + { + ListToolsResult result = await next(request, ct).ConfigureAwait(false); + foreach (Tool tool in result.Tools) + { + if (tool.Name != "vision_run_inference" || + tool.InputSchema.ValueKind != JsonValueKind.Object) + { + continue; + } + + JsonObject? schema = JsonNode.Parse(tool.InputSchema.GetRawText()) as JsonObject; + if (schema is null || !SetInferenceRequestContract(schema)) + { + continue; + } + + using JsonDocument document = JsonDocument.Parse(schema.ToJsonString()); + tool.InputSchema = document.RootElement.Clone(); + } + + return result; + }; + } + + private static bool SetInferenceRequestContract(JsonObject schema) + { + bool expectedKindChanged = SetStringEnum( + schema, + ["properties", "request", "properties", "expectedKind"], + kExpectedResultKinds, + "Auto"); + bool detailChanged = SetStringEnum( + schema, + ["properties", "request", "properties", "detail"], + kResultDetails, + "Summary"); + bool maxItemsChanged = SetIntegerRange( + schema, + ["properties", "request", "properties", "maxItems"], + 0, + 100, + 20); + return expectedKindChanged || detailChanged || maxItemsChanged; + } + + private static bool SetStringEnum( + JsonObject schema, + string[] path, + string[] values, + string defaultValue) + { + if (!TryGetProperty(schema, path, out JsonObject? property)) + { + return false; + } + + var enumValues = new JsonArray(); + for (int i = 0; i < values.Length; i++) + { + enumValues.Add(values[i]); + } + + property!["type"] = "string"; + property["enum"] = enumValues; + property["default"] = defaultValue; + return true; + } + + private static bool SetIntegerRange( + JsonObject schema, + string[] path, + int minimum, + int maximum, + int defaultValue) + { + if (!TryGetProperty(schema, path, out JsonObject? property)) + { + return false; + } + + property!["type"] = "integer"; + property["minimum"] = minimum; + property["maximum"] = maximum; + property["default"] = defaultValue; + return true; + } + + private static bool TryGetProperty( + JsonObject schema, + string[] path, + out JsonObject? property) + { + JsonNode? current = schema; + for (int i = 0; i < path.Length; i++) + { + if (current is not JsonObject currentObject || + currentObject[path[i]] is not JsonNode child) + { + property = null; + return false; + } + + current = child; + } + + property = current as JsonObject; + return property is not null; + } + + private static readonly string[] kExpectedResultKinds = + ["Auto", "Detection", "Inspection", "Segmentation"]; + + private static readonly string[] kResultDetails = + ["Summary", "HandleOnly"]; + } +} diff --git a/tools/Opc.Ua.Mcp/McpHostBuilder.cs b/tools/Opc.Ua.Mcp/McpHostBuilder.cs index 10bc4f95d0..bf44ee7092 100644 --- a/tools/Opc.Ua.Mcp/McpHostBuilder.cs +++ b/tools/Opc.Ua.Mcp/McpHostBuilder.cs @@ -46,7 +46,6 @@ namespace Opc.Ua.Mcp /// internal static class McpHostBuilder { - /// /// Registers the OPC UA client, session/PubSub managers and Pcap /// diagnostics services used by the MCP tools. @@ -62,6 +61,7 @@ public static void ConfigureServices( services.AddOpcUaMcpCore(OpcUaMcpOptions ?? CreateOpcUaMcpOptions()); services.AddOpcUaMcpPubSub(); services.AddOpcUaMcpRobotics(); + services.AddOpcUaMcpVision(); services.AddOpcUaMcpDiagnostics(options => { options.BaseFolder = pcapOptions.BaseFolder; @@ -81,38 +81,56 @@ public static OpcUaMcpOptions CreateOpcUaMcpOptions() } /// - /// Creates the from configuration and an optional CLI override. + /// Creates the from configuration and an optional CLI override + /// expressed as a comma or plus separated list of profile names, so a host can compose + /// several bounded profiles - vision,robotics, for instance - into one MCP server. /// public static OpcUaMcpOptions CreateOpcUaMcpOptions( IConfiguration configuration, - McpToolProfile? toolProfileOverride) + string? toolProfileOverride) { ArgumentNullException.ThrowIfNull(configuration); OpcUaMcpOptions options = CreateOpcUaMcpOptions(); - if (toolProfileOverride.HasValue) + if (!string.IsNullOrWhiteSpace(toolProfileOverride)) { - options.ToolProfile = toolProfileOverride.Value; + ApplyConfiguredProfile(options, toolProfileOverride); return options; } string? configuredProfile = configuration["McpServer:ToolProfile"] ?? Environment.GetEnvironmentVariable("OPCUA_MCP_TOOL_PROFILE"); + ApplyConfiguredProfile(options, configuredProfile); + return options; + } + + private static void ApplyConfiguredProfile(OpcUaMcpOptions options, string? configuredProfile) + { if (string.IsNullOrWhiteSpace(configuredProfile)) { - return options; + return; } - if (!Enum.TryParse(configuredProfile, ignoreCase: true, out McpToolProfile toolProfile) || - !Enum.IsDefined(toolProfile)) + if (!McpToolProfileSet.TryParse( + configuredProfile, + out McpToolProfileSet toolProfiles, + out string? error)) { - throw new InvalidOperationException( - $"Unknown MCP tool profile '{configuredProfile}'. " + - $"Valid profiles: {string.Join(", ", Enum.GetNames())}."); + throw new InvalidOperationException(error); } - options.ToolProfile = toolProfile; - return options; + if (toolProfiles.Count == 1) + { + foreach (McpToolProfile single in toolProfiles.Enumerate()) + { + options.ToolProfile = single; + } + options.ToolProfiles = McpToolProfileSet.Empty; + } + else + { + options.ToolProfiles = toolProfiles; + } } /// @@ -160,12 +178,56 @@ public static void ConfigureMcpTools( .WithOpcUaCoreTools(toolProfile) .WithOpcUaPubSubTools(toolProfile) .WithOpcUaRoboticsTools(toolProfile) + .WithOpcUaVisionTools(toolProfile) .WithOpcUaDiagnosticsTools(toolProfile, diagnosticsToolsEnabled) .WithOpcUaPubSubDiagnosticsTools(toolProfile, diagnosticsToolsEnabled); mcpServerBuilder.WithResources(); } + /// + /// Registers the MCP tool types selected by the composed + /// , so a single MCP server can carry the + /// tools of several bounded profiles at once - vision plus robotics for + /// a vision-guided pick-and-place agent, for example. + /// + /// + /// This calls each package's McpToolProfileSet overload in + /// turn. The core package's overload registers + /// ConnectionTools exactly once for any set that includes at + /// least one session-scoped profile, so composing Vision and + /// Robotics yields one set of connection tools rather than two. + /// + public static void ConfigureMcpTools( + IMcpServerBuilder mcpServerBuilder, + McpToolProfileSet toolProfiles, + bool diagnosticsToolsEnabled) + { + ArgumentNullException.ThrowIfNull(mcpServerBuilder); + + if (toolProfiles.Count <= 1) + { + McpToolProfile single = McpToolProfile.Full; + foreach (McpToolProfile profile in toolProfiles.Enumerate()) + { + single = profile; + } + ConfigureMcpTools(mcpServerBuilder, single, diagnosticsToolsEnabled); + return; + } + + mcpServerBuilder + .WithOpcUaMcpFilters() + .WithOpcUaCoreTools(toolProfiles) + .WithOpcUaPubSubTools(toolProfiles) + .WithOpcUaRoboticsTools(toolProfiles) + .WithOpcUaVisionTools(toolProfiles) + .WithOpcUaDiagnosticsTools(toolProfiles, diagnosticsToolsEnabled) + .WithOpcUaPubSubDiagnosticsTools(toolProfiles, diagnosticsToolsEnabled); + + mcpServerBuilder.WithResources(); + } + /// /// Emits a warning log entry when the Pcap diagnostics MCP tools are /// enabled, since they disclose symmetric channel keys. diff --git a/tools/Opc.Ua.Mcp/McpREADME.md b/tools/Opc.Ua.Mcp/McpREADME.md index ef857a0532..07f1075391 100644 --- a/tools/Opc.Ua.Mcp/McpREADME.md +++ b/tools/Opc.Ua.Mcp/McpREADME.md @@ -22,7 +22,13 @@ opcua-mcp --transport http --port 5100 ## Tools -The server exposes tools through a **tool profile** — a bounded, named catalog selected with `--profile core|services|administration|pubsub|diagnostics|full`. `full` is the default and currently registers every tool listed below; `core` and the other profiles expose a smaller, focused subset. See the [full documentation](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/docs/McpServer.md#tool-profiles) for the profile-to-tool mapping. +The server exposes tools through a **tool profile** — a bounded, named catalog +selected with +`--profile core|services|administration|pubsub|diagnostics|robotics|vision|full`. +Profiles can be composed, for example `--profile vision,robotics`. `full` is the +default; the other profiles expose smaller focused subsets. See the +[full documentation](https://github.com/OPCFoundation/UA-.NETStandard/blob/master/docs/McpServer.md#tool-profiles) +for the profile-to-tool mapping. Tools in the `full` profile cover all OPC UA Part 4 service sets: @@ -35,6 +41,10 @@ Tools in the `full` profile cover all OPC UA Part 4 service sets: - **MonitoredItem**: CreateMonitoredItems, ModifyMonitoredItems, SetMonitoringMode, SetTriggering, DeleteMonitoredItems - **Discovery**: FindServers, FindServersOnNetwork, GetEndpoints, RegisterServer, RegisterServer2 - **Convenience**: ReadValue, ReadValues, WriteValue, BrowseAll, CallMethod, ReadNode, Cancel +- **Robotics**: typed Robot Intent control and missions, paged monitoring, + bounded operation/mission waits, and same-session `robotics_vision_pick` +- **Vision**: image capture, structured one-shot inference, result monitoring, + feedback and frame-graph composition ## Embedding @@ -57,6 +67,8 @@ builder.Services.AddMcpServer() | `OPCFoundation.NetStandard.Opc.Ua.Mcp.PubSub` | PubSub runtime, actions, discovery | | `OPCFoundation.NetStandard.Opc.Ua.Mcp.Diagnostics` | UA-TCP capture, decode, replay | | `OPCFoundation.NetStandard.Opc.Ua.Mcp.PubSub.Diagnostics` | PubSub capture, decode | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Robotics` | Robot Intent control, missions, waits and Vision-guided Pick | +| `OPCFoundation.NetStandard.Opc.Ua.Mcp.Vision` | Vision discovery, seeing, inference, feedback and geometry | ## Documentation diff --git a/tools/Opc.Ua.Mcp/Opc.Ua.Mcp.csproj b/tools/Opc.Ua.Mcp/Opc.Ua.Mcp.csproj index 530be65eb8..ef2dd12910 100644 --- a/tools/Opc.Ua.Mcp/Opc.Ua.Mcp.csproj +++ b/tools/Opc.Ua.Mcp/Opc.Ua.Mcp.csproj @@ -37,6 +37,7 @@ + diff --git a/tools/Opc.Ua.Mcp/Program.cs b/tools/Opc.Ua.Mcp/Program.cs index cd61d81439..b10e9cfa08 100644 --- a/tools/Opc.Ua.Mcp/Program.cs +++ b/tools/Opc.Ua.Mcp/Program.cs @@ -57,9 +57,11 @@ DefaultValueFactory = _ => 5100 }; -var profileOption = new Option("--profile") +var profileOption = new Option("--profile") { - Description = "Tool profile: core, services, administration, pubsub, diagnostics, or full (default)" + Description = "Tool profiles: one profile or a comma-separated list of profiles - " + + "core, services, administration, pubsub, diagnostics, robotics, vision, or full (default). " + + "Compose profiles with ',' or '+', e.g. --profile vision,robotics for a vision-guided agent." }; var rootCommand = new RootCommand("OPC UA MCP Server - Exposes OPC UA Part 4 services as MCP tools") @@ -73,7 +75,14 @@ { string transport = parseResult.GetValue(transportOption)!; int port = parseResult.GetValue(portOption); - McpToolProfile? toolProfile = parseResult.GetValue(profileOption); + string? toolProfile = parseResult.GetValue(profileOption); + + if (!string.IsNullOrWhiteSpace(toolProfile) && + !McpToolProfileSet.TryParse(toolProfile, out _, out string? profileError)) + { + await Console.Error.WriteLineAsync(profileError).ConfigureAwait(false); + return 2; + } if (transport.Equals("stdio", StringComparison.OrdinalIgnoreCase)) { @@ -95,7 +104,7 @@ await Console.Error.WriteLineAsync( return await rootCommand.Parse(args).InvokeAsync().ConfigureAwait(false); -static async Task RunStdioServerAsync(McpToolProfile? toolProfileOverride, CancellationToken ct) +static async Task RunStdioServerAsync(string? toolProfileOverride, CancellationToken ct) { await Console.Error.WriteLineAsync( "Starting MCP server with stdio transport...").ConfigureAwait(false); @@ -115,7 +124,7 @@ await Console.Error.WriteLineAsync( .WithStdioServerTransport(); McpHostBuilder.ConfigureMcpTools( mcpServerBuilder, - OpcUaMcpOptions.ToolProfile, + OpcUaMcpOptions.EffectiveToolProfiles, diagnosticsToolsEnabled); IHost app = builder.Build(); @@ -125,7 +134,7 @@ await Console.Error.WriteLineAsync( static async Task RunHttpServerAsync( int port, - McpToolProfile? toolProfileOverride, + string? toolProfileOverride, CancellationToken ct) { await Console.Error.WriteLineAsync( @@ -146,7 +155,7 @@ await Console.Error.WriteLineAsync( .WithHttpTransport(); McpHostBuilder.ConfigureMcpTools( mcpServerBuilder, - OpcUaMcpOptions.ToolProfile, + OpcUaMcpOptions.EffectiveToolProfiles, diagnosticsToolsEnabled); WebApplication app = builder.Build(); diff --git a/tools/Opc.Ua.OpenUsd.Connector.Viewer/Opc.Ua.OpenUsd.Connector.Viewer.csproj b/tools/Opc.Ua.OpenUsd.Connector.Viewer/Opc.Ua.OpenUsd.Connector.Viewer.csproj index 0122d50418..839fd6f04d 100644 --- a/tools/Opc.Ua.OpenUsd.Connector.Viewer/Opc.Ua.OpenUsd.Connector.Viewer.csproj +++ b/tools/Opc.Ua.OpenUsd.Connector.Viewer/Opc.Ua.OpenUsd.Connector.Viewer.csproj @@ -25,6 +25,9 @@ $(PackageId).Debug + + + diff --git a/tools/Opc.Ua.OpenUsd.Connector/NugetREADME.md b/tools/Opc.Ua.OpenUsd.Connector/NugetREADME.md index 6d65c7d201..38f45fc0f1 100644 --- a/tools/Opc.Ua.OpenUsd.Connector/NugetREADME.md +++ b/tools/Opc.Ua.OpenUsd.Connector/NugetREADME.md @@ -53,9 +53,8 @@ into the server's `SpeedSetpoint` Variable. The stage also advertises a `RootLay **NVIDIA Omniverse** (USD Composer / Kit) for an RTX render with continuous `.live` updates. The `usd-core` PyPI wheel provides the `pxr` Python modules for validation but **not** the `usdview` GUI. -- The base USD asset (`Plant.usda`) and composed stage (`stage.usda`) from the companion - spec repo: `marcschier/opcua-drafts` → - `core-specs/extras/openusd-binding/examples/pumps/`. +- The base USD asset (`Plant.usda`) and composed stage (`stage.usda`) from the OpenUSD + binding pump example assets. ## Run it end-to-end @@ -70,8 +69,8 @@ dotnet build tools/Opc.Ua.OpenUsd.Connector/Opc.Ua.OpenUsd.Connector.csproj -c R ```bash mkdir ~/pump-live -cp /core-specs/extras/openusd-binding/examples/pumps/Plant.usda ~/pump-live/ -cp /core-specs/extras/openusd-binding/examples/pumps/stage.usda ~/pump-live/ +cp /core-specs/extras/openusd-binding/examples/pumps/Plant.usda ~/pump-live/ +cp /core-specs/extras/openusd-binding/examples/pumps/stage.usda ~/pump-live/ ``` `stage.usda` sublayers `live.usda` (stronger) over `Plant.usda`. @@ -184,5 +183,5 @@ PY modified. Live values live in a composed override layer (the equivalent of an Omniverse Nucleus `.live` layer, Part 3). -The generic companion specification and the full step-by-step guide live in -`marcschier/opcua-drafts` under `core-specs/openusd-binding/`. +The generic companion specification defines the binding model; the guide above +shows how the connector consumes the pump example's binding nodes and USD layers. diff --git a/tools/Opc.Ua.OpenUsd.Connector/Opc.Ua.OpenUsd.Connector.csproj b/tools/Opc.Ua.OpenUsd.Connector/Opc.Ua.OpenUsd.Connector.csproj index 2c7e44a59c..ae018b5d19 100644 --- a/tools/Opc.Ua.OpenUsd.Connector/Opc.Ua.OpenUsd.Connector.csproj +++ b/tools/Opc.Ua.OpenUsd.Connector/Opc.Ua.OpenUsd.Connector.csproj @@ -13,6 +13,9 @@ child window, which needs a supportedOS declaration to be granted. --> app.manifest + + +