OPC UA Vision: a vision-guided bin-picking demo an LLM agent can see and drive - #4235
Open
marcschier wants to merge 103 commits into
Open
OPC UA Vision: a vision-guided bin-picking demo an LLM agent can see and drive#4235marcschier wants to merge 103 commits into
marcschier wants to merge 103 commits into
Conversation
Branched from upstream/master. The sample exposes AI models through the OPC UA - AI Model Management and Inference draft, so a client can discover, call and audit them without knowing what is behind them. The riskiest unknown is settled first: the companion model source-generates from its NodeSet across every target framework, and the four types whose base lives in another assembly - ModelRegistryState, ModelPublisherState, ModelResourceState and AiResourceState - generate correctly against Opc.Ua.XRegistry. That is the arrangement Opc.Ua.Robotics already uses for DI and IA, so the sample follows a proven path rather than a hoped-for one. src/Opc.Ua.XRegistry moves from 0.1.0 to 0.3.0 because the AI model requires that version. The two NodeSets are node-identical - 66 nodes, same NodeIds, BrowseNames and classes - so this is a version stamp and nothing else; all three XRegistry projects still build. One IInferenceBackend covers a hosted service and an on-device runtime, because both speak the same REST contract. That is not a convenience of the sample: it is the property clause 8.1 asserts, that where inference runs changes the trust boundary and nothing else. Needing two shapes here would have meant the claim was wrong. The wire format is handled directly rather than through a vendor SDK. Microsoft.Extensions.AI would have forced System.Text.Json from the centrally pinned 10.0.9 to 10.0.10 across the whole repository, and for a sample against a specification whose ApiDialectEnum names this contract by shape, showing the shape is the point. Azure.Identity remains for the workload-identity path, and is why AOT stays off. ICredentialResolver is the only thing that ever holds a secret value. Clause 9.2 forbids exposing credential material through any Attribute and argues it from the address space being browsable, subscribable and historisable - a secret there is not exposed once, it is archived. So CredentialReference carries a name, and resolution happens in one place. The file resolver refuses a reference containing a path separator rather than sanitising it: a reference is configuration this Server controls, so one that could escape the mount is a mistake worth surfacing. Probing reports throttled separately from unreachable. The two look alike from outside and call for opposite responses - failing over a throttled endpoint moves load onto a weaker model for no reason, since it will serve again shortly.
Publishes the AI root, its models and its deployments, and serves the specification's own Methods: Invoke, InvokeAsync, GetCapabilities and BeginTransfer, plus a model source with TestConnection and ListModels. Three findings worth recording, because each was invisible until it failed: The AI NodeSet is not topologically ordered - LearningJobType (i=1005) subtypes AiJobType (i=1006) - and this stack's type table refuses a type whose supertype it has not seen. NodeIds are assigned in declaration order, so a model that gains an abstract base after its first concrete subtype will always look like this. The loader sorts rather than the NodeSet being reordered, which would have renumbered every id after it. Node managers that need the async pipeline derive from AsyncCustomNodeManager and override LoadPredefinedNodesAsync. Overriding the synchronous LoadPredefinedNodes compiles and is never called. A model's own namespace is not enough: every namespace its types reach into has to be passed to the base constructor, or the type table cannot translate the supertype and reports only BadNodeIdInvalid. AddPredefinedNodeAsync is overridden solely to name the node in that message. Diagnosing a model that will not load without it meant bisecting a NodeSet by hand. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1dab1302-19c5-4a9b-a50c-d97d389713aa
…space defects The client browses from the well-known entry point and hard-codes no NodeId, so it exercises the specification rather than this Server. It walks provenance the way an auditing client would: UsesModel to the artefact, then the digest the chain terminates at. Twenty tests, against a test-only fake backend. The fallback assertion was mutation-tested: making RunWithFallbackAsync report the requested model instead of the substituted one fails it, which is the point, since a fallback that answers without saying so is indistinguishable from a healthy primary at every layer above it. Running the client against the Server found three defects that no unit test and no clean build would have caught, all the same shape - a node the Server has and no client can see: Passing a parent to a NodeState constructor and then calling AddChild leaves ReferenceTypeId null, because AddChild only assigns it when the parent actually changes. The node is indexed, readable by NodeId and invisible to Browse. A child materialised by CreateChild is never initialised, so a Method has no InputArguments. It browses, accepts a call, and rejects it with BadTooManyArguments whatever is passed. A member created after AddPredefinedNode is not published at all. Every member a transfer or a job will ever carry is now materialised before the node is indexed, rather than when its value first arrives. ListModelsAsync now asks the endpoint instead of reporting configuration back to the caller. The question is what the source offers; configuration can only answer what this Server was told about. Verified end to end against a live HTTP endpoint: inline inference, chunked transfer, asynchronous job and connection test all round-trip, and every result names the model that produced it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1dab1302-19c5-4a9b-a50c-d97d389713aa
The chart defaults to an on-device runtime on loopback with no credential, because that is the shape that installs and runs without anything else existing. values-cloud.yaml is the hosted shape. It refuses four configurations outright. Each would come up green and describe itself wrongly, which is worse than failing to start because nobody investigates a healthy pod: ApiKey with no credential mounted, a fallback deployment with no fallback endpoint, a fallback pointing at the primary's endpoint, and EgressPermitted=false with a backend that is not on the machine. chart_tests.py renders the chart and asserts what the manifests must and must not say - including that a credential passed inline appears exactly once, inside the Secret, and in no env var or annotation. Every refusal is asserted to fire, because a guardrail that never triggers cannot be told apart from one that does not work. 23 checks. smoke-test.ps1 builds the image, creates a kind cluster, deploys a stub endpoint beside the Server and opens a real OPC UA session from outside. The probes are TCP, which proves the listener accepts connections and not that the Server is serving - so the smoke test drives a session rather than trusting readiness. Verified by running it: image built, chart installed, pod ready, and a session from outside the cluster completed an inline inference, a chunked transfer and an asynchronous job, each naming the model that produced it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1dab1302-19c5-4a9b-a50c-d97d389713aa
The client printed the model's ModelId but not its NodeId, which made the one claim worth checking impossible to check by reading the output: ModelUsed is a NodeId, so without it a reader cannot tell whether a result was attributed to the model that ran or to the one that was asked for. Verified live with the primary endpoint dead and the fallback reachable: primary model ns=2;i=8, fallback model ns=2;i=20, and ModelUsed came back as ns=2;i=20. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1dab1302-19c5-4a9b-a50c-d97d389713aa
… path ExecuteTransferAsync released the lock to run the inference and then wrote the result into the transfer's buffers without rechecking that the transfer was still live. Abort and expiry both remove the entry and dispose those buffers, so a transfer aborted while its inference was in flight threw ObjectDisposedException out of a Method call - which no client can do anything sensible with, and which only happens under a timing a single-threaded test never produces. The completing call now rechecks membership under the same lock that removal takes, and drops the answer if the transfer is gone. Dropping it is what the caller that aborted was asking for. The request payload is also read under the lock now. A client may keep writing to the request file while Execute runs, and an inference is entitled to a payload that does not change underneath it. Abort was never published at all. It was reached with FindChild, which does not create, so the optional Method was silently absent - the same shape of defect as the three in the previous commit, and the reason the new test found it: writing a test that calls Abort was the first thing that required Abort to exist. The fix is mutation-tested: removing the liveness recheck fails the new test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1dab1302-19c5-4a9b-a50c-d97d389713aa
A code review that browsed the Server rather than only reading it found eight defects. All of them build, start and answer calls. Critical - dynamic NodeIds collided with the model's own. The counter issued numeric identifiers in the namespace the NodeSet occupies (i=1001..i=7001), and the predefined-node index takes the last writer, so after a few hundred transfers AiRootType had been replaced by an inference job's FinishedAt property. String identifiers cannot collide with numeric ones at all, which is a stronger guarantee than a seed. The model already declares the entry point at i=7001, parented to the Server Object, and this created a second one. Two Objects named AiModelManagement under the Server, one populated and one empty, with which a client finds decided by browse order. The Jobs folder was created lazily on the first transfer, long after registration - present on the NodeState tree, absent from the index, so a Browse of the root listed it and browsing it returned BadNodeIdUnknown. The collection the specification defines was unreachable. Transfer buffers were reached under two different locks that do not exclude each other, so a Write concurrent with Execute was an unsynchronised MemoryStream access. StreamFileManager now owns them and exposes Snapshot/Replace; Replace also refreshes the published Size, which was fixed at zero and would make a client following Part 5 read an empty response. Jobs were never reclaimed. Any session that could call InvokeAsync could grow the address space without bound, in nodes and in retained payloads. FileType handles were global. Part 5 scopes them to the Session that opened them, and without that any session could reposition or inject bytes into another's in-flight upload - which for an inference payload means altering what a model is asked. Invoke reported Good with a null Transfer when BeginTransfer refused, which is the ordinary outcome once MaxConcurrentTransfers is reached. The fallback deployment published the primary's site, jurisdiction and egress, so pointing the fallback at a cloud endpoint gave a Server that routed payloads off the machine while telling clients OnServer and EgressPermitted=false. It now has its own options. The synthesised fallback model no longer inherits the primary's digest either: a provenance walk would have terminated at weights that never ran. Also: TryGetUInt64 and ValueKind checks on the untrusted response path, where a 200 carrying a negative token count threw FormatException out of a successful inference; OperationCanceledException handling, since HttpClient's own timeout is not HttpRequestException; Helm guardrails extended to every non-anonymous authentication kind and to an empty credential reference; TokenAudience wired into the workload-identity path rather than declared and never read; and the credential checksum annotation no longer digests the value, since Pod annotations are readable far more widely than Secrets. Four new tests cover the entry point, the id collision, the Jobs folder and job reclamation. The collision fix is mutation-tested. 25 tests on each of net8.0, net9.0 and net10.0; 28 chart checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1dab1302-19c5-4a9b-a50c-d97d389713aa
…ations Follow-ups to OPCFoundation#4165 from the compliance review, all of which that PR left open. A robot cell with two arms could not be built. IntentExecution carried no controller identity and there was effectively one IIntentExecutor per server, so an executor handed an execution could not tell which arm it was for. IntentExecution now carries ControllerId and ControllerName, so a single shared executor can discriminate, and DI gained per-controller registration - AddRobotIntentExecutor<T>(controllerBrowseName) and its instance overload - so two genuinely different executors can coexist. Resolution runs WithExecutor, then the per-controller registration, then the single DI executor, then a build-time throw; there is still no silent fallback to a rejecting executor. IIntentExecutor itself is unchanged. Accepts<T> silently discarded a redeclaration, so narrowing a capability after declaring it left the Server publishing the wider one with no signal - the same class of defect as the capability-honesty rules this work already fixed elsewhere. Identical redeclarations stay idempotent and conflicting ones now throw BadInvalidArgument. RequestAuthorityAsync returned a lease the caller had to remember to inspect; forgetting made every later submission fail with ControlNotOwned for reasons invisible at the call site. RequireAuthorityAsync throws instead and names the current owner. RI-Interop-40010 now requires exact set equality between the OPC 40010 TaskProgramName values and the Robot Intent ProgramType.ProgramId values, so Annex B item 3 is enforced rather than assumed. Adds tests for three normative statements that nothing pinned: a running intent surviving the session that submitted it, StopModeEnum.OnPath, and BlockingModeEnum.Single. Each was confirmed able to fail by temporarily breaking the behaviour it covers. FastenIntentDataType.Joint is deliberately not addressed here. A Server without an OPC 40450/40451 model cannot declare that narrowing at all, because capabilities are declared per intent type and the model has no member-level mechanism, so it is a specification gap rather than an implementation one; raised in marcschier/opcua-drafts#60. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
The server-certificate validation callbacks built their OPC UA CertificateCollection by wrapping the X509ChainElement.Certificate instances handed to them by the platform. Those handles are borrowed: the chain owns them and frees them when it is disposed, so wrapping them put two owners on one handle. One certificate instance was left outstanding at process exit as a result. That is what has been failing test-ubuntu-latest-Sessions on master. The suite itself passes - the run reports 777 passed and no failures - and the step then exits 1 because the certificate-leak check that runs after it finds the outstanding instance. It is timing-sensitive rather than deterministic, which is why the macOS leg of the same suite passes. Each callback now copies the raw DER and builds the collection from that, so the collection owns what it disposes and the chain keeps what it owns. The validation decision is unchanged - the same certificates are inspected in the same order - only their lifetime is. Also raises the leak detector's ceiling from 10s to 60s while keeping its quiescence semantics: progress still resets the clock, and a count that has stopped moving is still reported without waiting for the cap. The settled window goes from 0.5s to 5s, which is long enough not to race a loaded Linux agent's transport teardown and still prompt enough to report a real leak quickly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
…follow-ups branch test-ubuntu-latest-Sessions fails on master at 5a6bbcc, the commit this branch is cut from, so it is red here for reasons unrelated to the robot intent work. OPCFoundation#4196 fixes it: the TLS validation callbacks were wrapping borrowed X509ChainElement.Certificate handles, leaving one certificate outstanding at process exit and failing the post-suite leak check even though every test passed. Merged so this PR's checks reflect its own changes. The two commits are disjoint - OPCFoundation#4196 touches only the HTTPS/WebApi transports and the leak detector - so this drops out cleanly once OPCFoundation#4196 lands on master. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
marcschier/openusd-dotnet#12 is fixed in 0.6.0-alpha. A RID-less build no longer produces a silently broken output: it resolves to the host RID and copies that host's native payload, and errors clearly on an unsupported host rather than throwing DllNotFoundException later. The render smoke job was removed when it failed on exactly that bug. It comes back with the three problems that made it a poor citizen fixed: - it stays continue-on-error and out of the required summary check until it has passed unaided on CI, because a newly restored job that has never run green does not belong in a gate; - it asserts that a frame rendered rather than INCREMENTAL_GPU_UPLOAD, which was an upstream implementation detail that would redden our CI whenever the renderer changed how it batches uploads; - it is smaller, since less of the check needs to be inline C# now that the RID-less path works. The API we depend on is unchanged in 0.6.0-alpha - the pick callback, the viewer session's picking backend, camera and render state, and the colour and matrix accessors were all probed against the shipped assemblies rather than assumed. Documentation now states that a RID-less build uses the host payload and that an explicit RID is needed to publish for another platform. The supported RIDs themselves are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
marcschier/opcua-drafts#60 merged, carrying the four defects raised while implementing this. The NodeSet here is byte-identical to the specification's own generated artefact, which is how the two are known not to drift, so it is regenerated rather than edited: 216,777 bytes, matching the merged spec by hash. Existing NodeIds and IntentFailureEnum values 0-19 are unchanged; only NoTransition = 20 is appended. A mission whose branch point resolves to no transition, or to a step id that does not exist, already failed rather than reporting Succeeded. It now reports why, with NoTransition. FastenIntentDataType.Joint is refused with CapabilityNotSupported rather than ParameterInvalid. The merged clause 11.3 makes the distinction: with no OPC 40450/40451 joining model under the controller a non-null Joint is unsupported, whereas with such a model a Joint that does not resolve into it is malformed input. Clause 12.2 settles the discoverability question that prompted the defect - absence of the joining model under the controller is itself the structural statement, so no new declaration mechanism was needed. ActiveMission is now required where MissionsSupported is true. The implementation already satisfied that; the documentation did not say so. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
0.6.0-alpha did fix the packaging: the loader stopped reporting libopenusd_hdsilk.so as missing from the publish directory and instead failed on libOpenGL.so.0, which is a system library a bare ubuntu-latest image does not carry. The native is published and loadable; its dependency was absent. That distinction is invisible in the failure, which presents as a DllNotFoundException naming openusd_hdsilk with the real cause buried in the probe list, so the job now also reports what the publish produced and fails with a clear message if no native arrived. The next failure should explain itself instead of needing forensics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
The diagnostic listing added in the previous commit settled it: the publish produces six OpenUSD natives including libopenusd_hdsilk.so, so 0.6.0-alpha genuinely fixed the packaging and the remaining failure is purely a missing system dependency. libgl1 provides libGL.so.1, not libOpenGL.so.0, which is the name the Storm/Silk natives link against. On Ubuntu that comes from libopengl0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
The same 'build a CertificateCollection from the chain elements, else the leaf' shape was written out in five transports plus the HTTPS listener. Five copies of a decision about which certificates get validated is five places for them to drift, and none of them was reachable from a test without a real TLS handshake, so the shape was asserted nowhere. CertificateValidationHelpers.BuildValidationCertificateCollection is internal to Opc.Ua.Core and shared with Opc.Ua.Client through InternalsVisibleTo rather than widening the public surface. The call sites are not all identical, and the difference is preserved rather than tidied away. Four fall back to the leaf when the platform chain is null or empty. HttpsTransportChannel guards on the chain being non-null without checking that it has elements, so a non-null empty chain yields an empty collection there and only a null chain falls back. Those are not equivalent - an empty collection has nothing to validate and fails closed, whereas a collection holding a trusted leaf may be accepted - so unifying them changes what gets trusted. That is a validation-policy decision and does not belong in a lifetime fix. The helper takes an explicit EmptyChainHandling so each site keeps what it had, and both behaviours are now pinned by tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
The GitHub coverage job passed -RepoRoot '.' to check-coverage.ps1. The merged Cobertura report emits absolute filenames and a <source> root per top-level directory, and the code that puts 'src/' back only runs when the source root starts with the repository root - which a relative '.' never matches. Every changed file therefore normalised to a path with 'src/' stripped, matched nothing git reported, and the gate concluded there were no coverable changed lines and passed vacuously. It has been measuring nothing on every pull request. Azure passes an absolute root, which is the entire reason the two systems disagreed on the same commit. The invocation now passes an absolute root, the script resolves whatever it is given so a relative value cannot silently degrade again, and the vacuous case is no longer silent: a diff that changed no .cs file still passes quietly, but changed files that match nothing in the report is a path-matching failure and now says so, naming an example of each side. That distinction is what would have caught this. With the gate measuring honestly, it reports 71.20% on this branch against a 75% floor, so the uncovered changed lines are covered here: per-controller executor registration and its validation, the WithExecutor precedence chain, RequireAuthorityAsync refusing without an owner, the facet structural failures, capability redeclaration comparison, and the mission no-transition path. IntentControllerHost.cs line 1569 is left uncovered deliberately - it is defensive, and a fasten intent carrying a Joint is refused earlier by ReferencesUnsupportedFastenJoint, so nothing reaches it through the public admission path. Review feedback: A blank TaskProgramName was skipped when comparing the OPC 40010 task control programs against the published ProgramType instances, so RI-Interop-40010 could be claimed while the 40010 side carried an unnamed program. The published side already refused a blank ProgramId, so the two halves of one equality check disagreed about what a blank name meant. A blank name is now a structural failure on both sides. The second comment, that IntentExecution.ControllerId could be null, does not hold: NodeId is a readonly struct, so the parameter cannot be null, and NodeId.Null is a static readonly field left at its default, so an uninitialised NodeId already is NodeId.Null. No production change; the contract is now pinned by tests instead of only documented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Brings in the shared validation-collection helper and its tests, which cover 26 of this branch's uncovered changed lines. Drops out cleanly once OPCFoundation#4196 lands on master. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Folds OPCFoundation#4198 into this branch at the user's request so the OpenUSD dependency rev, the restored render smoke job and the platform documentation ship together with the robot intent follow-ups rather than as a separate pull request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Brings in OPCFoundation#4196 (the TLS certificate lifetime fix and its shared validation-collection helper), which this branch had merged in early so its checks would reflect its own changes. Now that it has landed on master the merge collapses, as intended. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Adds Opc.Ua.Mcp.Robotics, an MCP tool library over the Robot Intent client, so an agent can discover what a controller will accept, watch what it is doing, command it directly, and compile and submit missions. It follows the Opc.Ua.Mcp.PubSub template: a McpToolProfile.Robotics gates registration and contributes nothing when the profile does not name it, so a host can pass one profile to every OPC UA tool package. The layer owns no robot semantics. It translates JSON to the existing fluent builders and results back, and where it would have had to compute something about robot behaviour the client API gained it instead - a live state read, because RobotIntentControllerInfo described only what a controller can do and never what it is doing; controller-level cancel, cancel-all, pause, resume, retry and release, which previously existed only on the transport or a per-operation handle; enumeration of outstanding work, so an agent can see what it did not itself submit; and a bounded wait that reports current state rather than blocking forever. Refusals are the point rather than an afterthought. A NotPermittedInMode or SafetyLimitExceeded is a decision the Server made, so the tools report the specific IntentFailureEnum and the Server's message verbatim, never flatten a refusal into "failed", never retry one, and never acquire command authority as a side effect of something else. Tests assert each of those, since a translation layer is exactly where a refusal gets lost. IntentViewerClient hosts the tools behind --mcp, sharing its one session with the viewer so a human sees what the agent just commanded. Transport is stdio by default and HTTP when --view is on, because MCP stdio carries protocol on stdout and the Avalonia viewer shares that stream; the choice is announced, and asking for stdio with --view is warned about rather than silently overridden. MCP is conditioned on net8.0 and later, so the sample still builds and runs on net48 without it. The bench scene gains the payload it never had: parts in the bin, eight stack slots at the fixture, and a held part bound to the gripper. All of it is server-side simulation published through existing OpenUSD live bindings, so the viewport animates for an agent's commands for the same reason it does for anyone else's - no viewer or connector change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
The branch was merged with master remotely while the same merge was done locally, so both sides carry master independently. This reconciles them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
The static grader flags tool descriptions that leave an agent guessing: D004 for an undescribed parameter, D006 for two descriptions that are near-identical, and C001 for tools whose purpose appears to overlap. Every robotics_* control and monitoring description now states what the tool does, when to reach for it, which sibling tool to use instead when the two could be confused, and what it returns. Parameters that had no description have one. The behavioural guarantees the previous text carried - refusals are reported rather than retried, and command authority is never acquired as a side effect - are kept verbatim in meaning. No tool names, signatures or behaviour change; this is description text only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Moves ModelContextProtocol and ModelContextProtocol.AspNetCore from 1.4.0 to 2.1.0 across all five MCP tool libraries and the opcua-mcp executable. No code changes were required. The surface we use - AddMcpServer, WithStdioServerTransport, WithHttpTransport, MapMcp, McpServerToolType and McpServerTool - is unchanged in 2.x, and 2.1.0 ships net8.0, net9.0, net10.0 and netstandard2.0, so every target framework we build stays covered. The work was in central package management. CentralPackageTransitive- PinningEnabled is true, so a central pin overrides what a transitive dependency asks for and any shortfall surfaces as an NU1109 downgrade error. The 2.x packages pull a new dependency graph, which had to be resolved in a cascade: ModelContextProtocol.Core needed an exact pin of its own, Microsoft.Extensions.Caching.Abstractions is newly required, and Hosting.Abstractions, Configuration.Abstractions and System.Text.Json each had to move to 10.0.10 in turn. Each pin carries a comment saying why it is there. Both transports were verified against a running server rather than assumed: stdio and Streamable HTTP each answer tools/list with the same 114 tools in the full profile. The protocol change worth knowing about is that MCP 2.x is stateless by default - no initialize handshake and no Mcp-Session-Id. An OPC UA session is a different thing: it is application state, held by the process that opened it and addressed by a caller-supplied name, which is the shape the stateless model asks for. What it does assume is process affinity, so that assumption is now written down in the tool README instead of being implied. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Selecting --profile robotics produced 36 tools that could not do anything. Every robotics_* tool resolves a named OPC UA session through RoboticsIntentManager, and only the connection tools can open one, so with no Connect in the catalogue every tool in the profile failed. The host already knew about this pairing. It registers ConnectionTools explicitly for the diagnostics profile, with a comment noting that capturing traffic is only useful next to the connection tools that generate it. The robotics profile needs the same pairing and more strongly, since it cannot function at all without one; it was simply never given it when the profile was added. The profile now reports 40 tools with a working Connect, verified against a running server. A test had pinned the defect in place, asserting that the robotics profile does not contain Connect on the line directly below the diagnostics assertion requiring it. That assertion is corrected, and a new test states the invariant rather than the instance: any profile exposing session-scoped tools must also expose the tools that open a session. It was confirmed to fail without the fix. Also corrects docs/McpServer.md, which still described four MCP libraries when there are five. It was missing the Robotics package row, the robotics profile row, the library in the layout, and the robotics registration in the embedding example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Two problems, both surfaced by CI on the previous push. build-linux-all-tfm failed on the netstandard2.1 leg with NU1201. The Opc.Ua.Mcp.* projects opt into $(RestrictForLegacyTfm), so under a legacy $(CustomTestTarget) they build as empty reference-free net10.0 shells. targets.props validates the netstandard2.1 leg with a net8.0 app, and IntentViewerClient gated its MCP references on the target framework alone, so it tried to reference a net10.0 shell from net8.0. The sample now derives MCP support from both the target framework and whether the MCP projects are actually being built, and the source guards use a dedicated INTENT_VIEWER_MCP constant instead of overloading NET8_0_OR_GREATER, which UsdViewHostLoader.cs still uses for the viewer. The sample already had a no-MCP build shape for net48, so this is the shape it takes in the legacy legs. Verified by reproducing the exact CI error without the change and building clean with it, across every legacy CustomTestTarget and the full solution. The coverage gate then failed honestly at 72.79% against a 75% floor. The single largest gap was RoboticsIntentJson, which translates agent-supplied JSON into Robot Intent structures - untrusted input from a language model, previously with no tests at all. It now has 33 tests covering every intent kind, the common and process-specific fields, mission steps and transitions, and the rejection paths for a malformed document, a wrong root kind, a missing required property and an unknown intent kind. The file goes from uncovered to 329/329 lines. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
The test file was written but never committed. The working copy of this repository has the directory physically named "Tests" while git tracks it as "tests"; git therefore listed the new file as untracked under "Tests/..." and a "git add tests/..." matched nothing and silently did nothing. CI proved it: the Tools job ran 360 tests with no RoboticsIntentJsonTests among them, and the coverage gate reported exactly the same 551/757 as the run before. Staged through the index at the lowercase path so it joins the other 2,267 files under tests/ rather than creating a second, case-divergent directory that a case-sensitive Linux checkout would treat as separate and never compile. RoboticsIntentJson translates agent-supplied JSON into Robot Intent structures - untrusted input from a language model - and had no tests. These 33 cover every intent kind, the common and process-specific fields, mission steps and transitions, and the rejection paths for a malformed document, a non-object root, a missing required property and an unknown intent kind, taking the file to full line coverage locally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Three findings from a review pass over this branch. Two were real; the third was not, and the tests that disprove it are kept as a guard. RoboticsIntentJson parses JSON written by a language model, so every malformed shape it can be handed is reachable input. It indexed position[0..2] and orientation[0..3] straight out of caller-supplied arrays without checking their length, so a pose with a short array threw IndexOutOfRangeException from deep inside ArrayOf<T>, and it called EnumerateArray and GetDouble without checking the element kind, so a string or object where an array belonged threw InvalidOperationException. Neither is caught anywhere in the library, and both contradict the tool descriptions, which promise the agent a ParameterInvalid or argument error. Array-valued properties are now validated for kind, element type and - for a pose - exact arity, and GetUInt no longer lets a negative or oversized number escape as a FormatException. IntentOperationHandle.Merge treated QueuePosition zero as "not observed" and discarded it. That heuristic is right for a pump update, which carries one changed node, but wrong for RefreshAsync, which passes a complete server read. Zero is a meaningful value there: the server publishes it precisely to say the operation has left the queue. A handle therefore kept reporting the last queued position for the rest of the operation's life. Merge now takes fullyObserved, mirroring the flags the other fields already use, so a full read is authoritative and a partial update keeps the heuristic. The pose validation immediately caught a latent defect in the pallet scenario test: it interpolated doubles into JSON with the current culture, so under a comma-decimal culture such as this machine's en-DE "position": [0.12, 0.16, 0.28] became a six-element array. The test passed only because that particular submission is expected to be refused for want of authority, so the garbage coordinates were never used. It now formats invariantly. The third finding claimed RobotIntentControllerState hands out a null NodeId when default-constructed. It does not: NodeId, QualifiedName and LocalizedText are readonly structs here, so default is the null value and IsNull is true. This is the same mistake an earlier automated review made on this PR. No production change, but the three tests asserting the invariant are kept so the claim does not need re-litigating. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
…0-alpha Seven review threads on OPCFoundation#4195. The connection tools are no longer bolted on by the host. McpHostBuilder special-cased the Diagnostics and Robotics profiles and registered ConnectionTools for them itself, which put knowledge of what those packages need in the one place that should not have it. Each package now declares its own requirement in its registration extension, so an application embedding the package gets a usable catalogue without reproducing the host's special case. Full still takes them from the core package, so they are never registered twice - verified against a running server, where the catalogues are unchanged at 114 / 40 / 10 tools with no duplicate names. The name-set tests could not have caught a duplicate because they collapse to a HashSet, hence the live check. OpenUSD moves to 0.7.0-alpha, and the sentence recording which upstream issues the old workarounds tracked is gone now that the workarounds are. The openusd-render-smoke job is removed: the render path belongs to the openusd-dotnet repository and its packages, not to this repository's CI. It was provisional and continue-on-error, and was not part of the required status check, so nothing depended on it. Also drops the explanatory comments from the ModelContextProtocol, Hosting.Abstractions and Caching.Abstractions pins, and the reference to the removed render smoke job in the OpenUSD package comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
…to marcschier/vision-guided-picking # Conflicts: # UA.slnx
Clicking Help -> Shortcuts in the OpenUSD viewer throws inside the viewer package (marcschier/openusd-dotnet#16) and the exception surfaced at the await in Main, killing the whole client. The viewport is documented as optional and the same process serves MCP, so a stray menu click ended an agent's session. The viewport failure is now reported and swallowed, and when --mcp is active the client keeps serving MCP instead of exiting underneath the agent. Also logs when the live OpenUSD stream starts and stops. Without that it is impossible to tell a viewport that never subscribed from one whose subscription was torn down, which is exactly the distinction needed while diagnosing a static viewport. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
A subscription created through the classic Session.AddSubscription API is invisible to the V2 SubscriptionManager's registry, because SubscriptionBridge - the type written to join the two - is never constructed outside tests. The publish worker therefore resolved the first Publish response for that subscription to nothing, classified it as abandoned, and issued DeleteSubscriptions against the server. The result was a subscription that the server had destroyed and the client still reported as Created, publishing, with its monitored items attached. One notification arrived - the initial data change - and nothing ever again. There was no error to react to, so a consumer just went quiet: an OpenUSD viewport following a robot rendered the pose it happened to fetch at start-up and then froze, while the robot kept moving. The worker now asks the session whether it owns the identifier before treating it as abandoned, and keeps it if so. Genuinely orphaned identifiers are still cleaned up. This is a containment fix. Classic subscriptions on a V2 session still receive no notifications, because nothing wires SubscriptionBridge; that wiring is the remaining work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Follow-up to 42a2283, which stopped the publish worker deleting classic subscriptions but still dropped their notifications on the floor. Retaining a subscription that never delivers anything is only half a fix: the twin stayed frozen, just without the server-side deletion. The worker now hands the response to the session when the identifier belongs to a subscription the session holds outside the manager's registry. The session caches the message and raises the notification exactly as the classic engine does, so monitored item callbacks fire and the consumer sees live data. An identifier nothing claims is still treated as an orphan and deleted. Verified against the bin-picking cell: an OpenUSD viewport driven through this path now follows the robot as the arm moves, where before it rendered the start-up pose and never changed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
AddRobotIntentExecutor<TExecutor>() registered the interface against the type, so the container built one executor for IIntentExecutor and, whenever the application also injected TExecutor directly, a second one for that. The Robot Intent server drove instance A; the application observed instance B. In the bin-picking cell that meant intents were admitted, executed and reported Succeeded on an arm whose joint angles nobody published, while the cell mirrored a second arm that never moved. Its OpenUSD viewport therefore rendered a single pose - the values read once at subscribe time - no matter what the robot was told to do, which looked like a broken binding rather than two arms. The concrete type is now the registration and the interface resolves from it, matching what the per-controller overload already did. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
Four defects the end-to-end run surfaced. The eye-in-hand camera was pinned. Cell.usda reset the transform stack and placed the camera at a fixed world point, so it stayed put while the arm moved - the opposite of what the same file documents, and of what a sensor bolted to the flange does. It is now a plain child of the flange, turned onto the tool axis. The viewport no longer opens on it either: that camera shows what the tool sees, which is not what someone watching the cell wants. Camera frames could not be read. A 1280x1024 RGB frame is just under 4 MB, which is the default ByteString ceiling on the server and four times the default on the client, so GetClip refused the cell's own output with BadEncodingLimitsExceeded. Both ends now allow the frames this cell serves. Axis limits were published in radians while positions were published in degrees, telling a client the axis spanned +/-6.28 while it reported values up to +/-360. Limits and MaxSpeed are now degrees, like Position. Command authority was never released. IntentControllerHost.AttachSessionManager existed, and its own comment warns that without it "a crashed client locks the robot for good" - but nothing ever called it. Controllers now subscribe to Session lifetime, and the sample asks for a fifteen second Session timeout so a killed client stops blocking the next one for a minute. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
The ByteString bump alone was not enough: the response that carries a frame is larger than the frame, so the encoder still refused it. Raise MaxMessageSize too. GetClip is still refused, so the remaining limit is elsewhere. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a248a589-9a20-4372-868e-5d347e57001b
…/marcschier/UA-.NETStandard into marcschier/vision-guided-picking
…e stub The new ISubscriptionManagerContext member was added with the production adapter and the client-test fake updated, but StubSubscriptionManagerContext in Opc.Ua.Subscriptions.Tests was missed, so every TFM failed to compile with CS0535 and took the build, CodeQL, the Subscriptions tests and the coverage gate down with it. The stub holds no session-owned subscriptions, so false is the truthful answer rather than a placeholder: every identifier falls through to the manager's own handling, which is what these quiescent tests assert. It stays non-throwing like the other dispatch members on this stub. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
The sample does not build: two CS8604 nullable errors, and with TreatWarningsAsErrors that is a hard failure, so neither the cell server nor the end-to-end demo could be run at all. CI never caught it because CI never compiles this project. BinPickingCell pins RuntimeIdentifier=win-x64 and sets RestrictForLegacyTfm=true, so the Windows leg (net472/net48/ netstandard2.0) reduces it to the empty no-op shell and the Linux leg builds the modern TFMs without it. The project is in UA.slnx and looks covered, but nothing type-checks its sources. VisionDetectionDataType.ClassLabel is nullable and both log methods take a non-nullable string. The two sites are fixed differently because they know different things: - BinPickingOffServerProof builds its own detection with ClassLabel = RedCubeClass, so it passes that const directly. Provably non-null and clearer about intent than re-reading what it just wrote. - BinPickingInferenceProof logs detections published by the provider, where an absent label is possible, so it falls back the way the rest of the sample already does for nullable strings. Verified by running the demo end to end: cell server, scripted pick/place, MCP over HTTP (62 tools) and the OpenUSD viewport. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
Four defects found by running the demo end to end with the viewer.
The viewport opened after the demo finished. Program.cs ran the scripted
loop to completion and only then opened the viewport, so the robot had
already parked by the time there was anything to watch it in - the
opposite of what the README promises ("run the scripted pick-and-place
while the viewport is open"). The two are now sequenced: the viewport
opens, the client waits for the live OpenUSD stream to be subscribed,
and only then commands the robot. The window stays open afterwards so
the cell can still be inspected.
The bench top was 29 mm too low. Its slab reached z = 0.800 while the
OPC UA model puts robot_base and the Bin and Fixture locations at
z = 0.829, so everything the model placed on the bench floated above it.
The underside stays at 0.760 where the legs meet, so the correction is
carried by the thickness.
The address space reported joint positions the arm was never in.
ConfigureAxis set Position = 0 for all six axes, while the simulator
starts at its home configuration. Nothing corrected it until the first
motion, so a client - and the OpenUSD live binding, which renders from
these very nodes - saw a fabricated pose. The axes are now seeded from
the simulator's own snapshot.
The home configuration itself put the elbow through the bench. It was a
floor-standing pose: on a 0.829 m bench the elbow lands at z = 0.646,
183 mm under the table, which is the forearm the viewport was drawing
through the worktop. Replaced with the scan pose these cells already
document - it places the flange exactly on the Vision model's authored
flange transform (0.411, -0.062, 0.636 from robot_base), over the bin,
with 162 mm of clearance under the lowest joint. Cell.usda's authored
overrides carry the same pose in degrees, because the live binding
drives those ops from AxisState.Position and a still render that
disagrees with the first live update is just a different bug.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
…n quaternion The viewport never animated because nothing ever moved. ExecutePickAsync was a delay followed by ExecuteGraspAsync, and ExecutePlaceAsync a delay followed by ExecuteReleaseAsync: Pick closed the gripper, Place opened it, and no joint changed in between. The address space had nothing to report, so the OpenUSD live binding - which was correctly subscribed to all six axes, the server confirms ItemCount=6 - published one message across an entire pick-and-place and the viewport had nothing to draw. A Location arrives as a NodeId and the executor has no address space to resolve it with, so the host now supplies a resolver and the executor travels there before actuating the gripper. BinPickingCell resolves its own Bin and Fixture, converting from the world frame the Locations are authored in to the base frame the kinematics use, and lifting the target to an approach height so it is neither at bench level nor inside it. The travel solves inverse kinematics once and interpolates in joint space. Interpolating in Cartesian space re-solves at every step and abandons the move the first time the straight line between here and there passes through a pose the arm cannot hold, which for a reach across a bench it usually does - the first attempt did exactly that and still produced no motion. It is also best effort: no resolver, or a location out of reach, still actuates the gripper, because refusing a pick over unreachable scenery is a worse answer than picking in place. Separately, the flange scan pose and the hand-eye transform were written as 0.7071, leaving a quaternion norm of 0.99999041 - 9.6e-6 off unit, ten times the validator's 1e-6 tolerance. Composing a detection through those frames threw BadOutOfRange and killed the client. It went unseen because the default part is RedCube, which the startup proof has already removed, so the demo skipped the compose step; picking any part still in the bin hit it every time. Verified by capturing the viewport with PrintWindow during a run: 29 consecutive frames change while the arm travels, ramping and settling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
romanett
approved these changes
Aug 15, 2026
The bin-picking loop reported success while nothing in the world changed. Pick and Place travelled and actuated the gripper, but the cell's world model was only ever mutated by the startup proof, with RedCube hard-coded, so the ground-truth detector kept reporting a part the robot had just carried away and the demo printed its own confession that the executor does not update the world state. Three things were missing. The arm knew something was held but not what. SimulatedArmSnapshot carried HasObject and HeldPartPosition; it now carries HeldObjectClass too, set from the Pick intent's ObjectClass when the grasp closes and cleared on release, so a host can move the right item. The intent never said what to pick. PickIntentDataType has had ObjectClass all along - "what to pick, when the location can hold more than one kind" - and the bin holds five, which is exactly the case the field exists for. The Pick builder takes it as an optional argument and the demo passes the part it selected. Nothing joined the two. BinPickingRobotCell now tracks the carried part on every snapshot: while it is held it is marked at the tool position so it travels with the gripper rather than teleporting when the grasp opens, and on release it is left where the tool let go. The arm reports in its base frame and the world model works in the world frame, so the base height is added back. The demo's verification was also unfalsifiable. It reported a pass when the target was absent afterwards, including when the target had never been there - which was the default case, because the startup proof picked RedCube before any client connected and never put it back. The proof now restores the bin, an absent target is reported as inconclusive rather than as a pass, and the remaining failure message no longer explains away a mismatch that is now a real defect. Verified end to end: five parts detected, Pick and Place succeed, and the detector then reports four, without the part that was picked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
The robot moved the part in the cell's world model, but nothing could see it: only the six axes had OpenUSD live bindings, so a part that had been carried to the fixture still rendered in the bin. Each part now has a world position variable under a WorldState folder, published as this cell's simulation ground truth rather than as a standard OPC UA concept - a part lying in a bin is not something Robot Intent or Vision model, and the scene has to be drivable from the address space to be watchable. The cell pushes the carried position onto that variable on every snapshot, and one live binding per part follows it into the stage. The part prims had to be re-authored. The viewer sink writes a driven prim's pose as a single xformOp:transform matrix, and says why: authoring through SetLocalTransform rewrites xformOpOrder, which fails when a weaker layer already declares one. The parts declared translate/rotateZ/scale, so a translate binding against them would have been silently dropped. Each part is now an Xform carrying only that matrix, with its shape as a child holding the rotation and scale the binding must not overwrite. The composition and the authored positions are unchanged, which matters because BinPickingParts.InitialWorldPosition mirrors them exactly and the ground-truth detector projects from those numbers. Two ordering traps, both found by the server refusing to start: - The Server object belongs to the core node manager, so the WorldState folder cannot name i=2253 as its parent. It is added parentless and the forward reference is added afterwards, as the OpenUSD root already did. - The folder must not be registered until its representations are attached, or the position variables are registered twice, and its children need SymbolicName set - without it AssignInstanceChildNodeIds regenerates every id from an empty string and all five parts collide on one NodeId. Verified at the data level rather than by eye: the connector now creates 11 monitored items rather than 6 (six axes plus five parts), and its subscription delivered SeqNo=151, MessageCount=10 across a run that previously delivered SeqNo=3, MessageCount=1. Detections are byte-identical to before the re-authoring, and the scripted loop still passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
vision_list_sensors returned nothing on a server whose cameras were plainly there - vision_get_frame answered for the same sensor with a reason rather than an unknown-node error, and vision_list_pipelines listed its pipeline. An agent could reach the perception bound to a camera but never the camera, which is the node every frame and calibration tool takes as input. Sensors resolved the Sensors folder through the well-known NodeId in the Vision namespace, while Pipelines and Frames resolved theirs by browse path from the Vision root. The well-known identifier only holds for a server that materialises the Vision tree in the Vision namespace itself; one that builds it as instances in its own namespace - which is what the fluent builder produces, and what the bin-picking cell is - has a Sensors folder with its own NodeId, so the lookup landed on nothing and every sensor was invisible. GetSensorsFolderIdAsync now resolves by browse path like its siblings and falls back to the well-known identifier, so both shapes of server work. The existing SensorsFolderId property is unchanged. Verified against the cell: vision_list_sensors now returns BinPickingCameraTwin at ns=5;s=7001_Sensors_BinPickingCameraTwin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
…at the sibling arm The READMEs described a demo that no longer exists. The client README still carried a "known concurrent-agent-side wiring issue" note saying the client could not see the cell's pipeline, which was fixed by resolving Vision nodes by browse path; it promised the scripted loop ran while the viewport was open, which it now genuinely does; and it said nothing about the parts moving, because until now they did not. The cell README documented two proof services that mutate the world and a robot that does not, which is the wrong way round. Pick now travels to its Source, closes on the part named by ObjectClass and carries it, and the detector projects from the same positions, so the client's verification is real rather than a formality. The on-server proof also restores the bin, which is worth stating: it runs before any client connects, and leaving a part picked meant the demo started against a world it had not changed. Server/WorldState is documented as what it is - this cell's simulation ground truth, not a standard OPC UA concept, published because the scene has to be drivable from the address space to be watchable. Also seats the sibling sample's arm on its bench. IntentEnabledRobot references the arm at z = 0.820 on a bench whose top surface was 0.800, so it floated 20 mm - the same defect already fixed in the bin-picking cell, in the scene that shares the same arm asset. The underside stays at 0.760 where the legs meet, so the thickness carries the correction. Verified end to end on a third part: five detections, Pick and Place succeed, and OrangeBrick is gone from the detections afterwards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
vision_run_inference answered with the ResultId the Server assigned, and vision_read_detection_result requires a NodeId, so the sequence the client README documents - run a pass, then read what it produced - dead-ended. The only way through was to guess the Server's NodeId convention, and a wrong guess is worse than a failure: it resolves to some other node and reports an empty result rather than saying it looked in the wrong place. The two are one browse apart. A Server publishes each result under the pipeline's Results folder with the ResultId as its BrowseName, so VisionPipelineClient.ResolveResultNodeIdAsync turns one into the other, and the tool returns both along with a resolved flag for the case where a Server answers with a ResultId but publishes nothing addressable. The Part 4 method still returns what the specification says it returns; this is discovery layered over it, not a change to the method. The reads themselves were never broken. BinPickingGroundTruthInferenceProvider populates ResultId, CreationTime, Sensor, Pipeline, Detections and FrameId on every result it publishes, and assigns the NodeId from the node-id factory - which is exactly why a guessed identifier read nothing. Not yet verifiable end to end: RunInference currently fails against the bin-picking cell with BadMethodInvalid. That is pre-existing rather than a regression from this change - an MCP binary built before any of this work fails identically against the same cell, while reads such as vision_read_pipeline succeed - so it is method-node resolution on the typed proxy and wants its own investigation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
The ResultId-to-NodeId walk added in the previous commit resolved the Results folder by browse path, which was never exercised. The paired client already had a private resolver that enumerates the folder, and that one is proven - it is the path the scripted demo has been running all along. The shared method now uses it and the sample's copy is gone. The copy also carried a fallback worth losing: when no BrowseName matched it returned the most recently published result. That answers confidently with the wrong result, which for a tool an agent uses to decide what to grasp is worse than answering "not found". A Server may legitimately prefix the ResultId to keep BrowseNames unique in the folder, so that case is matched explicitly on the ResultId instead, and anything else resolves to null. Verified end to end: the demo runs inference, resolves the ResultId, reads the detections, and reports YellowSlab gone after the Pick - which also shows RunInference itself works on this path, narrowing the BadMethodInvalid seen over MCP to the MCP accessor rather than the pipeline or the method. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
vision_get_frame - the one tool that returns pixels, and the reason the sample claims a language model can look at the bin - failed on every call, and would have returned a corrupted image if it had not. Two independent faults, one behind the other. The Server could not encode its own camera output. GetClipAsync put the whole encoded frame into VisionImageReferenceDataType.Uri as a base64 data URI. Uri is a reference, not a container, and the bytes were already being returned separately in the inline ByteString, so every response carried the image twice - and a 1,311,400 byte PNG becomes a 1,748,536 character string against a MaxStringLength of 65,535, which is 26.7x over. The result was BadEncodingLimitsExceeded on a frame the renderer had produced successfully. It emits a reference now, the way the sibling ground-truth provider already did. This is also why raising MaxByteStringLength, MaxArrayLength and MaxMessageSize in Program.cs never helped: none of them govern a String. With that cleared the image arrived and was unusable. ImageContentBlock.Data was assigned raw bytes, but the protocol requires base64 there: every byte that was not valid UTF-8 serialised as U+FFFD, destroying it beyond recovery, and the response bloated to 6.8 MB for a 1.3 MB PNG because each byte became a \uXXXX escape. ImageContentBlock.FromBytes encodes properly. Verified on the wire: data now begins iVBORw0KGgo, decodes to exactly 1,311,400 bytes, and the response is 1.8 MB rather than 6.8 MB. The decoded PNG opens as a real 640x512 rendering of the cell. The frame shows the cell but not the bin - the eye-in-hand camera is aimed wrongly, because the home pose was solved for flange position while keeping whatever orientation the arm happened to hold, and orientation is what aims a camera. That is the next commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
Three things stood between an agent and a picture it could reason about. The camera pointed at nothing. The arm's home pose, the flange scan pose the Vision model declared and the joint angles authored into the USD stage were three independent claims about where the camera was, and they disagreed - the declared flange orientation aimed the view 86 degrees off the projection camera's. The home pose is now solved so the Camera prim lands exactly at the (0.38, 0, 1.35) the model declares for camera_eih, looking straight down, 1.7 degrees off the parts centroid at 0.50 m, with every joint clearing the bench. The same solution supplies the flange pose and the USD joint overrides, so the three cannot drift apart again. The camera also photographed its own gripper. The tool extends along the flange X axis, which is straight down at the scan pose, so the prim moves 0.16 m out along flange Z to clear it. The frame and the detections were different images. The sensor declared 2448x2048, the clip endpoint 1280x1024, and the renderer produced 640x512 - not even the same aspect ratio. The detector projects through the declared intrinsics, so its boxes lived in 2448x2048 space while an agent was handed a 640x512 picture: "pick the red cube you can see" came with coordinates that pointed off the image. Everything now agrees at 612x512, an exact 4x4 bin of the native sensor, with the intrinsics scaled to match; the native size stays in the model and serial number, where it describes the hardware rather than the image. LatestClip was created and then never written, so it reported Bad_NoDataAvailable for the life of the Server and a consumer following the model - read the published frame, call the method only if there is none - never got a frame. The dispatcher now publishes a clip it has just encoded, for every Vision server rather than this sample alone. Also adds OpcUaServerOptions.MaxStringLength. MaxByteStringLength, MaxArrayLength and MaxMessageSize were all settable and none of them govern a String, so a server needing a long one could not raise the limit through the hosting API at all. It is not the fix for anything here - the frame bug was a design bug - but the missing knob was real. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
The previous commit aimed the camera at the bin but the picture came back
showing the arm's own upper arm. Two separate faults, both structural
rather than numerical.
The elbow was in the way. Reaching the declared camera pose has several
IK branches, and the ones the solver kept finding park a link directly
under the camera - the elbow ended up 15 mm off the optical axis and
140 mm below it, filling the frame. Adding the camera's real field of
view to the solve, and rejecting any branch that puts geometry inside the
cone between camera and bin, picks the elbow-back branch, which comes in
from behind the base and stays above the camera the whole way.
The wrist was singular. Aiming a straight-down camera from a point on the
base's own X-Z plane lines J4 up with J6, and that is not a cosmetic
detail: the first IK solve of any motion away from home fails, so every
intent returns Kinematics. Opc.Ua.Robotics.Tests caught it -
ForceIntentWithoutContactFailsObjectNotFound got Kinematics instead - and
it was a regression I had just introduced; the two poses before this
session sat 24 and 43 degrees clear of the singularity.
The fix is to tilt the camera roll, which is free: the camera still looks
straight down at the bin, and the image and the detections roll together
so they still correspond. 15 degrees of roll puts the wrist 25 degrees
clear, matching the historical margin, and the camera still lands within
15 um of the declared (0.38, 0, 1.35).
Verified end to end rather than asserted:
- the rendered frame shows the bin and all five parts in their own
colours - red (212,47,47), green (61,193,71), blue (54,85,218),
yellow (215,196,58), orange (209,119,55) - where before this it was
two shades of grey;
- over MCP, vision_get_frame returns a 612x512 PNG that decodes from
base64 to 1,254,051 bytes, the descriptor beside the detections says
612x512, and all five BoundingBox2D centres land inside the frame on
the pixel colour their own detection claims;
- LatestClip and LatestClipMetadata carry the frame and its reference
after a GetClip;
- Pick and Place both succeed and the picked part leaves the bin;
- Opc.Ua.Robotics.Tests 541/541, Opc.Ua.Vision.Tests 445/445,
Opc.Ua.Server.Tests 4094/4094.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
The dispatcher publishing a clip it has just encoded was verified end to end but had no unit test, which is the wrong way round for behaviour every Vision server now inherits. Three tests: a successful inline GetClip publishes the bytes on LatestClip and the descriptor on LatestClipMetadata; a payload that overflows the endpoint's inline limit publishes nothing, because a clip the Server refused to deliver is not the latest one; and a Server with inline delivery disabled leaves the inline channel alone. Checked against the unfixed code rather than assumed - with the PublishLatestClip call removed the first test fails and the other two still pass, which is what tells them apart from assertions that would hold either way. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
The viewport framed the scene from wherever the viewer's own bounds fit put it, which is a view that changes with the scene and is not the one you want to watch the cell from. The stage now carries /World/ObserverCamera - front, slightly above, bench centred, fixture left, bin right, with headroom for the arm - and the client opens on it by default. --camera auto restores the old behaviour and --camera <primPath> opens on any other camera. The stage's other camera is the eye-in-hand sensor on the flange. It stays out of the default deliberately: opening on it shows what the tool sees, not the cell. The camera numbers are measured, not derived, and the comment in Cell.usda says so. Fitting the known scene dimensions to a reference framing gave a placement that rendered offset and over-scaled, so the final values come from correcting against successive captures. Two of those iterations were wasted on a fault in the capture harness rather than the camera: resizing the viewer window right after it appears leaves the renderer drawing nothing, which looks exactly like a camera pointing away from the scene. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
Three faults, each of which on its own made the cell look unphysical. A placed part was left wherever the tool centre point happened to be. A Place travelled to the location's approach height and opened the gripper, so the part stayed there: measured at z = 0.994 against a 0.829 bench, 165 mm in the air. There is now a resting model - what is the highest solid under this footprint - and a released part settles onto the bench, the fixture plate, a locating peg or another part. Stacking falls out of it: place a second part on the same spot and its base lands exactly on the first one's top. Measured over three parts on the fixture: bases 0.8380 / 0.8780 / 0.9080 against tops 0.8780 / 0.9080 / 0.9320, so no gaps and no intersections. A Place also no longer drops the part from the approach height. The cell knows it is carrying something - a Pick travels empty and closes, a Place travels loaded and opens - so it descends to the height that leaves the part on its support and the release is a release rather than a fall. The parts never moved in the viewport, however faithfully the server tracked them. The cell published each part's world position as a double[3]; the OpenUSD companion specification defines a translation source as a structured 3D coordinate and the connector's translation profile fails closed on anything else, so every part's target was left unresolved with no error raised anywhere. Publishing ThreeDCartesianCoordinates fixes it. Both halves of that rule are now pinned by tests, because a fail-closed profile that silently does nothing is exactly the kind of requirement that gets rediscovered the hard way - and it is what made this look finished before: the evidence recorded was subscription counters, which were flowing the whole time. Nothing stopped the arm reaching through its own bench either. The IK selector took the nearest solution regardless, and several solutions for a target near the surface pass a link through it. The solver now takes the nearest solution that clears the work surface and refuses when none does, rather than miming a move it cannot make. Also corrected: the Bin and Fixture locations were at (0.41, -0.28) and (0.48, 0.26) while Cell.usda puts the bin at (0.38, 0) and the fixture at (-0.32, 0). "Place it on the fixture" put the part down on bare bench a long way from the fixture, and the model disagreed with the render about where the cell's own furniture was. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
Two things stopped an MCP agent doing what the sample is for: pick parts from the camera and stack them. The Pick intent JSON had no objectClass. The typed builder has always taken one - it is what names the workpiece being grasped - but the MCP surface did not expose it, so an agent could only ask the robot to pick "something". A cell that tracks what it is holding then has nothing to track: the intent was accepted, the arm moved, and no part went anywhere. A local client could do it and an agent could not, which is the wrong way round for a sample whose point is the agent. The client took an exclusive command lease just to open a window. A viewer is an observer, and holding command authority to watch means every intent from an agent on its own session is refused while the viewport is open - the exact configuration the demo is meant to show. Authority is now requested only when this process is going to command the cell: the scripted demo, or an MCP host driving through this session. Verified by driving the whole thing over MCP as an agent would - look through the camera, take authority, pick and place three times - and then checking the result is a real stack: bases 0.8380 / 0.8780 / 0.9080 against tops 0.8780 / 0.9080 / 0.9320, nothing floating and nothing intersecting. Opc.Ua.Robotics.Tests 550/550, Opc.Ua.OpenUsd.Tests 916/916. Known and not fixed here: a viewer-only session renders a static scene while another client drives the cell. The arm does not move either, so it is the live stream in that configuration rather than anything to do with the parts - a single process that both views and commands updates correctly. Recorded rather than hidden. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
Chasing why placed parts did not move in the viewport was slow because the connector is silent: it logs nothing about what it bound, and a target it cannot resolve is dropped without a word. A prim that never moves then looks exactly like a prim nothing is trying to move, and every subscription counter agrees the data is flowing. It now reports what it bound at start-up, one line per applied update, and a warning naming any target it had to leave unresolved and why. The bin-picking client also has to thread telemetry into the connector - it was passing none, so every message went to NullLogger - and to install a console provider, which it did not have either. --verbose raises the level to Debug. Getting that right needed two goes. The first attempt logged applied updates at Debug while the level sat at Information, so the instrument read zero in a configuration I could see working, and an explicit AddFilter was needed because the console provider filters independently of SetMinimumLevel. Worth recording, because a diagnostic that quietly reads zero is worse than none. What it then established: a viewer-only session - one client watching while another drives - binds 11 bindings and monitors 11 items, exactly as a working one does, and receives zero live updates where the commanding session receives about 1,900 over the same sequence. So the subscription is created and the notifications never arrive; it is not specific to the parts, and the arm does not move either. Not root-caused, documented as a known limitation with the numbers rather than left for someone to rediscover. It predates this branch's changes - the same configuration failed the same way before the command-authority change. Opc.Ua.Robotics.Tests 550/550, Opc.Ua.OpenUsd.Tests 916/916. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
The scripted demo did one pick and one place, so watching the cell actually stack anything meant running it repeatedly - and since the viewport only follows the process that drives it, running it repeatedly restarted the window each time and there was nothing to watch. --stack-all picks every part the detector reported and places them all on the destination. The order is the order the camera reported them, so the stack is built out of what was actually seen rather than a hard-coded list, and each cycle waits for its intent to reach a terminal state before the next is submitted: overlapping them queues the next Pick while the arm is still carrying the last part and the parts end up wherever the arm happened to be. The Pick submission now takes the object class as an argument rather than always reading it from the options, which is what lets one run pick five different parts. Measured after a run: all five parts on the fixture, bases 0.8380 / 0.8780 / 0.9080 / 0.9560 / 0.9740 against tops 0.8780 / 0.9080 / 0.9560 / 0.9740 / 0.9980 - a 160 mm stack with no gaps and no intersections, and nothing left in the bin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 99d1ab42-f83a-41ab-bf3e-85bf81fadace
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements OPC UA — Vision and uses it to build the demo the companion specifications were drafted for: an eye-in-hand camera watches a bin of parts, a language model looks at the frame, decides which part matches the instruction, and commands the robot through Robot Intent — with the OpenUSD viewer showing the arm move and the bin empty out.
What is here
The specification, implemented.
Opc.Ua.Visionis generated from the spec's ownOpc.Ua.Vision.NodeSet2.xml, vendored byte-identically (SHA-25679C592C6…, verified against a fresh download and stored LF-preserved like the Robot Intent NodeSet).Opc.Ua.Vision.Serverbuilds the §4.2 discovery object, sensors, the coordinate-frame tree, calibrations, media endpoints, inference pipelines and the §9 feedback methods, with rendering and inference behind injectable providers.Opc.Ua.Vision.Clientis built on the generated ObjectType proxies and composes transforms across the frame tree, which is what turns a detection in camera coordinates into a pose a robot can act on.Two perception paths behind one contract, which is exactly what
InferenceLocationEnumis for.OnServeris a deterministic detector derived from the stage's ground truth — no model, no network, no GPU, so CI can exercise the whole loop.EdgeOffServeris the agent: it sees the frame over MCP and callsSubmitDetections, and the Server publishes results it did not compute. A client readsDetectionResultTypeidentically either way.The agent's eyes.
tools/Opc.Ua.Mcp.Visionadds 22vision_*tools, which with the four shared connection tools makes thevisionprofile 26;vision_get_framereturns the camera image as an MCPImageContentBlock, so the model looks at pixels rather than reading a description of them. Tool profiles now compose —--profile vision,roboticsyields 62 tools — so one agent can both see and act without dragging in the full 136-tool catalogue.The cell.
samples/Robotics/BinPickingCellfollows the spec's own Robotics-Vision Addendum: theworld → robot_base → flange → gripper_tcpframe tree withcamera_eihon the flange, theHandEyeextrinsics and theIntrinsics612x512intrinsics — the addendum's 2448×2048 calibration scaled to the 4×4-binned grid the camera actually delivers.samples/Robotics/BinPickingClienthosts the composed MCP catalogue and optionally the viewer.Documentation.
docs/Vision.md(~800 lines) covers the model, hosting, the build context, every topology builder, both perception paths, feedback, the MCP profile and the limitations; linked fromdocs/README.md, with the profile table indocs/McpServer.mdand READMEs for both samples.Evidence, not assertions
Each of these was measured on a running system rather than assumed:
(212,47,47), green(61,193,71), blue(54,85,218), yellow(215,196,58), orange(209,119,55)vision_get_framereturns a 612×512 PNG that decodes from base64 to 1,254,051 bytes, the descriptor beside the detections says 612×512, and all fiveBoundingBox2Dcentres land inside the frame on the pixel colour their own detection claimscamera_eih → worldlands on the authored position, residual 0.0000 m — on both the on-server and off-server pathsSucceededvision,robotics62, full 136, zero duplicatesGoodThat tool-count check matters because the profile tests collapse names into a
HashSetand cannot see a duplicate registration — so the new tests count registrations, not names.Things that were wrong, and are not now
Writing the tests found three defects of one kind — code that answered instead of refusing. A zero-norm quaternion was silently rewritten to identity, so a caller with an all-zero orientation got a confident wrong grasp pose.
SubmitCorrectionrewrote a nullResultIdto empty and called the sink anyway. A badStageIdentifierkilled the host process inside nativeUsdStage.Open, via an exception .NET cannot catch. All three now refuse loudly and are pinned by tests.Nodes that existed but could not be reached. The node manager indexes the Vision root before configurators run, so everything the fluent builder grafted on afterwards — both folders, the pipelines with their Results and Feedback, the frames, the sensors with their optics, calibrations and media endpoints — was never added to the index. Browsing forward from a parent worked, because that walks
NodeState.Childrenin memory; browsing any of those nodes by its own NodeId returnedBadNodeIdUnknown, which is how an ordinary client and the MCP discovery tools navigate. Registration is now deferred and flushed after each configurator. The first fix covered only the DI path and left the documented non-DI fallback broken in exactly the same way, soConfigureVisionAsyncnow exists and registers what it builds.That defect survived a green suite because every test browsed children. The new tests assert by NodeId, and were checked against the broken code — with the flush removed, they fail.
Every Method was uncallable. The Server passed 426 unit tests while not one of its Methods could be invoked over a real session —
RunInferencereturnedBad_TooManyArguments, and so did every other Method that takes arguments, which is all of them butStartContinuousandStop. Three independent faults, each sufficient on its own: the Method nodes were never created (they are Optional children, and the dispatcher guards every attachment with a null check, so it silently attached nothing); nothing declared the arguments, so the stack concluded none were expected; andMethodDeclarationIdpointed at a synthesised NodeId rather than the NodeSet declaration a client calls with, so the lookup missed the instance Method entirely. The builder now creates the Methods, theResultsfolder and theFeedbackobject when a provider or sink is configured, declares every signature, and names the right declaration.Unit tests could not have caught this: they invoke the handler delegates directly and never go through
MethodState.Call. It took an end-to-end test over a real session. Relatedly, children created by the generatedCreateOrReplacehelpers carry noReferenceTypeId, so nothing could reach them — fixed once in the build context rather than at the several dozen call sites.And one where I was wrong. Earlier in this branch I reversed my own agents' decision and made
SubmitDetectionsaccept an empty detection set, reasoning that "I looked and the bin is empty" is a correct observation and refusing it forces a correct agent to invent a detection. The reasoning is sound; it is also an argument against the specification rather than an implementation of it. §9.5 is normative and explicit — its StatusCode table listsBad_InvalidArgumentfor "Detectionsempty; orSubmitCorrectionsupplies both or neither corrected array".Worse, none of §9.5's argument rules were enforced anywhere — the dispatcher forwarded every submission straight to the feedback sink, so conformance depended on whichever sink a host installed. They are Server obligations, so they now live in the dispatcher, ahead of the sink, proven by strict mocks that would throw if the sink were consulted. The usability problem is real and does not go away by conforming, so it is raised where it can be fixed: opcua-drafts#70.
Tracking the updated drafts
The Vision, Robot Intent and AI Model Management NodeSets on
opcua-draftsmainhave moved, partly because of feedback this branch raised. All three are re-vendored byte-identically — verified by comparing git blob hashes against the upstream files rather than by eye — and the implementations follow.Two statements that were previously inexpressible now have a wire representation.
SubmitDetectionstakesSceneIsEmptyandSubmitCorrectiontakesRetractAll, so an agent can report "I examined this frame and there is nothing in it" — the terminating condition of the bin-picking task — and can retract a false positive by correcting a result down to nothing. Both are checked in both directions, because the flag is exactly what separates a deliberate empty observation from a lost payload: an empty array without the flag is refused, and the flag with an array attached is refused as two contradictory claims about one frame. Implemented in the dispatcher, the client, the MCP tools the agent drives, and the sample cell.Also absorbed:
LampTypeandLightingModebecame enums (§66),ProfileNamebecameDefaultProfileNameon the endpoint (distinct from theProfileNamea caller passes toGetStreamEndpoint, which keeps its name), AIInvoke/InvokeAsyncgainedPayloadUriunder the same exactly-one rule, and AIListModelsgained aContinuationPoint— becauseMaxResultsalone puts everything past the bound permanently out of reach.OpenUSD is on
0.9.0-alpha, verified on the running cell rather than by restore alone: D3D12 backend, 612×512 frame, the bin and all five parts in view and detected — the same result0.8.0-alphaproduced.The one local workaround that survives is
VisionMethodArguments, and its reason has changed rather than gone away: the builder materialises these Methods as Optional children throughCreateOrReplace, which constructs the state object directly and never runs the generated factory that now carries the signatures. Checked rather than assumed — removing the declarations fails five of the eightVisionMethodSurfaceTests.Review feedback
Three threads, all addressed in
0fe2803a3:src/Opc.Ua.AI(model),.Inference(IInferenceBackendand backends),.Server(node manager) and.Client(discovery, reads, calls, artefact transfer), in the same shape as the Robotics and Vision families; the samples becomesamples/AI/ModelManagementServerandModelManagementClient. The client library needed real work rather than relocation: its browsing surface wasprivateinside the sample's scenario runner, so moving the file alone would have produced a library nobody could call.Microsoft.Extensions.AI— anIChatClientis implemented by hosted services and on-device runtimes alike, which is exactly the property clause 8.1 asserts.Azure.Identityis gone: workload identity is not an Azure feature, every platform projects a token into a file and each SDK reads that file behind its own API, so the resolver reads it directly. The comment claimingMicrosoft.Extensions.AIwould force aSystem.Text.Jsonbump was stale — it pins 10.0.10, which this repository already centralises on.BinPickingCellandBinPickingClientmoved intosamples/Robotics/, which is where they belonged: the cell was already linking the arm, its kinematics and both USD assets out ofIntentEnabledRobot.Upstream
Work that belonged elsewhere went elsewhere rather than being worked around locally:
openusd-dotnet#13 — merged and shipped in0.8.0-alpha.SilkFrameCapture.Capturerendered only the first capture of a session and silently returned a blank frame for every one after it. Also filed #14 (no camera-by-prim API) and #15 (skipped tests reported as passed — which fooled me for several runs). Both are fixed; this branch takes0.8.0-alphaand deletes two workarounds, including 143 lines of hand-rolled projection maths.opcua-drafts#66–#69 — a close read of the spec produced 22 candidate ambiguities; 12 were discarded as misreadings with clause citations and the remaining 10 filed. One is concrete: five members exist in the NodeId CSV that no clause documents.opcua-drafts#70 — §9.5 could not express an empty observation or a false-positive retraction. Adopted, with the explicit-flag resolution argued for:SubmitDetectionsgainedSceneIsEmpty,SubmitCorrectiongainedRetractAll, and the corrected-array rule relaxed from exactly one to at most one.opcua-drafts#71 — the Vision NodeSet browse-named all 13InputArguments/OutputArgumentsProperties in the Vision namespace rather than the base namespace, so a stack could not find them and every Method in the specification was uncallable. Fixed; this branch now tracks the corrected artefact.The defect underneath it all
The end-to-end suite was removed earlier in this branch because nine of its
fourteen tests failed on one symptom: a client could not open a pipeline's
Feedbackobject or enumerate itsResults, even though browsing the sameparent listed both. Restoring it meant explaining that, and the explanation
was not in Vision.
RelativePathElementinitialisedIsInversetotrue. A caller thatsets a
ReferenceTypeIdand aTargetName— which reads as "the childreached by this Reference" — was therefore asking the Server for the
inverse Reference, and got
Bad_NoMatchwith nothing in the request tosuggest why.
ObjectTypeClient.ResolveChildNodeIdAsyncis such a caller, and it is whatevery source-generated Optional-child accessor in the stack is built on
— the accessor whose own documentation cites
AlarmConditionTypeClient.GetShelvingStateAsyncas an example. Vision is merely where it surfaced, because Vision puts two
Optional children on the path an off-Server agent must take. The sibling
helper in
StateMachineTypeClientExtensionsalready setIsInverse = falseby hand, which is the shape of a trap rather than a default.
The default is now forward,
ResolveChildNodeIdAsyncsays so explicitly,and the existing test that had recorded
trueas expected now pinsfalsewith the reasoning. Of the fortyRelativePathElementconstructionsites in the tree, that accessor was the only one relying on the default at
all — every deliberate caller already set it.
Two further defects were found underneath it, both real and both fixed:
CreateOrReplacehelpers construct a child state objectdirectly and leave
TypeDefinitionIdunset as well asReferenceTypeId.A child referenced by nothing cannot be browsed; one with no type definition
is a malformed Object that a client filtering by type silently skips — which
is exactly what
EnumerateResultsAsyncdoes. Normalisation moved toVisionNodeManager, so it also covers the case the builder cannot see: aresult published at runtime by an inference provider, long after the
address space was built. That path produced results a client could list but
not read.
Opc.Ua.AIshipped without anAssemblyInfo, so the solution built withsix
CA1014warnings.I had initially "fixed" this in the Vision client with Browse-based fallbacks,
on the strength of a probe that appeared to show
TranslateBrowsePathsToNodeIdsfailing even for
Server → ServerStatus. That probe was wrong: it used anunqualified
ReferenceTypeIds, which inside aOpc.Ua.Vision.*namespacebinds to Vision's
ReferenceTypeIdsand yields a nonsense NodeId. Chasingthat to the end is what turned up the real cause. The fallbacks are gone —
they would have hidden a stack-wide bug behind a Vision-shaped workaround, and
one of them matched on browse name while ignoring the namespace index.
tests/Opc.Ua.Vision.Intent.Testsis restored and back inUA.slnx, passing14/14 on its own merit rather than through a client-side workaround. A
registration test now pins the pipeline's
FeedbackandResultschildren,which nothing covered before.
Running it, and what that found
The demo's premise is that a language model looks at the bin and reasons
about what it sees. Running it end to end showed it could not, for reasons
no test covered, because every one of them lived between components.
The sample did not compile. Two
CS8604nullable errors withTreatWarningsAsErrors, so neither the cell server nor the demo could bestarted at all. Worth recording that my first explanation was wrong: CI does
build this project - the Linux leg's
net10.0pass compiles it, log-confirmed -so this is not a coverage gap in the workflow, and no workflow change was made.
PickandPlacenever moved the arm. They were a delay plus a gripperaction, so the operations reported
Succeededwhile nothing travelled. They nowsolve IK once and interpolate in joint space. And the parts never moved: the
arm swung but the bin stayed full, because nothing tied the held object to the
world state. The connector now creates 11 monitored items where it created 6,
and a scripted run delivers
SeqNo=151across 10 messages where it deliveredSeqNo=3across 1.The frame was smuggled through a string field. The media provider put a
base64 data URI of the PNG into
VisionImageReferenceDataType.Uri- a fieldmeant to be a reference, while the bytes were already returned in the inline
ByteString. Every response therefore shipped the image twice, and a1,311,400-byte PNG became a 1,748,536-character string against a
MaxStringLengthof 65,535: 26.7x over. That is why raisingMaxByteStringLength,MaxArrayLengthandMaxMessageSizenever helped - noneof them govern a
String. Separately the MCP tool assigned raw bytes whereImageContentBlockrequires base64, so the wire carried"data":"�PNG..."-every non-UTF-8 byte replaced by U+FFFD, destroying the image and inflating the
response to 6.8 MB.
The camera was aimed at nothing. The arm's home pose, the flange scan pose
the Vision model declared and the joint angles in the USD stage were three
independent claims about where the camera was, and they disagreed - the declared
flange orientation pointed the view 86 degrees away from the projection camera.
Solving them as one thing exposed two further constraints that are easy to miss:
the camera has to sit off the tool axis or it photographs its own gripper, and
the IK branch has to be elbow-back or a link parks under the camera and fills the
frame with the arm's own upper arm. A third was a regression this branch
introduced and
Opc.Ua.Robotics.Testscaught - aiming a straight-down camerafrom a point on the base's own X-Z plane puts the wrist on the J4/J6
singularity, so every motion away from home fails to solve. A 15-degree camera
roll, free because the camera still looks straight down, gets 25 degrees clear,
which is the margin the poses before this branch had.
The frame and the detections were different images. The sensor declared
2448x2048, the clip endpoint 1280x1024, and the renderer produced 640x512 - not
even the same aspect ratio. The ground-truth detector projects through the
declared intrinsics, so its boxes lived in 2448x2048 space while an agent was
handed a 640x512 picture: "pick the red cube you can see" arrived with
coordinates that pointed off the image. Everything now agrees at 612x512, an
exact 4x4 bin of the native sensor with the intrinsics scaled to match; the
native size stays in the model and serial number, where it describes the
hardware rather than the image.
LatestClipwas created and never written, so it reportedBad_NoDataAvailablefor the life of the Server and a consumer following themodel - read the published frame, call the method only if there is none - never
got a frame. The dispatcher now publishes a clip it has just encoded, for every
Vision server rather than this sample alone.
One genuine API gap turned up on the way:
OpcUaServerOptionsexposesMaxByteStringLength,MaxArrayLengthandMaxMessageSizeand noMaxStringLength, so a Server that legitimately needs a long string could notraise the limit through the hosting API at all. Added, with tests. It is not the
fix for anything above - that was a design bug - but the missing knob was real.
Not finished
Opc.Ua.Vision.OpenUsdis at 71.1% — what is left only executes with a real OpenUSD stage and plugin tree, which is not on every build agent.Opc.Ua.AI.Inferenceis now at 86.7%, up from 12.9%: the code this branch wrote is covered by 25 tests, andRestChatCompletionsBackend— previously the whole of the shortfall — is covered by 18 more against a stubbedHttpMessageHandler, exercising success, refusal, throttling, retry-after, timeout and cancellation without a network. Everything else clears the bar —Opc.Ua.Vision.Server92.1%,Opc.Ua.AI91.4%,.Vision.Client88.1%,Opc.Ua.Vision85.7%,Opc.Ua.AI.Server80.1%, all measured with coverlet. The CI coverage gate reports itself as advisory and non-blocking.SamplesCollectedis not counted here, anddocs/Vision.mdnow says so. Section 9.4 requires a Server to count a negative example (SceneIsEmpty/RetractAllcarrying aGroundTruthLabel) exactly as it counts one carrying geometry.VisionNodeManagerdoes not, and structurally cannot:SamplesCollectedis a property ofLearningJobType, which the AI Model Management companion defines, and Vision reaches the job through aNodeIdvalue rather than a Reference precisely so this model takes no dependency on the model that defines it. What the Server does guarantee is that the negative example survives the hop intact — both flags are carried verbatim toIVisionFeedbackSink— so a host that binds a learning job has everything it needs to satisfy the clause on the counter it owns. Stated in the limitations rather than left implied.Related Issues
Was stacked on #4195, now merged. No tracking issue yet for the Vision implementation itself — happy to open one if you would like the design recorded as an ADR before this goes further.
Checklist
.OpenUsd(71.1%) is below the 80% bar — see Not finished.dotnet build UA.slnx -c Release: 0 errors, 0 warnings.CreateUserManagementSeamBindsTheModelToTheServerAsync, andDurableDataValueQueueVerifyReferenceBatchingAsynconly under parallel load); both were confirmed pre-existing by re-running them with theIsInversechange reverted, which fails identically.ValueTask.FromResult/string.Create, which do not exist on .NET Framework; fixed infb57522daand verified building on every TFM.DisposeDrainsHeldConnectionCallbackBeforeListenerDisposalfailed once on the Ubuntu Client leg — a 13 ms timing-sensitive reverse-connect disposal test that passes locally 2123/2123 and touches no browse paths.