fix(azure-identityd): wait for a working resolver before starting identityd

[JanZachmann]

Summary

aziot-identityd.service gets a second ExecStartPre which waits until every endpoint host configured in /etc/aziot/config.toml resolves - cert_issuance.est.urls.default, provisioning.global_endpoint and provisioning.iothub_hostname. A single lookup is capped at 5 s and the whole wait at 30 s. No endpoint configured means no wait, and on timeout the daemon starts anyway.

Reason

identityd enrolls the device-id cert via EST and reaches out to DPS right at start. When the resolver cannot answer it gets EAI_AGAIN, exits 1 and is restarted; the second attempt succeeds. This hits the first boot after flashing or a factory reset - the state every CI DUT boots in. The failed start also fails the ADU agent health check and logs [ERR!] lines that are not in the smoke-test syslog whitelist, while neither the crash-loop check nor systemctl is-system-running can see it: a unit that failed once and recovered is active with NRestarts=1.

Seen twice on the rpi4 nightly, last in test-image-job 541. The lookup blocks for its full resolver timeout before it fails: aziot-certd starts at 18:09:14 and reports Temporary failure in name resolution at 18:09:34, identityd exits, and the restart at 18:09:40 succeeds. The rest of the network worked for the whole of those 20 s - the test runner opened dozens of ssh connections to the DUT between 18:09:15 and 18:09:34, and timesyncd had reached the lab time server at 18:09:08, before the window. So the queries were sent and stayed unanswered, which is a different thing from resolved having no server at all - that case fails immediately instead of blocking. Why the resolver was silent for those 20 s is not established; it is the lab resolver handed out by DHCP, and it answers in a few milliseconds when warm.

The unit already waits for network-online.target, and that target only says a link is configured. DHCP hands out the address of a DNS server, not a DNS server that answers, and no ordering can express "a name resolves" - which is why the wait has to attempt a lookup. It also means the gate does not depend on which cause it was: resolver not warm, resolved not ready or an upstream that is slow all end in the same wait.

Which endpoint identityd needs first depends on the provisioning method, and they are not the same domain, so every configured host is probed and each one still pending gets one try per pass - a host that never resolves must not starve the others. The single lookup is capped as well, because a failing lookup blocks for the resolver timeout: the unit sets no TimeoutStartSec, so its ceiling is the 90 s default, and running into it would produce the very failed start this gate exists to prevent.

The gate delays the start and never skips it. A condition-skipped activation would consume aziot-identityd.socket's trigger limit of 20 within 2 s, and a failed socket is never activated again on its own (#680).

[HarryWaschkeit]

why do we suppress a fail exit status of omnect_wait_dns_ready.sh?
as far as I can see it will always be 0? (ends explicitly with exit 0 only and doesn't terminate on failing commands)

[JanZachmann]

Correct that the script itself always exits 0 - it has no set -e and ends on exit 0. The - covers what the script cannot: systemd counts a failed exec as a failed ExecStartPre, so a missing, non-executable or killed script would fail the identityd start. That is the exact failure this PR removes, and the gate must never be able to cause it. It also matches the omnect_service_log.sh line above.

[HarryWaschkeit]

I don't buy it: this PR adds both, the script to the read-only rootfs as well as the ExecStartPre extension to the service file, a non-existing (executable) script would be a system error that cannot be handled at all imho.
so, why/how should this - matter here?

[JanZachmann]

You are right, dropped both prefixes in 2b31533.

- only ever covered a missing or non-executable script, which is a broken image and should fail loud rather than be swallowed. The script itself cannot return non-zero, so nothing else was being suppressed.

+ is gone too, for the same "buys nothing" reason: /etc/aziot is 0775 root:aziot and config.toml is 0664 root:aziot, identityd reads that same file as aziotid, and the unit has no sandboxing directives - so toml, timeout and getent all work without privileges. Say if you would rather keep + as a guard against the perms changing.

[HarryWaschkeit]

given that other service file changes do use + I would prefer leaving it here too for uniformity.

[JanZachmann]

Restored in a9548a0, so the line is now ExecStartPre=+/usr/bin/omnect_wait_dns_ready.sh - + back for uniformity, - gone.

[Copilot]

Pull request overview

This PR adds a startup gate for aziot-identityd.service to reduce first-boot flakiness by waiting (with caps) for DNS resolution of configured Azure IoT endpoint hostnames before identityd starts.

Changes:

• Added omnect_wait_dns_ready.sh, which extracts relevant endpoint hosts from /etc/aziot/config.toml and polls DNS resolution with per-lookup and overall timeouts.
• Updated the aziot-identityd recipe to install the script, add required runtime dependencies, and wire it into aziot-identityd.service via an additional ExecStartPre.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
recipes-azure-iot/azure-identityd/aziot-identityd/omnect_wait_dns_ready.sh	New DNS readiness wait script that probes configured endpoint hosts before identityd starts.
recipes-azure-iot/azure-identityd/aziot-identityd.inc	Installs the script, adds RDEPENDS, and injects an ExecStartPre call into the identityd unit.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[HarryWaschkeit]

see comment https://github.com/omnect/meta-omnect/pull/683/changes#r3810905041

[HarryWaschkeit]

see #683 (comment)

[JanZachmann]

Closing this without merging.

The only trigger we can point at is our own lab resolver, and the log does not say why it went quiet - so merging this would ship a guess to every device.

What the log does show (test-image-job 541): the lookup blocks for its full resolver timeout before failing - aziot-certd starts at 18:09:14, Temporary failure in name resolution at 18:09:34 - while the rest of the network was working the whole time: the test runner held ssh sessions to the DUT continuously from 18:09:15 to 18:09:34, and timesyncd had reached the lab time server at 18:09:08. So the queries went out and nothing came back from the resolver. That also rules out resolved starting with no server at all, which fails immediately instead of blocking.

Two things make the product-side wait not worth its cost:

• the failure is self-healing - identityd restarts after 5 s and the second attempt succeeds
• the one real consequence, AducIotAgent crashing on a failed health check, is already fixed in omnect/iot-hub-device-update#55

And the gate only ever helped when the resolver was silent for less than 30 s. Past that it starts the daemon anyway, which is exactly today's behaviour. That is a narrow win for a script plus bash, coreutils and toml-cli on every boot of every device.

Not solved by closing: the nightly still fails on found syslog errors not in whitelist when this happens. That needs a decision - either the identityd lines stay whitelisted as expected transient noise, or someone looks at why the lab resolver is silent for ~20 s after a DUT boots.

The review findings from this round were valid independently of the outcome and are worth keeping in mind for the next script of this shape: the URL parsing, the dedup array, and the - on ExecStartPre that would have hidden a broken image.