Persistent ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT — reproduces on two different EZSP radios (ZBT-1 and ZBT-2

[Heinrich-git]

Bug Report: Persistent ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT — reproduces on two different EZSP radios (ZBT-1 and ZBT-2)

Summary

bellows.ash.NcpFailure: NcpResetCode.ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT occurs
repeatedly during normal ZHA operation (triggered via set_extended_timeout inside
zigpy.zcl.reply → zigpy.device.request → bellows.zigbee.application.send_packet).
The error causes intermittent Home Assistant Core freezes/unresponsiveness
(Supervisor watchdog "missed API response", occasionally forcing a full Core
restart), and has now been reproduced identically after replacing the radio
hardware entirely — from a Home Assistant Connect ZBT-1 to a Home Assistant
Connect ZBT-2 — which rules out a defective individual USB stick.

Environment

• Home Assistant Core: 2026.7.3 (also reproduced on 2026.7.2)
• Installation type: Home Assistant OS 18.1, Home Assistant Green (aarch64, embedded)
• Supervisor: 2026.07.3
• ZHA: 2.0.0
• zigpy: 2.0.0
• bellows: 0.49.2
• zha-quirks: 2.1.1
• Radio (original): Home Assistant Connect ZBT-1, EmberZNet firmware 7.5.1.0 build 0 (20260224005823), chip EFR32MG21A020F512IM32, ezspVersion 13, stack_version 13
• Radio (after migration): Home Assistant Connect ZBT-2 — same error reproduces
• Network size: ~35 Zigbee devices (mix of Router/EndDevice, mostly Tuya/SONOFF)
• Connection: USB, official Nabu Casa 40cm extension cable (Nabu Casa-recommended)

Steps already ruled out

We spent significant time isolating this before concluding it's a software issue:

1. USB cable/port: tested multiple ports, tested with and without the official
   40cm extension cable. ASH_OVERFLOW_ERROR, ASH_FRAMING_ERROR, ASH_OVERRUN_ERROR
   counters are all 0 — no physical-layer transmission errors.
2. RF interference: energy scan showed the network's original channel (25) was
   heavily congested (score ~4/100) vs. channel 20 (~86/100). Migrated the network to
   channel 20 via zha_toolkit/ZHA network migration. Did not resolve the issue —
   confirms this is a USB/serial (ASH protocol) problem, not an over-the-air RF problem.
3. Known periodic-backup bug (load_network_info → xncp_get_route_table_entry burst during periodic backup causes ASH ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT / NcpFailure loop (bellows 0.49.1, Python 3.14, ZBT-2) #724): initially suspected this was the
   cause (xncp_get_route_table_entry burst during zigpy's periodic network backup
   overwhelming the ASH ACK budget). Applied the documented workaround:

   zha:
     zigpy_config:
       backup_enabled: false

   This did eliminate the specific load_network_info/route-table-dump crash-loop
   signature, but the underlying ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT still recurs
   via a different code path — triggered by ordinary set_extended_timeout calls
   during normal device communication (not the periodic backup task).
4. Hardware replacement: swapped the radio entirely (ZBT-1 → ZBT-2) using ZHA's
   built-in "automatic radio migration" flow (network key/PAN ID/channel/device list
   preserved, ~35 devices did not need re-pairing). The identical
   ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT traceback reappeared within an hour of
   migrating to the brand-new ZBT-2, ruling out a defective individual unit.
5. CPU/RAM/network/DNS/socket-count: all checked during "hang" windows and found
   normal (CPU ~24-29%, RAM ~2.6-2.9/4GB, ping/nslookup clean, /proc/net/sockstat
   unremarkable, no USB bus resets/re-enumerations in kernel log).

Observed traceback (identical on both ZBT-1 and ZBT-2)

Traceback (most recent call last):
  File "zigpy/zcl/__init__.py", line 891, in reply
    return await self._endpoint.reply(...)
  File "zigpy/endpoint.py", line 269, in reply
    return await self.device.reply(...)
  File "zigpy/device.py", line 996, in reply
    return await self.request(...)
  File "zigpy/device.py", line 693, in request
    await send_request(attempt=attempt)
  File "zigpy/application.py", line 1136, in request
    await self.send_packet(...)
  File "bellows/zigbee/application.py", line 1030, in send_packet
    await self._ezsp.set_extended_timeout(...)
  File "bellows/ezsp/v4/__init__.py", line 201, in set_extended_timeout
    (curr_extended_timeout,) = await self.getExtendedTimeout(remoteEui64=ieee)
  File "bellows/ezsp/protocol.py", line 124, in command
    await self._gw.send_data(data)
  File "bellows/uart.py", line 26, in send_data
    await self._transport.send_data(data)
  File "bellows/ash.py", line 735, in send_data
    await asyncio.shield(...)
  File "bellows/ash.py", line 660, in _send_data_frame
    raise NcpFailure(t.NcpResetCode.ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT)
bellows.ash.NcpFailure: NcpResetCode.ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT

Also observed around the same failures:

bellows.thread: Attempted to use a closed event loop
RuntimeWarning: coroutine 'Gateway.send_data' was never awaited
AttributeError: 'NoneType' object has no attribute 'set_extended_timeout'

suggesting the gateway/_ezsp object is being torn down (set to None) while
in-flight set_extended_timeout calls are still referencing it — a possible
race condition between connection-loss handling and concurrent outgoing requests.

Impact

• Supervisor watchdog periodically logs Watchdog missed a Home Assistant Core API response; in tight clusters this escalates to Watchdog missed 2 Home Assistant Core API responses in a row. Restarting Home Assistant Core! — forcing a full
  Core restart, several times a day on bad days.
• Frontend shows recurring "Connection lost. Reconnecting…" during these episodes.
• Home Assistant Cloud (Nabu Casa) full-instance backup uploads fail
  (pre_backup_actions_failed / Error during pre-backup: NcpResetCode. ERROR_EXCEEDED_MAXIMUM_ACK_TIMEOUT_COUNT) when the pre-backup hook happens to
  coincide with this condition.
• A secondary add-on (Automation Inspector) was observed cycling its WebSocket
  connection to Core every ~30-40s during bad episodes; stopping it correlated
  with a quieter period, but this may be a symptom rather than a cause and needs
  more data before concluding anything.

Request

Given this reproduces identically across two different EZSP radios (different
physical units, same chip family/firmware baseline) under the same Home Assistant
Core version, we suspect a bellows/zigpy-side issue (possibly related to the ASH
retry/timeout budget under load, or the _ezsp/gateway teardown race noted above)
rather than anything hardware-specific. Happy to provide full diagnostics
(config_entry-zha-*.json diagnostics download available, network keys redacted)
or run further debug-level logging on request.

Attachments to include when filing

• ZHA diagnostics JSON (download via Settings → Devices & Services → ZHA →
  ⋮ → Download diagnostics) — redact/remove network_key and
  tc_link_key fields before uploading, or use GitHub's private
  attachment flow
• Full Home Assistant Core log covering a "hang" window (Settings → System →
  Logs → download, or /config/home-assistant.log)
• Supervisor log covering the same window (Settings → System → Logs → pick
  "Supervisor" from the dropdown → download)

config_entry-zha-01JMN4W2RXRGJ6NRQVEDGNMN4J(2).json

bellows_log_excerpt_1.md