Skip to content

Commit 0d7ba8a

Browse files
authored
Release transient SYCL queues at the end of each test file to avoid OOM on low-memory GPUs (#3001)
Running the full test suite in a single process steadily drains device memory and can end in a `RuntimeError: ... UR_RESULT_ERROR_OUT_OF_RESOURCES` (seen intermittently on a low-memory iGPU, where the kernel logs `VM worker error: -12` / `exec queue reset detected`). Root cause: every distinct `dpctl.SyclQueue` a test creates is retained **for the whole session** as a key in dpctl's `SequentialOrderManager` (the map is keyed by queue identity and only ever grows). Each retained entry holds a host-task event that pins the backing USM allocation, so the memory is never released. Over a full run the map accumulates ~300 queues and free device memory bleeds from several GB down to ~1 GB, occasionally crossing the device limit. This is amplified on integrated GPUs, where "device memory" is system RAM, so the leaked USM competes directly with the host and the container's memory budget.
1 parent a41fd86 commit 0d7ba8a

1 file changed

Lines changed: 22 additions & 0 deletions

File tree

dpnp/tests/conftest.py

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
import warnings
3232

3333
import dpctl
34+
import dpctl.utils as dpu
3435
import numpy
3536
import pytest
3637

@@ -186,6 +187,27 @@ def pytest_collection_modifyitems(config, items):
186187
item.add_marker(skip_slow)
187188

188189

190+
def pytest_runtest_teardown(item, nextitem):
191+
"""
192+
Release transient SYCL queues at the end of each test file.
193+
194+
Every distinct ``SyclQueue`` a test creates is retained forever as a key in
195+
dpctl's ``SequentialOrderManager`` (keyed by queue identity), which pins its
196+
host-task events and the backing USM memory for the whole session. Over a
197+
full run this accumulates hundreds of queues and steadily drains device
198+
memory. Draining and dropping them at each file boundary keeps the footprint
199+
flat without paying the cost after every individual test.
200+
"""
201+
# Only act on the last test of the current file (``nextitem`` is None at the
202+
# very end of the session, or points at the first test of the next file).
203+
if nextitem is not None and item.path == nextitem.path:
204+
return
205+
try:
206+
dpu.SequentialOrderManager.clear()
207+
except Exception as e: # never let cleanup break the run
208+
warnings.warn(f"Failed to clear SequentialOrderManager: {e}")
209+
210+
189211
@pytest.fixture
190212
def allow_fall_back_on_numpy(monkeypatch):
191213
monkeypatch.setattr(

0 commit comments

Comments
 (0)