Commit 0d7ba8a
authored
Release transient SYCL queues at the end of each test file to avoid OOM on low-memory GPUs (#3001)
Running the full test suite in a single process steadily drains device
memory and can end in a `RuntimeError: ...
UR_RESULT_ERROR_OUT_OF_RESOURCES` (seen intermittently on a low-memory
iGPU, where the kernel logs `VM worker error: -12` / `exec queue reset
detected`).
Root cause: every distinct `dpctl.SyclQueue` a test creates is retained
**for the whole session** as a key in dpctl's `SequentialOrderManager`
(the map is keyed by queue identity and only ever grows). Each retained
entry holds a host-task event that pins the backing USM allocation, so
the memory is never released. Over a full run the map accumulates ~300
queues
and free device memory bleeds from several GB down to ~1 GB,
occasionally crossing the device limit.
This is amplified on integrated GPUs, where "device memory" is system
RAM, so the leaked USM competes directly with the host and the
container's memory budget.1 parent a41fd86 commit 0d7ba8a
1 file changed
Lines changed: 22 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
| |||
186 | 187 | | |
187 | 188 | | |
188 | 189 | | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
189 | 211 | | |
190 | 212 | | |
191 | 213 | | |
| |||
0 commit comments