fix[next]: make OTF build caches crash-consistent

docs/development/ADRs/next/0024-Crash_Consistent_Build_Caches.md

@@ -0,0 +1,103 @@
+---
+tags: []
+---
+
+# [Crash-Consistent Build Caches]
+
+- **Status**: valid
+- **Authors**: Hannes Vogt (@havogt)
+- **Created**: 2026-06-22
+- **Updated**: 2026-07-03
+
+In the context of the `gt4py.next` OTF build caches, facing user reports that
+an interrupted pipeline (Ctrl-C, `SIGKILL`/OOM, full disk) can leave a cache
+entry that a key lookup reports as a HIT but whose payload is truncated or
+missing — so the next run fails on load instead of rebuilding — we decided to
+make every cache write an **atomic publish** (temp sibling + `os.replace`) and
+every cache read **validate + self-heal** (any load failure → treat as miss →
+rebuild). We considered relying on the file lock, `fsync`-ing writes, and
+whole-directory renames, and accept that a hard power loss may still require
+regenerating the last in-flight entry.
+
+## Context
+
+[ADR 0023](0023-Fingerprinting.md) made cache *keys* correct, but a key only
+answers "is there an entry for this input?" — not whether the entry's payload
+is complete. Both backends use two independent fingerprint-keyed layers, each
+with its own present-but-broken states:
+
+| Layer                           | Broken state after an interruption                       | Old behavior on re-run                                                               |
+| ------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------ |
+| translation cache (`FileCache`) | truncated pickle                                         | `EOFError`/`UnpicklingError` aborts                                                  |
+| gtfn build folder               | truncated `gt4py.json`                                   | `JSONDecodeError` aborts                                                             |
+| gtfn build folder               | status `COMPILED`, module deleted (e.g. scratch cleanup) | `CompilationError`                                                                   |
+| dace build folder               | truncated `lib<name>.so` (kill mid-link)                 | accepted by dace's existence-only `use_cache` gate; `dlopen` fails on every `load()` |
+| compiledb template (shared)     | truncated `compile_commands.json`                        | `JSONDecodeError` for *every* program with that configuration                        |
+
+(A build interrupted *before* reaching `COMPILED` already resumes correctly via
+the gtfn status machine and must keep doing so.)
+
+The file lock is orthogonal: it serializes concurrent builders but a killed
+lock holder releases the lock and leaves its partial entry behind. Mature
+compiler caches (CPython `.pyc`, Triton, TorchInductor, Numba, ccache, sccache)
+all converge on the same write-atomically / validate-on-read combination used
+here.
+
+## Decision
+
+A shared helper `gt4py._core.file_utils.atomic_write_bytes/_text` writes to a
+uniquely-named sibling temp file and `os.replace`s it into place, so a reader
+sees old-or-new content, never a torn file. The temp file is created with a
+plain `open` so the result keeps umask-derived permissions. No `fsync`: it only
+adds power-loss durability, costs real latency on parallel filesystems, and
+validate-on-read absorbs the rare torn write.
+
+Per layer:
+
+- **Translation cache** — `FileCache.__setitem__` publishes atomically;
+  `__getitem__` treats any unpicklable entry (`EOFError`, `UnpicklingError`,
+  `OSError`, and `AttributeError`/`ImportError` from version skew) as a miss,
+  evicting the file and raising `KeyError` so `CachedStep` recomputes.
+- **gtfn build folder** — `build_data.write_data` publishes atomically, so the
+  `COMPILED` status is a proper commit marker written last; `read_data` returns
+  `None` for unreadable metadata. The cache-HIT gate is a single predicate
+  `is_usable(data, src_dir)` = status `COMPILED` *and* module file present;
+  anything else rebuilds (preserving the partial-build resume states).
+- **compiledb template** — published atomically; `_cc_find_compiledb`
+  validates the JSON on HIT and evicts an unreadable template so it is
+  regenerated under the existing lock.
+- **dace build folder** — dace's `use_cache` gate accepts a library on
+  mere existence, and the compile step never loads it
+  (`sdfg.compile(return_program_handle=False)` since #2587), so a truncated
+  library survives until `DaCeCompilationArtifact.load()` — which has neither
+  the build lock nor the dace config context to recompile. Instead, the locked
+  compile step writes a `.gt4py_compile_complete` marker **after** each
+  completed compile (and removes it before compiling); if the marker is absent,
+  the previous build was interrupted and the stale `lib<name>.*` /
+  `libdacestub_<name>.*` files are deleted so dace rebuilds them. A *missing*
+  library needs no handling: `SDFG.regenerate_code` defaults to `True`, so dace
+  regenerates it anyway.
+
+## Alternatives considered
+
+- **Lock-only / status quo**: gives no crash consistency (see Context).
+- **Whole-directory atomic publish**: strongest guarantee, but fights the
+  in-place CMake/Ninja incremental builds, dace's `build_folder` binding, and
+  the partial-build resume behavior. Remains an option if marker + self-heal
+  proves insufficient.
+- **`fsync` on every write**: protects only against power loss, not the
+  reported process-kill cases; rejected as default (a knob can be added if
+  power-loss corruption is ever observed).
+- **Self-heal on load failure for dace**: worked before #2587, but the
+  compile/load split moved loading out of the step that can rebuild.
+
+## Consequences
+
+- An interrupted run no longer poisons the cache; the affected entry is rebuilt
+  transparently on the next run. Version-skew pickles also become clean misses.
+- Corrupt entries are evicted silently (`_core` has no logging infrastructure,
+  and per-entry warnings would spam multi-rank runs); a persistent failure
+  still surfaces as rebuild-then-error, not a loop.
+- Known gaps: a gtfn module that is present but truncated *after* a completed
+  build (torn write at power loss) is not self-healed; orphaned `.tmp` files
+  from kills between write and rename are not swept.

src/gt4py/_core/file_utils.py

@@ -0,0 +1,46 @@
+# GT4Py - GridTools Framework
+#
+# Copyright (c) 2014-2024, ETH Zurich
+# All rights reserved.
+#
+# Please, refer to the LICENSE file in the root directory.
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""Helpers for atomic file writes."""
+
+from __future__ import annotations
+
+import os
+import pathlib
+import uuid
+
+
+def atomic_write_bytes(target: str | os.PathLike, data: bytes) -> None:
+    """Write ``data`` to ``target`` atomically.
+
+    The bytes are written to a temporary file in ``target``'s directory and then
+    renamed into place with ``os.replace``. A reader (or the next process after a
+    crash) therefore sees either the previous contents or the complete new
+    contents, never a half-written file. The temporary file is a sibling of
+    ``target`` so the rename stays on a single filesystem, and is created with a
+    plain ``open`` so the result gets the usual umask-derived permissions
+    (``tempfile.mkstemp`` would pin it to ``0600``).
+
+    This protects against process kill / Ctrl-C / OOM during the write. It does
+    *not* fsync, so a power loss may still lose the entry (callers tolerate this
+    by treating an unreadable entry as a cache miss).
+    """
+    target = pathlib.Path(target)
+    tmp = target.parent / f".{target.name}.{uuid.uuid4().hex}.tmp"
+    try:
+        with open(tmp, "wb") as f:
+            f.write(data)
+        os.replace(tmp, target)
+    except BaseException:
+        tmp.unlink(missing_ok=True)
+        raise
+
+
+def atomic_write_text(target: str | os.PathLike, text: str, *, encoding: str = "utf-8") -> None:
+    """Write ``text`` to ``target`` atomically (see `atomic_write_bytes`)."""
+    atomic_write_bytes(target, text.encode(encoding))

src/gt4py/_core/filecache.py

@@ -12,7 +12,7 @@
 import pickle
 from typing import Any, Hashable
 
-from gt4py._core import locking
+from gt4py._core import file_utils, locking
 from gt4py.eve import utils as eve_utils
 
 
@@ -36,14 +36,20 @@ def __getitem__(self, key: Hashable) -> Any:
         if key not in self:
             raise KeyError(key)
         with locking.lock(path := self._get_path(key)):
-            with open(path, "rb") as f:
-                return pickle.load(f)
+            try:
+                with open(path, "rb") as f:
+                    return pickle.load(f)
+            except (EOFError, pickle.UnpicklingError, OSError, AttributeError, ImportError) as e:
+                # An interrupted write can leave a truncated/corrupt entry that
+                # `__contains__` still reports as present. Treat it as a miss and
+                # drop the unusable entry so the caller recomputes it.
+                path.unlink(missing_ok=True)
+                raise KeyError(key) from e
 
     def __setitem__(self, key: Hashable, value: Any) -> None:
         self.path.mkdir(parents=True, exist_ok=True)
         with locking.lock(path := self._get_path(key)):
-            with open(path, "wb") as f:
-                pickle.dump(value, f, protocol=5)
+            file_utils.atomic_write_bytes(path, pickle.dumps(value, protocol=5))
 
     def __delitem__(self, key: Hashable) -> None:
         if key not in self:

src/gt4py/next/otf/compilation/build_data.py

@@ -14,6 +14,8 @@
 import pathlib
 from typing import Final, Optional
 
+from gt4py._core import file_utils
+
 
 _DATAFILE_NAME: Final = "gt4py.json"
 
@@ -76,12 +78,16 @@ def contains_data(path: pathlib.Path) -> bool:
 def read_data(path: pathlib.Path) -> Optional[BuildData]:
     try:
         return BuildData.from_json(json.loads((path / _DATAFILE_NAME).read_text()))
-    except FileNotFoundError:
+    except (OSError, ValueError, KeyError, AttributeError):
+        # Missing file (OSError/FileNotFoundError), or one left truncated/corrupt
+        # by an interrupted write (json.JSONDecodeError is a ValueError; a missing
+        # dict key raises KeyError, a bad status name AttributeError) is treated
+        # as "no build data" so the caller rebuilds rather than crashing.
         return None
 
 
 def write_data(data: BuildData, path: pathlib.Path) -> None:
-    (path / _DATAFILE_NAME).write_text(json.dumps(data.to_json()))
+    file_utils.atomic_write_text(path / _DATAFILE_NAME, json.dumps(data.to_json()))
 
 
 def update_status(new_status: BuildStatus, path: pathlib.Path) -> None:

src/gt4py/next/otf/compilation/build_systems/compiledb.py

@@ -16,7 +16,7 @@
 import subprocess
 from typing import Optional, TypeVar
 
-from gt4py._core import locking
+from gt4py._core import file_utils, locking
 from gt4py.next import config, errors, fingerprinting
 from gt4py.next.otf import code_specs, stages
 from gt4py.next.otf.binding import interface
@@ -294,6 +294,14 @@ def _cc_get_compiledb(
 def _cc_find_compiledb(path: pathlib.Path) -> Optional[pathlib.Path]:
     compile_db_path = path / "compile_commands.json"
     if compile_db_path.exists():
+        try:
+            json.loads(compile_db_path.read_text())
+        except (OSError, ValueError):
+            # The template is shared by every program built with the same
+            # configuration; one left truncated/corrupt by an interrupted write
+            # would poison all of them. Drop it so the caller regenerates.
+            compile_db_path.unlink(missing_ok=True)
+            return None
         return compile_db_path
     return None
 
@@ -368,6 +376,6 @@ def _cc_create_compiledb(
         )
 
     compile_db_path = path / "compile_commands.json"
-    compile_db_path.write_text(json.dumps(compile_db))
+    file_utils.atomic_write_text(compile_db_path, json.dumps(compile_db))
 
     return compile_db_path

src/gt4py/next/otf/compilation/compiler.py

@@ -10,7 +10,7 @@
 
 import dataclasses
 import pathlib
-from typing import Protocol, TypeVar
+from typing import Protocol, TypeGuard, TypeVar
 
 from gt4py._core import definitions as core_defs, locking
 from gt4py.next import config, fingerprinting
@@ -26,6 +26,17 @@ def module_exists(data: build_data.BuildData, src_dir: pathlib.Path) -> bool:
     return (src_dir / data.module).exists()
 
 
+def is_usable(
+    data: build_data.BuildData | None, src_dir: pathlib.Path
+) -> TypeGuard[build_data.BuildData]:
+    """Check that a cached build is marked compiled *and* its module artifact is present.
+
+    An interrupted run (or external cleanup) can leave a ``COMPILED`` marker
+    without the artifact; such a build folder must be rebuilt, not reused.
+    """
+    return data is not None and is_compiled(data) and module_exists(data, src_dir)
+
+
 CodeSpecT = TypeVar("CodeSpecT", bound=code_specs.SourceCodeSpec)
 TargetCodeSpecT = TypeVar("TargetCodeSpecT", bound=code_specs.SourceCodeSpec)
 CPPLikeCodeSpecT = TypeVar("CPPLikeCodeSpecT", bound=code_specs.CPPLikeCodeSpec)
@@ -103,14 +114,12 @@ def __call__(
         # If we are compiling the same program at the same time (e.g. multiple MPI ranks),
         # we need to make sure that only one of them accesses the same build directory for compilation.
         with locking.lock(src_dir):
-            data = build_data.read_data(src_dir)
-
-            if not data or not is_compiled(data) or self.force_recompile:
+            if self.force_recompile or not is_usable(build_data.read_data(src_dir), src_dir):
                 self.builder_factory(inp, self.cache_lifetime).build()
 
             new_data = build_data.read_data(src_dir)
 
-            if not new_data or not is_compiled(new_data) or not module_exists(new_data, src_dir):
+            if not is_usable(new_data, src_dir):
                 raise CompilationError(
                     f"On-the-fly compilation unsuccessful for '{inp.program_source.entry_point.name}'."
                 )

src/gt4py/next/program_processors/runners/dace/workflow/compilation.py

@@ -14,7 +14,7 @@
 import pathlib
 import warnings
 from collections.abc import Callable, MutableSequence, Sequence
-from typing import Any
+from typing import Any, Final
 
 import dace
 import dace.codegen.compiler as dace_compiler
@@ -30,6 +30,9 @@
 )
 
 
+_COMPILE_COMPLETE_MARKER: Final = ".gt4py_compile_complete"
+
+
 def _add_tx_markers(sdfg: dace.SDFG) -> None:
     has_gpu_schedule = any(
         getattr(node, "schedule", dace.dtypes.ScheduleType.Default) in dace.dtypes.GPU_SCHEDULES
@@ -220,7 +223,23 @@ def __call__(
 
             sdfg.build_folder = sdfg_build_folder
             with locking.lock(sdfg_build_folder):
+                # With `compiler.use_cache=True` dace reuses a cached library on mere
+                # *existence*, without validating it; an interrupted build can leave a
+                # truncated, unloadable library behind. The marker is written only
+                # after a completed compile: no marker -> drop the stale library so
+                # dace rebuilds it instead of handing it out.
+                marker = sdfg_build_folder / _COMPILE_COMPLETE_MARKER
+                if not marker.exists():
+                    for stale in (
+                        dace_compiler.get_binary_name(
+                            object_folder=sdfg_build_folder, sdfg_name=sdfg.name
+                        ),
+                        *sdfg_build_folder.glob(f"libdacestub_{sdfg.name}.*"),
+                    ):
+                        stale.unlink(missing_ok=True)
+                marker.unlink(missing_ok=True)
                 sdfg.compile(validate=False, return_program_handle=False)
+                marker.touch()
             # ``build_folder_mode`` is set by ``dace_context``; resolve the library
             # path here so ``get_binary_name`` sees the same mode dace built under.
             library_path = dace_compiler.get_binary_name(

tests/core_tests/unit_tests/test_filecache.py

@@ -46,3 +46,38 @@ def test_delete_item(temp_cache):
 def test_keyerror_on_missing(temp_cache):
     with pytest.raises(KeyError):
         _ = temp_cache["does_not_exist"]
+
+
+# --- Crash consistency ---
+# An interrupted write (Ctrl-C / SIGKILL / OOM during ``pickle.dump``) can leave a
+# truncated entry on disk. ``__contains__`` still reports a HIT, but reading must
+# behave like a cache miss (raise ``KeyError``) instead of propagating an
+# unpickling error, and the unusable entry must be evicted so the cache recovers.
+
+
+def test_truncated_entry_is_treated_as_missing(temp_cache):
+    key = "interrupted"
+    temp_cache[key] = {"a": 1}
+
+    # Simulate a write killed mid-``pickle.dump``: truncate the payload.
+    path = temp_cache._get_path(key)
+    path.write_bytes(path.read_bytes()[:4])
+
+    with pytest.raises(KeyError):
+        _ = temp_cache[key]
+
+
+def test_corrupt_entry_self_heals(temp_cache):
+    key = "interrupted"
+    temp_cache[key] = {"a": 1}
+
+    # A syntactically-invalid pickle (valid proto header, garbage body).
+    temp_cache._get_path(key).write_bytes(b"\x80\x05garbage-not-a-pickle")
+
+    with pytest.raises(KeyError):
+        _ = temp_cache[key]
+
+    # The unusable entry is evicted, so the cache is writable/usable again.
+    assert key not in temp_cache
+    temp_cache[key] = {"a": 2}
+    assert temp_cache[key] == {"a": 2}