Skip to content

Python 3.12 heap corruption #138

@swinter1

Description

@swinter1

Experience issue with python 3.12 crashing, but not 3.11 (not producing a traceback, but instead so internal error - see details below). Have reproduced the failure below.

Claude write up

Python 3.12 Heap Corruption: matplotlib + botorch + ax Import Order

Summary

On Python 3.12 (Windows), importing matplotlib, botorch, and ax-platform in a specific order causes a native heap corruption crash (STATUS_HEAP_CORRUPTION / 0xC0000374). The process terminates immediately with no Python traceback. In bash this surfaces as exit code 127.

This does not occur on Python 3.11 with identical package versions.

Environment

Package Version
Python 3.12.10
ax-platform 0.3.7
botorch 0.10.0
gpytorch 1.11
torch 2.4.1+cu124
matplotlib 3.10.8
numpy 1.26.4
scipy 1.17.1
OS Windows (MINGW64) . Note does not occur on Linux

Testing date: 2026-04-22

Reproduction

Crashes (exit code 0xC0000374)

# Any order where ax is imported AFTER both matplotlib and botorch
import matplotlib; import botorch; import ax       # CRASH
import botorch; import matplotlib; import ax       # CRASH

Works

# ax imported BEFORE matplotlib and/or botorch
import ax; import matplotlib; import botorch       # OK
import matplotlib; import ax; import botorch       # OK
import botorch; import ax; import matplotlib       # OK
import ax; import botorch; import matplotlib        # OK

Key observations

  • Two of three is fine. Any pair (matplotlib + botorch, matplotlib + ax, botorch + ax) works in isolation.
  • All three are required to trigger the crash, and ax must be the last one loaded.
  • import matplotlib alone (not matplotlib.pyplot) is sufficient to trigger it.
  • Every botorch submodule triggers it (botorch.acquisition, botorch.optim, botorch.utils, botorch.posteriors, botorch.sampling, botorch.generation), but gpytorch, torch, and linear_operator alone do not.
  • Backend selection (matplotlib.use("Agg")) does not help.
  • The lock files for Python 3.11 and 3.12 resolve to identical package versions — the difference is purely in the platform-specific wheel binaries selected for cp311 vs cp312.

Cause

The crash is a native C-level heap corruption, most likely caused by conflicting shared libraries (e.g. MKL, OpenMP, or BLAS DLLs) shipped inside the torch, numpy, and/or scipy wheels. When loaded in a particular order on Python 3.12, the DLLs from different packages collide.

botorch imports something beyond what gpytorch/torch alone load (likely triggering additional scipy or numpy native code paths), and when ax-platform subsequently loads its own native dependencies, the heap is already corrupted.

Impact on test suite

The project's tests/conftest.py imports in this order:

import gpytorch              # loads torch
import matplotlib.pyplot as plt
...
from botorch.fit import fit_gpytorch_mll    # loads botorch
from botorch.models import SingleTaskGP
from axtreme.plotting.gp_fit import plot_1d_model  # imports ax internally → CRASH

This means all tests fail silently on Python 3.12 — pytest exits with code 127 and produces no output.

Workarounds

1. Import ax early in conftest.py (recommended)

import ax  # Must come before matplotlib + botorch to avoid heap corruption on Python 3.12

import gpytorch
import matplotlib.pyplot as plt
import pytest
import torch
...

2. Pin Python < 3.12

Restrict the project to Python 3.11 until upstream packages fix the native library conflict.

3. Upgrade ax-platform and botorch

ax-platform==0.3.7 and botorch==0.10.0 are from early 2024. Newer versions may ship updated native dependencies that resolve the conflict. This requires verifying API compatibility.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions