Test regressions with numpy 2.4.0

[musicinmybrain]

Tests resize_using_exclusive_borrow and src/borrow/mod.rs - borrow::PyReadwriteArray<'py,T,Ix1>::resize (line 611) fail with numpy 2.4.0.

$ git clone https://github.com/PyO3/rust-numpy.git
$ cd rust-numpy
$ uv venv
$ . .venv/bin/activate
(rust-numpy) $ uv pip install numpy
Resolved 1 package in 40ms
Installed 1 package in 19ms
 + numpy==2.4.0
(rust-numpy) $ cargo test --test borrow
[…]
test resize_using_exclusive_borrow ... FAILED
[…]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: AlreadyBorrowed
[…]

failures:

---- resize_using_exclusive_borrow stdout ----

thread 'resize_using_exclusive_borrow' (755936) panicked at tests/borrow.rs:347:41:
called `Result::unwrap()` on an `Err` value: PyErr { type: <class 'ValueError'>, value: ValueError('cannot resize an array that references or is referenced\nby another array in this way.\nUse the np.resize function or refcheck=False'), traceback: None }


failures:
    resize_using_exclusive_borrow

test result: FAILED. 19 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s

error: test failed, to rerun pass `--test borrow`
(rust-numpy) $ cargo test --doc
[…]
failures:

---- src/borrow/mod.rs - borrow::PyReadwriteArray<'py,T,Ix1>::resize (line 611) stdout ----
Test executable failed (exit status: 101).

stderr:

thread 'main' (756617) panicked at src/borrow/mod.rs:13:39:
called `Result::unwrap()` on an `Err` value: PyErr { type: <class 'ValueError'>, value: ValueError('cannot resize an array that references or is referenced\nby another array in this way.\nUse the np.resize function or refcheck=False'), traceback: None }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace



failures:
    src/borrow/mod.rs - borrow::PyReadwriteArray<'py,T,Ix1>::resize (line 611)

test result: FAILED. 54 passed; 1 failed; 1 ignored; 0 measured; 0 filtered out; finished in 2.57s

error: doctest failed, to rerun pass `--doc`

[jakelishman]

I believe this is specific to Python 3.14 with the introduction of the new PyUnstable_Object_IsUniquelyReferenced, which Numpy uses as of Numpy 2.4. The previous implementation of refcheck in NumPy allowed two references to exist, I believe assuming that one was held in Python space in the Python interface passing it down to the C. The test writes the check as

let array = PyArray::<f64, _>::zeros(py, 3, false);
assert_eq!(array.shape(), [3]);

let mut array = array.readwrite();
assert_eq!(array.as_slice_mut().unwrap(), &[0.0; 3]);

let mut array = array.resize(5).unwrap();
assert_eq!(array.as_slice_mut().unwrap(), &[0.0; 5]);

If I insert a let array2 = array.clone(); after the first creation, I can reproduce the same failure on other versions of Python or previous versions of Numpy.

[Icxolu]

I think you are right. This looks like the numpy change: numpy/numpy@0957720

I believe this just shows that the test is slightly wrong in that we should drop the bound of the initial array after we created the readwrite guard.

let array = PyArray::<f64, _>::zeros(py, 3, false);
assert_eq!(array.shape(), [3]);

let mut array = {
    let guard = array.readwrite();
    drop(array);
    guard
};
assert_eq!(array.as_slice_mut().unwrap(), &[0.0; 3]);

let mut array = array.resize(5).unwrap();
assert_eq!(array.as_slice_mut().unwrap(), &[0.0; 5]);

cc @ngoldbaum Since you authored that change, maybe you can confirm that this is correct?

[jakelishman]

I'd considered making a patch to that effect, but I worry that the Numpy change implies a big caveat in the use of PyReadwriteArrayMethods::resize that will have knock-on effects for downstream users.

Perhaps we could ease the user pain by introducing a PyArray::into_readwrite that avoids the Bound::clone?

[ngoldbaum]

I agree that the original test was buggy. If you created that reference in Python then it wouldn't have been possible. Because you created it in an extension, the refcheck passed spuriously.

Now, NumPy is using a more stringent check based on new facilities in the CPython C API for this. It's correctly catching that the resized array is not uniquely referenced in the test.

[jakelishman]

I've opened #524 as an exploratory version of the fix, where the introduction of the try_into_readwrite method makes it rather simpler to write the test, as a proxy for the user experience of using resize with these changes.

[Icxolu]

Perhaps we could ease the user pain by introducing a PyArray::into_readwrite that avoids the Bound::clone?

I was also wondering that. However resize is kind of unique in that it requires a unique reference and a unique borrow is not sufficient. into_readwrite makes writing the test much nicer, but in the general case does not guarantee success. The initial array could already be cloned earlier in either Rust or Python increasing its refcount above 1 and failing to resize. Maybe it still worth adding, because the naive/easy way is always wrong?

[jakelishman]

There's a potential advantage to into_* anyway because of the lack of refcount update, which may make it worth it on its own?