WIP: Grudge array context

[nchristensen]

Needs inducer/meshmode#76

[nchristensen]

Somehow these three tests cause threads to crash in the CI. Any ideas what is happening here @inducer? The tests appear to run fine on Koelsch.

=================================== FAILURES ===================================
_____________________________ test/test_grudge.py ______________________________
[gw3] linux -- Python 3.8.6 /home/runner/work/grudge/grudge/.miniforge3/envs/testing/bin/python3
worker 'gw3' crashed while running "test/test_grudge.py::test_mass_surface_area[<array context factory for <pyopencl.Device 'pthread-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz' on 'Portable Computing Language'>-box3d]"
_____________________________ test/test_grudge.py ______________________________
[gw2] linux -- Python 3.8.6 /home/runner/work/grudge/grudge/.miniforge3/envs/testing/bin/python3
worker 'gw2' crashed while running "test/test_grudge.py::test_tri_diff_mat[<array context factory for <pyopencl.Device 'pthread-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz' on 'Portable Computing Language'>-3]"
_____________________________ test/test_grudge.py ______________________________
[gw1] linux -- Python 3.8.6 /home/runner/work/grudge/grudge/.miniforge3/envs/testing/bin/python3
worker 'gw1' crashed while running "test/test_grudge.py::test_convergence_maxwell[<array context factory for <pyopencl.Device 'pthread-Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz' on 'Portable Computing Language'>-4]"

[nchristensen]

It may be a memory issue. If I set ulimit -S -v 1000000 it fails in a similar manner on Koelsch.

[nchristensen]

It may be a memory issue. If I set ulimit -S -v 1000000 it fails in a similar manner on Koelsch.

As it turns out all of the tests fail with this ulimit so this isn't a good way to replicate the problem. Is there a way to run interactively in the CI environment @inducer?

[nchristensen]

I may have just stumbled on the problem. Let me check more.

[nchristensen]

It appears to be a problem with POCL in particular. The old AMD CPU backend works fine but the POCL gives a bus error and abandons the test.

edit: The Intel OpenCL implementation fails on the same tests but does not terminate the tests after the first failure.

[nchristensen]

It appears to be a problem with POCL in particular. The old AMD CPU backend works fine but the POCL gives a bus error and abandons the test.

edit: The Intel OpenCL implementation fails on the same tests but does not terminate the tests after the first failure.

The PoCL and Intel failures seemed to be due to over-allocation of local memory; apparently local and private memory are the same in these backends.

Unfortunately tests still break in CI but for different reasons than before.

[nchristensen]

We may want to consider making a GPU available for CI.