GPU docker builds are failing

[jrbourbeau]

Here's an example build https://github.com/coiled/examples/actions/runs/14764184328/job/41451797465?pr=52

Things are failing when we attempt to install the GPU version of pytorch

--------------------
  10 |     COPY pytorch.yml pytorch.yml
  11 | >>> RUN mamba env update -n base --file pytorch.yml \
  12 | >>>     && conda clean -afy \
  13 | >>>     && mamba uninstall -y pytorch torchvision \
  14 | >>>     && mamba install -y -n base -c pytorch -c nvidia -c conda-forge \
  15 | >>>         "cudatoolkit=${CUDA_VER%.*}.*" \
  16 | >>>         "cuda-version=${CUDA_VER%.*}.*" \
  17 | >>>         pytorch \
  18 | >>>         torchvision \
  19 | >>>         "pytorch-cuda=${CUDA_VER%.*}.*" \
  20 | >>>     && conda clean -afy \
  21 | >>>     && rm pytorch.yml
  22 |     
--------------------
ERROR: failed to solve: process "/bin/bash -euo pipefail -c mamba env update -n base --file pytorch.yml     && conda clean -afy     && mamba uninstall -y pytorch torchvision     && mamba install -y -n base -c pytorch -c nvidia -c conda-forge         \"cudatoolkit=${CUDA_VER%.*}.*\"         \"cuda-version=${CUDA_VER%.*}.*\"         pytorch         torchvision         \"pytorch-cuda=${CUDA_VER%.*}.*\"     && conda clean -afy     && rm pytorch.yml" did not complete successfully: exit code: 1

Just a hunch but there have been a lot of changes to how pytorch is distributed with conda lately. My guess is we need to install everything from conda-forge now.

[dchudz]

James, what's the impact here? Means our example is broken? (The CI job is for checking that the example works?) (Or is the docker build here something that users rely on to run the example?)

…

On Wed, Apr 30, 2025 at 5:11 PM James Bourbeau ***@***.***> wrote: *jrbourbeau* created an issue (coiled/examples#53) <#53> Here's an example build https://github.com/coiled/examples/actions/runs/14764184328/job/41451797465?pr=52 Things are failing when we attempt to install the GPU version of pytorch -------------------- 10 | COPY pytorch.yml pytorch.yml 11 | >>> RUN mamba env update -n base --file pytorch.yml \ 12 | >>> && conda clean -afy \ 13 | >>> && mamba uninstall -y pytorch torchvision \ 14 | >>> && mamba install -y -n base -c pytorch -c nvidia -c conda-forge \ 15 | >>> "cudatoolkit=${CUDA_VER%.*}.*" \ 16 | >>> "cuda-version=${CUDA_VER%.*}.*" \ 17 | >>> pytorch \ 18 | >>> torchvision \ 19 | >>> "pytorch-cuda=${CUDA_VER%.*}.*" \ 20 | >>> && conda clean -afy \ 21 | >>> && rm pytorch.yml 22 | -------------------- ERROR: failed to solve: process "/bin/bash -euo pipefail -c mamba env update -n base --file pytorch.yml && conda clean -afy && mamba uninstall -y pytorch torchvision && mamba install -y -n base -c pytorch -c nvidia -c conda-forge \"cudatoolkit=${CUDA_VER%.*}.*\" \"cuda-version=${CUDA_VER%.*}.*\" pytorch torchvision \"pytorch-cuda=${CUDA_VER%.*}.*\" && conda clean -afy && rm pytorch.yml" did not complete successfully: exit code: 1 Just a hunch but there have been a lot of changes to how pytorch is distributed with conda lately. My guess is we need to install everything from conda-forge now. — Reply to this email directly, view it on GitHub <#53>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJKQRXKMPFM4TXZTNQTZNL24E37TAVCNFSM6AAAAAB4GULGOGVHI2DSMVQWIX3LMV43ASLTON2WKOZTGAZTENJSGY4DQMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- <https://coiled.io/> David Chudzicki Director of Engineering

[jrbourbeau]

It means the coiled/gpu and coiled/gpu-examples Docker images are no longer getting updated. I'm not aware of coiled/gpu being used anywhere anymore. coiled/gpu-examples only gets used (to my knowledge) in the GPU notebook example on the "Get Started" page in the Coiled UI.

I just tried running the notebook command we point to there

coiled notebook start --vm-type g5.xlarge --container coiled/gpu-examples:latest --region us-west-2

which launches the notebook successfully. But when I then try running the pytorch notebook in that environment things broke (the error I'm seeing has been fixed in a more recent version of the Coiled client).

So overall not a large impact, but worth fixing. I'll look into this today.