Resumable downloads

[ericcurtin]

It's frustrating to have to start a download from scratch again:

bash-3.2$ docker model run hf.co/allenai/OLMoE-1B-7B-0924-Instruct-GGUF:Q4_K_M
Unable to find model 'hf.co/allenai/OLMoE-1B-7B-0924-Instruct-GGUF:Q4_K_M' locally. Pulling from the server.
Downloaded: 2993.61 MB
Failed to pull model: error pulling model: Error: writing blob: copy blob "sha256:8c310f1435a1222338fd2d3d974975be9cd908180b644bab0c2a94da1ac32f3f" to store: stream error: stream ID 1; INTERNAL_ERROR; received from peer
bash-3.2$ docker model run hf.co/allenai/OLMoE-1B-7B-0924-Instruct-GGUF:Q4_K_M
Unable to find model 'hf.co/allenai/OLMoE-1B-7B-0924-Instruct-GGUF:Q4_K_M' locally. Pulling from the server.
Downloaded: 441.57 MB

[ericcurtin]

The joys of being on not so reliable wifi, on my third attempt 😄

[xenoscopic]

I'll start working on some improvements here this week. We think it has less to do with the client side and more to do the CDN (and we've tried several different options). Or it might be the Go HTTP/2 stack - we haven't found the smoking gun yet. Right now the go-containerregistry package's abstractions don't give us much control over individual requests (e.g. Range isn't accessible to us) - the only thing we control is the net/http.Transport used, so we'll have to do something at that layer. Auto-resumption and concurrent downloads are on the roadmap. In terms of doing something higher-level (like storing partially downloaded blobs in case connectivity is fully disrupted), that's probably worth doing too, but having a Transport implementation that can use that is probably a pre-req.

[ericcurtin]

Here is another http stack to compare against:

https://github.com/ericcurtin/lm-pull/blob/main/lm-pull.py

Resumable huggingface downloads can be seen by doing:

./lm-pull.py hf://ggml-org/gemma-3-270m-GGUF/gemma-3-270m-Q8_0.gguf

Pressing ctrl-c and starting again.

Can pull Ollama ggufs also.

Dockerhub pulls aren't implemented here (but I think we could add if we just wanted something to compare against, Dockerhub and Ollama transport aren't completely different), not even sure they existed when this was written.

There's a C++ version in that repo too, using libcurl http stack.

It's tempting to port this progress bar to golang also for Docker Model Runner, shows more info than the current one. It's already been ported to python3 and C++

[ericcurtin]

@xenoscopic implemented resumable downloads with dockerhub but in python/c++, just to see if server-side supports it, it does:

ericcurtin/lm-pull@82c7a68

./lm-pull.py docker://ai/smollm2:135M-Q4_K_M

[xenoscopic]

I've got a transport-layer resumption implementation pending here: docker/model-distribution#127

We'll still do something at the application-layer later, so this doesn't satisfy the needs here, but it should smooth over a lot of the INTERNAL_ERROR issues (which we are also working to debug from the Hub side).

[ericcurtin]

I did implement a dockerhub puller directly in llama.cpp also :

ggml-org/llama.cpp#15790

it will likely never be as detailed as the docker model runner one or support things like pushing. But it's useful for testing. To be honest I want to enlighten people that Dockerhub is an option for storing models.

I was discussing AnythingLLM Android App with one of the founders and I asked would it be possible to pull models from Dockerhub and he said no because it wouldn't be possible to port Docker Engine to Android (which isn't necessary to pull models from Dockerhub).

I'm hoping that adding this feature will help open eyes around, pulling from Dockerhub and using Docker containers are two different things.

I don't think Docker Model Runner should use the llama.cpp version, it's not worth writing all this low-level code for http stuff.