feat: adopt mitodl/ol-python-base:3.11, BuildKit cache mounts, granian by blarghmatey · Pull Request #3655 · mitodl/mitxonline

blarghmatey · 2026-06-09T16:18:43Z

What are the relevant tickets?

N/A

Description (What does it do?)

⚠️ Draft — blocked on mitodl/ol-infrastructure#4750 merging and mitodl/ol-python-base:3.11 being published to DockerHub.

Adopts the new shared Python base image, adds BuildKit cache mounts, and migrates from uWSGI to granian (WSGI interface).

Changes:

FROM mitodl/ol-python-base:3.11 replaces the duplicated substrate stages. The shared base is defined in https://github.com/mitodl/ol-infrastructure/tree/main/dockerfiles/ol-python-base.
App-specific extras (libcairo2-dev, dnsutils, wget, operator tooling) installed with --mount=type=cache for apt.
--mount=type=cache,target=/opt/uv-cache,uid=1000,gid=1000 on uv sync for fast uv.lock-bump rebuilds.
CMD changed from uwsgi / uwsgi.ini to granian --interface wsgi main.wsgi:application on port 8013. uwsgi.ini is retained for reference but no longer used by the django-server/production targets.
The corresponding local-dev deployment.yaml is updated in feat: add ol-python-base shared image and migrate local-dev to granian ol-infrastructure#4750.
apt.txt trimmed to extras only with comments documenting what moved to the shared base.
Multi-stage targets (django-server, production, jupyter-notebook) preserved.

Image size (django-server target): 4.09 GB → 2.16 GB (−1.93 GB).

How can this be tested?

After mitodl/ol-infrastructure#4750 merges and the Concourse pipeline publishes the base image:

DOCKER_BUILDKIT=1 docker build --target django-server -t mitodl/mitxonline-app:test .
docker run --rm mitodl/mitxonline-app:test granian --version

In the local-dev Tilt environment: tilt up mitxonline, confirm the webapp comes up healthy under granian and the readiness probe at /health/readiness passes.

Additional Context

granian was already present as a dependency (granian>=2.5.4,<3 in pyproject.toml) but the Dockerfile and deployment were still running uWSGI. This PR completes the migration. Merge order: ol-infrastructure PR first → Concourse pipeline registered → this PR can be merged.

github-actions · 2026-06-09T16:19:05Z

OpenAPI Changes

Show/hide ## Changes for v0.yaml:

## Changes for v0.yaml:
No changes detected

## Changes for v1.yaml:
No changes detected

## Changes for v2.yaml:
No changes detected

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

Copilot

Pull request overview

This PR updates the container build/runtime setup for MITx Online by switching to the shared mitodl/ol-python-base:3.11 image, adding BuildKit cache mounts for faster rebuilds, and changing the default runtime command from uWSGI to granian.

Changes:

Adopt mitodl/ol-python-base:3.11 as the base stage and install only app-specific apt extras.
Add BuildKit cache mounts for both apt-get and uv sync to speed up iterative builds.
Update the django-server target to run granian --interface wsgi main.wsgi:application on port 8013.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
Dockerfile	Switches to shared base image, adds BuildKit cache mounts, and changes the runtime CMD to granian.
apt.txt	Trims/annotates the apt package list to app-specific extras intended for the new base image approach.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+# App-specific apt extras; common-core packages are in mitodl/ol-python-base:3.11.
+# Operator tooling (htop, ngrep, screen, etc.) is also kept here for parity
+# with the production image; trim this list if the production image diverges.
+RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
+    --mount=type=cache,target=/var/lib/apt,sharing=locked \
+    apt-get update && \
+    apt-get install -y --no-install-recommends \
+      dnsutils \
+      htop \
+      iputils-ping \
+      less \
+      libcairo2-dev \
+      lsof \
+      nano \
+      ngrep \
+      procps \
+      screen \
+      wget


+COPY pyproject.toml uv.lock /src/
+COPY mitol_*.gz /src/
+RUN chown -R mitodl:mitodl /src

 USER mitodl
 WORKDIR /src


 EXPOSE 8013
 ENV PORT=8013
-CMD ["uwsgi", "uwsgi.ini"]
+CMD ["granian", "--interface", "wsgi", "--host", "0.0.0.0", "--port", "8013", "--workers", "2", "main.wsgi:application"]


+# BuildKit cache mount keeps the uv download cache across builds.
+RUN --mount=type=cache,target=/opt/uv-cache,uid=1000,gid=1000 \
+    uv sync --frozen --no-install-project --group prod


blarghmatey

Addressing the bot review comments:

/var/media ownership (comment on line 78): no change needed. The base image (mitodl/ol-python-base:3.11) already runs mkdir -p /src /opt/venv /var/media && chown -R mitodl:mitodl /src /opt/venv /var/media, so the directory is created with the correct ownership before our layers run.
apt.txt two sources of truth: fixed — apt.txt has been deleted; the Dockerfile is now the single source of truth for app-specific packages.
ENV PORT ignored by granian CMD: fixed — CMD now uses shell form exec granian ... --port ${PORT:-8013} ... so the port can be overridden at runtime.
uv sync --group prod installs uwsgi/uwsgitop: fixed — removed the prod dependency group from pyproject.toml (and dropped --group prod from the Dockerfile uv sync call). uv.lock updated.
COPY certs/*.crt fails when no .crt files exist: fixed — changed to COPY --chmod=644 certs/ /usr/local/share/ca-certificates/ (copies the whole directory; update-ca-certificates no-ops if there are no .crt files).

rhysyngsun

The nginx configuration needs to be updated to route to granian instead of uwsgi.

https://github.com/mitodl/mitxonline/blob/main/config/nginx.conf.erb

blarghmatey · 2026-06-18T20:34:48Z

Is that template still used at all? My recollection is that the .erb config is a Heroku-only artifact that we should just delete.

rhysyngsun · 2026-06-18T20:55:46Z

It is, that's the template used locally for nginx too, I normalized this project to use a single nginx configuration for heroku and localdev so we didn't have drift between the two.

rhysyngsun · 2026-06-18T20:58:58Z

Here's my nginx logs showing that file getting compiled and the resulting 502 when it tries to connect to the upstream via uwsgi:

nginx-1  | 10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
nginx-1  | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
nginx-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-compile-nginx-conf-erb.sh
nginx-1  | 20-compile-nginx-conf-erb.sh: Running erb on /etc/nginx/templates/nginx.conf.erb to /etc/nginx/nginx.conf
nginx-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx-1  | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx-1  | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx-1  | 2026/06/18 20:57:07 [error] 29#29: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.21.0.9, server: _, request: "GET / HTTP/1.1", upstream: "uwsgi://172.21.0.7:8011", host: "mitxonline.odl.local"
nginx-1  | "18/Jun/2026:20:57:07 +0000" client=172.21.0.9 method=GET request="GET / HTTP/1.1" request_length=828 status=502 bytes_sent=300 body_bytes_sent=150 referer=- user_agent="Mozilla/5.0 (X11; Linux x86_64; rv:151.0) Gecko/20100101 Firefox/151.0" upstream_addr=172.21.0.7:8011 upstream_status=502 request_time=0.000 upstream_response_time=0.001 upstream_connect_time=- upstream_header_time=-

blarghmatey · 2026-06-23T19:01:43Z

Fixed in 4e828ba. Replaced uwsgi_pass / uwsgi_param block with proxy_pass http://... and standard HTTP proxy headers in config/nginx.conf.erb, and renamed NGINX_UWSGI_PASS → NGINX_UPSTREAM in docker-compose.yml to match.

rhysyngsun · 2026-06-24T21:11:20Z

I'm still getting a 502 error because the docker-compose.yml configuration overrides the command to use scripts/run-django-dev.sh which is hardcoded to run uwsgi still. I also noticed that uwsgi.ini is still in the root directory and that can be nuked.

rhysyngsun

LGTM

- FROM mitodl/ol-python-base:3.11 replaces the duplicated substrate stages. - App-specific extras (libcairo2-dev, dnsutils, wget, operator tooling) installed with BuildKit --mount=type=cache for apt. - BuildKit --mount=type=cache on uv sync (target=/opt/uv-cache uid=1000). - CMD changed from uwsgi / uwsgi.ini to granian --interface wsgi main.wsgi:application on port 8013. uwsgi.ini is retained for reference but no longer used. - apt.txt trimmed to extras only with comments documenting what moved to the shared base. - Image size (django-server target): 4.09 GB → 2.16 GB (-1.93 GB).

granian speaks HTTP, not the uWSGI binary protocol. Replace the uwsgi_pass block with proxy_pass and standard HTTP proxy headers. Rename NGINX_UWSGI_PASS -> NGINX_UPSTREAM in docker-compose.yml to match the new variable name in the template. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

uwsgi is no longer installed in the image; run-django-dev.sh still called uwsgi.ini which would fail on container start. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

nginx.conf.erb was already switched to proxy_pass http://localhost:8013 but the Procfile still ran uwsgi (which uses a unix socket, not TCP). Without this change the Heroku deployment would fail — nginx routing to port 8013 with nothing listening there. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The deps stage inherited default-groups = "all" from pyproject.toml, causing dev tools (pytest, ruff, ipdb, etc.) to be installed into the production image. Fix: add --no-dev to the production uv sync; add a development stage extending django-server that runs uv sync without --no-dev to layer in the dev group for local docker-compose workflows. Also: use COPY --chown instead of COPY + RUN chown to avoid an extra layer; update docker-compose to target the new development stage. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

blarghmatey marked this pull request as ready for review June 18, 2026 13:55

blarghmatey requested a review from Copilot June 18, 2026 13:55

Copilot started reviewing on behalf of blarghmatey June 18, 2026 13:55 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

blarghmatey force-pushed the tmacey/ol-python-base-image branch from 5a889b8 to 8f6b3ff Compare June 18, 2026 14:53

blarghmatey commented Jun 18, 2026

View reviewed changes

blarghmatey force-pushed the tmacey/ol-python-base-image branch from 8f6b3ff to 4aaefd2 Compare June 18, 2026 15:02

rhysyngsun self-assigned this Jun 18, 2026

rhysyngsun requested changes Jun 18, 2026

View reviewed changes

sentry Bot reviewed Jun 25, 2026

View reviewed changes

Comment thread Dockerfile Outdated

rhysyngsun approved these changes Jun 25, 2026

View reviewed changes

blarghmatey and others added 5 commits June 25, 2026 12:08

fix: switch dev startup script from uwsgi to granian

195290a

uwsgi is no longer installed in the image; run-django-dev.sh still called uwsgi.ini which would fail on container start. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

blarghmatey force-pushed the tmacey/ol-python-base-image branch from ffff114 to b5a96c9 Compare June 25, 2026 16:11

blarghmatey merged commit cbba19e into main Jul 1, 2026
12 checks passed

blarghmatey deleted the tmacey/ol-python-base-image branch July 1, 2026 10:24

odlbot mentioned this pull request Jul 1, 2026

Release 1.158.8 #3721

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: adopt mitodl/ol-python-base:3.11, BuildKit cache mounts, granian#3655

feat: adopt mitodl/ol-python-base:3.11, BuildKit cache mounts, granian#3655
blarghmatey merged 5 commits into
mainfrom
tmacey/ol-python-base-image

blarghmatey commented Jun 9, 2026

Uh oh!

github-actions Bot commented Jun 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

blarghmatey left a comment

Uh oh!

rhysyngsun left a comment

Uh oh!

blarghmatey commented Jun 18, 2026

Uh oh!

rhysyngsun commented Jun 18, 2026

Uh oh!

rhysyngsun commented Jun 18, 2026

Uh oh!

blarghmatey commented Jun 23, 2026

Uh oh!

rhysyngsun commented Jun 24, 2026

Uh oh!

Uh oh!

rhysyngsun left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

blarghmatey commented Jun 9, 2026

What are the relevant tickets?

Description (What does it do?)

How can this be tested?

Additional Context

Uh oh!

github-actions Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

OpenAPI Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

blarghmatey left a comment

Choose a reason for hiding this comment

Uh oh!

rhysyngsun left a comment

Choose a reason for hiding this comment

Uh oh!

blarghmatey commented Jun 18, 2026

Uh oh!

rhysyngsun commented Jun 18, 2026

Uh oh!

rhysyngsun commented Jun 18, 2026

Uh oh!

blarghmatey commented Jun 23, 2026

Uh oh!

rhysyngsun commented Jun 24, 2026

Uh oh!

Uh oh!

rhysyngsun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jun 9, 2026 •

edited

Loading