Automate Ubuntu dependency baseline generation by Aayushman-nvm · Pull Request #824 · precice/precice.github.io

Aayushman-nvm · 2026-02-28T11:52:40Z

Automate Ubuntu dependency baseline generation

addresses #617

Approach

Repology's API is rate-limited and the dumps are too large (noted in the issue), so this uses Docker + apt-cache policy to query package versions directly from Ubuntu's apt repositories instead.

What's included

scripts/dependencies.yml — package list (required/optional), derived from the Ubuntu system guide
scripts/generate_ubuntu_baseline.py — queries each package version via Docker and writes _data/ubuntu-baseline.yml. Run manually at release time with python3 scripts/generate_ubuntu_baseline.py --lts 22.04 24.04. A --mock flag is available to test without Docker.
_data/ubuntu-baseline.yml — generated output, committed so Jekyll can reference it at build time
.github/workflows/generate-baseline.yml — workflow_dispatch-only workflow for auto regenerating and adding ubuntu-baseline.yml list locally

How I tested

Ran debug.yml (a temporary workflow, not included in this PR) on the issue617 branch to verify all packages in dependencies.yml are actually available and resolvable on both Ubuntu 22.04 and 24.04 via apt.
Ran the generation script locally with Docker to produce the committed ubuntu-baseline.yml and confirmed the versions match what apt-cache policy reports inside the containers.

Notes

The package list in dependencies.yml can be updated independently of the script — easy to add or remove packages without touching any logic.
Page integration content/docs/installation/building-from-source/installation-source-dependencies is intentionally left untouched for now. Happy to wire the data file into the dependencies page however you'd prefer.

Screenshots:
Core:

Autogenerated:

precice-bot · 2026-03-16T20:13:13Z

This pull request has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/introduction-website-modernization/2852/1

MakisH

Thank you for contributing! We are still discussing in #617, but this PR already does something useful in the respective direction, and I think we can merge it independently of the conclusion.

This PR addresses #617 by automatically generating a list of relevant packages provided in an Ubuntu release with a Python script. A manually-triggered CI workflow executes that script and checks the respective YAML file into this repository.

The Python script executes an apt-cache policy for each package in a Docker container, which might be a bit overkill, but works. Inputs are specified in dependency.yml, the output is saved into _data/ubuntu-baseline.yml. This PR already adds such a file, which could in principle then be rendered on the website.

See some suggestions, which should all be easy to implement.

MakisH · 2026-05-05T15:51:37Z

+import subprocess
+import yaml
+import argparse
+from pathlib import Path
+from datetime import datetime, UTC


This Python script would still need a requirements.txt defining PyYAML as a dependency (probably PyYAML==6). And then change the CI to replace pip install PyYAML with pip install -r requirements.txt.

MakisH · 2026-05-05T15:55:29Z

+def get_version_mock(package):
+    return "0.0.0-mock"


What is the use case for this one? It just assigns 0.0.0-mock to every dependency.

MakisH · 2026-05-05T16:00:23Z

+        f"ubuntu:{ubuntu_version}",
+        "bash",
+        "-c",
+        f"apt-get update -qq && apt-cache policy {package} | grep Candidate | awk '{{print $2}}'",


While this will give the package name, which includes the respective upstream version, it also includes other elements, such as the Debian package revision. For example, 1.83.0.1ubuntu2 or 2.9.14+dfsg.

I am not sure if these would be useful for the user on the website, or rather confusing. In principle, the first three digits should work (all these packages use semantic versioning).

MakisH · 2026-05-05T16:02:40Z

+          git config user.name github-actions
+          git config user.email github-actions@github.com


In other workflows (update-submodules.yml and update-discourse-data.yml), we use:

Suggested change

git config user.name github-actions

git config user.email github-actions@github.com

git config --local user.name "precice-bot"

git config --local user.email "info@precice.org"

MakisH · 2026-05-05T16:04:20Z

+          python3 scripts/generate_ubuntu_baseline.py \
+            --lts 22.04 24.04


Since precice/precice#2564, the baseline has shifted to 24.04, and we have added 26.04. For the current version (3.4.x), we still need to keep 22.04 listed.

Let's make this:

Suggested change

python3 scripts/generate_ubuntu_baseline.py \

--lts 22.04 24.04

python3 scripts/generate_ubuntu_baseline.py \

--lts 22.04 24.04 26.04

MakisH · 2026-05-05T16:04:57Z

+
+jobs:
+  generate:
+    if: github.actor != 'github-actions[bot]'


Why this line?

MakisH · 2026-05-05T16:05:38Z

+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4


There is already a v6:

Suggested change

- uses: actions/checkout@v4

- uses: actions/checkout@v6

MakisH · 2026-05-05T16:20:28Z

Ideally, I would put this script (everything you currently have into scripts/) into a directory specific to this script (scripts/baseline/).

MakisH · 2026-05-05T16:22:00Z

+    parser.add_argument(
+        "--lts",
+        nargs="+",
+        required=True,
+        help="Ubuntu LTS versions to query (e.g. 22.04 24.04)",
+    )


Why restrict that to LTS releases? I would rather name the option --release. For example, --lts 25.10 also works, even though that is not an LTS release.

MakisH · 2026-05-05T16:25:14Z

+        "-c",
+        f"apt-get update -qq && apt-cache policy {package} | grep Candidate | awk '{{print $2}}'",
+    ]
+    raw = run_command(cmd)


Before running this command, I would expect some output with the command to run, especially since this might take a while. This would also be useful for debugging and for understanding what this script does.

fsimonis

As described in the issue, we have an API key for pkgs.org.
There is no reason to probe a Docker container to get the package versions.

Also, the distinction between optional and required packages is unnecessary. It is handled where we display the information.

Furthermore, we are interested in the version of the packaged software, not the package. Meaning that we want 1.83.0, not 1.83.0.1ubuntu2.

Aayushman-nvm and others added 13 commits February 28, 2026 09:32

Add baseline generation tooling

af45b7b

Trigger debug on push

5542513

Enable push trigger

c2ad8b8

Enable push trigger

ea4e00e

Update Ubuntu baseline

f4188ac

Fix dependency grouping logic

f8ba860

Update Ubuntu baseline

6efc1ed

Upstream semantic version

8b522db

Update Ubuntu baseline

7e3cc5b

Fix: preCICE core dependencies

995ed6f

Fix: preCICE core dependencies

0ee3dec

Update Ubuntu baseline

81d1815

Fix: Automate Ubuntu dependency baseline generation

f3ca2f5

Aayushman-nvm mentioned this pull request Feb 28, 2026

Automate dependency page #617

Open

MakisH added technical Technical issues on the website GSoC Contributed in the context of the Google Summer of Code labels Mar 2, 2026

MakisH changed the title ~~Automate Ubuntu dependency baseline generation (closes #617)~~ Automate Ubuntu dependency baseline generation Mar 14, 2026

MakisH requested changes May 5, 2026

View reviewed changes

Merge branch 'master' into issue617

34a61e5

MakisH reviewed May 5, 2026

View reviewed changes

fsimonis requested changes May 6, 2026

View reviewed changes

		git config user.name github-actions
		git config user.email github-actions@github.com

		python3 scripts/generate_ubuntu_baseline.py \
		--lts 22.04 24.04

Uh oh!

Conversation

Aayushman-nvm commented Feb 28, 2026 • edited by MakisH Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Automate Ubuntu dependency baseline generation

Approach

What's included

How I tested

Notes

Uh oh!

precice-bot commented Mar 16, 2026

Uh oh!

MakisH left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MakisH May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fsimonis left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Aayushman-nvm commented Feb 28, 2026 •

edited by MakisH

Loading

MakisH May 5, 2026 •

edited

Loading