diff --git a/.agents/skills/ams-build/SKILL.md b/.agents/skills/ams-build/SKILL.md new file mode 100644 index 00000000..26bc6aeb --- /dev/null +++ b/.agents/skills/ams-build/SKILL.md @@ -0,0 +1,169 @@ +--- +name: ams-build +description: >- + Build and install the AMS library (github.com/LLNL/AMS) from source on an HPC + cluster via CMake. Use this whenever the user wants to compile, configure, + build, or install AMS, or is choosing between build variants (with/without + RabbitMQ, GPU, MPI, Caliper, etc.). ALWAYS use this skill for AMS build/CMake + questions even when phrased loosely ("get AMS running on Tioga", "why can't + CMake find Torch", "AMS build with RabbitMQ"). It knows how to gather AMS's + dependencies two ways: via LLNL Livermore Computing's internal Spack + environment (the easy path on LC machines) or by pointing CMake at + manually-provided libraries on any other cluster. +--- + +# Installing AMS from source + +AMS is a C++ library built with CMake. The build itself is easy; the hard part +is **gathering the dependencies** so `find_package` succeeds. This skill's job +is to (1) figure out which dependency-provisioning path applies, and (2) hand +the user a correct CMake configure line for their chosen feature set. For help +regarding the installation process, please check `INSTALL.md`. + +## Where do the dependencies come from? + +Run the check and branch **before** writing any CMake command: + +- **On an LLNL Livermore Computing (LC) cluster** — the AMS team maintains a + prebuilt internal Spack environment. Source the repo's own + `scripts/gitlab/setup-env.sh`; it activates Spack, activates the per-host + environment, and exports `AMS_TORCH_PATH`, `AMS_HDF5_PATH`, + `AMS_CALIPER_PATH`, `AMS_AMQPCPP_PATH`, `AMS_CUDA_ARCH`, etc. Then you just + feed those into CMake. This is the path to prefer whenever it's available. + +- **On any other cluster** — Assume there is no shared environment unless + stated otherwise. + Each dependency must be installed (Spack, a module system, or by hand) + and CMake pointed at each one via its `*_DIR` config-package hint + or `CMAKE_PREFIX_PATH`. + See `references/manual-deps.md`. + +Detect LC by the presence of the internal environment directory: + +```bash +if [[ -d /usr/workspace/AMS/ams-spack-environments ]]; then + echo "LC cluster: use scripts/gitlab/setup-env.sh" +else + echo "Non-LC: provide dependencies manually (references/manual-deps.md)" +fi +``` + +`$SYS_TYPE` is a secondary signal on LC (`toss_4_x86_64_ib` = Dane/CTS-1, +`toss_4_x86_64_ib_cray` = Tuolumne/Tioga/El Capitan-class ROCm machines). + +## Dependencies at a glance (current `develop`) + +Always required: **nlohmann_json**, **HDF5** (C/CXX), **libTorch**, plus a C++17 +compiler and Threads. `fmt` and `tl::expected` are fetched automatically by +CMake — the user never installs them. + +Enabled only by a flag: **MPI** (`WITH_MPI`), **CUDA** (`WITH_CUDA`) or **HIP/ROCm** +(`WITH_HIP`), **Caliper** (`WITH_CALIPER`), **RabbitMQ** back end (`WITH_RMQ` → +pulls in `amqp-cpp`, `OpenSSL`, `libevent`), **PerfFlowAspect** +(`WITH_PERFFLOWASPECT`). + +The full option table, defaults, and the CMake hint variable for each package +are in `references/cmake-options.md`. Read it before answering flag-specific +questions — do not rely on the repo's `INSTALL.md`, which is stale (it lists +removed flags like `WITH_DB`, `WITH_TORCH`, `WITH_FAISS`, `WITH_EXAMPLES`). + +## Use cases (pick the smallest set that meets the need) + +1. **Minimal / CPU** — core library only. No MPI, no GPU, no RabbitMQ. + File-based (HDF5) storage. Good first build to prove the toolchain works. +2. **+ MPI** — add `-DWITH_MPI=On` for distributed applications. +3. **+ GPU** — add `-DWITH_CUDA=On` (NVIDIA) or `-DWITH_HIP=On` (AMD/ROCm). + Never both. On LC ROCm machines the setup script loads `rocm` and sets the + arch for you. +4. **+ RabbitMQ** — add `-DWITH_RMQ=On` to stream data to a running RabbitMQ + broker instead of (or alongside) HDF5 files. This is the "with RabbitMQ" + variant and requires a reachable broker at runtime; see the RMQ note below. +5. **+ profiling** — `-DWITH_CALIPER=On` and/or `-DWITH_PERFFLOWASPECT=On`. +6. **+ Python workflow drivers** — `-DWITH_WORKFLOW=On` installs the outer + `AMSWorkflow` Python drivers. + +These compose freely, e.g. MPI + CUDA + RabbitMQ for a distributed GPU run that +streams training data. + +## The workflow + +### Step 1 — clone + +```bash +git clone https://github.com/LLNL/AMS.git +cd AMS +``` + +### Step 2 — provision dependencies + +**LC cluster:** + +```bash +source scripts/gitlab/setup-env.sh # activates Spack + exports AMS_*_PATH +``` + +**Non-LC cluster:** install the required packages and export the hint +variables CMake expects (see `references/manual-deps.md`), e.g. + +```bash +export CMAKE_PREFIX_PATH=/path/to/installs:$CMAKE_PREFIX_PATH +export Torch_DIR=/path/to/libtorch/share/cmake/Torch +export AMS_HDF5_DIR=/path/to/hdf5 +``` + +### Step 3 — configure + +Use the helper script `scripts/ams-configure.sh` (it assembles the flags, maps the +`AMS_*_PATH` exports to the right `-D…_DIR` on LC, and prints the full command +before running). Examples: + +```bash +# Minimal CPU build +scripts/ams-configure.sh + +# With RabbitMQ + MPI +scripts/ams-configure.sh --rmq --mpi + +# GPU (CUDA) + Caliper, custom install prefix +scripts/ams-configure.sh --cuda --caliper --install-prefix $HOME/opt/ams + +# See the exact cmake line without running it +scripts/ams-configure.sh --rmq --mpi --dry-run +``` + +Or write the CMake command directly — see the annotated templates in +`INSTALL.md`. + +### Step 4 — build & install + +```bash +cmake --build build -j "$(nproc)" +cmake --install build # honors -DCMAKE_INSTALL_PREFIX +``` + +## RabbitMQ note + +`WITH_RMQ` compiles the AMQP client (`amqp-cpp` + `OpenSSL` + `libevent`). It +does **not** stand up a broker. At runtime AMS needs a reachable RabbitMQ +service and TLS/credentials config (host, port, vhost, cert). If the user only +wants local, file-based storage, leave `WITH_RMQ` off and AMS uses the HDF5 +back end — no broker required. + +## Common failure modes + +- **`Could NOT find Torch` / `HDF5` / `nlohmann_json`** — on LC you forgot to + `source scripts/gitlab/setup-env.sh` (or are on a login node with no + `$SYS_TYPE`). Off LC, the corresponding `*_DIR` / `CMAKE_PREFIX_PATH` isn't + set. Torch and HDF5 are mandatory on current `develop`, so a bare `cmake` + with no dependency hints will not configure. +- **`Could NOT find amqpcpp` / `libevent`** — only appears with `-DWITH_RMQ=On`; + provide `amqpcpp_DIR` (LC: `$AMS_AMQPCPP_PATH`) and libevent. +- **Both CUDA and HIP set** — CMake hard-errors; choose one. +- **C++ standard clashes** — AMS uses C++17; an application/Flux stack forcing + C++20 can conflict with the RMQ path. See the comment at the top of + `CMakeLists.txt`. + +## Files in this skill + +- `references/manual-deps.md` — how to obtain each dependency on a non-LC + cluster and which CMake variable points at it. diff --git a/.agents/skills/ams-build/agents/openai.yaml b/.agents/skills/ams-build/agents/openai.yaml new file mode 100644 index 00000000..41bdd5d2 --- /dev/null +++ b/.agents/skills/ams-build/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "AMS Build" + short_description: "Skill to build the AMS library" + default_prompt: "Use $ams-build to build AMS" diff --git a/.agents/skills/ams-build/references/manual-deps.md b/.agents/skills/ams-build/references/manual-deps.md new file mode 100644 index 00000000..434c3e15 --- /dev/null +++ b/.agents/skills/ams-build/references/manual-deps.md @@ -0,0 +1,75 @@ +# Providing AMS dependencies on a non-LC cluster + +Off the LLNL Livermore Computing systems there is no shared AMS Spack +environment, so you must provide each dependency yourself and point CMake at it. +Three broad approaches, roughly easiest → most manual: + +## Option A — Spack (recommended if available) + +If your site has Spack and an `ams` package in a reachable repo: + +```bash +spack install ams # resolves the whole dependency tree +spack load ams # or build against it as a dev package +``` + +For active development, `spack dev-build ams` builds from your working copy. +If your Spack does not have an `ams` package, install the dependencies +individually and use Option C: + +```bash +spack install nlohmann-json hdf5 py-torch # + amqp-cpp openssl libevent for RMQ +spack install caliper # if profiling +spack install mpi # if WITH_MPI (or use site MPI module) +``` + +Then locate each and pass its `*_DIR` (see the table below). + +## Option B — module system + +Many clusters expose these as modules: + +```bash +module load cmake gcc hdf5 cuda openmpi +``` + +libTorch and (usually) nlohmann_json are rarely modules — get those from +Option A or C. After loading, the module `*_ROOT`/`PATH` usually lets CMake find +the package; otherwise fall back to explicit `*_DIR` hints. + +## Option C — download / build by hand, then point CMake at each + +| Dependency | Required? | Where to get it | CMake hint | +|---|---|---|---| +| `nlohmann_json` | yes | header-only; package or GitHub release | `nlohmann_json_DIR=/lib/cmake/nlohmann_json` | +| `fmt` | yes | github.com/fmtlib/fmt | `fmt_DIR=/share` | +| `tl::expected` | yes | github.com/TartanLlama/optional | `tl_expected_DIR=/share` | +| HDF5 (C/CXX) | yes | hdfgroup.org, package manager, or Spack | `AMS_HDF5_DIR=` | +| libTorch | yes | pytorch.org "LibTorch" C++ zip (match CUDA/CPU) | `Torch_DIR=/share/cmake/Torch` | +| MPI | if `WITH_MPI` | OpenMPI/MPICH, or site module | found via compiler wrappers / `CMAKE_PREFIX_PATH` | +| CUDA | if `WITH_CUDA` | NVIDIA CUDA Toolkit | `CMAKE_CUDA_ARCHITECTURES=` | +| ROCm/HIP | if `WITH_HIP` | AMD ROCm install | set `ROCM_PATH`; HIP found via `CMAKE_PREFIX_PATH` | +| Caliper | if `WITH_CALIPER` | github.com/LLNL/Caliper | `caliper_DIR=/share/cmake/caliper` | +| amqp-cpp | if `WITH_RMQ` | github.com/CopernicaMarketingSoftware/AMQP-CPP | `amqpcpp_DIR=/cmake` | +| OpenSSL | if `WITH_RMQ` | system package | found via `OPENSSL_ROOT_DIR` / `CMAKE_PREFIX_PATH` | +| libevent | if `WITH_RMQ` | libevent.org | on `CMAKE_PREFIX_PATH` | +| PerfFlowAspect | if `WITH_PERFFLOWASPECT` | github.com/flux-framework/PerfFlowAspect | `perfflowaspect_DIR=/share` | + +## General technique + +Put all your hand-built installs under one prefix and add it to the search +path so most packages resolve without individual hints: + +```bash +export CMAKE_PREFIX_PATH=/opt/ams-deps:$CMAKE_PREFIX_PATH +``` + +Then supply explicit `*_DIR` only for the ones CMake still can't find — libTorch +almost always needs its explicit `Torch_DIR` because of its nested cmake path. + +## Sanity check before a full build + +Configure a **minimal CPU build first** (no MPI/GPU/RMQ). If that succeeds, the +mandatory trio (nlohmann_json, HDF5, Torch) is wired correctly and you can add +feature flags one at a time — much easier to localize a missing dependency than +debugging a fully-loaded configure line. diff --git a/.agents/skills/changelog/SKILL.md b/.agents/skills/changelog/SKILL.md new file mode 100644 index 00000000..0523e8ec --- /dev/null +++ b/.agents/skills/changelog/SKILL.md @@ -0,0 +1,105 @@ +--- +name: changelog +description: > + Maintain a CHANGELOG.md in Keep a Changelog format: record new features, changes, + and fixes under an Unreleased section, and cut releases. Use whenever a + user-facing change ships or the user says "update the changelog", "add a + changelog entry", "record this feature/fix", "cut a release", "what changed + since ", or mentions release notes or a CHANGELOG. Works for versioned and + unversioned projects — fall back to dates plus merge-request or commit IDs when + there are no version numbers. Trigger on any notable change worth recording, even + without the word "changelog". +--- + +# Changelog + +Maintain `CHANGELOG.md` at the repo root in **Keep a Changelog** format: a +human-readable, reverse-chronological list of notable changes, grouped by release. +A changelog is for humans — it is not a `git log` dump. The format follows +Keep a Changelog (https://keepachangelog.com). + +## Structure + +```markdown +# Changelog + +## [Unreleased] + +### Added +- User-facing description of a new feature (#123). + +## [1.2.0] - 2026-07-08 + +### Added +- ... +### Fixed +- ... +``` + +- Newest first. Keep an `## [Unreleased]` section at the top as a staging area. +- Dates are ISO 8601 (`YYYY-MM-DD`). +- Group each change under one of six headings; omit headings with no entries: + - **Added** — new features. + - **Changed** — changes to existing behavior. + - **Deprecated** — features slated for removal. + - **Removed** — features now removed. + - **Fixed** — bug fixes. + - **Security** — vulnerability fixes. + +Initialize on first use: if `CHANGELOG.md` is missing, create it with the header +above and an empty `## [Unreleased]`. Never overwrite existing history. + +## Adding an entry + +Add to `## [Unreleased]` when a change merges — not at release time — so nothing +is forgotten. + +1. Pick the right group heading (create it under Unreleased if absent). +2. Write one bullet per notable change in **plain, user-facing language**: what + changed and why it matters, not the commit subject. Rewrite + "fix: handle keydown in modal (#412)" as "Fixed the dialog not closing on Escape." +3. Reference the source for an audit trail: the PR/merge-request ID (`#123`, `!57`) + or a short commit SHA when there is no PR. Optionally lead with a bold name: + `- **CSV export:** feedback entries can now be exported to CSV (#234).` +4. Skip internal churn (refactors, formatting, test-only changes) unless it is + notable to users or integrators. + +Drafting from history is fine — inspect `git log ..HEAD` or the merged +PRs — but always curate and rewrite into user-facing wording; never paste raw +commit messages. + +## Cutting a release + +Move the `## [Unreleased]` entries into a new release section, then leave an empty +`## [Unreleased]` at the top. The release identifier is flexible: + +- **Versioned (SemVer):** `## [1.4.0] - YYYY-MM-DD`. Bump MAJOR for breaking + changes, MINOR for new features, PATCH for fixes. +- **Unversioned:** use a dated header, annotated with the merge request or commit + that marks the release point: + ``` + ## [2026-07-08] — mr !57 + ## [2026-07-08] — a1b2c3d + ``` + +Optionally, add comparison links at the bottom so headers are clickable. On a git +host these are compare URLs — by tag (`compare/v1.3.0...v1.4.0`) or, for +unversioned projects, by commit/MR range (`compare/...`): + +``` +[Unreleased]: https:////compare/...HEAD +[1.4.0]: https:////compare/v1.3.0...v1.4.0 +``` + +## Conventions + +- Write for readers who have never seen the code: no internal ticket shorthand or + component names without explanation. +- Be brief, we do not want the changelog to be millions of lines long. Not more than + one sentence for each change. +- Make "update the changelog" part of the definition of done — the person shipping + the change writes the entry, since they have the context. +- Do not reconstruct a changelog from memory; derive it from git history and curate. +- Do not update past entries of a changelog without very good reason and notify the user. +- If an architecture wiki is maintained (see the `codebase-map` skill), a change + notable enough for the changelog that also alters structure should update both. \ No newline at end of file diff --git a/.agents/skills/changelog/agents/openai.yaml b/.agents/skills/changelog/agents/openai.yaml new file mode 100644 index 00000000..bb65b823 --- /dev/null +++ b/.agents/skills/changelog/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Changelog" + short_description: "Skill to build and maintain a CHANGELOG file" + default_prompt: "Use $changelog to build and/or update the CHANGELOG of a repository" diff --git a/.agents/skills/codebase-map/SKILL.md b/.agents/skills/codebase-map/SKILL.md new file mode 100644 index 00000000..0c7940e2 --- /dev/null +++ b/.agents/skills/codebase-map/SKILL.md @@ -0,0 +1,217 @@ +--- +name: codebase-map +description: > + Build and maintain a living architecture wiki for a codebase: one markdown + article per module/subsystem plus an index that maps how they fit together. + Use whenever the user wants to document, map, or reason about the code's + architecture — "map the codebase", "update the architecture wiki", "what does + module X do", "how does X connect to Y", "reconstruct the architecture", or + onboarding to an unfamiliar repo. Trigger on any mention of an "architecture + map", "code wiki", "module map", or keeping architecture docs in sync with the + code, even without those exact words. +--- + +# Codebase architecture map + +Maintain a living wiki describing a codebase's architecture. The **code is the +source of truth** — read it, never invent — and the wiki is the compiled, +human-readable map you own and keep in sync. + +Principle: the LLM writes and maintains the map; the human reads it and asks +questions. Keep it at the architecture level — module responsibilities, interfaces, +and how things fit together — not a restatement of every function (that is what +the code and API docs are for). + +## Layout + +Under `architecture/` at the repo root (configurable; point elsewhere if it +clashes with generated docs): + +- `architecture/index.md` — the map: a header recording the commit the map was + last refreshed at, a short system overview, a module list with one-line + summaries grouped by layer/package (each row noting the commit that article + reflects), and a system-level Mermaid diagram of the main module dependencies / + data flow (see Diagrams below). +- `architecture/.md` — one article per module or subsystem. +- `architecture/log.md` — append-only log of updates. + +One level of articles only. A "module" is a package/directory or a cohesive +subsystem, not a single file. + +### Module article format + +```markdown +# + +- Path(s): `src/foo/`, `src/foo_utils.py` +- Reflects commit: Updated: YYYY-MM-DD + +## Purpose +What this module is responsible for, in 1-3 sentences. + +## Key files +- `path` — what it does. + +## Public interface +Classes / functions / endpoints other modules or users call into. + +## Depends on +- Internal: [Other Module](other-module.md) +- External: notable third-party libraries. + +## Used by +Modules that depend on this one (link them). + +## Data flow / interactions +What comes in, what goes out, how it talks to other modules. Add a Mermaid diagram +here when it makes the flow clearer than prose (see Diagrams below). + +## Gotchas / invariants +Non-obvious constraints, footguns, assumptions. +``` + +Initialize on first run: if missing, create `architecture/` with `index.md` +(heading `# Architecture Map`) and `log.md` (heading `# Architecture Log`). +Never overwrite existing files. + +## Recording the commit SHA + +Every write records the commit the repo was at, so you can tell which state of the +codebase the content was built from. Capture it once at the start of a Map or +Verify run: + +```bash +git rev-parse --short HEAD # e.g. a1b2c3d +git status --porcelain # if non-empty, the working tree is dirty +``` + +If the working tree has uncommitted changes, append `-dirty` (e.g. `a1b2c3d-dirty`) +so the SHA is not mistaken for a clean checkout; `git describe --always --dirty` is +a one-shot equivalent. Record the same value in three places: + +- each article's **Reflects commit** field — the state that article was written from; +- the **index header**, e.g. `Map reflects commit: — YYYY-MM-DD`, set to the + commit of the most recent Map run; +- every **log entry** (see below). + +If the project is not a git checkout, use `unknown` and note it. + +## Diagrams (Mermaid) + +Use Mermaid so diagrams live inside the markdown, stay diffable, and render on +GitHub/GitLab and in most wiki viewers. Put each diagram in a fenced ` ```mermaid ` +block. (If the wiki is published through Sphinx, enable a Mermaid extension such as +`sphinxcontrib-mermaid`; MkDocs needs the `mermaid2` plugin.) + +Where diagrams go: +- **index.md** — one system-level diagram: modules as nodes, dependencies or data + flow as edges, grouped into layers with `subgraph`. This is the visual map. +- **Module article** — a focused diagram for that module: its data flow, an + important call sequence, or its lifecycle. Prefer one clear diagram over several. + +Pick the type by what you are showing: +- **Module/dependency map or data flow** → `flowchart` (label edges with what flows). +- **Runtime interaction across components** → `sequenceDiagram`. +- **Lifecycle of a stateful component** → `stateDiagram-v2`. +- **Type/class relationships**, when they clarify → `classDiagram`. + +Grounding and legibility (same discipline as the prose): +- Nodes are real modules/files/symbols; edges are real dependencies, calls, or data + flows. Do not invent structure to make a diagram look complete. +- Keep node IDs stable and human-readable so diffs stay small as the code changes. +- Stay at the architecture level. If a diagram exceeds ~15-20 nodes, scope it to one + concern or split it rather than drawing the whole repo at once. + +Example — module data flow in an article: + +```mermaid +flowchart LR + caller[API layer] -->|request| svc[This module] + svc -->|reads / writes| db[(Store)] + svc -->|calls| dep[Other Module] +``` + +## Scope (what to read) + +Keep the scan cheap and focused on the real codebase: read only committed source, +and never walk build or generated output. + +- Enumerate files with `git ls-files` instead of walking the filesystem. It lists + exactly the tracked files and automatically excludes untracked files and anything + in `.gitignore` (build dirs, caches, artifacts). For strictly the state committed + at HEAD — excluding staged-but-uncommitted files — use + `git ls-tree -r --name-only HEAD`. +- Skip build/generated/vendored trees even when a project commits them: `build/`, + `dist/`, `out/`, `target/`, `node_modules/`, `.venv/`, `venv/`, `__pycache__/`, + `*.egg-info/`, `site-packages/`, minified assets, generated code, large data + files, lockfiles, and binaries — none of these describe architecture. +- Do not read every file. Per module, read the entry points and public interface + (`__init__.py`, headers, `main`, service entrypoints) and sample a few + representative implementation files; skip tests/fixtures unless they are the + clearest description of behavior. Use `git ls-files ` to list a module's + files, then open only what you need. +- If the project is not a git checkout, fall back to a filesystem walk but apply + the same ignore list and honor `.gitignore`. + +## Map (build / refresh) + +Scan the repo — or a named module — and create or update articles. + +1. Identify modules from the directory/package structure and entry points + (`pyproject.toml`, `CMakeLists.txt`, `__init__.py`, `main`, service configs). + List files with `git ls-files` and stay within Scope — do not walk the tree. +2. Read enough of the actual code to describe each module accurately. **Every + claim must trace to a real file or symbol** — read the code rather than guess; + if you cannot verify something, say so instead of inventing it. +3. Write/update the article in the format above, including a Mermaid diagram where + it aids understanding (see Diagrams). Record the commit the article reflects + (see Recording the commit SHA) and today's date. +4. Cascade: if a module's public interface, dependencies, or responsibilities + changed, update the "Depends on" / "Used by" sections of affected articles and + the index. Refresh the Updated date on every article you materially change. +5. Update `index.md` — refresh its `Map reflects commit` header, the module list, + summaries, and the system-level Mermaid diagram — and append to `log.md`: + ``` + ## [YYYY-MM-DD] map | | + ``` + +## Query + +Answer architecture questions from the wiki. + +1. Read `index.md` to locate relevant articles, then read those articles. +2. Prefer wiki content; if it is thin or possibly stale, fall back to reading the + code and note that you did so. +3. Cite articles and the underlying files, e.g. `[Module](architecture/module.md)` + and `src/foo/bar.py`. Answer in the conversation; do not write files unless asked. + +## Verify (lint against the code) + +Check the map against the real codebase (enumerate files with `git ls-files`; see +Scope). + +Auto-fix when unambiguous: +- Article references a path/symbol that moved → update it if there is exactly one + clear match; otherwise report. +- A module directory exists with no article → add a stub entry to the index. +- An index entry points to a missing article → mark `[MISSING]`; do not delete. + +Report only (needs judgment): +- Articles whose recorded commit is behind `HEAD` **and** whose paths changed since + → flag as possibly stale. Check with + `git log --oneline ..HEAD -- ` (non-empty = changed). +- Described modules/interfaces that no longer exist, or new ones undocumented. +- Mermaid diagram nodes referencing modules/files that no longer exist, or missing + edges for dependencies now present in the code. +- Contradictions between articles; missing cross-references. + +Append to `log.md`: +``` +## [YYYY-MM-DD] verify | | issues, auto-fixed +``` + +## Conventions + +- Standard markdown with relative links between articles. +- Update the map in the same PR that changes a module's structure or interface — + the map is only useful if it stays trustworthy. \ No newline at end of file diff --git a/.agents/skills/codebase-map/agents/openai.yaml b/.agents/skills/codebase-map/agents/openai.yaml new file mode 100644 index 00000000..19744a61 --- /dev/null +++ b/.agents/skills/codebase-map/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Codebase Map" + short_description: "Skill to build and maintain a knowledge base of a code repository" + default_prompt: "Use $codebase-map to build and/or update the code knowledge database" diff --git a/.agents/skills/flux/SKILL.md b/.agents/skills/flux/SKILL.md new file mode 100644 index 00000000..c8491938 --- /dev/null +++ b/.agents/skills/flux/SKILL.md @@ -0,0 +1,97 @@ +--- +name: flux +description: > + Commands for running and monitoring jobs on a Flux (flux-framework) HPC + scheduler: submitting jobs, checking whether a job is running, reading job + output and exit codes, and cancelling jobs. Use this whenever a task involves + submitting, monitoring, inspecting, or cancelling work on a Flux cluster, or + running builds / test suites / model serving on compute nodes rather than the + login node. Trigger it on any mention of `flux submit`, `flux run`, + `flux jobs`, `flux batch`, Flux job IDs, or "is my job running / done" on an + HPC system where Flux is the main scheduler. +--- + +# Flux job management + +Flux is a resource manager / scheduler used on HPC clusters. Use it to run +anything heavy — builds, test suites, model serving, data processing — as a job +on **compute nodes**. Do **not** run heavy work directly on the login node. + +Queue names, bank/account, and node counts are cluster-specific. This skill +covers the generic commands only; get the per-machine values (queue, bank, +typical `-N`/`-n`) from the project's machine-specific setup notes before +submitting. In general, you want to use `--exclusive` when you request nodes. + +## Interactive allocations + +```bash +flux alloc -B --exclusive -N1 -q pdebug -t 1h # open a new shell with the allocated nodes +flux alloc -B --exclusive -N4 -t 8h +``` + +## Submitting jobs + +```bash +flux submit ./script.sh # queue a job; prints a job ID and returns immediately +flux run ./script.sh # run interactively and block until it finishes +flux batch ./batch.sh # submit a batch script (directives via '# flux:' lines) +flux submit -N2 -n8 ./script.sh # request 2 nodes, 8 tasks +flux submit --queue= --name= ./script.sh # target a queue, name the job +``` + +A batch script declares its resources with `# flux:` directive lines, e.g.: + +```bash +#!/bin/sh +# flux: -N4 -n16 +flux run -n16 ./my_step.sh +``` + +## Checking whether a job is running + +```bash +flux jobs # your active jobs (pending + running) +flux jobs -a # include completed / inactive jobs +flux jobs --filter=running # only running jobs (also: pending, inactive) +flux job last # job ID of your most recent submission +``` + +## Reading output and exit code + +```bash +flux submit --watch ./script.sh # stream output live as it runs +flux job attach $(flux job last) # attach to / print output of the last job +flux submit --output=job-{{id}}.out ./script.sh # write stdout to a file named per job ID +flux jobs --no-header -o '{status}:{returncode}' # status + exit code of one job +``` + +A non-zero `returncode` means the job failed — inspect its output before assuming +the step succeeded. Do not report a job as "passed" until you have confirmed both +that it is `inactive` and that its return code is `0`. + +## Cancelling jobs + +```bash +flux cancel # cancel a single job +flux cancel --all # cancel all of your jobs +flux cancel --states=RUN # cancel jobs in a given state +``` + +## Inspecting resources + +```bash +flux resource list # nodes available to you and their state +flux uptime # is the Flux instance up, and for how long +flux overlay status # health of the Flux overlay network +``` + +## Notes for agents + +- Prefer `flux submit` (non-blocking) for long work, then poll with `flux jobs`; + use `flux run` only for quick interactive checks. +- Always capture the job ID from `flux submit` (or use `flux job last`) so you can + check status and output later. +- Never launch a full model-serving stack or a long test run on the login node — + submit it as a Flux job. + +Full command reference: https://flux-framework.org/cheat-sheet/ diff --git a/.agents/skills/flux/agents/openai.yaml b/.agents/skills/flux/agents/openai.yaml new file mode 100644 index 00000000..63feb0b6 --- /dev/null +++ b/.agents/skills/flux/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Flux Resource Manager" + short_description: "Skill to use the Flux resource manager" + default_prompt: "Use $flux to let the agents interact with Flux" diff --git a/.claude/skills/ams-build b/.claude/skills/ams-build new file mode 120000 index 00000000..4cc67435 --- /dev/null +++ b/.claude/skills/ams-build @@ -0,0 +1 @@ +../../.agents/skills/ams-build/ \ No newline at end of file diff --git a/.claude/skills/changelog b/.claude/skills/changelog new file mode 120000 index 00000000..5dffccaa --- /dev/null +++ b/.claude/skills/changelog @@ -0,0 +1 @@ +../../.agents/skills/changelog/ \ No newline at end of file diff --git a/.claude/skills/codebase-map b/.claude/skills/codebase-map new file mode 120000 index 00000000..0c5253c1 --- /dev/null +++ b/.claude/skills/codebase-map @@ -0,0 +1 @@ +../../.agents/skills/codebase-map/ \ No newline at end of file diff --git a/.claude/skills/flux b/.claude/skills/flux new file mode 120000 index 00000000..9a71d17e --- /dev/null +++ b/.claude/skills/flux @@ -0,0 +1 @@ +../../.agents/skills/flux/ \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..e4a56a08 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,256 @@ +# AGENTS.md + +This file provides guidance to LLMs (Claude Code, Codex etc) when working with code in this repository. + +## Project Overview + +AMS (Autonomous MultiScale Library) is a library to simplify machine learning surrogate model integration in HPC codes. +It enables scientific applications to use ML models as surrogates for expensive physics computations with uncertainty quantification. + +**Key components:** +- **AMSLib (C++)**: Core library providing the AMS API for scientific applications +- **AMSWorkflow (Python)**: Workflow orchestration components (AMSBroker, AMSTrain, AMSDeploy, AMSStore, AMSOrchestrator, AMSDBStage) +- **ML Integration**: PyTorch-based surrogate models with uncertainty quantification +- **Data Management**: HDF5 and optional RabbitMQ backends for storing/retrieving training data + + +## Executable Commands +- **Load Dependencies on Livermore Computing**: `source scripts/gitlab/setup-env.sh` +- **Test**: `ctest --test-dir build --output-on-failure` +- **Lint**: `clang-tidy -p build src/**/*.cpp` +- **Format**: `find src/ -regex '.*\.\(cpp\|hpp\|cu\|cuh\|c\|h\)' -exec clang-format -i {} \;` + +## Build System + +### CMake Configuration + +Standard build on Dane or on machine **without** GPU: +```bash +mkdir build && cd build +cmake \ + -DWITH_HIP=Off \ + -DWITH_CALIPER=On \ + -Dcaliper_DIR=$AMS_CALIPER_PATH \ + -DTorch_DIR=$AMS_TORCH_PATH \ + -DWITH_MPI=On \ + -DWITH_HDF5=On \ + -DWITH_RMQ=On \ + -Damqpcpp_DIR=$AMS_AMQPCPP_PATH \ + -DWITH_TESTS=On \ + -DWITH_WORKFLOW=On \ + -DWITH_AMS_DEBUG=On \ + .. +make -j6 +make install +``` + +Standard build on Tioga/Tuo or on machine with AMD GPUs: + +```bash +cmake \ + -DWITH_HIP=On \ + -DWITH_CALIPER=On \ + -Dcaliper_DIR=$AMS_CALIPER_PATH \ + -DTorch_DIR=$AMS_TORCH_PATH \ + -DWITH_MPI=On \ + -DWITH_HDF5=On \ + -DWITH_RMQ=On \ + -Damqpcpp_DIR=$AMS_AMQPCPP_PATH \ + -DWITH_TESTS=On \ + -DWITH_WORKFLOW=On \ + -DWITH_AMS_DEBUG=On \ + .. +``` + +If you want to build on a system with NVIDIA GPU you can just use `-DWITH_CUDA=On` and `-DAMS_CUDA_ARCH`. + +### CMake Options + +Required dependencies: +- `WITH_MPI`: Enable MPI support (mandatory for distributed execution) +- `WITH_TORCH`: Enable PyTorch support (optional) + +Optional features: +- `WITH_CUDA` / `WITH_HIP`: GPU acceleration (mutually exclusive) +- `WITH_CALIPER`: Caliper profiling support +- `WITH_PERFFLOWASPECT`: PerfFlowAspect profiling (requires PFA-enabled clang/llvm) +- `WITH_RMQ`: RabbitMQ backend for distributed data management +- `WITH_HDF5`: HDF5 file I/O (effectively required despite being "optional") +- `WITH_AMS_DEBUG`: Enable verbose debug output (defines `LIBAMS_VERBOSE` and `__AMS_DEBUG__`) +- `WITH_TESTS`: Build test suite (uses Catch2) +- `WITH_WORKFLOW`: Install Python workflow drivers +- `WITH_AMS_LIB`: Build C++ library (defaults to ON) + +## Running Tests + +```bash +cd build +make test +# or for detailed output: +ctest --output-on-failure +# or +CTEST_OUTPUT_ON_FAILURE=1 make test +# or to run a specific test +ctest --output-on-failure -R "testName" +``` + +Tests use Catch2 framework (v3.11.0, fetched automatically during build). + +Test directory structure: +- `tests/AMSlib/ams_interface/`: End-to-end AMS interface tests +- `tests/AMSlib/db/`: Database backend tests (HDF5) +- `tests/AMSlib/torch/`: PyTorch model inference tests +- `tests/AMSlib/wf/`: Workflow component tests +- `tests/AMSlib/models/`: Test model generation scripts + +## Code Architecture + +### Core AMS API (`src/AMSlib/`) + +Main API is defined in `src/AMSlib/include/AMS.h`: + +1. **Initialization**: `AMSInit()` / `AMSFinalize()` - Setup and teardown +2. **Model Registration**: `AMSRegisterAbstractModel()` - Register a surrogate model with domain name, threshold, and model path +3. **Executor Creation**: `AMSCreateExecutor()` - Create an executor for a registered model +4. **Execution**: `AMSExecute()` / `AMSCExecute()` - Execute with surrogate model or physics fallback +5. **Cleanup**: `AMSDestroyExecutor()` - Destroy executor + +**Key concepts:** +- **Uncertainty Quantification**: Models return `Tuple[[Tensor[N, ...], Tensor[N, 1]]` where second tensor contains uncertainty scores (lower = more confident) +- **Threshold**: Controls when to use surrogate vs physics (based on uncertainty) +- **Hybrid Execution**: Automatically falls back to physics computation when uncertainty exceeds threshold + +### Workflow System (`src/AMSlib/wf/`) + +The `AMSWorkflow` class orchestrates hybrid execution: + +- **Evaluation Pipeline**: + 1. Predict using surrogate model + 2. Check uncertainty against threshold + 3. For high-uncertainty samples: execute physics and store data + 4. For low-uncertainty samples: use ML predictions + +- **Model Updates**: Supports dynamic model updates via RabbitMQ +- **Data Storage**: Stores training data to HDF5 or RabbitMQ backends +- **Distributed Execution**: MPI-aware for parallel processing + +Key files: +- `src/AMSlib/wf/workflow.hpp`: Main workflow class +- `src/AMSlib/wf/action.hpp`: Action concept for data transformations +- `src/AMSlib/wf/eval_context.hpp`: Evaluation context management +- `src/AMSlib/wf/basedb.hpp`: Database backend interface + +### ML Components (`src/AMSlib/ml/`) + +- `surrogate.hpp`: Surrogate model wrapper around PyTorch models +- `Model.hpp`: PyTorch model loading and inference +- `AbstractModel.hpp`: Abstract interface for ML models + +### Python Workflow (`src/AMSWorkflow/`) + +Components for outer training/deployment loop: +- `AMSBroker`: Message broker for distributed coordination +- `AMSTrain`: Training orchestration +- `AMSDeploy`: Model deployment +- `AMSStore`: Data storage management +- `AMSOrchestrator`: Workflow orchestration +- `AMSDBStage`: Database staging + +Install with: `pip install -e .` from project root + +## Code Style +- **Standards**: C++17, strictly. Prefer standard library over external dependencies where possible. +- **Ownership**: Use smart pointers or value semantics. NO raw `new`/`delete`. +- **Safety**: Use `tl::expected` for error handling; avoid raw exceptions in performance-critical paths. +- **Headers**: Prefer `#pragma once` over traditional include guards. +- **Formatting**: Strictly follow the project's `.clang-format`. Run it after every file modification. + + +Configuration (`.clang-format`): +- Based on Google style +- 80 character line limit +- 2-space indentation +- Linux brace style + +Format Python code with: +```bash +black --line-length 120 +# or +ruff format +``` + +Python configuration in `pyproject.toml`: +- 120 character line limit +- Black formatting profile +- Ruff linting with ignore rules: E501, E226, E203 + +## Development Workflow + +**Main branch**: `develop` (not `main`) + +**Creating PRs**: Always target `develop` as the base branch. + +**Python requirements**: Tests require `h5py` installed (`pip install h5py`) + +## Installation + +Recommended: Use Spack for dependency management: +```bash +spack install ams +# or for development: +spack dev-build ams +``` + +See INSTALL.md for manual installation details. + +## Repository Structure + +``` +src/ +├── AMSlib/ # C++ library +│ ├── include/ # Public API headers +│ ├── ml/ # ML model components +│ └── wf/ # Workflow system +└── AMSWorkflow/ # Python workflow tools + ├── ams/ # Python package + └── ams_wf/ # Workflow drivers + +tests/ +├── AMSlib/ # C++ tests (Catch2) +└── AMSWorkflow/ # Python tests + +examples/ +├── ideal_gas/ # Example: ideal gas law application +└── bnm_opt/ # Example: optimization application + +cmake/ # CMake modules +docs/ # Sphinx documentation +``` + +## Common Patterns + +**Type aliases in AMS.h:** +- `AMSExecutor`: Executor handle (int64_t) +- `AMSCAbstrModel`: Model handle (int) +- `DomainLambda`: C++ lambda callback type +- `DomainCFn`: C function pointer callback type + +**Device support:** +- AMS uses custom resource manager for memory management across CPU/GPU +- Set allocator: `AMSSetAllocator(AMSResourceType resource, const char* name)` +- Supported resources: Host, Device (CUDA/HIP) + +**Database configuration:** +- File system DB: `AMSConfigureFSDatabase(AMSDBType db_type, const char* db_path)` +- RabbitMQ DB: Enable with `-DWITH_RMQ=On` at build time + +## Boundaries & Guardrails +- **Always**: Run tests that are impacted by your changes. For example, to re-run the + core tests: `ctest --output-on-failure -R "CORE::"` or `ctest --output-on-failure -R "CORE::TENSOR_INT"` + to re-run one specific test. +- **Ask First**: Before adding new external dependencies to `CMakeLists.txt`. +- **Never**: Use C-style casts; instead use `static_cast` or `reinterpret_cast`. +- **Never**: Do not over-engineer solutions with superfluous safety checking. +- **Never**: Use modifying git commands unless explicitly asked to by the user. +- **Never**: Run all the tests with `ctest` unless explicitly asked to by the user or before + commiting to a branch. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 120000 index 00000000..47dc3e3d --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/INSTALL.md b/INSTALL.md index 2240b226..60234a4d 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,85 +1,214 @@ # Setup and Build -AMSLib depends on the following packages: -* UMPIRE (Mandatory) -* MPI (Mandatory) -* CALIPER (Optional) -* FAISS (Optional) -* MFEM (Optional) -* PY-TORCH (Optional) -* MFEM (Optional) -* REDIS (Optional) -* HDF5 (Optional) -* CUDA (Optional) -* ADIAK (Optional) +AMSLib is a CMake (>= 3.18, C++17) project. The build itself is straightforward; +the effort is in providing its dependencies so that `find_package` succeeds. + +## Dependencies + +**Always required:** + +* HDF5 (C and CXX components) +* LibTorch (PyTorch C++ API) +* A C++17 compiler and a threading library (i.e., pthreads) +* `nlohmann_json` (https://github.com/nlohmann/json) +* `{fmt}` (https://github.com/fmtlib/fmt) +* `tl::expected` (header-only, https://github.com/TartanLlama/optional) + +**Optional, enabled per build flag:** + +| Dependency | Enabled by | +| --- | --- | +| MPI | `WITH_MPI` | +| CUDA (NVIDIA) | `WITH_CUDA` | +| HIP / ROCm (AMD) | `WITH_HIP` | +| Caliper | `WITH_CALIPER` | +| amqp-cpp, OpenSSL, libevent | `WITH_RMQ` (RabbitMQ back end) | +| PerfFlowAspect | `WITH_PERFFLOWASPECT` | + +`WITH_CUDA` and `WITH_HIP` are mutually exclusive. + +## Build options + +| Option | Default | Description | Extra dependencies pulled in | +| --- | --- | --- | --- | +| `WITH_AMS_LIB` | `ON` | Build and install the C++ library. | — (core: nlohmann_json, HDF5, Torch, Threads) | +| `WITH_MPI` | `OFF` | Enable MPI support. | MPI | +| `WITH_CUDA` | `OFF` | Enable CUDA (NVIDIA GPUs). | CUDA | +| `WITH_HIP` | `OFF` | Enable HIP (AMD GPUs). | HIP/ROCm | +| `WITH_CALIPER` | `OFF` | Enable Caliper profiling. | caliper | +| `WITH_RMQ` | `OFF` | Use RabbitMQ as a database back end (requires a reachable, running RabbitMQ server at runtime). | amqp-cpp, OpenSSL, libevent | +| `WITH_PERFFLOWASPECT` | `OFF` | Enable PerfFlowAspect profiling. | perfflowaspect | +| `WITH_WORKFLOW` | `OFF` | Install the Python drivers used by the outer workflow. | Python side deps (see `pyproject.toml`) | +| `WITH_AMS_DEBUG` | `OFF` | Enable verbose logging. | — | +| `WITH_TESTS` | `OFF` | Build the test suite. | — | + +Notes: +- `WITH_CUDA` and `WITH_HIP` are mutually exclusive — enabling both is a fatal + CMake error. +- HDF5 is always required. Supply `-DAMS_HDF5_DIR=` to pin a specific + install (config-mode search looks in `` and `/share/cmake`); set + `-DHDF5_USE_STATIC_LIBRARIES=On` to link the static libs. + + +### Dependency location hints + +When a dependency is installed outside the default search paths, point CMake at +its config package (or add its prefix to `CMAKE_PREFIX_PATH`). On LC these are exported by +`scripts/gitlab/setup-env.sh` under the `AMS_*_PATH` names shown. + +| Package | CMake variable | LC export from setup-env.sh | +|---|---|---| +| libTorch | `Torch_DIR` | `$AMS_TORCH_PATH` (…/torch/share/cmake/Torch) | +| HDF5 | `AMS_HDF5_DIR` | `$AMS_HDF5_PATH` | +| Caliper | `caliper_DIR` | `$AMS_CALIPER_PATH` (…/share/cmake/caliper) | +| amqp-cpp | `amqpcpp_DIR` | `$AMS_AMQPCPP_PATH` (…/cmake) | +| nlohmann_json | `nlohmann_json_DIR` | provided by active Spack env | +| CUDA arch | `CMAKE_CUDA_ARCHITECTURES` | `$AMS_CUDA_ARCH` | +| ROCm arch | (loaded via module) | `$AMS_HIP_ARCH` | + +Anything not found via a `*_DIR` hint is searched on `CMAKE_PREFIX_PATH`. ## Spack Installation -AMS depends on multiple complex external libraries, our preferred and suggested mechanism to install AMS is through [spack](https://github.com/spack/spack) as follows: +AMS depends on multiple complex external libraries, so our preferred and +suggested mechanism to install AMS is through [Spack](https://github.com/spack/spack): ```bash spack install ams ``` -If you are a developer and would like to extend AMS you can do so by using the `spack dev-build' command. -For more instructions look [here](https://spack-tutorial.readthedocs.io/en/lanl19/tutorial_developer_workflows.html) +If you are a developer and would like to extend AMS, you can build from a +working copy with `spack dev-build`. See the Spack +[developer workflows tutorial](https://spack-tutorial.readthedocs.io/en/latest/tutorial_developer_workflows.html) +for details. + +### Livermore Computing (LC) clusters + +On LC systems, the AMS team maintains a prebuilt internal Spack environment. +Instead of installing dependencies yourself, source the provided setup script +from the repository root; it loads the appropriate compiler/MPI/ROCm modules, +activates the per-host Spack environment, and exports the dependency locations +(`AMS_TORCH_PATH`, `AMS_HDF5_PATH`, `AMS_CALIPER_PATH`, `AMS_AMQPCPP_PATH`, +`AMS_CUDA_ARCH`, ...) used by the CMake command below: + +```bash +source scripts/gitlab/setup-env.sh +``` + +## Python Virtual Environnement -## Manual cmake installation -Below you can find a `cmake` command to configure to configure AMS, build and install it. +## Manual CMake installation + +Below is a `cmake` command that configures, builds, and installs AMS. Most +options are optional; the example enables a representative feature set. On LC, +the `$AMS_*_PATH` variables are set by `scripts/gitlab/setup-env.sh`; otherwise +substitute the paths to your own installs. ```bash -$ mkdir build; cd build -$ cmake \ - -DWITH_DB=On -DWITH_RMQ=On \ - -Damqpcpp_DIR=$AMS_AMQPCPP_PATH \ +cmake -S . -B build \ + -DCMAKE_BUILD_TYPE=Release \ -DBUILD_SHARED_LIBS=On \ - -DCMAKE_PREFIX_PATH=$INSTALL_DIR \ - -DWITH_CALIPER=On \ - -DWITH_HDF5=On \ - -DWITH_EXAMPLES=On \ - -DHDF5_Dir=$AMS_HDF5_PATH \ -DCMAKE_INSTALL_PREFIX=./install \ - -DCMAKE_BUILD_TYPE=Release \ - -DWITH_CUDA=On \ - -DUMPIRE_DIR=$AMS_UMPIRE_PATH \ - -DMFEM_DIR=$AMS_MFEM_PATH \ - -DWITH_FAISS=On \ -DWITH_MPI=On \ - -DWITH_TORCH=On \ - -DWITH_TESTS=Off \ - -DTorch_DIR=$AMS_TORCH_PATH \ - -DFAISS_DIR=$AMS_FAISS_PATH \ - -DAMS_CUDA_ARCH=${AMS_CUDA_ARCH} \ + -DWITH_CUDA=On \ + -DCMAKE_CUDA_ARCHITECTURES=${AMS_CUDA_ARCH} \ + -DWITH_CALIPER=On \ + -DWITH_RMQ=On \ -DWITH_AMS_DEBUG=On \ - ../ + -DTorch_DIR=${AMS_TORCH_PATH} \ + -DAMS_HDF5_DIR=${AMS_HDF5_PATH} \ + -Dcaliper_DIR=${AMS_CALIPER_PATH} \ + -Damqpcpp_DIR=${AMS_AMQPCPP_PATH} -$ make -j6 -$ make install +cmake --build build -j 6 +cmake --install build ``` -Most of the compile time options are optional. +## Example builds -## Building AMS with PerfFlowAspect +### 1. Minimal / CPU (LC) -To built AMS with [PFA](https://github.com/flux-framework/PerfFlowAspect) support you first need to install a PFA clang/llvm version and add it to `$PATH`. Next to configure, built and install perform the following: +For a minimal, CPU-only first build, drop the feature flags and keep only the +mandatory hints: +```bash +source scripts/gitlab/setup-env.sh +cmake -S . -B build \ + -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_SHARED_LIBS=On \ + -DTorch_DIR="$AMS_TORCH_PATH" \ + -DAMS_HDF5_DIR="$AMS_HDF5_PATH" \ + -DCMAKE_INSTALL_PREFIX=./install ``` -$ cd $CODE_ROOT/setup -$ mkdir build; cd build -$ cmake \ - -DCMAKE_CXX_COMPILER=clang++ \ - -DCMAKE_C_COMPILER=clang \ - -DMFEM_DIR=$AMS_MFEM_PATH \ - -DUMPIRE_DIR=$AMS_UMPIRE_PATH \ - -DWITH_CUDA=On \ - -DWITH_CALIPER=On \ - -DWITH_TORCH=On -DTorch_DIR=$AMS_TORCH_PATH \ - -DWITH_FAISS=On -DFAISS_DIR=$AMS_FAISS_PATH \ - -DWITH_PERFFLOWASPECT=On \ - -Dperfflowaspect_DIR=$AMS_PFA_PATH/share \ - ../ -$ make -j6 + +### 2. With RabbitMQ + MPI (LC) + +```bash +source scripts/gitlab/setup-env.sh +cmake -S . -B build \ + -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_SHARED_LIBS=On \ + -DWITH_MPI=On \ + -DWITH_RMQ=On \ + -DTorch_DIR="$AMS_TORCH_PATH" \ + -DAMS_HDF5_DIR="$AMS_HDF5_PATH" \ + -Damqpcpp_DIR="$AMS_AMQPCPP_PATH" \ + -DCMAKE_INSTALL_PREFIX=./install ``` +### 3. GPU (CUDA) + Caliper (LC) + +```bash +source scripts/gitlab/setup-env.sh +cmake -S . -B build \ + -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_SHARED_LIBS=On \ + -DWITH_CUDA=On \ + -DCMAKE_CUDA_ARCHITECTURES="$AMS_CUDA_ARCH" \ + -DWITH_CALIPER=On \ + -Dcaliper_DIR="$AMS_CALIPER_PATH" \ + -DTorch_DIR="$AMS_TORCH_PATH" \ + -DAMS_HDF5_DIR="$AMS_HDF5_PATH" \ + -DCMAKE_INSTALL_PREFIX=./install +``` + +### 4. GPU (HIP/ROCm) (LC Cray/ROCm machine) + +```bash +source scripts/gitlab/setup-env.sh # loads rocm, sets AMS_HIP_ARCH +cmake -S . -B build \ + -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_SHARED_LIBS=On \ + -DWITH_HIP=On \ + -DTorch_DIR="$AMS_TORCH_PATH" \ + -DAMS_HDF5_DIR="$AMS_HDF5_PATH" \ + -DCMAKE_INSTALL_PREFIX=./install +``` + +### 5. Minimal / CPU (non-LC, manual deps) + +```bash +export CMAKE_PREFIX_PATH=/opt/deps:$CMAKE_PREFIX_PATH +cmake -S . -B build \ + -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_SHARED_LIBS=On \ + -DTorch_DIR=/opt/libtorch/share/cmake/Torch \ + -DAMS_HDF5_DIR=/opt/hdf5 \ + -Dnlohmann_json_DIR=/opt/deps/lib/cmake/nlohmann_json \ + -DCMAKE_INSTALL_PREFIX=./install +``` + +Add feature flags (`-DWITH_RMQ=On`, `-DWITH_MPI=On`, …) and the corresponding +`*_DIR` hints from `references/manual-deps.md` as needed. + +## Build & install + +```bash +cmake --build build -j "$(nproc)" +cmake --install build +``` diff --git a/scripts/ams-configure.sh b/scripts/ams-configure.sh new file mode 100755 index 00000000..6d0e1dbe --- /dev/null +++ b/scripts/ams-configure.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash +# +# ams-configure.sh — assemble and run the CMake configure step for AMS. +# +# We have two dependency-provisioning paths: +# * On an LLNL Livermore Computing (LC) cluster it sources the repo's own +# scripts/gitlab/setup-env.sh (internal Spack env) and maps the exported +# AMS_*_PATH variables onto the right -D..._DIR hints. +# * Elsewhere it relies on *_DIR / CMAKE_PREFIX_PATH you set yourself +# (see references/manual-deps.md). +# +# The full cmake command is always printed before it runs. Use --dry-run to +# print without configuring. +# +# Run from the AMS repo root, or pass --src . + +set -euo pipefail + +# ---- defaults -------------------------------------------------------------- +SRC="." +BUILD="build" +BUILD_TYPE="Release" +INSTALL_PREFIX="./install" +SHARED="On" +DRY_RUN=0 +FORCE_MODE="" # "lc" | "nolc" | "" (auto) + +# feature toggles (all off by default -> minimal CPU build) +WITH_MPI="Off" +WITH_CUDA="Off" +WITH_HIP="Off" +WITH_CALIPER="Off" +WITH_RMQ="Off" +WITH_PERFFLOWASPECT="Off" +WITH_WORKFLOW="Off" +WITH_DEBUG="Off" +WITH_TESTS="Off" +CUDA_ARCH="" # override; else AMS_CUDA_ARCH after setup-env + +usage() { + cat <<'EOF' +Usage: ams-configure.sh [feature flags] [options] + +Feature flags (compose freely): + --mpi enable MPI (-DWITH_MPI=On) + --cuda enable CUDA (NVIDIA) (-DWITH_CUDA=On) + --hip enable HIP (AMD) (-DWITH_HIP=On) [mutually exclusive with --cuda] + --caliper Caliper profiling (-DWITH_CALIPER=On) + --rmq RabbitMQ back end (-DWITH_RMQ=On) + --perfflowaspect PerfFlowAspect (-DWITH_PERFFLOWASPECT=On) + --workflow Python drivers (-DWITH_WORKFLOW=On) + --debug verbose logging (-DWITH_AMS_DEBUG=On) + --tests build tests (-DWITH_TESTS=On) + +Options: + --src PATH AMS repo root (default: .) + --build DIR build directory (default: build) + --build-type TYPE Release|Debug|RelWithDebInfo (default: Release) + --install-prefix PATH install prefix (default: ./install) + --static build static libs (default: shared) + --cuda-arch ARCH CUDA arch, e.g. 80,90 (default: $AMS_CUDA_ARCH on LC) + --lc | --no-lc force LC / non-LC dependency path (default: auto-detect) + --dry-run print the cmake command but do not run it + -h, --help this help + +Examples: + ams-configure.sh # minimal CPU build + ams-configure.sh --rmq --mpi # RabbitMQ + MPI + ams-configure.sh --cuda --caliper # GPU + profiling + ams-configure.sh --rmq --dry-run # show the command only +EOF +} + +# ---- parse args ------------------------------------------------------------ +while [[ $# -gt 0 ]]; do + case "$1" in + --mpi) WITH_MPI="On";; + --cuda) WITH_CUDA="On";; + --hip) WITH_HIP="On";; + --caliper) WITH_CALIPER="On";; + --rmq|--rabbitmq) WITH_RMQ="On";; + --perfflowaspect|--pfa) WITH_PERFFLOWASPECT="On";; + --workflow) WITH_WORKFLOW="On";; + --debug) WITH_DEBUG="On";; + --tests) WITH_TESTS="On";; + --src) SRC="$2"; shift;; + --build) BUILD="$2"; shift;; + --build-type) BUILD_TYPE="$2"; shift;; + --install-prefix) INSTALL_PREFIX="$2"; shift;; + --static) SHARED="Off";; + --cuda-arch) CUDA_ARCH="$2"; shift;; + --lc) FORCE_MODE="lc";; + --no-lc) FORCE_MODE="nolc";; + --dry-run) DRY_RUN=1;; + -h|--help) usage; exit 0;; + *) echo "Unknown argument: $1" >&2; usage; exit 2;; + esac + shift +done + +if [[ "$WITH_CUDA" == "On" && "$WITH_HIP" == "On" ]]; then + echo "Error: --cuda and --hip are mutually exclusive." >&2 + exit 2 +fi + +if [[ ! -f "$SRC/CMakeLists.txt" ]]; then + echo "Error: '$SRC' does not look like the AMS repo root (no CMakeLists.txt)." >&2 + echo " cd into the AMS clone or pass --src ." >&2 + exit 2 +fi + +# ---- decide dependency path ------------------------------------------------ +LC_ENV_DIR="/usr/workspace/AMS/ams-spack-environments" +MODE="$FORCE_MODE" +if [[ -z "$MODE" ]]; then + if [[ -d "$LC_ENV_DIR" ]]; then MODE="lc"; else MODE="nolc"; fi +fi + +# extra -D hints accumulated from the environment +declare -a DEP_ARGS=() + +if [[ "$MODE" == "lc" ]]; then + echo ">> LC cluster detected — sourcing $SRC/scripts/gitlab/setup-env.sh" + # shellcheck disable=SC1091 + source "$SRC/scripts/gitlab/setup-env.sh" + + [[ -n "${AMS_TORCH_PATH:-}" ]] && DEP_ARGS+=("-DTorch_DIR=${AMS_TORCH_PATH}") + [[ -n "${AMS_HDF5_PATH:-}" ]] && DEP_ARGS+=("-DAMS_HDF5_DIR=${AMS_HDF5_PATH}") + if [[ "$WITH_CALIPER" == "On" && -n "${AMS_CALIPER_PATH:-}" ]]; then + DEP_ARGS+=("-Dcaliper_DIR=${AMS_CALIPER_PATH}") + fi + if [[ "$WITH_RMQ" == "On" && -n "${AMS_AMQPCPP_PATH:-}" ]]; then + DEP_ARGS+=("-Damqpcpp_DIR=${AMS_AMQPCPP_PATH}") + fi + if [[ "$WITH_CUDA" == "On" ]]; then + ARCH="${CUDA_ARCH:-${AMS_CUDA_ARCH:-}}" + [[ -n "$ARCH" ]] && DEP_ARGS+=("-DCMAKE_CUDA_ARCHITECTURES=${ARCH}") + fi +else + echo ">> Non-LC cluster — using your *_DIR / CMAKE_PREFIX_PATH hints." + echo " (see references/manual-deps.md; a bare configure will fail if" + echo " Torch/HDF5/nlohmann_json can't be found)" + # pass through anything the user already exported, if present + [[ -n "${Torch_DIR:-}" ]] && DEP_ARGS+=("-DTorch_DIR=${Torch_DIR}") + [[ -n "${AMS_HDF5_DIR:-}" ]] && DEP_ARGS+=("-DAMS_HDF5_DIR=${AMS_HDF5_DIR}") + [[ -n "${nlohmann_json_DIR:-}" ]] && DEP_ARGS+=("-Dnlohmann_json_DIR=${nlohmann_json_DIR}") + [[ "$WITH_CALIPER" == "On" && -n "${caliper_DIR:-}" ]] && DEP_ARGS+=("-Dcaliper_DIR=${caliper_DIR}") + [[ "$WITH_RMQ" == "On" && -n "${amqpcpp_DIR:-}" ]] && DEP_ARGS+=("-Damqpcpp_DIR=${amqpcpp_DIR}") + if [[ "$WITH_CUDA" == "On" && -n "$CUDA_ARCH" ]]; then + DEP_ARGS+=("-DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH}") + fi +fi + +# ---- assemble cmake command ------------------------------------------------ +CMAKE_ARGS=( + -S "$SRC" -B "$BUILD" + -DCMAKE_BUILD_TYPE="$BUILD_TYPE" + -DBUILD_SHARED_LIBS="$SHARED" + -DCMAKE_INSTALL_PREFIX="$INSTALL_PREFIX" + -DWITH_MPI="$WITH_MPI" + -DWITH_CUDA="$WITH_CUDA" + -DWITH_HIP="$WITH_HIP" + -DWITH_CALIPER="$WITH_CALIPER" + -DWITH_RMQ="$WITH_RMQ" + -DWITH_PERFFLOWASPECT="$WITH_PERFFLOWASPECT" + -DWITH_WORKFLOW="$WITH_WORKFLOW" + -DWITH_AMS_DEBUG="$WITH_DEBUG" + -DWITH_TESTS="$WITH_TESTS" + "${DEP_ARGS[@]}" +) + +echo +echo ">> cmake command:" +printf ' cmake' +for a in "${CMAKE_ARGS[@]}"; do printf ' \\\n %q' "$a"; done +printf '\n\n' + +if [[ "$DRY_RUN" == "1" ]]; then + echo ">> --dry-run set; not configuring." + exit 0 +fi + +cmake "${CMAKE_ARGS[@]}" + +echo +echo ">> Configured. Next:" +echo " cmake --build $BUILD -j \"\$(nproc)\"" +echo " cmake --install $BUILD" +