diff --git a/docs/design_decisions/DR-008-infra.rst b/docs/design_decisions/DR-008-infra.rst new file mode 100644 index 00000000000..90a025720c4 --- /dev/null +++ b/docs/design_decisions/DR-008-infra.rst @@ -0,0 +1,210 @@ +.. + Copyright (c) 2026 Contributors to the Eclipse Foundation + + See the NOTICE file(s) distributed with this work for additional + information regarding copyright ownership. + + This program and the accompanying materials are made available under the + terms of the Apache License Version 2.0 which is available at + https://www.apache.org/licenses/LICENSE-2.0 + + SPDX-License-Identifier: Apache-2.0 + +DR-008-Infra: Generating documentation sources via Bazel +======================================================== + +- **Date:** 2026-05-13 + +.. dec_rec:: Generating documentation sources via Bazel + :id: dec_rec__infra__docs_src_dir + :status: accepted + :context: Infrastructure + :decision: Option B because Option N loses wrt flexibility + +Context / Problem +----------------- + +The docs-as-code system builds documentation by reading from a static ``source_dir`` (default ``"docs/"``) on the workspace filesystem. +Three build paths coexist: + +1. **Live preview** — Local development via `sphinx-autobuild `_. +2. **Direct Sphinx** — Sphinx invoked in the same venv for fast iteration or CI. +3. **Bazel sandbox** — ``needs_json`` and similar targets run Sphinx in a hermetic sandbox. + +We have no generic solution for generating parts of the documentation source directory via Bazel. +See `docs-as-code issue #423 `_ for an open feature request +to implement "Extra docs pages from artifacts". + +Workarounds we already have in place are: + +* Use ``.`` as source directory to place sources anywhere. + This implies a careful maintenance of include/exclude patterns in ``conf.py``. +* Generate json files for special inputs like source links or test reports. + This is limiting because we cannot generate whole pages or directories with this approach. +* The ``:docs_combo`` does compose a sources directory via `sphinx-collections `_. + It allows no control over the folder hierarchy + and symlinks in the git workspace can be confusing. + +We look for a solution which is simpler and easier to maintain, +so we don't have to keep adding more and more workarounds for each new use case. + +Additionally, we repeatedly has issues with caching. +Since we don't rely on Bazel sandbox for docs building, Bazel cannot help with hermeticity and determinism. +We need incremental builds locally and determinism with caching in CI. +See `rules_python sphinxdocs `_ +how an idea how to achieve this using Bazel. + +The "Module API" proposal +(somewhat implemented in `tooling PR 95 `_) +fully relies on Bazel. +It is not compatible with the docs-as-code live preview as of now. +`Another exploration by Useblocks `_ +is available but does not cover live preview either. + +Goals +^^^^^ + +- **Effort**: Minimise one-time implementation and ongoing maintenance cost. +- **Flexibility**: Minimise the effort for potential future extensions. +- **Speed**: Minimise the build time for documentation builds, especially for live preview. +- **UX**: Minimize efforts necessary to documentation work. + +Non-Goals +^^^^^^^^^ + +- Replacing Sphinx or Sphinx-Needs as documentation tools. +- Keep Esbonio language server alive as we assume nobody is using it. + +Options Considered +------------------ + +Option N: No change (status quo) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Keep the current architecture. +The ``docs()`` macro in ``docs.bzl`` accepts a ``source_dir`` parameter and reads +documentation sources directly from that directory on the workspace filesystem. + +.. mermaid:: + + graph LR + docs@{ shape: docs, label: "docs/" } + docs --> :docs + docs --> :live_preview + :live_preview -- watch --> docs + + +Effort 💚: No implementation work required. + +Flexibility 😡: More workarounds instead of generic solution. + +Speed 💚: Fast but only covers source updates (not test result updates, for example). + +UX 💚: Status quo + +While ``sphinx-autobuild --pre-build`` is available to trigger some build steps before each rebuild, +this does not work with Bazel: +If you ``bazel run :live_preview`` and do a ``bazel build`` inside, +that build will wait for the run to finish, thus deadlock. + +Option B: Introduce ``:docs_src_dir`` Bazel target +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In short: sphinx-autobuild from a Bazel target that re-materializes the sources continuously via +`ibazel / bazel-watcher `_. + +We add an ``extra_docs`` attribute to the ``docs()`` macro +for additional sources specified via `sphinx_docs_library `_, +which allows to adapt path prefixes. +The source tree is materialized using hardlinks (``ln``) inside a Bazel ``declare_directory`` action. +Symlinks fail Bazel's output tree validation (dangling link detection), +while copies are unnecessarily expensive for large doc sets. + +.. mermaid:: + + graph LR + docs@{ shape: docs, label: "docs/" } + extradocs@{ shape: docs, label: "extra_srcs" } + preview@{ shape: subproc, label: "live_preview" } + allsrc@{ label: ":docs_src_dir" } + docs --> allsrc --> :docs --> preview + extradocs --> allsrc + preview -- rebuild --> :docs + allsrc --> :needs_json + +The live preview is replaced by a custom implementation. +This live preview cannot be executed via ``bazel run`` because of the need to rebuild via Bazel internally. +Thus, there is no ``:live_preview`` target but a ``live_preview`` script. +We cannot rely on watching file system changes to trigger rebuilds because the source directory is composed by Bazel +and may contain generated files. + +The ``live_preview`` script runs two concurrent processes: + +1. ``ibazel build :docs_src_dir`` — watches workspace sources and re-materializes the tree on change. +2. ``sphinx-autobuild`` — watches the materialized tree and serves HTML with websocket-based browser reload. + +The ``score_sync_toml`` extension writes a ``ubproject.toml`` file to the source directory +but Bazel sandboxing makes this fail. +The ``score_sync_toml`` extension's write to the source directory is redirected via +``--define=needscfg_outpath=/docs/ubproject.toml``, +which works without modifications to the extension itself. + +Effort 💛: Some implementation effort but prototype already works. + +Flexibility 💚: Generic solution for all build paths and future extensions. + +Speed 💛: Overall latency is comparable to the status quo for edit-preview cycles, but the initial cold start is a little slower due to the extra Bazel invocation. + +UX 😡: Requires a two-step setup: ``bazel run //:ide_support`` (venv) then +``bazel run //:gen_live_preview`` (script). +The generated script is workspace-specific and should be gitignored. + +The generally idea is also described in `the rules_python documentation `_: + +.. code-block:: bash + + bazel run //docs:docs.serve # Run in separate terminal + ibazel build //docs:docs # Automatically rebuilds docs + +This ``docs.serve`` target implemented in `rules_python` does not have +auto-refresh in the browser though. + +Option D: Dual-path — keep ``:live_preview``, add hermetic ``:docs`` build +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Keep the existing ``bazel run :live_preview`` target unchanged (sphinx-autobuild watching ``docs/`` on the workspace filesystem). +In parallel, introduce a separate hermetic ``bazel build :docs`` target +that materialises a composed source directory inside the Bazel sandbox before invoking Sphinx. + +Effort 😡: By definition requires nearly the effort for option N and B combined. + +Flexibility 😡: Still requires all the workarounds of option N. + +Speed 💚: No slowdown. + +UX 😡: Live-preview UX is unchanged, but risk of downstream breaks. + + +Evaluation +---------- + +In order of importance, most important first. + +.. csv-table:: + :header: Goals, Option N, Option B, Option D + :widths: 2, 1, 1, 1 + + Flexibility, 😡, 💚, 😡 + Effort, 💚, 💛, 😡 + Speed, 💚, 💛, 💚 + UX, 💚, 😡, 😡 + +**Decision: Option B** because Option N loses wrt flexibility. Option D has no advantage over B. + +Appendix: any_folder experiment +------------------------------- + +For a brief moment, we had an ``any_folder`` extension but removed it before the docs-as-code release. +It breaks when using such documentation in ``:docs_combo``: +It relied on configuration in ``conf.py`` but with ``:docs_combo`` +the modules' ``conf.py`` is ignored and only the root ``conf.py`` is used.