[Tracking] BE build-time optimization: header-closure surgery, unity builds, and instantiation cuts

[morningman]

Description

Umbrella issue for the BE build-time optimization series: make a cold BE build several times faster and shrink the incremental rebuild radius of the hottest headers, with zero runtime behavior change.

The batch was developed and measured on one branch and published as the combined reference PR #66510, which stays open as the whole-batch reference carrying the end-to-end measurements. It is being landed as the focused PR series below; the split PRs supersede it.

Headline numbers

• [opt](build) 1/4: Speed up BE full build ~22% by cutting hot-header include edges #66400 (part 1, merged): cold full BE build ~-22% by cutting hot-header include edges.
• The rest of the batch, measured as one pairwise A/B at ./build.sh --compile-bench -j 6 on the same idle machine (details in [opt](build) 2/4: Halve the BE cold build (21m45s -> 11m10s, -48.7%) via unity builds, instantiation cuts and header-closure surgery #66510): 21m45s → 11m10s (-48.7%) end-to-end, sum of per-TU wall -51.3%, compile units 8380 → 7281.
• Incremental rebuild radius: today touching storage/olap_common.h rebuilds all 318 first-party TUs (through the PCH) plus the 219MB PCH itself; after the final PR in the series it rebuilds 182, and the PCH depends on 9 doris headers instead of 31.

PR series

Landed

• [opt](build) 1/4: Speed up BE full build ~22% by cutting hot-header include edges #66400 — 1/x: cut hot include edges under exec_env.h / runtime_state.h / thread_context.h (~-22% cold build)
• [fix](build) Fix the BE unit-test and benchmark build on macOS arm64 #66615 — fix the BE unit-test & benchmark build on macOS arm64; also fixes -fno-access-control never actually reaching doris_be_test on any platform
• [opt](build) Add build-timing and header-closure sweep tooling #66616 — build-timing and header-closure sweep tooling (closure-sweep.sh --no-pch, rebuild_radius.py, tu-bench.sh): the measurement gate the rest of the series points at
• [opt](exprs) Remove unreachable mixed-width decimal registrations for add/subtract/mod #66617 — remove unreachable mixed-width decimal registrations and hand-written constant-constant paths in binary arithmetic (sentinel TU -39% compile time, registration surface pinned by a contract UT)
• [opt](build) Cut three more waves of hot include edges in the BE header graph #66672 — 3/x: three more waves of hot include-edge cuts (column.h, RLE decoders, pipeline dependency.h), same seed-then-cut shape as [opt](build) 1/4: Speed up BE full build ~22% by cutting hot-header include edges #66400
• [opt](build) Add ENABLE_UNITY_BUILD and pilot unity builds on three glue targets #66712 — introduce ENABLE_UNITY_BUILD (default ON) and pilot unity builds on three glue targets (-11.8% end-to-end at -j14; information_schema segment 13×, http 10.4×)

In review

• [opt](build) Enable unity builds for the Exec and Exprs targets #66776 — unity builds for Exec and Exprs, the two heaviest targets (-12.9% and -11.2% end-to-end on the combined branch)

Planned — numbers measured on the combined branch, to be re-validated in each PR

• [opt](build) Enable unity builds for the remaining BE targets #66789 — unity builds for the remaining targets (Cloud / Storage / Runtime / Load / IO / Util / …, -20.7% on top; the conversion also surfaced 4 latent bugs that get fixed on the way)
• [opt](build) Declare extern template for existing explicit instantiations #66807 — extern template declarations for the existing explicit instantiations (~-2% cold build, doris_be -47MB, cross-TU duplicate weak symbols -68%)
• [opt](build) Take olap_common.h off the dependency base and slim the PCH #66826 — Decouple storage/olap_common.h from the bottom of the include graph and slim the PCH — the incremental-rebuild-radius PR (318 → 182 dependents; PCH trigger line 31 → 9 doris headers)
• Small standalone: DORIS_DEV_DEBUG_INFO=line-tables developer knob (-10.8% per TU for local debug builds; default output bit-identical)

How each PR is written

Every PR body carries its own measured numbers with the measurement conditions (-j level, PCH on/off, paired runs), its verification evidence (no-PCH closure sweep over all TUs, incremental rebuilds, BE UT, regression/microbenchmarks where semantics are touched), and a disclosure section for negative results and known blind spots — e.g. all local measurements are macOS/clang20, so Linux gcc/clang coverage relies on the PR CI lines.