ci(build-all): build two targets per job concurrently

Each build job compiled its group one target at a time, and the serial
parts of every build (configure, the NuttX make, linking) left the 4-vCPU
runners about half idle. Two concurrent builds keep them around 80% busy
and cut total build-step time about 24% and runner cost about 17%.
Larger runners and more slots were measured and cost more than they save.

build_all_runner.py replaces build_all_runner.sh. NuttX compiles inside
its source tree, so each extra slot is a git worktree under .build_slots/.
All submodules are fetched once before building, each slot clones them
locally from the checkout, and builds run with GIT_SUBMODULES_ARE_EVIL=1
so CMake's per-configure submodule check does not sync and update them
concurrently on the shared .git/config. Targets that write the same
build/ directory (<board>_deb with <board>_default, the metadata targets
with px4_sitl_default) always share a slot, and the runner refuses a plan
that would split them. Every target is built even after a failure. Build
output streams with a [seconds|slot target] prefix, each target prints its
memory usage or, on failure, the excerpt from the first error, and a Build
Logs step prints each target's full log as one collapsed group. Slot build
directories are moved into build/, so packaging is unchanged.

The matrix now carries targets as a list plus a slot count from
build_all_config.yml; the comma-joined string existed only because
workflow expressions could not pass an array to a shell command, which
join() now does. The scan job runs new tests for the generated matrix and
the slot assignment.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ramon Roche <mrpollo@gmail.com>
This commit is contained in:
Ramon Roche
2026-09-28 09:45:09 -07:00
parent 9a90020437
commit 89c68cd37f
8 changed files with 616 additions and 39 deletions
+12 -3
View File
@@ -90,6 +90,10 @@ jobs:
- name: Install Python Dependencies
run: pip3 install pyyaml kconfiglib 'empy>=3.3,<4' pyros-genmsg
- name: Test Build Matrix and Slot Assignment
working-directory: Tools/ci
run: python3 -m unittest -v test_generate_board_targets_json test_build_all_runner
# The generator runs as a standalone assignment so a non-zero exit
# fails the step; embedded in `echo "x=$(...)"` it would be masked
# and the empty output silently skips the whole build matrix.
@@ -236,9 +240,14 @@ jobs:
ccache -s
ccache -z
- name: Building Artifacts for [${{ matrix.targets }}]
run: |
./Tools/ci/build_all_runner.sh ${{matrix.targets}} ${{matrix.arch}}
- name: Building Artifacts
run: ./Tools/ci/build_all_runner.py --slots ${{ matrix.slots }} ${{ join(matrix.targets, ' ') }}
# Full per-target compiler output, one collapsed group per target, kept
# out of the build step so a running job shows progress, not logs.
- name: Build Logs
if: ${{ !cancelled() }}
run: ./Tools/ci/build_all_runner.py --print-logs
- name: Arrange Build Artifacts
run: |