mirror of
https://github.com/PX4/PX4-Autopilot.git
synced 2026-10-06 09:02:52 +08:00
ci(build-all): seed ccache only for chip families with cold caches (#28278)
* ci(build-all): seed ccache only for chip families with cold caches Measured across recent runs, the seeder stage contributes nothing in steady state: all 264 build-job cache restores in the 8 most recent successful runs came from group-level caches, zero from seeder caches, yet the stage gates the build matrix for ~3.4 min of every run's critical path and burns ~22 8cpu-runner-minutes warming caches nobody reads. Its fail-fast value is similarly thin: 2 of the last 40 failed runs failed at seed stage; the other 34 passed seeding and failed in the matrix anyway. Probe each family's ccache namespace in the RunsOn cache bucket from the group_targets job and emit only cold families into the seed matrix. Warm runs skip the stage entirely (the matrix starts right after the scan); a cold family still seeds first and gates the matrix, which is the one case the warmup pays for. The probe fails open: if the bucket env, the listing, or the assumed object layout is wrong, every family reads as cold and the workflow behaves exactly as before. Assisted-by: Claude:claude-fable-5 Signed-off-by: Ramon Roche <mrpollo@gmail.com> * ci(build-all): fail open when seeder probe output is unparseable The has_cold computation ran unguarded under bash -e: malformed probe output would fail the whole workflow instead of falling back to the full seeder matrix. Guard it so every failure path degrades to seeding everything, on RunsOn and on downstream forks without it alike. Assisted-by: Claude:claude-fable-5 Signed-off-by: Ramon Roche <mrpollo@gmail.com> * ci(build-all): move seeder probe logic into the helper script The probe step carried fail-open shell logic inline in the workflow. Move output writing, has_cold computation, and all fail-open handling into filter_cold_seeders.py; the workflow step is now a single script invocation. The script always exits zero and degrades every failure (missing bucket env, aws error, unparseable input) to the full seeder matrix. Assisted-by: Claude:claude-fable-5 Signed-off-by: Ramon Roche <mrpollo@gmail.com> * ci(build-all): probe the magic cache's real object layout The first probe guessed cache/{key} for the magic-cache object layout and read every family COLD, degrading to unconditional seeding. The real layout is cache/v1/{org}/{repo}/{ref}/{version-hash}/{cache-key}; the version hash is the actions/cache digest and not knowable a priori, so list the default-branch scope once (main-scope caches are visible to every ref) and match cache-key basenames. Verified locally against the live bucket: all 10 families read warm and the seed stage skips. Assisted-by: Claude:claude-fable-5 Signed-off-by: Ramon Roche <mrpollo@gmail.com> * ci(build-all): single-source ccache key roots in the matrix generator The ccache key convention was encoded three times: hand-assembled in the workflow's restore/save steps, again in the seeder's cache-key-prefix argument, and a third time in the probe script. Drift between them is silent thanks to the probe's fail-open design: a renamed key scheme would read every family COLD forever and quietly revert to unconditional seeding. Mint the namespace root once in generate_board_targets_json.py as cache_prefix on every group and seeder matrix entry, with seeders occupying the reserved 'seeder' group inside their family namespace. The workflow composes {cache_prefix}-{group}-{ref}-{sha} from matrix fields only, and the probe greps the root it is handed, knowing nothing of the format. Composed keys are byte-identical to the old scheme (asserted for all 33 groups and 10 seeders), so no cache is invalidated. Assisted-by: Claude:claude-fable-5 Signed-off-by: Ramon Roche <mrpollo@gmail.com> --------- Signed-off-by: Ramon Roche <mrpollo@gmail.com>
This commit is contained in:
Ramon Roche
@@ -66,13 +66,16 @@ jobs:
|
||||
group_targets:
|
||||
name: Scan for Board Targets
|
||||
# runs-on: ubuntu-latest
|
||||
runs-on: [runs-on,runner=1cpu-linux-x64,image=ubuntu24-full-x64,"run-id=${{ github.run_id }}",spot=false]
|
||||
runs-on: [runs-on,runner=1cpu-linux-x64,image=ubuntu24-full-x64,"run-id=${{ github.run_id }}",spot=false,extras=s3-cache]
|
||||
outputs:
|
||||
matrix: ${{ steps.set-matrix.outputs.matrix }}
|
||||
seeders: ${{ steps.set-seeders.outputs.seeders }}
|
||||
cold_seeders: ${{ steps.filter-seeders.outputs.cold_seeders }}
|
||||
has_cold: ${{ steps.filter-seeders.outputs.has_cold }}
|
||||
timestamp: ${{ steps.set-timestamp.outputs.timestamp }}
|
||||
branchname: ${{ steps.set-branch.outputs.branchname }}
|
||||
steps:
|
||||
- uses: runs-on/action@v2
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- name: Cache Python pip
|
||||
@@ -109,6 +112,15 @@ jobs:
|
||||
echo "EOF"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Seed only chip families whose ccache namespace has no warm cache.
|
||||
# Fail-open semantics live in the script: any probe problem emits
|
||||
# the full matrix, reproducing unconditional seeding.
|
||||
- id: filter-seeders
|
||||
name: Probe Family Caches
|
||||
env:
|
||||
ALL_SEEDERS: ${{ steps.set-seeders.outputs.seeders }}
|
||||
run: ./Tools/ci/filter_cold_seeders.py <<< "$ALL_SEEDERS"
|
||||
|
||||
- id: set-timestamp
|
||||
name: Save Current Timestamp
|
||||
run: |
|
||||
@@ -133,20 +145,25 @@ jobs:
|
||||
echo "$(./Tools/ci/generate_board_targets_json.py --group --verbose)"
|
||||
|
||||
# ===========================================================================
|
||||
# CACHE SEEDER JOBS
|
||||
# CACHE SEEDER JOBS (conditional)
|
||||
# ===========================================================================
|
||||
# Build one representative target per chip family to warm the ccache.
|
||||
# Matrix jobs fall back to these caches via restore-keys when no
|
||||
# group-specific cache exists yet. If any seeder fails, the build matrix
|
||||
# does not start, catching common build errors early.
|
||||
# Build one representative target per chip family to warm the ccache,
|
||||
# but only for families whose cache namespace came up cold in the
|
||||
# group_targets probe. Measured steady state is zero seeder restores
|
||||
# (group jobs live on their own rolling caches), so on warm runs the
|
||||
# whole stage skips and the build matrix starts immediately. When a
|
||||
# family IS cold (matrix regrouping, cache expiry, new chip family),
|
||||
# its seeder runs and still gates the matrix, which is the only case
|
||||
# where the warmup is worth waiting for.
|
||||
# ===========================================================================
|
||||
|
||||
seed:
|
||||
name: Seed [${{ matrix.chip_family }}]
|
||||
needs: group_targets
|
||||
if: needs.group_targets.outputs.has_cold == 'true'
|
||||
runs-on: [runs-on,"runner=8cpu-linux-${{ matrix.runner }}","image=ubuntu24-full-${{ matrix.runner }}","run-id=${{ github.run_id }}",spot=false,extras=s3-cache]
|
||||
strategy:
|
||||
matrix: ${{ fromJson(needs.group_targets.outputs.seeders) }}
|
||||
matrix: ${{ fromJson(needs.group_targets.outputs.cold_seeders) }}
|
||||
fail-fast: false
|
||||
container:
|
||||
image: ${{ matrix.container }}
|
||||
@@ -163,7 +180,7 @@ jobs:
|
||||
- uses: ./.github/actions/setup-ccache
|
||||
id: ccache
|
||||
with:
|
||||
cache-key-prefix: ccache-${{ matrix.chip_family }}-${{ matrix.runner }}-seeder
|
||||
cache-key-prefix: ${{ matrix.cache_prefix }}-${{ matrix.group }}
|
||||
max-size: 400M
|
||||
- name: Build seed target
|
||||
run: make ${{ matrix.target }}
|
||||
@@ -199,12 +216,12 @@ jobs:
|
||||
uses: actions/cache/restore@v5
|
||||
with:
|
||||
path: ~/.ccache
|
||||
key: ccache-${{ matrix.chip_family }}-${{ matrix.runner }}-${{ matrix.group }}-${{ github.ref_name }}-${{ github.sha }}
|
||||
key: ${{ matrix.cache_prefix }}-${{ matrix.group }}-${{ github.ref_name }}-${{ github.sha }}
|
||||
restore-keys: |
|
||||
ccache-${{ matrix.chip_family }}-${{ matrix.runner }}-${{ matrix.group }}-${{ github.ref_name }}-
|
||||
ccache-${{ matrix.chip_family }}-${{ matrix.runner }}-${{ matrix.group }}-${{ github.base_ref || 'main' }}-
|
||||
ccache-${{ matrix.chip_family }}-${{ matrix.runner }}-${{ matrix.group }}-
|
||||
ccache-${{ matrix.chip_family }}-${{ matrix.runner }}-
|
||||
${{ matrix.cache_prefix }}-${{ matrix.group }}-${{ github.ref_name }}-
|
||||
${{ matrix.cache_prefix }}-${{ matrix.group }}-${{ github.base_ref || 'main' }}-
|
||||
${{ matrix.cache_prefix }}-${{ matrix.group }}-
|
||||
${{ matrix.cache_prefix }}-
|
||||
|
||||
- name: Cache - Configure ccache
|
||||
run: |
|
||||
|
||||
Executable
+95
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Emit the seeder matrix filtered to chip families with no warm ccache.
|
||||
|
||||
Reads the full seeder matrix JSON on stdin and writes two GitHub Actions
|
||||
outputs (to $GITHUB_OUTPUT, or stdout when unset for local runs):
|
||||
|
||||
cold_seeders matrix containing only the families with no cache entry
|
||||
under their ccache namespace in the RunsOn magic-cache
|
||||
S3 bucket
|
||||
has_cold "true"/"false", whether cold_seeders has any entries
|
||||
|
||||
Fails open, never non-zero: on any probe problem (missing bucket env,
|
||||
aws cli error, unexpected input) the full input matrix is emitted with
|
||||
has_cold=true, reproducing unconditional seeding. The probe can only
|
||||
skip work, never leave a truly cold family unseeded.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def scope_key_basenames(bucket, scope):
|
||||
"""Cache-key basenames in one ref scope of the magic-cache bucket.
|
||||
|
||||
The magic cache stores GitHub cache entries as
|
||||
cache/v1/{org}/{repo}/{ref}/{version-hash}/{cache-key}; the version
|
||||
hash is the actions/cache path+compression digest and not knowable
|
||||
here, so list the whole ref scope (auto-paginated) and match on the
|
||||
final path component.
|
||||
"""
|
||||
prefix = f"cache/v1/{os.environ['GITHUB_REPOSITORY']}/{scope}/"
|
||||
result = subprocess.run(
|
||||
["aws", "s3", "ls", f"s3://{bucket}/{prefix}", "--recursive"],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(result.stderr.strip())
|
||||
return [line.split()[-1].rsplit("/", 1)[-1]
|
||||
for line in result.stdout.splitlines() if line.strip()]
|
||||
|
||||
|
||||
def filter_cold(matrix):
|
||||
bucket = os.environ["RUNS_ON_S3_BUCKET_CACHE"]
|
||||
# Default-branch caches are visible to every ref, so main-scope
|
||||
# warmth is what the build jobs' last restore-keys fallback can
|
||||
# always reach regardless of the branch being built.
|
||||
scope = os.environ.get("SEEDER_PROBE_SCOPE", "refs/heads/main")
|
||||
basenames = scope_key_basenames(bucket, scope)
|
||||
cold = []
|
||||
for entry in matrix["include"]:
|
||||
# The namespace root is minted by generate_board_targets_json.py
|
||||
# and arrives on every matrix entry; no key format knowledge here.
|
||||
prefix = entry["cache_prefix"] + "-"
|
||||
warm = any(name.startswith(prefix) for name in basenames)
|
||||
print(f"::notice title=seeder probe::{entry['chip_family']}/"
|
||||
f"{entry['runner']}: {'warm' if warm else 'COLD'}", file=sys.stderr)
|
||||
if not warm:
|
||||
cold.append(entry)
|
||||
return {"include": cold}
|
||||
|
||||
|
||||
def write_outputs(cold_json, has_cold):
|
||||
payload = (f"cold_seeders<<EOF\n{cold_json}\nEOF\n"
|
||||
f"has_cold={'true' if has_cold else 'false'}\n")
|
||||
path = os.environ.get("GITHUB_OUTPUT")
|
||||
if path:
|
||||
with open(path, "a") as out:
|
||||
out.write(payload)
|
||||
else:
|
||||
sys.stdout.write(payload)
|
||||
|
||||
|
||||
def main():
|
||||
raw = sys.stdin.read()
|
||||
try:
|
||||
matrix = json.loads(raw)
|
||||
except ValueError as exc:
|
||||
# Unparseable input passes through verbatim so the seed matrix
|
||||
# fails loudly downstream instead of being silently skipped.
|
||||
print(f"::notice title=seeder probe::fail-open, bad input: {exc}",
|
||||
file=sys.stderr)
|
||||
write_outputs(raw, True)
|
||||
return 0
|
||||
try:
|
||||
cold = filter_cold(matrix)
|
||||
except Exception as exc:
|
||||
print(f"::notice title=seeder probe::fail-open, seeding all "
|
||||
f"families: {exc}", file=sys.stderr)
|
||||
cold = matrix
|
||||
write_outputs(json.dumps(cold), bool(cold["include"]))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -493,9 +493,14 @@ if (args.group):
|
||||
})
|
||||
chunk_counter += 1
|
||||
|
||||
# Add cache_size to each group based on chip family
|
||||
# Add cache_size and the shared ccache namespace root to each group.
|
||||
# cache_prefix is the single source of the cache key convention: the
|
||||
# workflow composes keys as {cache_prefix}-{group}-{ref}-{sha} with a
|
||||
# final restore fallback of {cache_prefix}-, and the seeder probe
|
||||
# (Tools/ci/filter_cold_seeders.py) greps the bucket for the same root.
|
||||
for g in final_groups:
|
||||
g['cache_size'] = CHIP_CACHE_SIZES.get(g['chip_family'], DEFAULT_CACHE_SIZE)
|
||||
g['cache_prefix'] = f"ccache-{g['chip_family']}-{g['runner']}"
|
||||
|
||||
if(verbose):
|
||||
import pprint
|
||||
@@ -553,6 +558,13 @@ if (args.group):
|
||||
'runner': 'x64',
|
||||
})
|
||||
|
||||
# Seeders share the family namespace root with the build groups and
|
||||
# occupy the reserved "seeder" group inside it, so seeder caches are
|
||||
# always reachable from the build jobs' {cache_prefix}- fallback.
|
||||
for s in seeders:
|
||||
s['group'] = 'seeder'
|
||||
s['cache_prefix'] = f"ccache-{s['chip_family']}-{s['runner']}"
|
||||
|
||||
print(json.dumps({ "include": seeders }, **extra_args))
|
||||
else:
|
||||
print(json.dumps({ "include": final_groups }, **extra_args))
|
||||
|
||||
Reference in New Issue
Block a user