Files
threadx/.github/workflows/regression_template.yml
T
Frédéric Desbiens 7959aef3bc Kept the coverage report from the runs that most need one (#659)
A failing test threw away coverage that had already been collected, and the
run whose behaviour changed is exactly the run whose coverage is worth
reading. Measured on the failing run of 2026-08-18: it uploaded test_reports
for all three suites and no coverage_report artifact at all.

Two causes, and the workflow one is the smaller of them.

cmake_bootstrap.sh runs under set -e, so a failing ctest aborted test()
before ./coverage.sh was reached. The gcda files exist by that point, so
nothing was missing except the step that reads them. ctest's status is now
captured and returned at the end, and the summary grep is allowed to fail
rather than being the thing that stops the coverage behind it.

The serial branch of the test dispatch collected no status either, so under
set -e the first failing configuration stopped the remaining four from being
tested at all -- and their coverage from being collected. That was cheap
while the suites ran in parallel, because the parallel branch already
collects exit codes from its background jobs. Moving to serial execution in
#643 quietly made one failure cost the other four configurations. The serial
branch now collects status the same way the parallel branch does.

With those fixed the report exists, so the workflow steps that publish it no
longer skip on failure. They are guarded with !cancelled() rather than
always(), so a cancelled run still stops promptly, which is the idiom
deploy_code_coverage already uses. The ${{ }} wrapping is required and not
decoration: a bare ! opens a YAML tag, and the file will not parse without
it.

Verified locally by replacing one test binary with a stub that exits 1:

  before  the failing run of 2026-08-18 produced no coverage_report artifact
  after   run.sh test default_build_coverage exits 8, and produces
          coverage_report/default_build_coverage.xml with 177 files and
          3804 of 3827 lines
  after   run.sh test all exits 8, and all five configurations run rather
          than stopping at the first

The failure still fails. Only the reporting around it changed.

Assisted-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-24 16:36:18 -04:00

251 lines
9.4 KiB
YAML

# This is a basic workflow that is manually triggered
name: regression_template
on:
workflow_call:
inputs:
install_script:
default: './scripts/install.sh'
required: false
type: string
build_script:
default: './scripts/build.sh'
required: false
type: string
test_script:
default: './scripts/test.sh'
required: false
type: string
cmake_path:
default: './test/cmake'
required: false
type: string
skip_test:
default: false
required: false
type: boolean
skip_coverage:
default: false
required: false
type: boolean
coverage_name:
default: 'default_build_coverage'
required: false
type: string
skip_deploy:
default: false
required: false
type: boolean
deploy_list:
default: ''
required: false
type: string
result_affix:
default: ''
required: false
type: string
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "linux_job"
run_tests:
if: ${{ !inputs.skip_test}}
permissions:
contents: read
issues: read
checks: write
pull-requests: write
# The type of runner that the job will run on
runs-on: ubuntu-24.04
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
- name: Check out the repository
uses: actions/checkout@v4.2.2
with:
submodules: true
# apt and PyPI have stalled this step twice, for 55 minutes and for over two
# hours, against a normal 27 to 152 seconds. Nothing here had a timeout, so a
# stall ran until the six hour job limit and cost a whole run rather than
# failing and being retried.
- name: Install softwares
timeout-minutes: 10
run: ${{ inputs.install_script }}
- name: Build
timeout-minutes: 15
run: ${{ inputs.build_script }}
- name: Test
timeout-minutes: 60
run: ${{ inputs.test_script }}
- name: Publish Test Results
uses: EnricoMi/publish-unit-test-result-action@v2.11.0
if: always()
with:
check_name: Test Results ${{ inputs.result_affix }}
files: |
${{ inputs.cmake_path }}/build/*/*.xml
- name: Upload Test Results
if: success() || failure()
uses: actions/upload-artifact@v4.6.2
with:
name: test_reports ${{ inputs.result_affix }}
path: |
${{ inputs.cmake_path }}/build/*.txt
${{ inputs.cmake_path }}/build/*/Testing/**/*.xml
${{ inputs.cmake_path }}/build/**/regression/output_files/*.bin
- name: Configure GitHub Pages
uses: actions/configure-pages@v5.0.0
# The coverage steps below run even when a test failed. cmake_bootstrap.sh
# now produces the report in that case, and the run whose behaviour changed
# is the one whose coverage is worth reading; previously a single flaky test
# suppressed the report for the whole run. !cancelled() rather than always(),
# so a cancelled run still stops promptly -- the same idiom the
# deploy_code_coverage job below already uses. The ${{ }} is required: a bare
# ! opens a YAML tag, and the expression will not parse without it.
- name: Generate Code Coverage Results Summary
if: ${{ !cancelled() && (!inputs.skip_coverage) }}
uses: irongut/CodeCoverageSummary@v1.3.0
with:
filename: ${{ inputs.cmake_path }}/coverage_report/${{ inputs.coverage_name }}.xml
format: markdown
badge: true
hide_complexity: true
output: file
- name: Write Code Coverage Summary
if: ${{ !cancelled() && (!inputs.skip_coverage) }}
run: |
echo "## Coverage Report ${{ inputs.result_affix }}" >> $GITHUB_STEP_SUMMARY
cat code-coverage-results.md >> $GITHUB_STEP_SUMMARY
- name: Create CheckRun for Code Coverage
if: ${{ !cancelled() && ((github.event_name == 'push') || (github.event_name == 'workflow_dispatch') || (github.event.pull_request.head.repo.full_name == github.repository)) && (!inputs.skip_coverage) }}
uses: LouisBrunner/checks-action@v2.0.0
with:
token: ${{ secrets.GITHUB_TOKEN }}
name: Code Coverage ${{ inputs.result_affix }}
conclusion: ${{ job.status }}
output: |
{"summary":"Coverage Report"}
output_text_description_file: code-coverage-results.md
- name: Add Code Coverage PR Comment
if: ${{ !cancelled() && ((github.event_name == 'push') || (github.event.pull_request.head.repo.full_name == github.repository)) && (!inputs.skip_coverage) }}
uses: marocchino/sticky-pull-request-comment@v2.9.4
with:
header: Code Coverage ${{ inputs.result_affix }}
path: code-coverage-results.md
# Add sudo to move coverage folder created by root user
- name: Prepare Coverage GitHub Pages
if: ${{ !cancelled() && (!inputs.skip_coverage) }}
run: >-
if [ "${{ inputs.result_affix }}" != "" ] && ${{ inputs.skip_deploy }}; then
sudo mv ${{ inputs.cmake_path }}/coverage_report/${{ inputs.coverage_name }} \
${{ inputs.cmake_path }}/coverage_report/${{ inputs.result_affix }}
fi
- name: Coverage Report name
id: artifact
if: ${{ !cancelled() && (!inputs.skip_coverage) }}
run: echo "coverage_report=coverage_report-$(date +%s)" >> $GITHUB_OUTPUT
- name: Upload Code Coverage Artifacts
uses: actions/upload-artifact@v4.6.2
if: ${{ !cancelled() && (inputs.skip_deploy && !inputs.skip_coverage) }}
with:
name: ${{ steps.artifact.outputs.coverage_report }}
path: ${{ inputs.cmake_path }}/coverage_report
retention-days: 1
- name: Upload Code Coverage Pages
uses: actions/upload-pages-artifact@v3.0.1
if: ${{ !cancelled() && (!inputs.skip_deploy && !inputs.skip_coverage) }}
with:
path: ${{ inputs.cmake_path }}/coverage_report/${{ inputs.coverage_name }}
deploy_code_coverage:
runs-on: ubuntu-24.04
if: ((github.event_name == 'push') || (github.event_name == 'workflow_dispatch')) && !inputs.skip_coverage && !inputs.skip_deploy && !failure() && !cancelled()
needs: run_tests
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
permissions:
pages: write
id-token: write
steps:
# Selects the coverage artifacts by pattern rather than by name.
#
# This used to ask for ${{ steps.artifact.outputs.coverage_report }},
# which is set by the "Coverage Report name" step of run_tests -- a
# different job. The steps context does not cross jobs, so that
# expression evaluated to the empty string and the action fell back to
# its documented behaviour for an unspecified name: "No input name,
# artifact-ids or pattern filtered specified, downloading all
# artifacts". It pulled down four, the two coverage reports and the two
# test_reports bundles of JUnit XML, each into a directory named after
# the artifact -- so the published site carried the test reports as well
# as the coverage, under paths containing a run timestamp that changed
# on every publish.
#
# merge-multiple puts the contents of both coverage artifacts directly
# into coverage_report rather than under a directory named for each
# artifact. Each one holds a single directory named for its suite,
# ThreadX or SMP, renamed from default_build_coverage by the "Prepare
# Coverage GitHub Pages" step, so the merge yields exactly the two
# suite directories the deploy expects, and the artifact name -- with
# its timestamp -- stops appearing in the published path at all.
#
# The two artifacts also each carry a default_build_coverage.xml, and
# the merge means one overwrites the other. That file is consumed by
# CodeCoverageSummary back in run_tests and is not read here, so this is
# untidy rather than wrong.
- uses: actions/download-artifact@v4.3.0
if: ${{ inputs.skip_test }}
with:
pattern: coverage_report-*
merge-multiple: true
path: ${{ inputs.cmake_path }}/coverage_report
- name: Upload Code Coverage Pages
uses: actions/upload-pages-artifact@v3.0.1
if: ${{ inputs.skip_test }}
with:
path: .
# The artifacts are named coverage_report-<epoch> by run_tests, so the
# bare name matched nothing: useGlob defaults to true in this action, and
# as a glob "coverage_report" matches only the literal string. The step
# has therefore been deleting nothing. It does not fail on a miss, which
# is why that went unnoticed; retention-days: 1 on the upload is what has
# actually been clearing these up.
- name: Delete Duplicate Code Coverage Artifact
uses: geekyeggo/delete-artifact@v5.1.0
with:
name: coverage_report-*
- name: Deploy GitHub Pages site
id: deployment
uses: actions/deploy-pages@v4.0.5
- name: Write Code Coverage Report URL
run: >-
if [ "${{ inputs.deploy_list }}" != "" ]; then
for i in ${{ inputs.deploy_list }}; do
echo 'Coverage report for ' $i ':${{ steps.deployment.outputs.page_url }}'$i >> $GITHUB_STEP_SUMMARY
done
else
echo 'Coverage report: ${{ steps.deployment.outputs.page_url }}' >> $GITHUB_STEP_SUMMARY
fi