From dbb4212ff3247fa84a388d84eef3247ee2f602fa Mon Sep 17 00:00:00 2001 From: Ramon Roche Date: Tue, 7 Jul 2026 11:05:11 -0700 Subject: [PATCH] docs(test_and_ci): add hardware bench testing guide New page in Platform Testing and CI covering px4bench: why on-hardware verification exists alongside SITL and CI, the firmware traceability gate, the bench test inventory, the SIH hardware-in-the-loop flight test with Hawkeye, baseline comparison for upgrade regression and golden-unit workflows, production end-of-line usage, and CI integration. Linked from the section index and the sidebar. Signed-off-by: Ramon Roche --- docs/en/SUMMARY.md | 1 + docs/en/test_and_ci/bench_testing.md | 127 +++++++++++++++++++++++++++ docs/en/test_and_ci/index.md | 1 + 3 files changed, 129 insertions(+) create mode 100644 docs/en/test_and_ci/bench_testing.md diff --git a/docs/en/SUMMARY.md b/docs/en/SUMMARY.md index f859e306965..6d85ff4fccd 100644 --- a/docs/en/SUMMARY.md +++ b/docs/en/SUMMARY.md @@ -913,6 +913,7 @@ - [Fixed Wing](test_and_ci/test_flights.md#fixed-wing) - [Test FW_01 - Manual Modes](test_cards/fw_01_manual_modes.md) - [Test FW_02 - Full Autonomous](test_cards/fw_02_full_autonomous.md) + - [Hardware Bench Testing (px4bench)](test_and_ci/bench_testing.md) - [Unit Tests](test_and_ci/unit_tests.md) - [Fuzz Tests](test_and_ci/fuzz_tests.md) - [Sanitizers](test_and_ci/sanitizers.md) diff --git a/docs/en/test_and_ci/bench_testing.md b/docs/en/test_and_ci/bench_testing.md new file mode 100644 index 00000000000..a4fabcc62f0 --- /dev/null +++ b/docs/en/test_and_ci/bench_testing.md @@ -0,0 +1,127 @@ +# Hardware Bench Testing (px4bench) + +px4bench is the PX4 toolkit for automated verification on real flight-controller hardware. +It qualifies a firmware build on a board sitting on a bench: no props, no airframe, just a USB cable and optionally a telemetry radio. + +Use this guide if you are: + +- **Qualifying a release**: verify a release candidate on real NuttX hardware before cutting a branch or handing a beta to a test team. +- **Sanity-checking a change**: prove that a PR or local build boots, communicates, stores parameters and missions, and survives reboots on a real board. +- **Testing flight logic hardware-in-the-loop**: fly a full auto mission in simulation on the FMU itself (SIH) and verify arming, takeoff, mission progression, landing, and disarm on real RTOS scheduling. +- **Running end-of-line checks in production**: manufacturers can run the same suite against every unit that comes off the line, with machine-readable pass/fail results and full firmware traceability per unit. + +The toolkit lives in the PX4-Autopilot source tree at [`Tools/bench_test/`](https://github.com/PX4/PX4-Autopilot/tree/main/Tools/bench_test), and its [README](https://github.com/PX4/PX4-Autopilot/blob/main/Tools/bench_test/README.md) is the complete reference. +This page explains what it does and how to apply it. + +## Why Bench Testing + +CI builds every NuttX target but never boots one, and [SITL](../simulation/index.md) runs flight code on a host operating system whose threading, scheduling, and C library behave differently from the RTOS on a flight controller. +Some defects therefore only exist on real hardware: boot-time initialization ordering, link lifecycle across USB re-enumeration, storage and filesystem behavior, loop-rate drift, and concurrency bugs that are invisible on a desktop OS. +A build can pass every SITL test and still ship with a dead telemetry link. + +Failures on embedded targets are frequently silent hangs rather than crashes, so px4bench is built around one rule: **every operation has a timeout, and a timeout is reported as a failure naming exactly what stalled**. +Every check prints `[PASS]` or `[FAIL]`, every test exits nonzero on failure, and every run writes a timestamped report directory containing the raw evidence. + +## Firmware Traceability + +Qualification means knowing what you tested. +Before any test runs, the suite reads the board's identity (git hash, version, hardware) and records it in `firmware.json` in every report directory. +You state where the firmware under test comes from, and the suite verifies the flashed board actually reports the expected git hash: + +```sh +# flash a local .px4 file, verify its identity, then test +./run_bench_suite.py /dev/ttyACM0 --firmware px4_fmu-v6xrt_default.px4 + +# build the current source tree for the connected board, flash, verify, test +./run_bench_suite.py /dev/ttyACM0 --build + +# download a GitHub release artifact, flash, verify, test +./run_bench_suite.py /dev/ttyACM0 --release v1.17.0 + +# no flashing: assert the board already runs the expected build +./run_bench_suite.py /dev/ttyACM0 --expect-hash 0c000d59 + +# explicit opt-out: test whatever is on the board (still recorded) +./run_bench_suite.py /dev/ttyACM0 --any-firmware +``` + +Run interactively with none of these flags, the suite shows what is on the board and asks what to do. +Run non-interactively (CI, production scripting), it refuses to start without an explicit firmware expectation, so automation can never silently test the wrong build. + +## The Bench Suite + +The bench tests run against normal firmware and never arm the vehicle. + +| Test | What it verifies | +| ------------------ | ----------------------------------------------------------------------------------------------------------------------- | +| `boot_health` | System snapshot over the MAVLink shell: no task in error state, required work queues present, uORB topics publishing. Has a baseline mode (see below). | +| `reboot_loop` | Repeated reboot/reconnect cycles; catches boards that boot once but not every time. | +| `usb_replug` | USB unplug/replug cycles (operator-assisted); catches link-teardown leaks in mavlink instance count or RAM. | +| `link_forwarding` | Simultaneous heavy traffic on two links with forwarding; catches deadlocks and stalls in the MAVLink send path. Requires a second link. | +| `param_stress` | Full parameter download, 50 set/readback cycles, save, reboot, persistence verification. | +| `mission_stress` | Repeated 220-waypoint upload/download/compare/clear transactions, alternating links when two are available. | +| `log_transfer` | Short log capture, MAVFTP download, ULog integrity verification. | + +```sh +cd Tools/bench_test +./run_bench_suite.py /dev/ttyACM0 --build # everything, single link +./run_bench_suite.py /dev/ttyACM0 /dev/ttyUSB0 --expect-hash # + radio: adds dual-link tests +``` + +A per-test watchdog turns a hung board into a named failure instead of a stuck terminal. + +## Simulated Flight on the FMU (SIH) + +`sih/flight_mission.py` performs a complete hardware-in-the-loop flight without any host-side simulator: it switches the board to a [SIH airframe](../sim_sih/index.md) (`SYS_HITL=2`, physics computed on the autopilot), uploads a mission, arms through the NuttX shell, and asserts takeoff, waypoint progression, RTL, landing, and auto-disarm against per-phase timeouts. +The flight log is downloaded into the report directory automatically for post-flight analysis, and the original board configuration is restored afterwards. + +Real outputs are replaced by `pwm_out_sim` in this mode, so nothing is driven on the output rails; still, run it only on a bench board with nothing connected. +The firmware must be built with `CONFIG_MODULES_SIMULATION_SIMULATOR_SIH=y`. + +To watch the flight live in 3D, install [Hawkeye](https://github.com/PX4/Hawkeye) and run the test with `--viewer`: + +```sh +hawkeye -udp 19410 -mc & +./sih/flight_mission.py /dev/ttyACM0 --viewer +``` + +## Baseline Comparison + +`boot_health` can diff two report directories, comparing uORB publication rates, work queues, and topics between runs: + +```sh +./bench/boot_health.py /dev/ttyACM0 # capture (prints report dir) +# ... flash a different build ... +./bench/boot_health.py /dev/ttyACM0 +./bench/boot_health.py --baseline # FAIL on rate drift or lost topics +``` + +This supports two workflows: + +- **Upgrade regression**: capture on the current release, capture on the candidate, diff. A control-loop rate that quietly dropped between versions is a finding long before it becomes mushy flight behavior. +- **Golden unit** (manufacturing): capture a known-good reference unit once, then diff every produced unit against it. A board with a marginal sensor or misloaded configuration deviates from the golden baseline even when it nominally boots. + +## Production End-of-Line Usage + +For manufacturers the suite is designed to run unattended per unit: + +```sh +./run_bench_suite.py "$PORT" --firmware "$RELEASE_PX4" --report-dir "reports/$SERIAL" \ + || echo "UNIT $SERIAL FAILED" +``` + +- Exit code is the verdict: `0` all checks passed, nonzero otherwise, with the failing check named in the output. +- `reports//` retains the full evidence per unit: firmware identity, system snapshot, parameter and mission transaction results, downloaded logs. +- The firmware gate guarantees each unit was tested against the intended production image, not whatever was in flash. +- `usb_replug` (connector/enumeration exercise) is the only operator-assisted test and can be included where a physical connector check is part of the procedure. + +## CI Integration + +The suite is built to sit behind a self-hosted runner with a board permanently attached: non-interactive runs demand an explicit firmware source, all results are machine-readable exit codes plus report artifacts, and a wedged board fails fast with a named check rather than hanging the job. +Pairing `--build` (current revision) with the bench suite plus a SIH flight closes the gap where CI compiles NuttX firmware without ever executing it. + +## Further Information + +- [`Tools/bench_test/README.md`](https://github.com/PX4/PX4-Autopilot/blob/main/Tools/bench_test/README.md): complete reference (all flags, per-test details, troubleshooting, how to add a test) +- [SIH on Hardware](../sim_sih/hardware.md) +- [Test Flights](../test_and_ci/test_flights.md): real flight testing, which bench testing complements but does not replace diff --git a/docs/en/test_and_ci/index.md b/docs/en/test_and_ci/index.md index 5ecf545d81f..218f298936b 100644 --- a/docs/en/test_and_ci/index.md +++ b/docs/en/test_and_ci/index.md @@ -6,6 +6,7 @@ Live flight testing is also performed by the development team and the broader co Test topics include: - [Test Flights](../test_and_ci/test_flights.md) - How to make test flights (e.g. to [test PRs](../contribute/code.md#pull-requests)) +- [Hardware Bench Testing (px4bench)](../test_and_ci/bench_testing.md) - Automated verification on real flight-controller hardware: release qualification, hardware-in-the-loop flight testing, and production end-of-line checks - [Unit Tests](../test_and_ci/unit_tests.md) - [Sanitizers](../test_and_ci/sanitizers.md) - Build SITL with ASan/TSan to catch memory errors and data races - [Continuous Integration (CI)](../test_and_ci/continous_integration.md)