diff --git a/Tools/bench_test/README.md b/Tools/bench_test/README.md index 5fa37e44b0c..d8a5c2d4e31 100644 --- a/Tools/bench_test/README.md +++ b/Tools/bench_test/README.md @@ -21,10 +21,13 @@ than crashes, so the suite is built around one rule: every operation has a timeout, and a timeout is reported as FAIL naming exactly what stalled. A hang is the finding, not a nuisance. -The bench tests never arm the vehicle; logging is triggered with -`logger on` / `logger off`. The one test that flies does so in simulation on -the FMU (SIH, `SYS_HITL=2`) with `pwm_out_sim` in place of real outputs, and -it lives in its own directory so there is no ambiguity about which is which. +The tests under `bench/` never arm the vehicle; logging is triggered with +`logger on` / `logger off`. The default suite ends with one test that flies +in simulation on the FMU (SIH, `SYS_HITL=2`, `pwm_out_sim` in place of real +outputs): it arms the flight controller, so it runs behind an explicit +arming confirmation (or `--allow-arming` for automation) and the board must +be bare, with nothing on the output rails. It lives in `sih/` so the +simulation/no-simulation boundary stays obvious. ## Layout @@ -47,7 +50,9 @@ Tools/bench_test/ link_forwarding.py param_stress.py mission_stress.py + storage_stress.py log_transfer.py + serial_loopback.py sih/ # simulation-in-hardware (SYS_HITL=2) tests flight_mission.py ``` @@ -86,13 +91,23 @@ first; it holds the serial port. # alternating mission uploads) ./run_bench_suite.py /dev/tty.usbmodem01 /dev/tty.usbserial-RADIO --expect-hash 0c000d59 -# operator-assisted USB re-enumeration test (not part of the suite) +# operator-assisted tests (not in the suite: need an operator or a fixture) ./bench/usb_replug.py /dev/tty.usbmodem01 +./bench/serial_loopback.py /dev/tty.usbmodem01 --device /dev/ttyS2 -# simulated flight on the FMU (reconfigures the board, run separately) +# the simulated flight can also run standalone ./sih/flight_mission.py /dev/tty.usbmodem01 --expect-hash 0c000d59 ``` +The default suite sequence is boot_health, param_stress, mission_stress +(link_forwarding after it when a second link is given), storage_stress, +log_transfer, reboot_loop, and finally the simulated flight +(flight_mission). The flight arms the flight controller: on a TTY you are +asked to type 'arm' to confirm (declining records a skip, not a failure); +non-interactive runs skip it unless `--allow-arming` is passed. Tests that +probe the firmware and find a needed command missing record SKIP with a +warning instead of failing. + ## Firmware gate Qualification means knowing what you tested. Before any test starts the @@ -135,6 +150,16 @@ detected identity and asks: continue on the current firmware, flash a local abort. In automation (stdin not a TTY) the gate refuses to guess and exits with an error listing the flags; CI must always state what it is testing. +With `--build`, after resolving the target the gate reads the board config +(default.px4board plus the label overlay for labeled targets) and prints a +bench capability report: which bench-relevant modules the image will +contain (simulator_sih, sd_bench, sd_stress, serial_test) and which suite +tests will therefore run or skip. For anything missing it names a sibling +variant of the same board that has it (for example +`--target px4_fmu-v6xrt_bench`) and the exact config line to add; +interactively it offers to append the line to the local board config, +which stays local and uncommitted (commit it yourself for it to stick). + Wedged-board note: a board whose mavlink is hung cannot soft-reboot into the bootloader, so the uploader sits in its reboot-request loop. The gate detects this and prints an operator instruction to unplug and replug USB; @@ -169,7 +194,9 @@ why the stress tests loop. | bench/link_forwarding | reworked mavlink nested-send lock path (silently deadlockable on NuttX) | Silent traffic gap > 5s on either link, a stalled param download, or a dead link after heavy simultaneous two-link use | | bench/param_stress | parameter storage concurrency from the 2026-07-02 TSAN batch (DynamicSparseLayer races, AtomicTransaction) | Echo/readback mismatch under repetition, a download that stalls mid-list, or a save that does not survive reboot | | bench/mission_stress | new mission shared-state mutex (#27813) + dataman TSAN fixes | Upload/download round-trip corruption or a handshake that stalls; with two links the mutex is hit from two channels | +| bench/storage_stress | SD card and FAT filesystem under the logger's feet | Sequential write/read below the floor, fsync latency spikes that cause log dropouts, or file churn failures | | bench/log_transfer | logger + MAVFTP (FILE_TRANSFER_PROTOCOL replies are nested sends) | A burst-read download that stops mid-file, or a ULog that fails integrity checks | +| bench/serial_loopback | UART driver TX/RX path through a physical loopback jumper (fixture) | Lost or corrupted bytes through the jumper, or a port that cannot sustain the byte rate | | sih/flight_mission | commander/navigator/land-detector flight logic on real NuttX scheduling | Arming, takeoff, waypoint progression, RTL, land detection or auto-disarm failing per-phase timeouts | ## Bench tests (real firmware, no simulation) @@ -241,6 +268,38 @@ Full param download, then a set/readback loop (default 50) on 220-item mission upload/download/item-by-item compare/clear/verify-cleared, default 10 iterations, alternating links when two are given. +### storage_stress + +``` +./bench/storage_stress.py CONNECTION [-b BAUD] [--report-dir DIR] + [--bench-runs N] [--bench-duration-ms MS] [--bench-block B] + [--min-write-kbs X] [--min-read-kbs Y] [--max-fsync-ms Z] + [--stress-runs N] [--stress-bytes B] +``` + +Drives the firmware's own storage tools over the shell: `sd_bench` with +data verification for sequential write/read throughput and fsync latency, +then `sd_stress` for file create/rename/delete churn (100 files per +iteration). Probes first: firmware without the commands, or a board +without an SD card, records SKIP with a warning; the two phases degrade +independently (sd_bench present but sd_stress absent runs only the bench +phase). + +Threshold defaults are generous FAIL floors, chosen so a slow but working +card warns instead of flapping: + +- `--min-write-kbs 100`: the default logger profile needs roughly + 50-100 KB/s sustained; below 100 KB/s sequential write at 4 KB blocks a + card cannot log reliably. Healthy cards do an order of magnitude more. +- `--min-read-kbs 200`: far below any usable card; catches only genuinely + broken read paths. +- `--max-fsync-ms 500`: fsync stalls beyond a few hundred ms are what + cause logger dropouts; occasional 100-300 ms peaks on cheap cards are + normal and stay under this ceiling. + +Anything above a floor but within 4x of it additionally prints a WARNING +line so marginal cards are visible without failing the suite. + ### log_transfer ``` @@ -251,6 +310,24 @@ default 10 iterations, alternating links when two are given. .ulg via MAVFTP and verify it (pyulog when available, ULog magic bytes otherwise). A stalled transfer reports the byte count it reached. +### serial_loopback (operator/fixture, not in the suite) + +``` +./bench/serial_loopback.py CONNECTION --device /dev/ttySn [-b BAUD] + [--test-baud BAUD] [--seconds S] [--no-prompt] + [--report-dir DIR] +``` + +Manufacturing/end-of-line check for the UART hardware path. The operator +(or the test fixture) installs a loopback jumper wiring TX to RX on the +UART under test, then `serial_test` transmits a known pattern for +`--seconds` and verifies its own received bytes. PASS means bytes flowed +both ways with zero pattern errors and matching rx/tx counts. The port +must not be claimed by a running service (mavlink instance, GPS, RC); +`--no-prompt` skips the jumper pause for fixtures that pre-install it. +Skips with a warning when the firmware lacks `serial_test` +(CONFIG_SYSTEMCMDS_SERIAL_TEST=y). + ## SIH test (simulation on the FMU) ### flight_mission @@ -258,6 +335,7 @@ otherwise). A stalled transfer reports the byte count it reached. ``` ./sih/flight_mission.py CONNECTION [-b BAUD] [--airframe N] [--alt M] [--viewer] [--viewer-port PORT] [--keep-config] + [--allow-arming] [--expect-hash PREFIX] [--board-dev DEV] [--report-dir DIR] ``` @@ -270,9 +348,16 @@ failure. `--viewer` tees every MAVLink frame to UDP so Hawkeye (`hawkeye -udp 19410 -mc`) renders the flight live from the serial-connected board. -This is deliberately separated from `bench/`: it reconfigures the board and -exercises simulated flight logic, while the bench tests exercise real -firmware paths without simulation. +Runs last in the default suite and works standalone. Before doing anything +it probes the live firmware for the SIH module (`simulator_sih status`); if +nsh replies command not found, it warns and records SKIP instead of +failing. It then passes the arming gate: the flight controller WILL ARM +(simulated flight, but the board must be bare), so on a TTY the operator +must type 'arm' to proceed, and non-interactive runs skip unless +`--allow-arming` is given. Reconfiguring the board (airframe, `SYS_HITL`) +is expected and accepted; everything is restored afterwards, `--keep-config` +opts out. The `sih/` directory split only marks the simulation/no-simulation +boundary. ## Baseline workflow @@ -299,7 +384,12 @@ Conventions the suite relies on; new tests should follow all of them: test hides the hang it was supposed to expose. - Restore any board state you change (params, airframe), in a `finally`, even when the test failed. -- No arming in `bench/`. Anything that flies (simulated) goes in `sih/`. +- No arming in `bench/`. Anything that flies (simulated) goes in `sih/` + and must pass `px4bench.arming_gate` first. +- Depend on optional firmware features via probe-and-skip: check with + `px4bench.shell_command_exists` and exit `px4bench.EXIT_SKIP` (recorded + as SKIP by the orchestrator) with a warning naming the config option, + never FAIL for a feature the build simply does not have. - Reuse the library: connection and shell primitives from `px4bench`, protocol helpers from `px4bench.params` / `.missions` / `.ftp`. If two tests need the same helper, it moves into the library. @@ -315,9 +405,11 @@ previous command can arrive late). ## Safety - Bench use only. Props off. -- `bench/` never arms the vehicle. `sih/flight_mission.py` arms only in - simulation (`SYS_HITL=2`, `pwm_out_sim`); nothing is driven on the output - rails. +- `bench/` never arms the vehicle. `sih/flight_mission.py` (last in the + default suite) arms only in simulation (`SYS_HITL=2`, `pwm_out_sim`); + nothing is driven on the output rails, but the board must be bare. It + asks for confirmation on a TTY and requires `--allow-arming` in + automation. - `param_stress`, `reboot_loop`, and `flight_mission` reboot the board. Expect the link to drop and return. - Scratch params and airframe configuration are restored automatically, diff --git a/docs/en/test_and_ci/bench_testing.md b/docs/en/test_and_ci/bench_testing.md index a4fabcc62f0..87e49fcd4ad 100644 --- a/docs/en/test_and_ci/bench_testing.md +++ b/docs/en/test_and_ci/bench_testing.md @@ -48,9 +48,14 @@ You state where the firmware under test comes from, and the suite verifies the f Run interactively with none of these flags, the suite shows what is on the board and asks what to do. Run non-interactively (CI, production scripting), it refuses to start without an explicit firmware expectation, so automation can never silently test the wrong build. +With `--build`, the gate also prints a bench capability report before compiling: whether the image will contain `simulator_sih`, `sd_bench`, `sd_stress`, and `serial_test`, and which suite tests will therefore run or skip. +For anything missing it names a sibling board variant that includes it (for example `--target px4_fmu-v6xrt_bench`) and the exact config line, and can append the line to your local board config on request (the edit stays local and uncommitted). + ## The Bench Suite -The bench tests run against normal firmware and never arm the vehicle. +The tests under `bench/` run against normal firmware and never arm the vehicle. +The default suite sequence is `boot_health`, `param_stress`, `mission_stress` (plus `link_forwarding` when a second link is given), `storage_stress`, `log_transfer`, `reboot_loop`, and finally the simulated SIH flight (see below), which does arm and therefore sits behind a confirmation. +Tests that need a firmware feature the build does not contain probe for it over the shell and record SKIP with a warning instead of failing. | Test | What it verifies | | ------------------ | ----------------------------------------------------------------------------------------------------------------------- | @@ -60,7 +65,9 @@ The bench tests run against normal firmware and never arm the vehicle. | `link_forwarding` | Simultaneous heavy traffic on two links with forwarding; catches deadlocks and stalls in the MAVLink send path. Requires a second link. | | `param_stress` | Full parameter download, 50 set/readback cycles, save, reboot, persistence verification. | | `mission_stress` | Repeated 220-waypoint upload/download/compare/clear transactions, alternating links when two are available. | +| `storage_stress` | SD card health via the firmware's own `sd_bench` and `sd_stress`: write/read throughput, fsync latency, file churn. Skips without an SD card. | | `log_transfer` | Short log capture, MAVFTP download, ULog integrity verification. | +| `flight_mission` | Simulated SIH flight on the FMU, last in the sequence; arms behind a confirmation (`--allow-arming` for automation). | ```sh cd Tools/bench_test @@ -73,10 +80,13 @@ A per-test watchdog turns a hung board into a named failure instead of a stuck t ## Simulated Flight on the FMU (SIH) `sih/flight_mission.py` performs a complete hardware-in-the-loop flight without any host-side simulator: it switches the board to a [SIH airframe](../sim_sih/index.md) (`SYS_HITL=2`, physics computed on the autopilot), uploads a mission, arms through the NuttX shell, and asserts takeoff, waypoint progression, RTL, landing, and auto-disarm against per-phase timeouts. +It runs as the last test of the default bench suite and also works standalone. +Because it arms the flight controller, the suite asks for explicit confirmation on a terminal (type `arm`), and non-interactive runs skip it unless `--allow-arming` is passed. +It first probes the running firmware for the SIH module and records a skip, not a failure, when the build does not include it. The flight log is downloaded into the report directory automatically for post-flight analysis, and the original board configuration is restored afterwards. -Real outputs are replaced by `pwm_out_sim` in this mode, so nothing is driven on the output rails; still, run it only on a bench board with nothing connected. -The firmware must be built with `CONFIG_MODULES_SIMULATION_SIMULATOR_SIH=y`. +Real outputs are replaced by `pwm_out_sim` in this mode, so nothing is driven on the output rails; still, run it only on a bare bench board with nothing connected to the outputs. +The firmware must be built with `CONFIG_MODULES_SIMULATION_SIMULATOR_SIH=y` (boards may provide a bench variant that adds it, such as `px4_fmu-v6xrt_bench`). To watch the flight live in 3D, install [Hawkeye](https://github.com/PX4/Hawkeye) and run the test with `--viewer`: @@ -111,9 +121,11 @@ For manufacturers the suite is designed to run unattended per unit: ``` - Exit code is the verdict: `0` all checks passed, nonzero otherwise, with the failing check named in the output. +- A skipped test does not fail the suite (individual tests exit `75` for SKIP), but a skip is absent coverage, not a pass: production procedures should treat unexpected SKIP lines in the summary as findings. Fixture design (SD card present, `--allow-arming` set, firmware built with the needed modules per the capability report) determines which tests are expected to run. - `reports//` retains the full evidence per unit: firmware identity, system snapshot, parameter and mission transaction results, downloaded logs. - The firmware gate guarantees each unit was tested against the intended production image, not whatever was in flash. -- `usb_replug` (connector/enumeration exercise) is the only operator-assisted test and can be included where a physical connector check is part of the procedure. +- Pass `--allow-arming` so the simulated flight runs unattended (the line fixture must leave the output rails unconnected). +- `usb_replug` (connector/enumeration exercise) and `serial_loopback` (UART verification through a loopback jumper on the fixture) are operator/fixture-assisted and can be added where the procedure includes a connector or UART check. ## CI Integration