Saibernard 436cc71c89 fix(ekf2): avoid failover to an instance with a failing test ratio (#28418)
* fix(ekf2): keep evaluating instance selection while the primary is unhealthy

The selection block only ran when UpdateErrorScores() reported a change:
a new instance appearing or a health transition. A primary that stops
publishing produces exactly one such transition, on the cycle its
timeout is detected. If no switch happens on that cycle, nothing sets
updated again: the stable alternatives do not count as primary updates
and the timed out instance is skipped thereafter, so the fallback logic
is never evaluated again even though the module keeps being scheduled.

Today that single evaluation always resolves the situation, because the
fallback switches unconditionally to the best healthy candidate on that
same cycle. But any selection policy that can decline to switch on the
transition cycle, for example one that waits out a transient fault,
needs the decision re-evaluated while the primary remains unhealthy.
Re-enter the selection block whenever the selected instance is
unhealthy. The decisions inside are unchanged and switching is
idempotent, so behaviour today is identical.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>

* fix(ekf2): do not fail over to an instance with a sustained test ratio failure

When the primary EKF goes unhealthy the selector falls back to the best
instance that is healthy, and healthy only requires zero filter fault
flags and a positive combined test ratio. An instance whose test ratio
has been failing for a long time therefore remains a first class switch
target even though its state can be far from the truth.

b7efd4f947 introduced this on purpose for the switch-away direction: a
test ratio at or above one became a warning rather than ill health, so a
transient ratio spike cannot hard fail an instance, and a warned primary
is left through the lower relative error path once the warning has been
sustained for one second. What that commit did not do is apply the same
reasoning to the switch-to direction. The candidate loop only filters on
healthy, so a brief hard fault on the primary, for example transient
accelerometer clipping, sends the selector straight to a diverged
instance.

That is the mechanism behind the repeated altitude jumps in issue 27013:
one instance had stopped fusing baro, its vertical state up to 155.6 m
from the other instance while its combined test ratio sat pegged at 2,
and each of the seven short clipping faults on the good instance bounced
the selector back to it (15 instance switches in total counting the
returns), the worst switch stepping the published altitude by 128.5 m
and provoking a hard TECS reaction.

Classify fallback candidates with the same sustained warning test the
switch-away trigger already uses. When the primary goes unhealthy, fail
over immediately to the best candidate without a sustained warning; the
different IMU preference is kept within each tier, and a candidate
without a sustained warning is preferred even over a warned candidate on
a different IMU, since a warned instance is the one known to be
diverging. A sustained warned candidate is accepted in two cases only:
the primary has timed out entirely, where frozen attitude and position
outputs are worse than any live alternative, or the primary has been
continuously unhealthy for kWarnedFallbackDelay (five seconds), so a
brief fault rides out on the current state while a persistently faulted
primary still gets the least bad alternative rather than none. The
re-evaluation of this decision while the primary stays unhealthy is
provided by the previous commit.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>

* test(ekf2): add a functional test for the instance selector

The selector had no test at any level. This drives EKF2Selector through
published multi instance estimator_status messages on the real work
queue and observes estimator_selector_status, so the selection policy is
exercised without a simulator.

The scenarios encode the failure pattern from issue 27013 and the
no-whipsaw property discussed there: a clean fallback on a hard primary
fault stays immediate, the switch away from a degraded primary through
the sustained warning path still works, three separate brief hard faults
on the primary no longer bounce the selector to an instance whose test
ratio has been failing for seconds, a primary that stops publishing
falls back to the degraded instance without delay, and a primary that
stays hard faulted for longer than the ride-out window still falls back
rather than being kept forever.

The scenario setup helpers run until the selector reaches the intended
starting state rather than assuming fixed timings, since the health
hysteresis and warning windows run on wall clock time. The fault-clear
windows exceed the selector's one second healthy hysteresis so the
faults are genuinely separate and the unhealthy-since tracking restarts
between them. The harness waits until the work queue manager actually
serves queues before constructing the selector: a fixed delay races the
manager startup on a loaded runner.

to run: make tests TESTFILTER=EKF2Selector

Assisted-by: Claude:claude-fable-5
Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>

---------

Signed-off-by: Saibernard Yogendran <bernie97@seas.upenn.edu>
2026-08-29 15:23:11 -06:00
…
…
…

PX4 Autopilot

The autopilot stack the industry builds on.

Release DOI Discord

OpenSSF Best Practices LFX Health Score LFX Contributors LFX Active Contributors


About

PX4 is an open-source autopilot stack for drones and unmanned vehicles. It supports multirotors, fixed-wing, VTOL, rovers, and many more experimental platforms from racing quads to industrial survey aircraft. It runs on NuttX, Linux, and macOS. Licensed under BSD 3-Clause.

Why PX4

Modular architecture. PX4 is built around uORB, a DDS-compatible publish/subscribe middleware. Modules are fully parallelized and thread safe. You can build custom configurations and trim what you don't need.

Wide hardware support. PX4 runs on a wide range of autopilot boards and supports an extensive set of sensors, telemetry radios, and actuators through the Pixhawk ecosystem.

Developer friendly. First-class support for MAVLink and DDS / ROS 2 integration. Comprehensive SITL simulation, hardware-in-the-loop testing, and log analysis tools. An active developer community on Discord and the weekly dev call.

Vendor neutral governance. PX4 is hosted under the Dronecode Foundation, part of the Linux Foundation. Business-friendly BSD-3 license. No single vendor controls the roadmap.

Supported Vehicles

Multicopter
Multicopter
Fixed Wing
Fixed Wing
VTOL
VTOL
Rover
Rover

…and many more: helicopters, autogyros, airships, submarines, boats, and other experimental platforms. These frames have basic support but are not part of the regular flight-test program. See the full airframe reference.

Try PX4

Run PX4 in simulation with a single command. No build tools, no dependencies beyond Docker:

docker run --rm -it -p 14550:14550/udp px4io/px4-sitl:latest

Open QGroundControl and fly. See PX4 Simulation Quickstart for more options.

Build from Source

git clone https://github.com/PX4/PX4-Autopilot.git --recursive
cd PX4-Autopilot
make px4_sitl

Note

See the Development Guide for toolchain setup and build options.

Documentation & Resources

Resource Description
User Guide Build, configure, and fly with PX4
Developer Guide Modify the flight stack, add peripherals, port to new hardware
Airframe Reference Full list of supported frames
Autopilot Hardware Compatible flight controllers
Release Notes What's new in each release
Contribution Guide How to contribute to PX4

Community

Contributing

We welcome contributions of all kinds — bug reports, documentation, new features, and code reviews. Please read the Contribution Guide to get started.

Citation

If you use PX4 in academic work, please cite it. BibTeX:

@software{px4_autopilot,
  author    = {Meier, Lorenz and {The PX4 Contributors}},
  title     = {{PX4 Autopilot}},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.595432},
  url       = {https://px4.io}
}

The DOI above is a Zenodo concept DOI that always resolves to the latest release. For a version-pinned citation, see the Zenodo record or our CITATION.cff.

Governance

The PX4 Autopilot project is hosted by the Dronecode Foundation, a Linux Foundation Collaborative Project. Dronecode holds all PX4 trademarks and serves as the project's legal guardian, ensuring vendor-neutral stewardship — no single company owns the name or controls the roadmap. The source code is licensed under the BSD 3-Clause license, so you are free to use, modify, and distribute it in your own projects.

Dronecode Logo

Languages
C++ 50.1%
C 36.5%
CMake 4.4%
Python 4.1%
Linker Script 3.2%
Other 1.5%