One real fault, and a log full of everything else

Two situations keep coming back when we debug robots. The first is startup: while a robot boots, drivers are still connecting and nodes are waiting for each other, so the log fills with errors that go away on their own a few seconds later. The second is the chain reaction, where one thing breaks and a minute later a handful of other nodes complain about missing data, timeouts and a failed bringup. In both cases there is one real fault, and everything else is noise or an effect of it that somebody has to sort out by hand.

We wanted to see both on a concrete case, so we staged a startup with Nav2 Jazzy and AMCL in which the Hokuyo Ethernet lidar never connects. That is what happens with a disconnected data cable or a wrong subnet, even though the sensor still looks powered and healthy.

The log on the left replays the real driver and Nav2 texts at their real repeat rates. The fault list on the right comes from a real ros2_medkit run with one correlation rule, in which fifteen nodes reported 19 faults. Twelve were startup faults that recovered within about 14 seconds and left the list on their own. Of the seven that stayed, one was the lidar and six were effects of it, which the rule filtered out of the list as noise.

From the lidar to the rejected goal

urg_node reports the diagnostic status Not Connected and keeps retrying. AMCL publishes the map to odom transform only after it processes a scan, so it never publishes it here, and it logs nothing about the missing scans. The global costmap needs that transform and starts repeating this every half second:

Timed out waiting for transform from base_link to map to become available ...

After 60 seconds it fails to activate, the lifecycle manager reports Failed to bring up all requested nodes. Aborting bringup., and bt_navigator rejects the goal because its action server is inactive.

The launch log ends up with 164 messages on 222 lines. The lidar error is in it 29 times, between 120 costmap messages, and nothing in the log says that the costmap messages depend on it. The most frequent message describes a downstream effect, which is why the log pulls attention towards Nav2.

Cars already filter consecutive faults

Automotive ECUs have had this for years. In the AUTOSAR Classic Diagnostic Event Manager, engineers declare which component depends on which. When an event fails while a parent component has already failed, the specification calls it a consecutive fault and does not store it in the failure memory (section 7.3.6 of the specification). Correlation in ros2_medkit brings the same idea to ROS 2, except that a muted symptom stays stored and can still be listed with its cause.

One rule for the chain reaction

Correlation is off by default and ships without rules, so we wrote the dependency down for this run. The rule names the lidar fault as the cause and the scan, costmap, lifecycle and goal rejection codes as its symptoms:

correlation:
  enabled: true
  default_window_ms: 500
  patterns:
    scan_dependent_localization:
      codes: ["LIDAR_SCAN_TOPIC_NO_EVENTS"]
    nav2_global_bringup:
      codes: ["GLOBAL_COSTMAP_*", "NAV_LIFECYCLE_*", "BT_NAVIGATOR_GOAL_REJECTED"]
  rules:
    - id: lidar_not_connected_blocks_localization
      name: "Lidar driver is not connected"
      mode: hierarchical
      window_ms: 5000
      root_cause:
        codes: ["LIDAR_NOT_CONNECTED"]
      symptoms:
        - pattern: scan_dependent_localization
        - pattern: nav2_global_bringup
      mute_symptoms: true
      auto_clear_with_root: true

When a matching symptom arrives after a report of the cause and inside the window, correlation links it to the cause, and mute_symptoms: true takes it out of the default fault list. The rule works on fault codes that your nodes report to the fault manager, so it does not turn Nav2 log lines into faults by itself.

At the end of the run, the default query returns one fault and muted_count 6:

curl http://localhost:8080/api/v1/faults

Adding ?include_muted=true lists all seven, and x-medkit.muted_faults says which cause each symptom belongs to:

{
  "fault_code": "GLOBAL_COSTMAP_TF_WAIT",
  "root_cause_code": "LIDAR_NOT_CONNECTED",
  "rule_id": "lidar_not_connected_blocks_localization",
  "delay_ms": 3508
}

What to know before you write your own

The window is measured from the latest report of the cause. Costmap activation fails more than a minute after the lidar fault first appears, and the rule still caught it because our staged /urg_node reporter sent the lidar fault every second. When we reported the cause once with a ten second window and the symptoms came twelve seconds later, nothing was muted.

A symptom that arrives before its cause stays visible, because hierarchical correlation only mutes what comes after the cause.

Clearing the cause through the fault manager also clears its symptoms. Healing the cause through a PASSED report does not, so when the lidar comes back, clear the cause or let the symptom reporters send their own PASSED.

Run it yourself

ros2_medkit installs with apt on Humble, Jazzy and Lyrical, and the installation guide covers the prerequisites:

sudo apt install ros-jazzy-ros2-medkit-gateway   # or ros-humble- / ros-lyrical-

Save the rule above as correlation.yaml and point the fault manager at it:

fault_manager:
  ros__parameters:
    confirmation_threshold: -1
    healing_enabled: true
    healing_threshold: 0
    correlation.config_file: "/absolute/path/to/correlation.yaml"
ros2 launch ros2_medkit_gateway bringup.launch.py params_file:=/absolute/path/to/fault_manager.yaml

The fault manager logs Correlation engine enabled with the number of rules it loaded. Your nodes report faults with the FaultReporter client library from ros2_medkit_fault_reporter, or by calling /fault_manager/report_fault directly.

We build ros2_medkit as an open source ROS 2 diagnostics stack. If this would help on your robot, give ros2_medkit a star on GitHub.