Failure-Mode Decomposition

Companion page: where the missing success rate goes on the mixed specification in the Dubins Car environment. See the main page for the full results.

Summary

At $N=32$, the shortfall of every planner is dominated by collisions (STLPY 34.4%, single-agent diffusion 27.5%, ours 10.9%), whereas the entire STL violation umbrella stays at or below 6.9% for every method. Joint multi-agent sampling keeps the collision channel roughly $3\times$ below both baselines, which is the source of the gap. At $N\le16$ all methods reach at least 86% success and the residual failures are about spec satisfaction rather than safety.

Setting: Dubins Car environment, mixed heterogeneous specification, where each agent samples one component (sequence, branch, cover, loop, or signal) per episode, with randomized predicate locations. The fixed-goal variant of the same sweep, at goal scale 1.0, is milder (STLPY at $N=32$ reaches 85.9% success with 8.8% collisions), so the collapse at $N=32$ is specific to randomizing predicate locations.

What the channels mean

Every agent is attributed to exactly one outcome, so each row of the table below sums to 100%. Attribution is exclusive and follows a priority order: collision first, then resampling exhaustion, then tracking error. No agent carries two labels.

  • Success. The agent never collided and its executed trajectory satisfies the specification.
  • Collision. The agent collided at any point. Collision claims the agent first, so an agent that collided and later satisfied its specification still counts here, consistent with success being defined as safe and satisfied.
  • STL violation. The agent stayed safe but its executed trajectory violates the specification. This is not a sibling of the two channels below, it is their union. It splits further by asking whether the plan was already bad at the moment the planner handed it over.
    • Resampling exhaustion. The reject-and-resample loop hit its iteration cap and delivered a plan that already violated the specification at plan time. This is the planner-side channel.
    • Tracking error. The delivered plan satisfied the specification and execution failed to realize it. This is the controller-side channel.

Resampling exhaustion is a diffusion concept and carries no meaning for STLPY, which has no reject-and-resample loop. The MILP delivers plans that satisfy the specification, so a safe STLPY agent that still violates the specification did so because execution failed to realize a satisfying plan. Its STL violation is therefore entirely tracking error. Either way this does not move the comparison, because STLPY's shortfall is collision (34.4% at $N=32$) and its whole STL violation umbrella is only 6.9%.

Results

Note: values are percentages of all agents. Rows sum to 100%.

Per-agent failure-mode decomposition for the Dubins Car environment on the mixed specification with randomized predicate locations ($N\in\{8,16,32\}$). D-MA: Diffusion-based Multi-Agent (Ours), D-SA: Diffusion-based Single-Agent, STLPY: MILP planner. Resampling exhaustion and tracking error are the two halves of STL violation, not extra channels, and resampling exhaustion does not apply to STLPY (see the note below the table). Best success rate for each $N$ is in bold.
Planner N Success (↑) Collision (↓) STL Violation (↓)
Total ↳ Resampling Exhaustion ↳ Tracking Error
D-MA (Ours) 8 91.20.0 8.85.03.8
16 92.53.8 3.82.51.2
32 84.110.9 5.01.93.1
D-SA 8 92.51.2 6.21.25.0
16 90.02.5 7.51.95.6
32 66.927.5 5.61.64.1
STLPY 8 86.22.5 11.2 0.011.2
16 87.55.6 6.9 0.06.9
32 58.834.4 6.9 0.06.9

Resampling exhaustion is a diffusion concept and does not apply to STLPY, which has no reject-and-resample loop. The MILP delivers plans that satisfy the specification, so an STLPY agent that stays safe and still violates the specification did so because execution failed to realize a satisfying plan. Its STL violation is therefore attributed entirely to tracking error.