# How VLAs Fail Differently: Black-Box Action Monitoring Reveals Architecture-Specific Failure Signatures

arXiv: https://arxiv.org/abs/2605.28726 · HTML: https://mlsanity.com/paper/2605.28726/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 18 of 18)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Behavior Generation with Latent Actions](https://mlsanity.com/paper/2403.03181/) (2024)
- [SAFE: Multitask Failure Detection for Vision-Language-Action Models](https://mlsanity.com/paper/2506.09937/) (2025)
- [SafeDiffuser: Safe Planning with Diffusion Probabilistic Models](https://mlsanity.com/paper/2306.00148/) (2023)
- [SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning](https://mlsanity.com/paper/2503.03480/) (2025)
- [Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress](https://mlsanity.com/paper/2410.04640/) (2024)
- [Failure Prediction at Runtime for Generative Robot Policies](https://mlsanity.com/paper/2510.09459/) (2025)
- [VLSA: Vision-Language-Action Models with Plug-and-Play Safety Constraint Layer](https://mlsanity.com/paper/2512.11891/) (2025)
- [RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbations](https://mlsanity.com/paper/2510.00037/) (2025)
- [Modular Safety Guardrails Are Necessary for Foundation-Model-Enabled Robots in the Real World](https://mlsanity.com/paper/2602.04056/) (2026)
- [Towards Safe Robot Foundation Models Using Inductive Biases](https://mlsanity.com/paper/2505.10219/) (2025)
- [Conformal Safety Monitoring for Flight Testing: A Case Study in Data-Driven Safety Learning](https://mlsanity.com/paper/2511.20811/) (2025)

## Abstract (arXiv)

We discover that VLA architectures fail in fundamentally different, predictable ways at the motor-command level. Running VQ-BeT, Diffusion Policy, and ACT on identical evaluation protocols (n=450 episodes across PushT and ALOHA 14-DOF bimanual manipulation), we find: (1) direction reversal rate is a universal failure predictor across all three architectures (AUROC=0.93, 0.79, 0.91; p<0.001); (2) jerk monitoring is predictive only for discrete-token architectures, following a discrete-to-continuous gradient (0.88, 0.69, 0.41); (3) velocity violations alone are non-predictive everywhere (AUROC 0.41-0.69), yet velocity checking is the most common safety mechanism in VLA deployment code; and (4) for continuous-family VLAs, velocity monitoring provides effectively zero predictive signal (AUROC=0.52 on ACT, 0.41 on Diffusion), proving that architecture-matched monitor selection is essential. These results quantify a monitoring consequence of the well-known discrete/continuous VLA distinction: the two families produce qualitatively different failure signatures that require different monitors. No single monitor works universally; architecture-matched selection is required. This finding was enabled by SafeContract, a training-free, black-box action monitoring toolkit with conformal calibration. Code: https://github.com/krishnam94/vla-edge