# Beyond Single-Axis Testing: Paired Evaluation of Compound Robustness in Vision-Language-Action Policies

arXiv: https://arxiv.org/abs/2609.15940 · HTML: https://mlsanity.com/paper/2609.15940/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models.
- Builds on 18 of its 19 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Placed in

- Vision-language-action models / [Failure recovery and robustness](https://mlsanity.com/vla/failure-recovery-and-robustness/)

## Builds on (most cited first, 18 of 18)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [UniVLA: Learning to Act Anywhere with Task-centric Latent Actions](https://mlsanity.com/paper/2505.06111/) (2025)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models](https://mlsanity.com/paper/2510.13626/) (2025)
- [LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization](https://mlsanity.com/paper/2510.03827/) (2025)
- [VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks](https://mlsanity.com/paper/2412.18194/) (2024)
- [VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model](https://mlsanity.com/paper/2602.10098/) (2026)
- [RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots](https://mlsanity.com/paper/2603.04356/) (2026)
- [MINERVA: How Small Can a Manipulation Policy Be and Still Solve LIBERO?](https://mlsanity.com/paper/2609.03715/) (2026)

## Built on by (most cited first, 2 of 2)

- [What Paired Evaluations Reveal under Visual Perturbations](https://mlsanity.com/paper/2609.35583/) (2026)
- [VLA-GAUGE: Systematically Evaluating Generalization and Robustness in Vision-Language-Action Models](https://mlsanity.com/paper/2610.16567/) (2026)

## Abstract (arXiv)

Vision-language-action policies are typically evaluated one perturbation at a time, providing a useful diagnosis of their sensitivity to individual distribution shifts. Real-world deployment, however, may involve several shifts simultaneously, and it remains unclear how these individual robustness measurements compose. We ask whether compound robustness can be inferred from single-axis evaluations. We introduce LIBERO-CTRL, a six-axis benchmark that pairs each initial state across single-axis conditions and a matched simultaneous condition. This design reveals two opposing outcome changes that aggregate success rates cannot distinguish: emergent failures, where all single-axis rollouts succeed but the simultaneous rollout fails, and compensated successes, where at least one single-axis rollout fails but the simultaneous rollout succeeds. Because one transition decreases compound success while the other increases it, they can cancel, making aggregate compound performance appear consistent with single-axis measurements even when individual outcomes differ substantially. These opposing transitions can largely cancel in aggregate: even when the difference between the two transition rates is not statistically distinguishable from zero, as many as 29.0% of matched initial states still change outcome. Across six policies and three severity levels, such outcome changes reach 34.5% in the most affected condition. The relative prevalence of the two transitions varies across policies and severities, while the transition rates remain similar under independent re-evaluation of stochastic policies. Compound robustness therefore cannot be characterized from aggregate single-axis success rates alone; matched per-instance evaluation is needed to reveal how joint perturbations alter behavior.