# When Does Language Matter? Multilingual Instructions Reveal Step-wise Language Sensitivity in Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2606.11906 · HTML: https://mlsanity.com/paper/2606.11906/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models.
- Builds on 18 of its 20 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- Vision-language-action models / [Failure recovery and robustness](https://mlsanity.com/vla/failure-recovery-and-robustness/)

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [PaLI-X: On Scaling up a Multilingual Vision and Language Model](https://mlsanity.com/paper/2305.18565/) (2023)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning](https://mlsanity.com/paper/2406.11815/) (2024)
- [PALO: A Polyglot Large Multimodal Model for 5B People](https://mlsanity.com/paper/2402.14818/) (2024)
- [Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalization](https://mlsanity.com/paper/2505.15660/) (2025)
- [RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbations](https://mlsanity.com/paper/2510.00037/) (2025)
- [Eva-VLA: Evaluating Vision-Language-Action Models' Robustness Under Real-World Physical Variations](https://mlsanity.com/paper/2509.18953/) (2025)
- [Improving Pre-Trained Vision-Language-Action Policies with Model-Based Search](https://mlsanity.com/paper/2508.12211/) (2025)
- [CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention](https://mlsanity.com/paper/2506.11073/) (2025)
- [A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages](https://mlsanity.com/paper/2510.09555/) (2025)
- [Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering](https://mlsanity.com/paper/2511.23231/) (2025)
- [Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Models](https://mlsanity.com/paper/2508.18381/) (2025)

## Built on by (most cited first, 4 of 4)

- [Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models](https://mlsanity.com/paper/2608.02497/) (2026)
- [Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition](https://mlsanity.com/paper/2609.07474/) (2026)
- [Outcome-Conditioned End-Effector Geometry Across Vision-Language-Action Policies](https://mlsanity.com/paper/2609.21659/) (2026)
- [Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies](https://mlsanity.com/paper/2610.06235/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) models have shown strong performance in language-conditioned robotic manipulation, yet their robustness to linguistic variation remains poorly understood. In this work, we present the first systematic multilingual evaluation of VLA models by translating the LIBERO benchmark into ten languages, revealing severe performance degradation under non-English instructions, with success rates dropping by 30-50%. Through fine-grained analysis of task executions, we find that language influence is highly non-uniform across steps: certain steps exhibit strong language dependence and dominate overall task failure, while others are largely language-agnostic. Based on this insight, we propose a step-wise inference-time intervention that aligns representations according to step language sensitivity, substantially improving performance under linguistic variation. Our results indicate that language robustness in VLA models is fundamentally a step-wise control problem, highlighting the importance of temporally structured analysis for reliable embodied agents.