# RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models

arXiv: https://arxiv.org/abs/2609.34384 · HTML: https://mlsanity.com/paper/2609.34384/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning and chain-of-thought in VLA, one of 13 directions in Vision-language-action models.
- Builds on 19 of its 21 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- Vision-language-action models / [Reasoning and chain-of-thought in VLA](https://mlsanity.com/vla/reasoning-and-chain-of-thought-in-vla/)

## Builds on (most cited first, 19 of 19)

- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2506.18088/) (2025)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators](https://mlsanity.com/paper/2309.13037/) (2023)
- [VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks](https://mlsanity.com/paper/2412.18194/) (2024)
- [ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes](https://mlsanity.com/paper/2304.04321/) (2023)
- [MemER: Scaling Up Memory for Robot Control via Experience Retrieval](https://mlsanity.com/paper/2510.20328/) (2025)
- [SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation](https://mlsanity.com/paper/2501.18564/) (2025)
- [RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies](https://mlsanity.com/paper/2603.04639/) (2026)
- [RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design](https://mlsanity.com/paper/2603.01229/) (2026)
- [Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning](https://mlsanity.com/paper/2502.10550/) (2025)
- [RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation](https://mlsanity.com/paper/2506.06677/) (2025)
- [RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark](https://mlsanity.com/paper/2605.10921/) (2026)
- [REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?](https://mlsanity.com/paper/2505.10872/) (2025)
- [LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models](https://mlsanity.com/paper/2609.24350/) (2026)

## Abstract (arXiv)

Vision-Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing benchmarks typically assume that task-relevant information is explicitly specified in the instruction. In practice, however, humans frequently refer to objects, quantities, and relations implicitly, requiring robots to recover the intended target from linguistic and perceptual context. We study this capability as Implicit Referential Grounding (IRG) and introduce RoboIRG-Bench, a manipulation benchmark designed to systematically evaluate it. Built upon RoboMME, RoboIRG-Bench contains 40 variants derived from 11 tasks and covers four challenges, including direct, reasoning-mediated, spatial, and contextual referential grounding. As IRG often requires retaining and retrieving previously established context, we evaluate representative VLAs spanning different memory mechanisms. Our evaluation reveals a noticeable referential robustness gap. Models that perform well under explicit instructions can degrade sharply when the same task-relevant information must be recovered from context. Reasoning-mediated and spatial references are particularly challenging, while models using external VLMs show greater robustness but still exhibit significant failures. Moreover, replacing the external VLM with a stronger model does not eliminate these gaps. We further validate these findings on a Franka Research 3 robot arm, where the gap persists under real-world manipulation and manifests as both incorrect referent grounding and downstream execution failures. These results establish IRG as a distinct and underexplored capability for reliable robotic instruction following and highlight the need for VLAs that can robustly integrate language, perception, reasoning, and action.