# InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

arXiv: https://arxiv.org/abs/2608.22990 · HTML: https://mlsanity.com/paper/2608.22990/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models.
- Builds on 23 of its 25 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Placed in

- Vision-language-action models / [Failure recovery and robustness](https://mlsanity.com/vla/failure-recovery-and-robustness/)

## Builds on (most cited first, 23 of 23)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2506.18088/) (2025)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [Motus: A Unified Latent Action World Model](https://mlsanity.com/paper/2512.13030/) (2025)
- [VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks](https://mlsanity.com/paper/2412.18194/) (2024)
- [RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics](https://mlsanity.com/paper/2411.16537/) (2024)
- [RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins](https://mlsanity.com/paper/2504.13059/) (2025)
- [InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policy](https://mlsanity.com/paper/2510.13778/) (2025)
- [InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation](https://mlsanity.com/paper/2507.17520/) (2025)
- [RoboGround: Robotic Manipulation with Grounded Vision-Language Priors](https://mlsanity.com/paper/2504.21530/) (2025)
- [EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence](https://mlsanity.com/paper/2506.10600/) (2025)
- [When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs](https://mlsanity.com/paper/2602.17659/) (2026)
- [Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation](https://mlsanity.com/paper/2607.13653/) (2026)

## Built on by (most cited first, 2 of 2)

- [PerturBot: Breaking Shortcut Priors in Vision-Language-Action Models with Perturbative Training](https://mlsanity.com/paper/2610.04616/) (2026)
- [Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies](https://mlsanity.com/paper/2610.06235/) (2026)

## Abstract (arXiv)

Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim