# Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets

arXiv: https://arxiv.org/abs/2505.15517 · HTML: https://mlsanity.com/paper/2505.15517/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- First posted on arXiv 21 May 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Built on by (most cited first, 30 of 40)

- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models](https://mlsanity.com/paper/2601.03309/) (2026)
- [Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain](https://mlsanity.com/paper/2510.17801/) (2025)
- [Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning](https://mlsanity.com/paper/2510.11027/) (2025)
- [HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents](https://mlsanity.com/paper/2604.07430/) (2026)
- [Towards Cross-View Point Correspondence in Vision-Language Models](https://mlsanity.com/paper/2512.04686/) (2025)
- [RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation](https://mlsanity.com/paper/2602.09973/) (2026)
- [NanoVLA: Routing Decoupled Vision-Language Understanding for Nano-sized Generalist Robotic Policies](https://mlsanity.com/paper/2510.25122/) (2025)
- [Wall-OSS-0.5 Technical Report](https://mlsanity.com/paper/2605.30877/) (2026)
- [MoTVLA: A Vision-Language-Action Model with Unified Fast-Slow Reasoning](https://mlsanity.com/paper/2510.18337/) (2025)
- [Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Models](https://mlsanity.com/paper/2605.13119/) (2026)
- [Mind to Hand: Purposeful Robotic Control via Embodied Reasoning](https://mlsanity.com/paper/2512.08580/) (2025)
- [Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-training](https://mlsanity.com/paper/2512.24125/) (2025)
- [Green-VLA: Staged Vision-Language-Action Model for Generalist Robots](https://mlsanity.com/paper/2602.00919/) (2026)
- [GEM: Generative Supervision Helps Embodied Intelligence](https://mlsanity.com/paper/2605.28548/) (2026)
- [Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models](https://mlsanity.com/paper/2606.11324/) (2026)
- [Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment](https://mlsanity.com/paper/2607.13429/) (2026)
- [Vesta: A Generalist Embodied Reasoning Model](https://mlsanity.com/paper/2606.20905/) (2026)
- [HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interaction](https://mlsanity.com/paper/2506.20566/) (2025)
- [The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents](https://mlsanity.com/paper/2605.20544/) (2026)
- [Demystifying the Visual Quality Paradox in Multimodal Large Language Models](https://mlsanity.com/paper/2506.15645/) (2025)
- [MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machines](https://mlsanity.com/paper/2507.07818/) (2025)
- [ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models](https://mlsanity.com/paper/2603.13033/) (2026)
- [SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning](https://mlsanity.com/paper/2603.28730/) (2026)
- [Towards Predictive, Aligned, and Scalable Robot Learning](https://mlsanity.com/paper/2607.11270/) (2026)
- [AURA: Autonomous Upskilling with Retrieval-Augmented Agents](https://mlsanity.com/paper/2506.02507/) (2025)
- [EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training](https://mlsanity.com/paper/2604.20012/) (2026)
- [RoboJailBench: Benchmarking Adversarial Attacks and Defenses in Embodied Robotic Agents](https://mlsanity.com/paper/2605.19328/) (2026)
- [How Good are Foundation Models in Step-by-Step Embodied Reasoning?](https://mlsanity.com/paper/2509.15293/) (2025)
- [Vision-Language Models on the Edge for Real-Time Robotic Perception](https://mlsanity.com/paper/2601.14921/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) acquire real-world knowledge and general reasoning ability through Internet-scale image-text corpora. They can augment robotic systems with scene understanding and task planning, and assist visuomotor policies that are trained on robot trajectory data. We explore the reverse paradigm - using rich, real, multi-modal robot trajectory data to enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual Question Answering (VQA) dataset generation framework for VLMs. Given a human tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual and non-descriptive sensory modalities, such as end-effector pose, gripper aperture, and force sensing. Based on these modalities, it segments the robot trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses scene and interaction understanding to identify 3D properties of the robot, task goal, and the target object. The properties are used to generate representative VQA queries - images with textural multiple-choice questions - based on spatial, goal-conditioned, and interaction reasoning question templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710 questions covering 463 distinct scenes and 3,396 robotic manipulation tasks from 176k real robot trajectories. Results suggest that Robo2VLM-1 can benchmark and improve VLM capabilities in spatial and interaction reasoning.