mlsanityData to 14 Oct 2026

Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets

arXiv 2505.15517 ↗First posted 21 May 2025cs.RO cs.AI cs.CLcited by 40

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) acquire real-world knowledge and general reasoning ability through Internet-scale image-text corpora. They can augment robotic systems with scene understanding and task planning, and assist visuomotor policies that are trained on robot trajectory data. We explore the reverse paradigm - using rich, real, multi-modal robot trajectory data to enhance and evaluate VLMs. In this paper, we present Robo2VLM, a Visual Question Answering (VQA) dataset generation framework for VLMs. Given a human tele-operated robot trajectory, Robo2VLM derives ground-truth from non-visual and non-descriptive sensory modalities, such as end-effector pose, gripper aperture, and force sensing. Based on these modalities, it segments the robot trajectory into a sequence of manipulation phases. At each phase, Robo2VLM uses scene and interaction understanding to identify 3D properties of the robot, task goal, and the target object. The properties are used to generate representative VQA queries - images with textural multiple-choice questions - based on spatial, goal-conditioned, and interaction reasoning question templates. We curate Robo2VLM-1, a large-scale in-the-wild dataset with 684,710 questions covering 463 distinct scenes and 3,396 robotic manipulation tasks from 176k real robot trajectories. Results suggest that Robo2VLM-1 can benchmark and improve VLM capabilities in spatial and interaction reasoning.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. First posted on arXiv 21 May 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 27.

40 papers.

Sort:

By area:

  1. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  2. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 39
  3. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 25
  4. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 23
  5. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  6. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 14
  7. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulationcited by 14
  8. 2025NanoVLA: Routing Decoupled Vision-Language Understanding for Nano-sized Generalist Robotic Policiescited by 12
  9. 2026Wall-OSS-0.5 Technical Reportcited by 12
  10. 2025MoTVLA: A Vision-Language-Action Model with Unified Fast-Slow Reasoningcited by 9
  11. 2026Towards Long-horizon Embodied Agents with Tool-Aligned Vision-Language-Action Modelscited by 9
  12. 2025Mind to Hand: Purposeful Robotic Control via Embodied Reasoningcited by 8
28 more
  1. 2025Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-trainingcited by 7
  2. 2026Green-VLA: Staged Vision-Language-Action Model for Generalist Robotscited by 6
  3. 2026GEM: Generative Supervision Helps Embodied Intelligencecited by 6
  4. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 6
  5. 2026Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignmentcited by 6
  6. 2026Vesta: A Generalist Embodied Reasoning Modelcited by 5
  7. 2025HRIBench: Benchmarking Vision-Language Models for Real-Time Human Perception in Human-Robot Interactioncited by 4
  8. 2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agentscited by 4
  9. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Modelscited by 3
  10. 2025MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machinescited by 3
  11. 2026ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Modelscited by 3
  12. 2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learningcited by 3
  13. 2026Towards Predictive, Aligned, and Scalable Robot Learningcited by 3
  14. 2025AURA: Autonomous Upskilling with Retrieval-Augmented Agentscited by 2
  15. 2026EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-trainingcited by 2
  16. 2026RoboJailBench: Benchmarking Adversarial Attacks and Defenses in Embodied Robotic Agentscited by 2
  17. 2025How Good are Foundation Models in Step-by-Step Embodied Reasoning?cited by 1
  18. 2026Vision-Language Models on the Edge for Real-Time Robotic Perceptioncited by 1
  19. 2026Learning Multi-View Spatial Reasoning from Cross-View Relationscited by 1
  20. 2026CometVLA: Co-Training on an Embodied Data Pyramid towards Physical Understandingcited by 1
  21. 2026UniWAM: Unified World-Action Modelcited by 1
  22. 2025LAVQA: A Latency-Aware Visual Question Answering Framework for Shared Autonomy in Self-Driving Vehiclescited by 0
  23. 2026Rethinking VLM Representation for VLA Initializationcited by 0
  24. 2026When Video Misreads: Closed-Loop Distillation of Reading Heuristics for Exploratory Manipulation Trace QAcited by 0
  25. 2026Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligencecited by 0
  26. 2026Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoningcited by 0
  27. 2026YUBI-STAG: Contact and Semantic-Rich Alignment for VLAs via Automated Video-Language Groundingcited by 0
  28. 2026ABot-ER: Unified Embodied Reasoning through Agentic Data-Model Co-Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
  2. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models
  3. 2025AntiGrounding: Executable Robot Trajectories as Visual Prompts for VLM-Guided Manipulation
  4. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
  5. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
  7. 2024Task-oriented Robotic Manipulation with Vision Language Models
  8. 2026PROBE: Manipulation-Grounded Visual Question Answering with VLM Agents