mlsanityData to 14 Oct 2026

RoboVQA: Multimodal Long-Horizon Reasoning for Robotics

arXiv 2311.00899 ↗First posted 1 Nov 2023cs.ROcited by 120

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We present a scalable, bottom-up and intrinsically diverse data collection scheme that can be used for high-level reasoning with long and medium horizons and that has 2.2x higher throughput compared to traditional narrow top-down step-by-step collection. We collect realistic data by performing any user requests within the entirety of 3 office buildings and using multiple robot and human embodiments. With this data, we show that models trained on all embodiments perform better than ones trained on the robot data only, even when evaluated solely on robot episodes. We find that for a fixed collection budget it is beneficial to take advantage of cheaper human collection along with robot collection. We release a large and highly diverse (29,520 unique instructions) dataset dubbed RoboVQA containing 829,502 (video, text) pairs for robotics-focused visual question answering. We also demonstrate how evaluating real robot experiments with an intervention mechanism enables performing tasks to completion, making it deployable with human oversight even if imperfect while also providing a single performance metric. We demonstrate a single video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is capable of performing a variety of grounded high-level reasoning tasks in broad realistic settings with a cognitive intervention rate 46% lower than the zero-shot state of the art visual language model (VLM) baseline and is able to guide real robots through long-horizon tasks. The performance gap with zero-shot state-of-the-art models indicates that a lot of grounded data remains to be collected for real-world deployment, emphasizing the critical need for scalable data collection approaches. Finally, we show that video VLMs significantly outperform single-image VLMs with an average error rate reduction of 19% across all VQA tasks. Data and videos available at https://robovqa.github.io

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 3 of its 42 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

3 of 42 references.

References · 3

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 26; 2025: 47; 2026: 46.

120 papers.

Sort:

By area:

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  2. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  3. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  4. 2024RT-H: Action Hierarchies Using Languagecited by 210
  5. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 175
  6. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  7. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 164
  8. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  9. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
  10. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 103
  11. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  12. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
38 more
  1. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  2. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 71
  3. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  4. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 67
  5. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 66
  6. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  7. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  8. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  9. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 50
  10. 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
  11. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  12. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 39
  13. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  14. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 34
  15. 2025Generating Robot Constitutions & Benchmarks for Semantic Safetycited by 33
  16. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Taskscited by 31
  17. 2025EO-1: An Open Unified Embodied Foundation Model for General Robot Controlcited by 31
  18. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 30
  19. 2024EVA: An Embodied World Model for Future Video Anticipationcited by 28
  20. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  21. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  22. 2025MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Modelscited by 27
  23. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 25
  24. 2025Embodied AI: From LLMs to World Modelscited by 24
  25. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 24
  26. 2025Locate 3D: Real-World Object Localization via Self-Supervised Learning in 3Dcited by 23
  27. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  28. 2024GenEx: Generating an Explorable Worldcited by 21
  29. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 21
  30. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 20
  31. 2025ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmarkcited by 19
  32. 2025PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachabilitycited by 18
  33. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  34. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  35. 2025RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulationcited by 16
  36. 2025SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agentscited by 15
  37. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  38. 2024DKPROMPT: Domain Knowledge Prompting Vision-Language Models for Open-World Planningcited by 14

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applications
  2. 2025ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks
  3. 2025Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets
  4. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models
  5. 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answering
  6. 2025JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics
  7. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
  8. 2026STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multimodal Robot Memory