mlsanityData to 14 Oct 2026

RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete

arXiv 2502.21257 ↗First posted 28 Feb 2025cs.RO cs.CVcited by 135

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advancements in Multimodal Large Language Models (MLLMs) have shown remarkable capabilities across various multimodal contexts. However, their application in robotic scenarios, particularly for long-horizon manipulation tasks, reveals significant limitations. These limitations arise from the current MLLMs lacking three essential robotic brain capabilities: Planning Capability, which involves decomposing complex manipulation instructions into manageable sub-tasks; Affordance Perception, the ability to recognize and interpret the affordances of interactive objects; and Trajectory Prediction, the foresight to anticipate the complete manipulation trajectory necessary for successful execution. To enhance the robotic brain's core capabilities from abstract to concrete, we introduce ShareRobot, a high-quality heterogeneous dataset that labels multi-dimensional information such as task planning, object affordance, and end-effector trajectory. ShareRobot's diversity and accuracy have been meticulously refined by three human annotators. Building on this dataset, we developed RoboBrain, an MLLM-based model that combines robotic and general multi-modal data, utilizes a multi-stage training strategy, and incorporates long videos and high-resolution images to improve its robotic manipulation capabilities. Extensive experiments demonstrate that RoboBrain achieves state-of-the-art performance across various robotic tasks, highlighting its potential to advance robotic brain capabilities.

Key facts. Cited by 135 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 96 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

75 of 96 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  2. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning17 Jun
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  4. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 71

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2023Mistral 7Bcited by 3,164
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2015VQA: Visual Question Answeringcited by 2,506
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  9. 2019Towards VQA Models That Can Readcited by 1,718
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  14. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  15. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  16. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  18. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  19. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  20. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  21. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  22. 2016A Diagram Is Worth A Dozen Imagescited by 804
  23. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  24. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  25. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  26. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  27. 2021OCR-free Document Understanding Transformercited by 407
  28. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
  29. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  30. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  31. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  32. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
  33. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  34. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  35. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  36. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  37. 2024RT-H: Action Hierarchies Using Languagecited by 210
  38. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 55; 2026: 80.

135 papers.

Sort:

By area:

  1. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Roboticscited by 127
  2. 2026VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Modelcited by 118
  3. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  4. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  5. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  6. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 66
  7. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 61
  8. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  9. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 42
  10. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  11. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  12. 2025VLA-R1: Enhancing Reasoning in Vision-Language-Action Modelscited by 38
38 more
  1. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  2. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  3. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 35
  4. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spacescited by 33
  5. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 30
  6. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  7. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Modelscited by 27
  8. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  9. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 25
  10. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 24
  11. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 24
  12. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  13. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedbackcited by 20
  14. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 20
  15. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 20
  16. 2025RoVer: Robot Reward Model as Test-Time Verifier for Vision-Language-Action Modelcited by 19
  17. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  18. 2025From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidancecited by 18
  19. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  20. 2026G0.5: One Autoregressive Stream for Robot Reasoning and Actioncited by 18
  21. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  22. 2025Contrastive Representation Regularization for Vision-Language-Action Modelscited by 17
  23. 2026AtomicVLA: Unlocking the Potential of Atomic Skill Learning in Robotscited by 16
  24. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 14
  25. 2026pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoningcited by 14
  26. 2025Euclid's Gift: Enhancing Spatial Perception and Reasoning in Vision-Language Models via Geometric Surrogate Taskscited by 13
  27. 2025Mantis: A Versatile Vision-Language-Action Model with Disentangled Visual Foresightcited by 13
  28. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 13
  29. 2025Reinforced Reasoning for Embodied Planningcited by 12
  30. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  31. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  32. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Groundingcited by 10
  33. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  34. 2025Pelican-VL 1.0: A Foundation Brain Model for Embodied Intelligencecited by 10
  35. 2025Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigationcited by 10
  36. 2026Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesiscited by 10
  37. 2026HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensingcited by 10
  38. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigationcited by 10

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation
  2. 2025RoboBERT: An End-to-end Multimodal Robotic Manipulation Model
  3. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation
  4. 2025MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulation
  5. 2026RoboBrain 2.5: Depth in Sight, Time in Mind
  6. 2026AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
  7. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models
  8. 2024Affordance-based Robot Manipulation with Flow Matching