mlsanityData to 7 Oct 2026

RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation

arXiv 2412.13877 ↗First posted 18 Dec 2024cs.RO cs.AIcited by 166

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

In this paper, we introduce RoboMIND (Multi-embodiment Intelligence Normative Data for Robot Manipulation), a dataset containing 107k demonstration trajectories across 479 diverse tasks involving 96 object classes. RoboMIND is collected through human teleoperation and encompasses comprehensive robotic-related information, including multi-view observations, proprioceptive robot state information, and linguistic task descriptions. To ensure data consistency and reliability for imitation learning, RoboMIND is built on a unified data collection platform and a standardized protocol, covering four distinct robotic embodiments: the Franka Emika Panda, the UR5e, the AgileX dual-arm robot, and a humanoid robot with dual dexterous hands. Our dataset also includes 5k real-world failure demonstrations, each accompanied by detailed causes, enabling failure reflection and correction during policy learning. Additionally, we created a digital twin environment in the Isaac Sim simulator, replicating the real-world tasks and assets, which facilitates the low-cost collection of additional training data and enables efficient evaluation. To demonstrate the quality and diversity of our dataset, we conducted extensive experiments using various imitation learning methods for single-task settings and state-of-the-art Vision-Language-Action (VLA) models for multi-task scenarios. By leveraging RoboMIND, the VLA models achieved high manipulation success rates and demonstrated strong generalization capabilities. To the best of our knowledge, RoboMIND is the largest multi-embodiment teleoperation dataset collected on a unified platform, providing large-scale and high-quality robotic training data. Our project is at https://x-humanoid-robomind.github.io/.

Key facts. Cited by 166 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 93 of its 119 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

93 of 119 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedback1 Jul
  2. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation12 Mar
  3. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators22 Sep
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scale24 Aug
  5. 2023XSkill: Cross Embodiment Skill Discovery19 Jul
  6. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  7. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordance5 Apr
  8. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  9. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation7 Nov

Other references · 84

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
38 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  12. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  13. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  14. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  15. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  16. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  17. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  18. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  19. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  20. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  21. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  22. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  23. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  24. 2020Rescaling Egocentric Visioncited by 529
  25. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  26. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  27. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  28. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  29. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  30. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  31. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  32. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  33. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  34. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  35. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  36. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  37. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  38. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 63; 2026: 103.

166 papers.

Sort:

By area:

  1. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  2. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  3. 2026Causal World Modeling for Robot Controlcited by 289
  4. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  5. 2025Motus: A Unified Latent Action World Modelcited by 231
  6. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  7. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  8. 2026GigaWorld-Policy: An Efficient Action-Centered World--Action Modelcited by 129
  9. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  10. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  11. 2026ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learningcited by 94
  12. 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 90
38 more
  1. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  4. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  5. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  6. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  7. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  8. 2025Vidar: Embodied Video Diffusion Model for Generalist Manipulationcited by 58
  9. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  10. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulationcited by 48
  11. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 47
  12. 2025InternData-A1: Pioneering High-Fidelity Synthetic Data for Pre-training Generalist Policycited by 46
  13. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  14. 2026Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodimentscited by 42
  15. 2025H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulationcited by 41
  16. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  17. 2026DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulationcited by 33
  18. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentationcited by 31
  19. 2025EO-1: An Open Unified Embodied Foundation Model for General Robot Controlcited by 30
  20. 2026Rethinking Video Generation Model for the Embodied Worldcited by 30
  21. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  22. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 29
  23. 2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 28
  24. 2026ABot-M0.5: Unified Mobility-and-Manipulation World Action Modelcited by 28
  25. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  26. 2026StarVLA-α: Reducing Complexity in Vision-Language-Action Systemscited by 27
  27. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  28. 2025AnyPos: Automated Task-Agnostic Actions for Bimanual Manipulationcited by 23
  29. 2025SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overheadcited by 23
  30. 2026ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignmentcited by 23
  31. 2025MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulationcited by 22
  32. 2025RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Modelscited by 20
  33. 2025Is Diversity All You Need for Scalable Robotic Manipulation?cited by 19
  34. 2025Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approachcited by 18
  35. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  36. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  37. 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Modelscited by 17
  38. 2026LIBERO-X: Robustness Litmus for Vision-Language-Action Modelscited by 17

50 most cited of 166 All 166 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RoboMIND 2.0: A Multimodal, Bimanual Mobile Manipulation Dataset for Generalizable Embodied Intelligence
  2. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments
  3. 2026Monkey See, Can Monkey Do? A Benchmark for Evaluating Robot Skill Learning by Observation ↗
  4. 2026RoboTacDex: A Dexterous Visual-Tactile-Action Dataset for Humanoid Manipulation ↗
  5. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation
  6. 2026RoboSynChallenge: Mastering Real-World Dexterity via Generalizing Synthesized Manipulation Skills ↗
  7. 2026AXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation
  8. 2026ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation