mlsanityData to 14 Oct 2026

RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

arXiv 2410.07864 ↗First posted 10 Oct 2024cs.RO cs.AI cs.CVcited by 691

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Bimanual manipulation is essential in robotics, yet developing foundation models is extremely challenging due to the inherent complexity of coordinating two robot arms (leading to multi-modal action distributions) and the scarcity of training data. In this paper, we present the Robotics Diffusion Transformer (RDT), a pioneering diffusion foundation model for bimanual manipulation. RDT builds on diffusion models to effectively represent multi-modality, with innovative designs of a scalable Transformer to deal with the heterogeneity of multi-modal inputs and to capture the nonlinearity and high frequency of robotic data. To address data scarcity, we further introduce a Physically Interpretable Unified Action Space, which can unify the action representations of various robots while preserving the physical meanings of original actions, facilitating learning transferrable physical knowledge. With these designs, we managed to pre-train RDT on the largest collection of multi-robot datasets to date and scaled it up to 1.2B parameters, which is the largest diffusion-based foundation model for robotic manipulation. We finally fine-tuned RDT on a self-created multi-task bimanual dataset with over 6K+ episodes to refine its manipulation capabilities. Experiments on real robots demonstrate that RDT significantly outperforms existing methods. It exhibits zero-shot generalization to unseen objects and scenes, understands and follows language instructions, learns new skills with just 1~5 demonstrations, and effectively handles complex, dexterous tasks. We refer to https://rdt-robotics.github.io/rdt-robotics/ for the code and videos.

Key facts. Cited by 691 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 75 of its 108 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

75 of 108 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play7 Dec
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  3. 2023Imitating Human Behaviour with Diffusion Models25 Jan

Other references · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023Segment Anythingcited by 8,234
  10. 2022Scalable Diffusion Models with Transformerscited by 6,054
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  11. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  12. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  13. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  14. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  15. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  16. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  17. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  18. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  19. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  20. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  21. 2020Query-Key Normalization for Transformerscited by 280
  22. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  23. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  24. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 154
  25. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 154
  26. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  27. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 136
  28. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 133
  29. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 108
  30. 2023On Bringing Robots Homecited by 101
  31. 2021Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulationcited by 83
  32. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  33. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 80
  34. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  35. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  36. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
  37. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  38. 2024ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperationcited by 63

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 269; 2026: 412.

691 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  2. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  3. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  4. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 518
  5. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  6. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
  7. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 349
  8. 2026Causal World Modeling for Robot Controlcited by 321
  9. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
  10. 2025Motus: A Unified Latent Action World Modelcited by 251
  11. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  12. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
38 more
  1. 2025Real-Time Execution of Action Chunking Flow Policiescited by 216
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  3. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 198
  4. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 197
  5. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 192
  6. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  7. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 175
  8. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 175
  9. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 167
  10. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  11. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  12. 2025Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Bettercited by 146
  13. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 145
  14. 2026GigaWorld-Policy: An Efficient Action-Centered World--Action Modelcited by 139
  15. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
  16. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 134
  17. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 123
  18. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 112
  19. 2025GR-3 Technical Reportcited by 111
  20. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 108
  21. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  22. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  23. 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 92
  24. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  25. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  26. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  27. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 82
  28. 2025Humanoid Policy ~ Human Policycited by 78
  29. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 75
  30. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 74
  31. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 72
  32. 2026Unified 4D World Action Modeling from Video Priors with Asynchronous Denoisingcited by 71
  33. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 70
  34. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  35. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 68
  36. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 66
  37. 2025Training-Time Action Conditioning for Efficient Real-Time Chunkingcited by 64
  38. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 64

50 most cited of 691 All 691 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation
  2. 2024Planning-Guided Diffusion Policy Learning for Generalizable Contact-Rich Bimanual Manipulation
  3. 2024PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks
  4. 2025Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulation
  5. 2025Learning Coordinated Bimanual Manipulation Policies using State Diffusion and Inverse Dynamics Models
  6. 2026FineART: Fine-Grained Annotated Robotic Trajectory Dataset and Vision-Language-Action Model for Bimanual Manipulation
  7. 2025ManiDP: Manipulability-Aware Diffusion Policy for Posture-Dependent Bimanual Manipulation
  8. 2023Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation