mlsanityData to 14 Oct 2026

The Ingredients for Robotic Diffusion Transformers

arXiv 2410.10088 ↗First posted 14 Oct 2024cs.RO cs.AI cs.CVcited by 73

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

In recent years roboticists have achieved remarkable progress in solving increasingly general tasks on dexterous robotic hardware by leveraging high capacity Transformer network architectures and generative diffusion models. Unfortunately, combining these two orthogonal improvements has proven surprisingly difficult, since there is no clear and well-understood process for making important design choices. In this paper, we identify, study and improve key architectural design decisions for high-capacity diffusion transformer policies. The resulting models can efficiently solve diverse tasks on multiple robot embodiments, without the excruciating pain of per-setup hyper-parameter tuning. By combining the results of our investigation with our improved model components, we are able to present a novel architecture, named method, that significantly outperforms the state of the art in solving long-horizon (1500+ time-steps) dexterous tasks on a bi-manual ALOHA robot. In addition, we find that our policies show improved scaling performance when trained on 10 hours of highly multi-modal, language annotated ALOHA demonstration data. We hope this work will open the door for future robot learning techniques that leverage the efficiency of generative diffusion modeling with the scalability of large scale transformer architectures. Code, robot dataset, and videos are available at: https://dit-policy.github.io

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 56 of its 72 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

56 of 72 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Diffusion-Reward Adversarial Imitation Learning25 May
  2. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillation13 May
  3. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representations16 Feb
  4. 2023DiffAIL: Diffusion Adversarial Imitation Learning11 Dec
  5. 2023C3DM: Constrained-Context Conditional Diffusion Models for Imitation Learning2 Nov
  6. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  8. 2023Diffusion Model-Augmented Behavioral Cloning26 Feb
  9. 2023Imitating Human Behaviour with Diffusion Models25 Jan

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2023Segment Anythingcited by 8,234
  10. 2020End-to-End Object Detection with Transformerscited by 7,414
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2022Scalable Diffusion Models with Transformerscited by 6,054
35 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  11. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  12. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  13. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  14. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  15. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  16. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  17. 2021Implicit Behavioral Cloningcited by 452
  18. 2021Early Convolutions Help Transformers See Bettercited by 396
  19. 2019Learning Latent Plans from Playcited by 316
  20. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  21. 2021How to Train Your Energy-Based Modelscited by 274
  22. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  23. 2023ViNT: A Foundation Model for Visual Navigationcited by 222
  24. 2014Simultaneous Detection and Segmentationcited by 159
  25. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 138
  26. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 124
  27. 2021Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objectscited by 119
  28. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 114
  29. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  30. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  31. 2020Transformers for One-Shot Visual Imitationcited by 78
  32. 2015Multimodal Deep Learning for Robust RGB-D Object Recognitioncited by 57
  33. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 42
  34. 2020Bridging the Imitation Gap by Adaptive Insubordinationcited by 38
  35. 2017Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environmentscited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 36; 2026: 33.

73 papers.

Sort:

By area:

  1. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  2. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  4. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 142
  5. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 113
  6. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 106
  7. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 105
  8. 2025Humanoid Policy ~ Human Policycited by 78
  9. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 74
  10. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  11. 2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 57
  12. 2025Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policiescited by 52
38 more
  1. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  2. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Trainingcited by 49
  3. 2026World Model for Robot Learning: A Comprehensive Surveycited by 46
  4. 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Controlcited by 36
  5. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 32
  6. 2025VITA: Vision-to-Action Flow Matching Policycited by 25
  7. 2024ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Taskscited by 22
  8. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
  9. 2025ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstrationcited by 20
  10. 2025ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrationscited by 19
  11. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 17
  12. 2026CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansioncited by 17
  13. 2024GHIL-Glue: Hierarchical Control with Filtered Subgoal Imagescited by 15
  14. 2025MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videoscited by 12
  15. 2026Training-Free Adaptation of Diffusion Models via Doob's h-Transformcited by 11
  16. 2025Point What You Mean: Visually Grounded Instruction Policycited by 10
  17. 2026Long-Horizon Manipulation via Trace-Conditioned VLA Planningcited by 9
  18. 2025Grounding Actions in Camera Space: Observation-Centric Vision-Language-Action Policycited by 8
  19. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  20. 2026A1: A Fully Transparent Open-Source, Adaptive and Efficient Truncated Vision-Language-Action Modelcited by 8
  21. 2025Hybrid Diffusion for Simultaneous Symbolic and Continuous Planningcited by 7
  22. 2026Green-VLA: Staged Vision-Language-Action Model for Generalist Robotscited by 6
  23. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 5
  24. 2025KDPE: A Kernel Density Estimation Strategy for Diffusion Policy Trajectory Selectioncited by 4
  25. 2025D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AIcited by 4
  26. 2026From Flow to One Step: Real-Time Multi-Modal Trajectory Policies via Implicit Maximum Likelihood Estimation-based Distribution Distillationcited by 4
  27. 2026FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Spacecited by 4
  28. 2025Train-Once Plan-Anywhere Kinodynamic Motion Planning via Diffusion Treescited by 3
  29. 2026SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flowscited by 3
  30. 2026Drift is a Sampling Error: SNR-Aware Power Distributions for Long-Horizon Robotic Planningcited by 3
  31. 2025PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playingcited by 2
  32. 2025Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learningcited by 2
  33. 2025M4Diffuser: Multi-View Diffusion Policy with Manipulability-Aware Control for Robust Mobile Manipulationcited by 2
  34. 2026Moving On, Even When You're Broken: Fail-Active Trajectory Generation via Diffusion Policies Conditioned on Embodiment and Taskcited by 2
  35. 2026Frequency-Guided Action Diffusion via Sub-Frequency Manifold Traversalcited by 2
  36. 2026Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latencycited by 2
  37. 2025FRMD: Fast Robot Motion Diffusion with Consistency-Distilled Movement Primitives for Smooth Action Generationcited by 1
  38. 2025U-DiT Policy: U-shaped Diffusion Transformers for Robotic Manipulationcited by 1

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation
  2. 2025MTDP: A Modulated Transformer based Diffusion Policy Model
  3. 2024Diffusion Transformer Policy
  4. 2024Advances in Transformers for Robotic Applications: A Review
  5. 2025On-Device Diffusion Transformer Policy for Efficient Robot Manipulation
  6. 2026Diffusion Sequence Models for Generative In-Context Meta-Learning of Robot Dynamics
  7. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation
  8. 2025Hybrid Diffusion Policies with Projective Geometric Algebra for Efficient Robot Manipulation Learning