mlsanityData to 7 Oct 2026

Diffusion Transformer Policy

arXiv 2410.15959 ↗First posted 21 Oct 2024cs.RO cs.CVcited by 41

On the map: Vision-language-action models → Efficient VLA inference and adaptation

also in 1 more
Abstract (arXiv)

Recent large vision-language-action models pretrained on diverse robot datasets have demonstrated the potential for generalizing to new environments with a few in-domain data. However, those approaches usually predict individual discretized or continuous action by a small action head, which limits the ability in handling diverse action spaces. In contrast, we model the continuous action sequence with a large multi-modal diffusion transformer, dubbed as Diffusion Transformer Policy, in which we directly denoise action chunks by a large transformer model rather than a small action head for action embedding. By leveraging the scaling capability of transformers, the proposed approach can effectively model continuous end-effector actions across large diverse robot datasets, and achieve better generalization performance. Extensive experiments demonstrate the effectiveness and generalization of Diffusion Transformer Policy on Maniskill2, Libero, Calvin and SimplerEnv, as well as the real-world Franka arm, achieving consistent better performance on Real-to-Sim benchmark SimplerEnv, real-world Franka Arm and Libero compared to OpenVLA and Octo. Specifically, without bells and whistles, the proposed approach achieves state-of-the-art performance with only a single third-view camera stream in the Calvin task ABC->D, improving the average number of tasks completed in a row of 5 to 3.6, and the pretraining stage significantly facilitates the success sequence length on the Calvin by over 1.2.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Efficient VLA inference and adaptation, one of 13 directions in Vision-language-action models. Builds on 66 of its 79 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

66 of 79 references.

References · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
38 more
  1. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  5. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  10. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  11. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  12. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  13. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  14. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  15. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  16. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  17. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  18. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  19. 2019Learning Latent Plans from Playcited by 310
  20. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  21. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  22. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  23. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 234
  24. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
  25. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  26. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  27. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  28. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  29. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  30. 2023Language-Driven Representation Learning for Roboticscited by 163
  31. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  32. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  33. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  34. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  35. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  36. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  37. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  38. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 20; 2026: 20.

41 papers.

Sort:

By area:

  1. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  2. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  4. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  5. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  6. 2025Think Twice, Act Once: Token-Aware Compression and Action Reuse for Efficient Inference in Vision-Language-Action Modelscited by 30
  7. 2025dVLA: Diffusion Vision-Language-Action Model with Multimodal Chain-of-Thoughtcited by 30
  8. 2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 28
  9. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  10. 2025RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedbackcited by 19
  11. 2026World-Value-Action Model: Implicit Planning for Vision-Language-Action Systemscited by 13
  12. 2025Block-wise Adaptive Caching for Accelerating Diffusion Policycited by 11
29 more
  1. 2025Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videoscited by 10
  2. 2026MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generationcited by 10
  3. 2026Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wildcited by 8
  4. 2026Pri4R: Learning World Dynamics for Vision-Language-Action Models with Privileged 4D Representationcited by 7
  5. 2026Training-Free Adaptation of Diffusion Models via Doob's h-Transformcited by 6
  6. 2025Learning to See and Act: Task-Aware Virtual View Exploration for Robotic Manipulationcited by 5
  7. 2025RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulationcited by 5
  8. 2025EveryDayVLA: A Vision-Language-Action Model for Affordable Robotic Manipulationcited by 5
  9. 2026RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulationcited by 5
  10. 2025ADPro: a Test-time Adaptive Diffusion Policy via Manifold-constrained Denoising and Task-aware Initialization for Robotic Manipulationcited by 4
  11. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  12. 2026CF-VLA: Efficient Coarse-to-Fine Action Generation for Vision-Language-Action Policiescited by 4
  13. 2025Hybrid Diffusion Policies with Projective Geometric Algebra for Efficient Robot Manipulation Learningcited by 3
  14. 2025Robotic Manipulation via Imitation Learning: Taxonomy, Evolution, Benchmark, and Challengescited by 3
  15. 2025MTDP: A Modulated Transformer based Diffusion Policy Modelcited by 2
  16. 2026Continuous Reasoning for Vision-Language-Actioncited by 2
  17. 2025Dynamic Objects Relocalization in Changing Environments with Flow Matchingcited by 1
  18. 2026FlowMaps: Modeling Long-Term Multimodal Object Dynamics with Flow Matchingcited by 1
  19. 2026PHR-VLA: Planning Horizon Reasoning for Vision-Language-Action Modelscited by 1
  20. 2026KAN We Flow? Advancing Robotic Manipulation with 3D Flow Matching via KAN & RWKVcited by 0
  21. 2026Test-time Adversarial Takeover: A Real-time Hijacking Interface against Robotic Diffusion Policiescited by 0
  22. 2026IMPACT: Learning Internal-Model Predictive Control for Forceful Robotic Manipulationcited by 0
  23. 2026WorldBagel: Uncovering the Power of Unified Multimodal Models for Vision-Language-Action-World Modelingcited by 0
  24. 2026Patch Policy: Efficient Embodied Control via Dense Visual Representationscited by 0
  25. 2026WorldDiT: A Unified Diffusion Architecture for World and Action Modelingcited by 0
  26. 2026Not All Layers Need Tuning: Diagnosing and Directing Adaptation in Vision-Language-Action Modelscited by 0
  27. 2026A Comprehensive Review of Generative Physical Artificial Intelligencecited by 0
  28. 2026Efficient World Action Model Inference with Adaptive Intermediate Statescited by 0
  29. 2026A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platformcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy
  2. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation
  3. 2024The Ingredients for Robotic Diffusion Transformers
  4. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Process
  5. 2026DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
  6. 2025Mini Diffuser: Fast Multi-task Diffusion Policy Training Using Two-level Mini-batches
  7. 2024RoLD: Robot Latent Diffusion for Multi-task Policy Modeling
  8. 2026Self-Improving VLA Policies: Selected Diffusion Noise for Spurious-Robust Action Smoothing