mlsanityData to 7 Oct 2026

Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals

arXiv 2407.05996 ↗First posted 8 Jul 2024cs.ROcited by 131

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

This work introduces the Multimodal Diffusion Transformer (MDT), a novel diffusion policy framework, that excels at learning versatile behavior from multimodal goal specifications with few language annotations. MDT leverages a diffusion-based multimodal transformer backbone and two self-supervised auxiliary objectives to master long-horizon manipulation tasks based on multimodal goals. The vast majority of imitation learning methods only learn from individual goal modalities, e.g. either language or goal images. However, existing large-scale imitation learning datasets are only partially labeled with language annotations, which prohibits current methods from learning language conditioned behavior from these datasets. MDT addresses this challenge by introducing a latent goal-conditioned state representation that is simultaneously trained on multimodal goal instructions. This state representation aligns image and language based goal embeddings and encodes sufficient information to predict future states. The representation is trained via two self-supervised auxiliary objectives, enhancing the performance of the presented transformer backbone. MDT shows exceptional performance on 164 tasks provided by the challenging CALVIN and LIBERO benchmarks, including a LIBERO version that contains less than 2% language annotations. Furthermore, MDT establishes a new record on the CALVIN manipulation challenge, demonstrating an absolute performance improvement of 15% over prior state-of-the-art methods that require large-scale pretraining and contain 10× more learnable parameters. MDT shows its ability to solve long-horizon manipulation from sparsely annotated data in both simulated and real-world environments. Demonstrations and Code are available at https://intuitive-robots.github.io/mdtpolicy/.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 65 of its 71 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

65 of 71 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play7 Dec
  2. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learning4 Jul
  3. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2023Consistency Modelscited by 1,778
38 more
  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  3. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
  4. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  5. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  6. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  7. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  8. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  10. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  12. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  13. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  14. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  15. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  16. 2019Learning Latent Plans from Playcited by 310
  17. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  18. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  19. 2022Masked Visual Pre-training for Motor Controlcited by 246
  20. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  21. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  22. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  23. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  24. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  25. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  26. 2023Language-Driven Representation Learning for Roboticscited by 163
  27. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  28. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  29. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  30. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  31. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  32. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  33. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  34. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 107
  35. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  36. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  37. 2023On Bringing Robots Homecited by 101
  38. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 66; 2026: 50.

131 papers.

Sort:

By area:

  1. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  2. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  3. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  4. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  5. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  6. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  7. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  8. 2025Interactive Post-Training for Vision-Language-Action Modelscited by 129
  9. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  10. 2025GR-3 Technical Reportcited by 110
  11. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  12. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
38 more
  1. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  2. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  3. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 65
  4. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 64
  5. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 59
  6. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  7. 2025FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policiescited by 56
  8. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  9. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  10. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulationcited by 48
  11. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  12. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  13. 2025ManiFlow: A General Robot Manipulation Policy via Consistency Flow Trainingcited by 43
  14. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 42
  15. 2024Diffusion Transformer Policycited by 41
  16. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  17. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  18. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  19. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  20. 2025Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Processcited by 37
  21. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  22. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  23. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 33
  24. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
  25. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Taskscited by 30
  26. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 27
  27. 2024CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Predictioncited by 26
  28. 2025BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learningcited by 25
  29. 2025WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulationcited by 21
  30. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  31. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 18
  32. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  33. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 17
  34. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  35. 2025ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrationscited by 17
  36. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  37. 2025Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Modelscited by 14
  38. 2025LUMOS: Language-Conditioned Imitation Learning with World Modelscited by 13

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Diffusion Forcing for Forceful Manipulation
  2. 2025Latent Diffusion Planning for Imitation Learning
  3. 2025Diffusion Trajectory-guided Policy for Long-horizon Robot Manipulation
  4. 2024DNAct: Diffusion Guided Multi-Task 3D Policy Learning
  5. 2025MTDP: A Modulated Transformer based Diffusion Policy Model
  6. 2024Unpacking the Individual Components of Diffusion Policy
  7. 2025D2PPO: Diffusion Policy Policy Optimization with Dispersive Loss
  8. 2025Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternatives