mlsanityData to 7 Oct 2026

MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion

arXiv 2306.03083 ↗First posted 5 Jun 2023cs.RO cs.AIcited by 210

On the map: World models → World models for autonomous driving

Abstract (arXiv)

We present MotionDiffuser, a diffusion based representation for the joint distribution of future trajectories over multiple agents. Such representation has several key advantages: first, our model learns a highly multimodal distribution that captures diverse future outcomes. Second, the simple predictor design requires only a single L2 loss training objective, and does not depend on trajectory anchors. Third, our model is capable of learning the joint distribution for the motion of multiple agents in a permutation-invariant manner. Furthermore, we utilize a compressed trajectory representation via PCA, which improves model performance and allows for efficient computation of the exact sample log probability. Subsequently, we propose a general constrained sampling framework that enables controlled trajectory sampling based on differentiable cost functions. This strategy enables a host of applications such as enforcing rules and physical priors, or creating tailored simulation scenarios. MotionDiffuser can be combined with existing backbone architectures to achieve top motion forecasting results. We obtain state-of-the-art results for multi-agent motion prediction on the Waymo Open Motion Dataset.

Key facts. Cited by 210 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 48 of its 57 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

48 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networks12 Jul

Other references · 47

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2018Neural Ordinary Differential Equationscited by 3,733
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2022Video Diffusion Modelscited by 2,280
35 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2017Decoupled Weight Decay Regularizationcited by 1,722
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  4. 2022Denoising Diffusion Restoration Modelscited by 951
  5. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  6. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  7. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  8. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  9. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 501
  10. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  11. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 306
  12. 2021MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Predictioncited by 246
  13. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  14. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  15. 2021MP3: A Unified Model to Map, Perceive, Predict and Plancited by 211
  16. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  17. 2022Stochastic Trajectory Prediction via Motion Indeterminacy Diffusioncited by 184
  18. 2019Multiple Futures Predictioncited by 178
  19. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  20. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  21. 2021HOME: Heatmap Output for future Motion Estimationcited by 122
  22. 2022M2I: From Factored Marginal Trajectory Prediction to Interactive Predictioncited by 81
  23. 2020PiP: Planning-informed Trajectory Prediction for Autonomous Drivingcited by 80
  24. 2020What-If Motion Prediction for Autonomous Drivingcited by 80
  25. 2021Generating Smooth Pose Sequences for Diverse Human Motion Predictioncited by 66
  26. 2020DSDNet: Deep Structured self-Driving Networkcited by 61
  27. 2021Identifying Driver Interactions via Conditional Behavior Predictioncited by 48
  28. 2020Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approachcited by 47
  29. 2022JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Drivingcited by 40
  30. 2021RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecastingcited by 36
  31. 2022A generic diffusion-based approach for 3D human pose prediction in the wildcited by 33
  32. 2021FloMo: Tractable Motion Prediction with Normalizing Flowscited by 21
  33. 2021DenseTNT: Waymo Open Dataset Motion Prediction Challenge 1st Place Solutioncited by 14
  34. 2022CVAE-H: Conditionalizing Variational Autoencoders via Hypernetworks and Trajectory Forecasting for Autonomous Drivingcited by 13
  35. 2019Real Time Trajectory Prediction Using Deep Conditional Generative Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 54; 2025: 90; 2026: 52.

210 papers.

Sort:

By area:

  1. 2024DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Drivingcited by 324
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 138
  4. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 136
  5. 2025GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Drivingcited by 117
  6. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 110
  7. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  8. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  9. 2023DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcastingcited by 57
  10. 2024UniTraj: A Unified Framework for Scalable Vehicle Trajectory Predictioncited by 51
  11. 2024SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rolloutcited by 47
  12. 2023DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Drivingcited by 46
38 more
  1. 2024Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Followingcited by 41
  2. 2024Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuningcited by 41
  3. 2024DiffuserLite: Towards Real-time Diffusion Planningcited by 40
  4. 2024Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Modelscited by 40
  5. 2025Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Trainingcited by 39
  6. 2024Multi-Robot Motion Planning with Diffusion Modelscited by 37
  7. 2024SingularTrajectory: Universal Trajectory Predictor Using Diffusion Modelcited by 35
  8. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  9. 2024Variational Diffusion Posterior Sampling with Midpoint Guidancecited by 32
  10. 2024Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Predictioncited by 31
  11. 2025CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Drivingcited by 31
  12. 2025MoFlow: One-Step Flow Matching for Human Trajectory Forecasting via Implicit Maximum Likelihood Estimation based Distillationcited by 31
  13. 2024Model-Based Diffusion for Trajectory Optimizationcited by 30
  14. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  15. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  16. 2024Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioningcited by 29
  17. 2024DriveGPT: Scaling Autoregressive Behavior Models for Drivingcited by 29
  18. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 27
  19. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 26
  20. 2025DiffAD: A Unified Diffusion Modeling Approach for Autonomous Drivingcited by 26
  21. 2023Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023cited by 25
  22. 2024SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Predictioncited by 25
  23. 2023SceneDM: Scene-level Multi-agent Trajectory Generation with Consistent Diffusion Modelscited by 24
  24. 2025Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Drivingcited by 24
  25. 2023Large Trajectory Models are Scalable Motion Predictors and Plannerscited by 23
  26. 2024Adapting to Length Shift: FlexiLength Network for Trajectory Predictioncited by 22
  27. 2024Causal Composition Diffusion Model for Closed-loop Traffic Generationcited by 22
  28. 2025Flow Matching-Based Autonomous Driving Planning with Advanced Interactive Behavior Modelingcited by 22
  29. 2024T4P: Test-Time Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memorycited by 20
  30. 2024AdvDiffuser: Generating Adversarial Safety-Critical Driving Scenarios via Guided Diffusioncited by 20
  31. 2025SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Modelcited by 20
  32. 2024Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sportscited by 19
  33. 2023DICE: Diverse Diffusion Model with Scoring for Trajectory Predictioncited by 18
  34. 2024DifFUSER: Diffusion Model for Robust Multi-Sensor Fusion in 3D Object Detection and BEV Segmentationcited by 18
  35. 2024Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generationcited by 18
  36. 2025BEVDiffuser: Plug-and-Play Diffusion Model for BEV Denoising with Ground-Truth Guidancecited by 17
  37. 2025DIVER: Reinforced Diffusion Breaks Imitation Bottlenecks in End-to-End Autonomous Drivingcited by 17
  38. 2025Learning Personalized Driving Styles via Reinforcement Learning from Human Feedbackcited by 16

50 most cited of 210 All 210 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making
  2. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modeling
  3. 2024Motion Planning Diffusion: Learning and Adapting Robot Motion Planning with Diffusion Models
  4. 2022End-to-End Trajectory Distribution Prediction Based on Occupancy Grid Maps
  5. 2023Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models
  6. 2025CoPlanner: An Interactive Motion Planner with Contingency-Aware Diffusion for Autonomous Driving
  7. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectories
  8. 2024DeMo: Decoupling Motion Forecasting into Directional Intentions and Dynamic States