# Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation

arXiv: https://arxiv.org/abs/2408.00766 · HTML: https://mlsanity.com/paper/2408.00766/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 46 of its 52 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 46)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [Fast Sampling of Diffusion Models with Exponential Integrator](https://mlsanity.com/paper/2204.13902/) (2022)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction](https://mlsanity.com/paper/2112.05146/) (2021)
- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Wayformer: Motion Forecasting via Simple & Efficient Attention Networks](https://mlsanity.com/paper/2207.05844/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion](https://mlsanity.com/paper/2203.13777/) (2022)
- [Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality](https://mlsanity.com/paper/2202.05830/) (2022)
- [Scene Transformer: A unified architecture for predicting multiple agent trajectories](https://mlsanity.com/paper/2106.08417/) (2021)
- [MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying](https://mlsanity.com/paper/2306.17770/) (2023)
- [Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion](https://mlsanity.com/paper/2304.01893/) (2023)
- [THOMAS: Trajectory Heatmap Output with learned Multi-Agent Sampling](https://mlsanity.com/paper/2110.06607/) (2021)
- [Noise Estimation for Generative Diffusion Models](https://mlsanity.com/paper/2104.02600/) (2021)

## Built on by (most cited first, 18 of 18)

- [DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving](https://mlsanity.com/paper/2411.15139/) (2024)
- [TrajDiff: End-to-end Autonomous Driving without Perception Annotation](https://mlsanity.com/paper/2512.00723/) (2025)
- [Continuous Locomotive Crowd Behavior Generation](https://mlsanity.com/paper/2504.04756/) (2025)
- [Projected Coupled Diffusion for Test-Time Constrained Joint Generation](https://mlsanity.com/paper/2508.10531/) (2025)
- [Hypergraph-based Motion Generation with Multi-modal Interaction Relational Reasoning](https://mlsanity.com/paper/2409.11676/) (2024)
- [Generative Active Learning for Long-tail Trajectory Prediction via Controllable Diffusion Model](https://mlsanity.com/paper/2507.22615/) (2025)
- [Unified Human Localization and Trajectory Prediction with Monocular Vision](https://mlsanity.com/paper/2503.03535/) (2025)
- [ParkDiffusion++: Ego Intention Conditioned Joint Multi-Agent Trajectory Prediction for Automated Parking using Diffusion Models](https://mlsanity.com/paper/2602.20923/) (2026)
- [Solving Prior Distribution Mismatch in Diffusion Models via Optimal Transport](https://mlsanity.com/paper/2410.13431/) (2024)
- [TrafficGamer: Reliable and Flexible Traffic Simulation for Safety-Critical Scenarios with Game-Theoretic Oracles](https://mlsanity.com/paper/2408.15538/) (2024)
- [EP-Diffuser: An Efficient Diffusion Model for Traffic Scene Generation and Prediction via Polynomial Representations](https://mlsanity.com/paper/2504.05422/) (2025)
- [SocialTraj: Two-Stage Socially-Aware Trajectory Prediction for Autonomous Driving via Conditional Diffusion Model](https://mlsanity.com/paper/2509.17850/) (2025)
- [Path Diffuser: Diffusion Model for Data-Driven Traffic Simulator](https://mlsanity.com/paper/2509.24995/) (2025)
- [Manifold-Optimal Guidance: A Unified Riemannian Control View of Diffusion Guidance](https://mlsanity.com/paper/2603.11509/) (2026)
- [ECTraj: Enhanced Consistency Training for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2605.08572/) (2026)
- [Guiding Neuro-Symbolic Scenario Generation with Spatio-Temporal Logic](https://mlsanity.com/paper/2605.19038/) (2026)
- [Guided Diffusion with Distilled Vision-Language Reliability for Aerial Navigation](https://mlsanity.com/paper/2606.13883/) (2026)
- [Proposal-Conditioned Latent Diffusion for Closed-Loop Traffic Scenario Generation](https://mlsanity.com/paper/2606.27123/) (2026)

## Abstract (arXiv)

Diffusion models are promising for joint trajectory prediction and controllable generation in autonomous driving, but they face challenges of inefficient inference steps and high computational demands. To tackle these challenges, we introduce Optimal Gaussian Diffusion (OGD) and Estimated Clean Manifold (ECM) Guidance. OGD optimizes the prior distribution for a small diffusion time T and starts the reverse diffusion process from it. ECM directly injects guidance gradients to the estimated clean manifold, eliminating extensive gradient backpropagation throughout the network. Our methodology streamlines the generative process, enabling practical applications with reduced computational overhead. Experimental validation on the large-scale Argoverse 2 dataset demonstrates our approach's superior performance, offering a viable solution for computationally efficient, high-quality joint trajectory prediction and controllable generation for autonomous driving. Our project webpage is at https://yixiaowang7.github.io/OptTrajDiffPage/.