# SceneDM: Scene-level Multi-agent Trajectory Generation with Consistent Diffusion Models

arXiv: https://arxiv.org/abs/2311.15736 · HTML: https://mlsanity.com/paper/2311.15736/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Motion Transformer with Global Intention Localization and Local Movement Refinement](https://mlsanity.com/paper/2209.13508/) (2022)
- [MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction](https://mlsanity.com/paper/2111.14973/) (2021)
- [MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion](https://mlsanity.com/paper/2306.03083/) (2023)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion](https://mlsanity.com/paper/2203.13777/) (2022)
- [TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios](https://mlsanity.com/paper/2210.06609/) (2022)
- [The Waymo Open Sim Agents Challenge](https://mlsanity.com/paper/2305.12032/) (2023)
- [SceneGen: Learning to Generate Realistic Traffic Scenes](https://mlsanity.com/paper/2101.06541/) (2021)
- [SimNet: Learning Reactive Self-driving Simulations from Real-world Observations](https://mlsanity.com/paper/2105.12332/) (2021)
- [Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation](https://mlsanity.com/paper/2009.08311/) (2020)
- [Modeling Human Driving Behavior through Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.06412/) (2020)
- [Dynamic Scenario Representation Learning for Motion Forecasting with Heterogeneous Graph Convolutional Recurrent Networks](https://mlsanity.com/paper/2303.04364/) (2023)
- [Generating Driving Scenes with Diffusion](https://mlsanity.com/paper/2305.18452/) (2023)
- [Diverse Critical Interaction Generation for Planning and Planner Evaluation](https://mlsanity.com/paper/2103.00906/) (2021)
- [CVAE-H: Conditionalizing Variational Autoencoders via Hypernetworks and Trajectory Forecasting for Autonomous Driving](https://mlsanity.com/paper/2201.09874/) (2022)
- [TAE: A Semi-supervised Controllable Behavior-aware Trajectory Generator and Predictor](https://mlsanity.com/paper/2203.01261/) (2022)
- [The 2nd Place Solution for 2023 Waymo Open Sim Agents Challenge](https://mlsanity.com/paper/2306.15914/) (2023)
- [A New Multi-vehicle Trajectory Generator to Simulate Vehicle-to-Vehicle Encounters](https://mlsanity.com/paper/1809.05680/) (2018)

## Built on by (most cited first, 24 of 24)

- [DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving](https://mlsanity.com/paper/2408.00415/) (2024)
- [Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving](https://mlsanity.com/paper/2408.14197/) (2024)
- [Robust Autonomy Emerges from Self-Play](https://mlsanity.com/paper/2502.03349/) (2025)
- [SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout](https://mlsanity.com/paper/2412.12129/) (2024)
- [Gen-Drive: Enhancing Diffusion Generative Driving Policies with Reward Modeling and Reinforcement Learning Fine-tuning](https://mlsanity.com/paper/2410.05582/) (2024)
- [Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training](https://mlsanity.com/paper/2503.12030/) (2025)
- [DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes](https://mlsanity.com/paper/2409.04003/) (2024)
- [WcDT: World-centric Diffusion Transformer for Traffic Scene Generation](https://mlsanity.com/paper/2404.02082/) (2024)
- [Towards Knowledge-driven Autonomous Driving](https://mlsanity.com/paper/2312.04316/) (2023)
- [Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments](https://mlsanity.com/paper/2503.22496/) (2025)
- [CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning](https://mlsanity.com/paper/2403.19918/) (2024)
- [SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction](https://mlsanity.com/paper/2405.15677/) (2024)
- [SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model](https://mlsanity.com/paper/2506.21976/) (2025)
- [Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation](https://mlsanity.com/paper/2408.00766/) (2024)
- [Decoupled Diffusion Sparks Adaptive Scene Generation](https://mlsanity.com/paper/2504.10485/) (2025)
- [LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation](https://mlsanity.com/paper/2504.11521/) (2025)
- [Integration of Mixture of Experts and Multimodal Generative AI in Internet of Vehicles: A Survey](https://mlsanity.com/paper/2404.16356/) (2024)
- [DragTraffic: Interactive and Controllable Traffic Scene Generation for Autonomous Driving](https://mlsanity.com/paper/2404.12624/) (2024)
- [DiffRoad: Realistic and Diverse Road Scenario Generation for Autonomous Vehicle Testing](https://mlsanity.com/paper/2411.09451/) (2024)
- [Rolling Ahead Diffusion for Traffic Scene Simulation](https://mlsanity.com/paper/2502.09587/) (2025)
- [Direct Preference Optimization-Enhanced Multi-Guided Diffusion Model for Traffic Scenario Generation](https://mlsanity.com/paper/2502.12178/) (2025)
- [Controllable Motion Generation via Diffusion Modal Coupling](https://mlsanity.com/paper/2503.02353/) (2025)
- [SceneStreamer: Continuous Scenario Generation as Next Token Group Prediction](https://mlsanity.com/paper/2506.23316/) (2025)
- [Robust Planning for Autonomous Driving via Mixed Adversarial Diffusion Predictions](https://mlsanity.com/paper/2505.12327/) (2025)

## Abstract (arXiv)

Realistic scene-level multi-agent motion simulations are crucial for developing and evaluating self-driving algorithms. However, most existing works focus on generating trajectories for a certain single agent type, and typically ignore the consistency of generated trajectories. In this paper, we propose a novel framework based on diffusion models, called SceneDM, to generate joint and consistent future motions of all the agents, including vehicles, bicycles, pedestrians, etc., in a scene. To enhance the consistency of the generated trajectories, we resort to a new Transformer-based network to effectively handle agent-agent interactions in the inverse process of motion diffusion. In consideration of the smoothness of agent trajectories, we further design a simple yet effective consistent diffusion approach, to improve the model in exploiting short-term temporal dependencies. Furthermore, a scene-level scoring function is attached to evaluate the safety and road-adherence of the generated agent's motions and help filter out unrealistic simulations. Finally, SceneDM achieves state-of-the-art results on the Waymo Sim Agents Benchmark. Project webpage is available at https://alperen-hub.github.io/SceneDM.