mlsanityData to 7 Oct 2026

Imagining The Road Ahead: Multi-Agent Trajectory Prediction via Differentiable Simulation

arXiv 2104.11212 ↗First posted 22 Apr 2021stat.ML cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

We develop a deep generative model built on a fully differentiable simulator for multi-agent trajectory prediction. Agents are modeled with conditional recurrent variational neural networks (CVRNNs), which take as input an ego-centric birdview image representing the current state of the world and output an action, consisting of steering and acceleration, which is used to derive the subsequent agent state using a kinematic bicycle model. The full simulation state is then differentiably rendered for each agent, initiating the next time step. We achieve state-of-the-art results on the INTERACTION dataset, using standard neural architectures and a standard variational training objective, producing realistic multi-modal predictions without any ad-hoc diversity-inducing losses. We conduct ablation studies to examine individual components of the simulator, finding that both the kinematic bicycle model and the continuous feedback from the birdview image are crucial for achieving this level of performance. We name our model ITRA, for "Imagining the Road Ahead".

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

30 of 42 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2016Variational Inference: A Review for Statisticianscited by 1,466
  6. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  7. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  8. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  9. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  10. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  11. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  12. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
18 more
  1. 2018SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraintscited by 296
  2. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  3. 2018The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systemscited by 276
  4. 2020One Thousand and One Hours: Self-driving Motion Prediction Datasetcited by 239
  5. 2019Human Motion Trajectory Prediction: A Surveycited by 231
  6. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  7. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  8. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  9. 2019Multiple Futures Predictioncited by 178
  10. 2019Multi-Agent Tensor Fusion for Contextual Trajectory Predictioncited by 131
  11. 2020LGSVL Simulator: A High Fidelity Simulator for Autonomous Drivingcited by 119
  12. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  13. 2018Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Drivingcited by 90
  14. 2016An Uncertain Future: Forecasting from Static Images using Variational Autoencoderscited by 86
  15. 2020openDD: A Large-Scale Roundabout Drone Datasetcited by 29
  16. 2020ReCoG: A Deep Learning Framework with Heterogeneous Graph for Interaction-Aware Trajectory Predictioncited by 20
  17. 2019Amortized Rejection Sampling in Universal Probabilistic Programmingcited by 5
  18. 2020Coping With Simulators That Don't Always Returncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 5; 2023: 8; 2024: 10; 2025: 4; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 158
  2. 2021GOHOME: Graph-Oriented Heatmap Output for future Motion Estimationcited by 146
  3. 2022KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradientscited by 95
  4. 2024GPUDrive: Data-driven, multi-agent driving simulation at 1 million FPScited by 43
  5. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  6. 2021PredictionNet: Real-Time Joint Probabilistic Traffic Prediction for Planning, Control, and Simulationcited by 29
  7. 2024CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learningcited by 26
  8. 2023Learning Realistic Traffic Agents in Closed-loopcited by 24
  9. 2022Critic Sequential Monte Carlocited by 16
  10. 2023A Diffusion-Model of Joint Interactive Navigationcited by 16
  11. 2024Learning to Drive via Asymmetric Self-Playcited by 11
  12. 2023Bridging the Gap Between Multi-Step and One-Shot Trajectory Prediction via Self-Supervisioncited by 7
20 more
  1. 2023Conditional Unscented Autoencoders for Trajectory Predictioncited by 7
  2. 2023Don't be so negative! Score-based Generative Modeling with Oracle-assisted Guidancecited by 6
  3. 2024TorchDriveEnv: A Reinforcement Learning Benchmark for Autonomous Driving with Reactive, Realistic, and Diverse Non-Playable Characterscited by 6
  4. 2022DiPA: Probabilistic Multi-Modal Interactive Prediction for Autonomous Drivingcited by 5
  5. 2022Beyond Simple Meta-Learning: Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learningcited by 4
  6. 2023Interpretable Goal-Based model for Vehicle Trajectory Prediction in Interactive Scenarioscited by 3
  7. 2025Getting SMARTER for Motion Planning in Autonomous Driving Systemscited by 3
  8. 2022Vehicle Type Specific Waypoint Generationcited by 2
  9. 2024MAP-Former: Multi-Agent-Pair Gaussian Joint Predictioncited by 2
  10. 2024Gradient-based Trajectory Optimization with Parallelized Differentiable Traffic Simulationcited by 2
  11. 2025Control-ITRA: Controlling the Behavior of a Driving Modelcited by 2
  12. 2025Rolling Ahead Diffusion for Traffic Scene Simulationcited by 2
  13. 2026Improved Constrained Generation by Bridging Pretrained Generative Modelscited by 2
  14. 2023Video Killed the HD-Map: Predicting Multi-Agent Behavior Directly From Aerial Imagescited by 1
  15. 2024Interpretable Interaction Modeling for Trajectory Prediction via Agent Selection and Physical Coefficientcited by 1
  16. 2024Analyzing Closed-loop Training Techniques for Realistic Traffic Agent Models in Autonomous Highway Driving Simulationscited by 1
  17. 2025Toward Efficient and Robust Behavior Models for Multi-Agent Driving Simulationcited by 1
  18. 2026TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scalecited by 1
  19. 2024MRIC: Model-Based Reinforcement-Imitation Learning with Mixture-of-Codebooks for Autonomous Driving Simulationcited by 0
  20. 2026ROSA-RL: Uncertainty-Aware Roundabout Optimized Speed Advisory with Reinforcement Learningcited by 0