mlsanityData to 7 Oct 2026

Scene Transformer: A unified architecture for predicting multiple agent trajectories

arXiv 2106.08417 ↗First posted 15 Jun 2021cs.CV cs.LG cs.ROcited by 165

On the map: World models → World models for autonomous driving

Abstract (arXiv)

Predicting the motion of multiple agents is necessary for planning in dynamic environments. This task is challenging for autonomous driving since agents (e.g. vehicles and pedestrians) and their associated behaviors may be diverse and influence one another. Most prior work have focused on predicting independent futures for each agent based on all past motion, and planning against these independent predictions. However, planning against independent predictions can make it challenging to represent the future interaction possibilities between different agents, leading to sub-optimal planning. In this work, we formulate a model for predicting the behavior of all agents jointly, producing consistent futures that account for interactions between agents. Inspired by recent language modeling approaches, we use a masking strategy as the query to our model, enabling one to invoke a single model to predict agent behavior in many ways, such as potentially conditioned on the goal or full future trajectory of the autonomous vehicle or the behavior of other agents in the environment. Our model architecture employs attention to combine features across road elements, agent interactions, and time steps. We evaluate our approach on autonomous driving datasets for both marginal and joint motion prediction, and achieve state of the art performance across two popular datasets. Through combining a scene-centric approach, agent permutation equivariant model, and a sequence masking strategy, we show that our model can unify a variety of motion prediction tasks from joint motion predictions to conditioned prediction.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). Placed in World models for autonomous driving, one of 11 directions in World models. Builds on 55 of its 76 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

55 of 76 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks18 Sep

Other references · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2019Invariant Risk Minimizationcited by 1,832
  11. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 728
  12. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
38 more
  1. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  2. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  3. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  4. 2020VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representationcited by 528
  5. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  6. 2020Learning Lane Graph Representations for Motion Forecastingcited by 380
  7. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  8. 2019Attention Augmented Convolutional Networkscited by 363
  9. 2019Axial Attention in Multidimensional Transformerscited by 350
  10. 2020TNT: Target-driveN Trajectory Predictioncited by 345
  11. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 327
  12. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  13. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  14. 2021End-to-end Interpretable Neural Motion Plannercited by 240
  15. 2019INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Mapscited by 231
  16. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  17. 2020Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Predictioncited by 218
  18. 2021Multimodal Motion Prediction with Stacked Transformerscited by 207
  19. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  20. 2020Transformer Networks for Trajectory Forecastingcited by 180
  21. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  22. 2019Multiple Futures Predictioncited by 178
  23. 2019Multi-Agent Tensor Fusion for Contextual Trajectory Predictioncited by 131
  24. 2021HOME: Heatmap Output for future Motion Estimationcited by 122
  25. 2019Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactionscited by 117
  26. 2020Implicit Latent Variable Model for Scene-Consistent Motion Forecastingcited by 116
  27. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  28. 2019Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Datacited by 104
  29. 2021TPCN: Temporal Point Cloud Networks for Motion Forecastingcited by 94
  30. 2018TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactionscited by 82
  31. 2020What-If Motion Prediction for Autonomous Drivingcited by 80
  32. 2020Trajectory Forecasts in Unknown Environments Conditioned on Grid-Based Planscited by 72
  33. 2020MANTRA: Memory Augmented Networks for Multiple Trajectory Predictioncited by 69
  34. 2020Diverse and Admissible Trajectory Forecasting through Multimodal Context Understandingcited by 68
  35. 2020End-to-end Contextual Perception and Prediction with Interaction Transformercited by 54
  36. 2021Identifying Driver Interactions via Conditional Behavior Predictioncited by 48
  37. 2019Learning an Uncertainty-Aware Object Detector for Autonomous Drivingcited by 34
  38. 2020PRANK: motion Prediction based on RANKingcited by 22

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 12; 2023: 29; 2024: 42; 2025: 51; 2026: 29.

165 papers.

Sort:

By area:

  1. 2023VAD: Vectorized Scene Representation for Efficient Autonomous Drivingcited by 613
  2. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  3. 2024VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planningcited by 287
  4. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  5. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 158
  6. 2025Diffusion-Based Planning for Autonomous Driving with Flexible Guidancecited by 138
  7. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  8. 2024PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Drivingcited by 84
  9. 2022M2I: From Factored Marginal Trajectory Prediction to Interactive Predictioncited by 81
  10. 2023Scenario Diffusion: Controllable Driving Scenario Generation With Diffusioncited by 46
  11. 2025Distilling Multi-modal Large Language Models for Autonomous Drivingcited by 46
  12. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 43
38 more
  1. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  2. 2024Solving Motion Planning Tasks with a Scalable Generative Modelcited by 40
  3. 2024Closed-Loop Supervised Fine-Tuning of Tokenized Traffic Modelscited by 40
  4. 2023ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptationcited by 39
  5. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  6. 2023BEVNeXt: Reviving Dense BEV Frameworks for 3D Object Detectioncited by 35
  7. 2024SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Predictioncited by 33
  8. 2023QCNeXt: A Next-Generation Framework For Joint Multi-Agent Trajectory Predictioncited by 31
  9. 2024WcDT: World-centric Diffusion Transformer for Traffic Scene Generationcited by 31
  10. 2024GaussianAD: Gaussian-Centric End-to-End Autonomous Drivingcited by 31
  11. 2023SEPT: Towards Efficient Scene Representation Learning for Motion Predictioncited by 27
  12. 2023SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Predictioncited by 27
  13. 2024SIMPL: A Simple and Efficient Multi-agent Motion Prediction Baseline for Autonomous Drivingcited by 27
  14. 2025DiffAD: A Unified Diffusion Modeling Approach for Autonomous Drivingcited by 26
  15. 2023Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023cited by 25
  16. 2024SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Predictioncited by 25
  17. 2022Addressing Optimism Bias in Sequence Modeling for Reinforcement Learningcited by 22
  18. 2024Causal Composition Diffusion Model for Closed-loop Traffic Generationcited by 22
  19. 2021COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modalitycited by 19
  20. 2025Scaling Laws of Motion Forecasting and Planning -- Technical Reportcited by 19
  21. 2024VisionTrap: Vision-Augmented Trajectory Prediction Guided by Textual Descriptionscited by 18
  22. 2024Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generationcited by 18
  23. 2025Learning Personalized Driving Styles via Reinforcement Learning from Human Feedbackcited by 16
  24. 2024DeTra: A Unified Model for Object Detection and Trajectory Forecastingcited by 15
  25. 2025Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modelingcited by 15
  26. 2025Co-MTP: A Cooperative Trajectory Prediction Framework with Multi-Temporal Fusion for Autonomous Drivingcited by 14
  27. 2023MoDAR: Using Motion Forecasting for 3D Object Detection in Point Cloud Sequencescited by 13
  28. 2024Large Language Models Powered Context-aware Motion Prediction in Autonomous Drivingcited by 13
  29. 2025Trajectory Prediction Meets Large Language Models: A Surveycited by 13
  30. 2024Learning to Drive via Asymmetric Self-Playcited by 11
  31. 2024TranSPORTmer: A Holistic Approach to Trajectory Understanding in Multi-Agent Sportscited by 11
  32. 2023Holistic Graph-based Motion Predictioncited by 10
  33. 2023Categorical Traffic Transformer: Interpretable and Diverse Behavior Prediction with Tokenized Latentcited by 10
  34. 2022Domain Knowledge Driven Pseudo Labels for Interpretable Goal-Conditioned Interactive Trajectory Predictioncited by 9
  35. 2023Deception Game: Closing the Safety-Learning Loop in Interactive Robot Autonomycited by 9
  36. 2023MGTR: Multi-Granular Transformer for Motion Prediction with LiDARcited by 9
  37. 2021TIP: Task-Informed Motion Prediction for Intelligent Vehiclescited by 8
  38. 2022STrajNet: Multi-modal Hierarchical Transformer for Occupancy Flow Field Prediction in Autonomous Drivingcited by 8

50 most cited of 165 All 165 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Driving
  2. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modeling
  3. 2025Self-supervised Pretraining for Integrated Prediction and Planning of Automated Vehicles ↗
  4. 2024Diffusion-Based Environment-Aware Trajectory Prediction
  5. 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives
  6. 2022JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Driving
  7. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networks
  8. 2024Controllable Diverse Sampling for Diffusion Based Motion Behavior Forecasting