mlsanityData to 7 Oct 2026

AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting

arXiv 2103.14023 ↗First posted 25 Mar 2021cs.AI cs.CV cs.LGcited by 327

Not on the map. Why?

Abstract (arXiv)

Predicting accurate future trajectories of multiple agents is essential for autonomous systems, but is challenging due to the complex agent interaction and the uncertainty in each agent's future behavior. Forecasting multi-agent trajectories requires modeling two key dimensions: (1) time dimension, where we model the influence of past agent states over future states; (2) social dimension, where we model how the state of each agent affects others. Most prior methods model these two dimensions separately, e.g., first using a temporal model to summarize features over time for each agent independently and then modeling the interaction of the summarized features with a social model. This approach is suboptimal since independent feature encoding over either the time or social dimension can result in a loss of information. Instead, we would prefer a method that allows an agent's state at one time to directly affect another agent's state at a future time. To this end, we propose a new Transformer, AgentFormer, that jointly models the time and social dimensions. The model leverages a sequence representation of multi-agent trajectories by flattening trajectory features across time and agents. Since standard attention operations disregard the agent identity of each element in the sequence, AgentFormer uses a novel agent-aware attention mechanism that preserves agent identities by attending to elements of the same agent differently than elements of other agents. Based on AgentFormer, we propose a stochastic multi-agent trajectory prediction model that can attend to features of any agent at any previous timestep when inferring an agent's future position. The latent intent of all agents is also jointly modeled, allowing the stochasticity in one agent's behavior to affect other agents. Our method substantially improves the state of the art on well-established pedestrian and autonomous driving datasets.

Key facts. Cited by 327 papers in the map (data through 7 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

46 of 63 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  10. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  11. 2013Auto-Encoding Variational Bayescited by 3,980
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
34 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  3. 2015Variational Inference with Normalizing Flowscited by 2,161
  4. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  6. 2015Gated Graph Sequence Neural Networkscited by 792
  7. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 687
  8. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  9. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  10. 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
  11. 2018Neural Relational Inference for Interacting Systemscited by 339
  12. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  13. 2018SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraintscited by 296
  14. 2018Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networkscited by 289
  15. 2019CoverNet: Multimodal Behavior Prediction using Trajectory Setscited by 259
  16. 2020It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Predictioncited by 235
  17. 2019Human Motion Trajectory Prediction: A Surveycited by 231
  18. 2019Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networkscited by 228
  19. 2020Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Predictioncited by 218
  20. 2017Social Attention: Modeling Attention in Human Crowdscited by 185
  21. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
  22. 2020Transformer Networks for Trajectory Forecastingcited by 180
  23. 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
  24. 2019Multiple Futures Predictioncited by 178
  25. 2019Multi-Agent Tensor Fusion for Contextual Trajectory Predictioncited by 131
  26. 2019SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Predictioncited by 111
  27. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  28. 2019Diverse Trajectory Forecasting with Determinantal Point Processescited by 80
  29. 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
  30. 2020Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesiscited by 63
  31. 2020End-to-end Contextual Perception and Prediction with Interaction Transformercited by 54
  32. 2017The Microsoft 2017 Conversational Speech Recognition Systemcited by 51
  33. 2019Generative Hybrid Representations for Activity Forecasting with No-Regret Learningcited by 18
  34. 2020PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Samplingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 52; 2023: 75; 2024: 79; 2025: 62; 2026: 50.

327 papers.

Sort:

By area:

  1. 2022Planning-oriented Autonomous Drivingcited by 1,068
  2. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  3. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  4. 2021Scene Transformer: A unified architecture for predicting multiple agent trajectoriescited by 165
  5. 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
  6. 2023GameFormer: Game-theoretic Modeling and Learning of Transformer-based Interactive Prediction and Planning for Autonomous Drivingcited by 141
  7. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 137
  8. 2021Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Priorcited by 136
  9. 2023Leapfrog Diffusion Model for Stochastic Trajectory Predictioncited by 129
  10. 2022GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction with Relational Reasoningcited by 128
  11. 2023Language-Guided Traffic Simulation via Scene-Level Diffusioncited by 110
  12. 2021THOMAS: Trajectory Heatmap Output with learned Multi-Agent Samplingcited by 101
38 more
  1. 2023EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoningcited by 97
  2. 2022On Adversarial Robustness of Trajectory Prediction for Autonomous Vehiclescited by 89
  3. 2022Remember Intentions: Retrospective-Memory-based Trajectory Predictioncited by 86
  4. 2025ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Drivingcited by 66
  5. 2022SocialVAE: Human Trajectory Prediction using Timewise Latentscited by 65
  6. 2022ScePT: Scene-consistent, Policy-based Trajectory Predictions for Planningcited by 63
  7. 2022MUSE-VAE: Multi-Scale VAE for Environment-Aware Long Term Trajectory Predictioncited by 59
  8. 2022Motion Inspired Unsupervised Perception and Prediction in Autonomous Drivingcited by 59
  9. 2023Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Trackingcited by 54
  10. 2022Adaptive Trajectory Prediction via Transferable GNNcited by 51
  11. 2023LAformer: Trajectory Prediction for Autonomous Driving with Lane-Aware Scene Constraintscited by 50
  12. 2022AdvDO: Realistic Adversarial Attacks for Trajectory Predictioncited by 48
  13. 2023Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Predictioncited by 47
  14. 2023Leveraging Future Relationship Reasoning for Vehicle Trajectory Predictioncited by 47
  15. 2022Non-Probability Sampling Network for Stochastic Human Trajectory Predictioncited by 46
  16. 2023DTPP: Differentiable Joint Conditional Prediction and Cost Evaluation for Tree Policy Planning in Autonomous Drivingcited by 46
  17. 2023Scenario Diffusion: Controllable Driving Scenario Generation With Diffusioncited by 46
  18. 2021View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrumscited by 44
  19. 2022Learning Pedestrian Group Representations for Multi-modal Trajectory Predictioncited by 44
  20. 2023Traj-MAE: Masked Autoencoders for Trajectory Predictioncited by 42
  21. 2023EigenTrajectory: Low-Rank Descriptors for Multi-Modal Trajectory Forecastingcited by 41
  22. 2022Social-Implicit: Rethinking Trajectory Prediction Evaluation and The Effectiveness of Implicit Maximum Likelihood Estimationcited by 40
  23. 2024HPNet: Dynamic Trajectory Forecasting with Historical Prediction Attentioncited by 37
  24. 2022Graph-based Spatial Transformer with Memory Replay for Multi-future Pedestrian Trajectory Predictioncited by 36
  25. 2022How many Observations are Enough? Knowledge Distillation for Trajectory Forecastingcited by 35
  26. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  27. 2024SingularTrajectory: Universal Trajectory Predictor Using Diffusion Modelcited by 35
  28. 2023Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecastingcited by 33
  29. 2023Social-Transmotion: Promptable Human Trajectory Predictioncited by 33
  30. 2024Producing and Leveraging Online Map Uncertainty in Trajectory Predictioncited by 32
  31. 2023FEND: A Future Enhanced Distribution-Aware Contrastive Learning Framework for Long-tail Trajectory Predictioncited by 31
  32. 2024Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Predictioncited by 31
  33. 2025MoFlow: One-Step Flow Matching for Human Trajectory Forecasting via Implicit Maximum Likelihood Estimation based Distillationcited by 31
  34. 2023The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Reviewcited by 29
  35. 2022Goal-driven Self-Attentive Recurrent Networks for Trajectory Predictioncited by 28
  36. 2022Dynamic-Group-Aware Networks for Multi-Agent Trajectory Prediction with Relational Reasoningcited by 28
  37. 2023Fast Inference and Update of Probabilistic Density Estimation on Trajectory Predictioncited by 27
  38. 2023SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Predictioncited by 27

50 most cited of 327 All 327 (JSON)