# Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction

arXiv: https://arxiv.org/abs/2203.04421 · HTML: https://mlsanity.com/paper/2203.04421/

## Key facts

- Cited by 9 papers in the map (data through 7 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [One Thousand and One Hours: Self-driving Motion Prediction Dataset](https://mlsanity.com/paper/2006.14480/) (2020)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Implicit Latent Variable Model for Scene-Consistent Motion Forecasting](https://mlsanity.com/paper/2007.12036/) (2020)
- [Deep Coordination Graphs](https://mlsanity.com/paper/1910.00091/) (2019)
- [Learning Latent Representations to Influence Multi-Agent Interaction](https://mlsanity.com/paper/2011.06619/) (2020)
- [Multi-Agent Imitation Learning for Driving Simulation](https://mlsanity.com/paper/1803.01044/) (2018)
- [DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Sampling](https://mlsanity.com/paper/1911.12736/) (2019)
- [A Regularized Framework for Sparse and Structured Neural Attention](https://mlsanity.com/paper/1705.07704/) (2017)
- [Sparse and Continuous Attention Mechanisms](https://mlsanity.com/paper/2006.07214/) (2020)
- [Multi-Head Attention with Disagreement Regularization](https://mlsanity.com/paper/1810.10183/) (2018)
- [Forecasting Trajectory and Behavior of Road-Agents Using Spectral Clustering in Graph-LSTMs](https://mlsanity.com/paper/1912.01118/) (2019)
- [When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans](https://mlsanity.com/paper/2001.04377/) (2020)
- [Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.02926/) (2018)
- [On the Critical Role of Conventions in Adaptive Human-AI Collaboration](https://mlsanity.com/paper/2104.02871/) (2021)
- [The Importance of Prior Knowledge in Precise Multimodal Prediction](https://mlsanity.com/paper/2006.02636/) (2020)
- [Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving](https://mlsanity.com/paper/2007.00178/) (2020)
- [Deep Local Trajectory Replanning and Control for Robot Navigation](https://mlsanity.com/paper/1905.05279/) (2019)
- [Multi-Agent Safe Planning with Gaussian Processes](https://mlsanity.com/paper/2008.04452/) (2020)
- [Generic Tracking and Probabilistic Prediction Framework and Its Application in Autonomous Driving](https://mlsanity.com/paper/1908.09031/) (2019)

## Built on by (most cited first, 9 of 9)

- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Joint Metrics Matter: A Better Standard for Trajectory Forecasting](https://mlsanity.com/paper/2305.06292/) (2023)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds](https://mlsanity.com/paper/2405.16439/) (2024)
- [Learning Preferences for Interactive Autonomy](https://mlsanity.com/paper/2210.10899/) (2022)
- [Benchmark for Models Predicting Human Behavior in Gap Acceptance Scenarios](https://mlsanity.com/paper/2211.05455/) (2022)
- [iPLAN: Intent-Aware Planning in Heterogeneous Traffic via Distributed Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.06236/) (2023)
- [Estimating cognitive biases with attention-aware inverse planning](https://mlsanity.com/paper/2510.25951/) (2025)
- [Smooth-Trajectron++: Augmenting the Trajectron++ behaviour prediction model with smooth attention](https://mlsanity.com/paper/2305.19678/) (2023)

## Abstract (arXiv)

Multi-agent interactions are important to model for forecasting other agents' behaviors and trajectories. At a certain time, to forecast a reasonable future trajectory, each agent needs to pay attention to the interactions with only a small group of most relevant agents instead of unnecessarily paying attention to all the other agents. However, existing attention modeling works ignore that human attention in driving does not change rapidly, and may introduce fluctuating attention across time steps. In this paper, we formulate an attention model for multi-agent interactions based on a total variation temporal smoothness prior and propose a trajectory prediction architecture that leverages the knowledge of these attended interactions. We demonstrate how the total variation attention prior along with the new sequence prediction loss terms leads to smoother attention and more sample-efficient learning of multi-agent trajectory prediction, and show its advantages in terms of prediction accuracy by comparing it with the state-of-the-art approaches on both synthetic and naturalistic driving data. We demonstrate the performance of our algorithm for trajectory prediction on the INTERACTION dataset on our website.