# Interaction Modeling with Multiplex Attention

arXiv: https://arxiv.org/abs/2208.10660 · HTML: https://mlsanity.com/paper/2208.10660/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 51 references that are in the map.
- First posted on arXiv 23 Aug 2022.

## Builds on (most cited first, 29 of 29)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2103.07719/) (2021)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [Conditional Generative Neural System for Probabilistic Trajectory Prediction](https://mlsanity.com/paper/1905.01631/) (2019)
- [Coordinated Multi-Agent Imitation Learning](https://mlsanity.com/paper/1703.03121/) (2017)
- [World Model as a Graph: Learning Latent Landmarks for Planning](https://mlsanity.com/paper/2011.12491/) (2020)
- [Relational Graph Learning for Crowd Navigation](https://mlsanity.com/paper/1909.13165/) (2019)
- [Generating Multi-Agent Trajectories using Programmatic Weak Supervision](https://mlsanity.com/paper/1803.07612/) (2018)
- [Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Tracking](https://mlsanity.com/paper/2102.09117/) (2021)
- [Progressive Learning and Disentanglement of Hierarchical Representations](https://mlsanity.com/paper/2002.10549/) (2020)
- [Generating Long-term Trajectories Using Deep Hierarchical Networks](https://mlsanity.com/paper/1706.07138/) (2017)
- [PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception](https://mlsanity.com/paper/2103.01933/) (2021)
- [Relational Forward Models for Multi-Agent Learning](https://mlsanity.com/paper/1809.11044/) (2018)
- [Multi-Modal Trajectory Prediction of NBA Players](https://mlsanity.com/paper/2008.07870/) (2020)
- [Factorised Neural Relational Inference for Multi-Interaction Systems](https://mlsanity.com/paper/1905.08721/) (2019)
- [Learning World Graphs to Accelerate Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1907.00664/) (2019)
- [Disentangled Representations in Neural Models](https://mlsanity.com/paper/1602.02383/) (2016)

## Built on by (most cited first, 14 of 14)

- [UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving](https://mlsanity.com/paper/2503.24381/) (2025)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints](https://mlsanity.com/paper/2306.01075/) (2023)
- [MATRIX: Multi-Agent Trajectory Generation with Diverse Contexts](https://mlsanity.com/paper/2403.06041/) (2024)
- [EqDrive: Efficient Equivariant Motion Forecasting with Multi-Modality for Autonomous Driving](https://mlsanity.com/paper/2310.17540/) (2023)
- [Disentangled Neural Relational Inference for Interpretable Motion Prediction](https://mlsanity.com/paper/2401.03599/) (2024)
- [Equivariant Map and Agent Geometry for Autonomous Driving Motion Prediction](https://mlsanity.com/paper/2310.13922/) (2023)
- [Towards Generalizable Safety in Crowd Navigation via Conformal Uncertainty Handling](https://mlsanity.com/paper/2508.05634/) (2025)
- [Learning Heterogeneous Interaction Strengths by Trajectory Prediction with Graph Neural Network](https://mlsanity.com/paper/2208.13179/) (2022)
- [Trajectory Imputation in Multi-Agent Sports with Derivative-Accumulating Self-Ensemble](https://mlsanity.com/paper/2408.10878/) (2024)
- [CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition](https://mlsanity.com/paper/2410.07153/) (2024)
- [CCF: Cross Correcting Framework for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2406.00749/) (2024)
- [SSL-Interactions: Pretext Tasks for Interactive Trajectory Prediction](https://mlsanity.com/paper/2401.07729/) (2024)
- [PathCRF: Ball-Free Soccer Event Detection via Possession Path Inference from Player Trajectories](https://mlsanity.com/paper/2602.12080/) (2026)

## Abstract (arXiv)

Modeling multi-agent systems requires understanding how agents interact. Such systems are often difficult to model because they can involve a variety of types of interactions that layer together to drive rich social behavioral dynamics. Here we introduce a method for accurately modeling multi-agent systems. We present Interaction Modeling with Multiplex Attention (IMMA), a forward prediction model that uses a multiplex latent graph to represent multiple independent types of interactions and attention to account for relations of different strengths. We also introduce Progressive Layer Training, a training strategy for this architecture. We show that our approach outperforms state-of-the-art models in trajectory forecasting and relation inference, spanning three multi-agent scenarios: social navigation, cooperative task achievement, and team sports. We further demonstrate that our approach can improve zero-shot generalization and allows us to probe how different interactions impact agent behavior.