# Remember Intentions: Retrospective-Memory-based Trajectory Prediction

arXiv: https://arxiv.org/abs/2203.11474 · HTML: https://mlsanity.com/paper/2203.11474/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 26 of 26)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Social Attention: Modeling Attention in Human Crowds](https://mlsanity.com/paper/1710.04689/) (2017)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [Ask Me Anything: Dynamic Memory Networks for Natural Language Processing](https://mlsanity.com/paper/1506.07285/) (2015)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction](https://mlsanity.com/paper/2006.03340/) (2020)
- [Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data](https://mlsanity.com/paper/2001.03093/) (2020)
- [Collaborative Motion Prediction via Neural Motion Message Passing](https://mlsanity.com/paper/2003.06594/) (2020)
- [Conditional Flow Variational Autoencoders for Structured Sequence Prediction](https://mlsanity.com/paper/1908.09008/) (2019)
- [RAIN: Reinforced Hybrid Attention Inference Network for Motion Forecasting](https://mlsanity.com/paper/2108.01316/) (2021)
- [Visual Question Answering with Memory-Augmented Networks](https://mlsanity.com/paper/1707.04968/) (2017)
- [Online Multi-Agent Forecasting with Interpretable Collaborative Graph Neural Network](https://mlsanity.com/paper/2107.00894/) (2021)
- [Collaborative Uncertainty in Multi-Agent Trajectory Forecasting](https://mlsanity.com/paper/2110.13947/) (2021)

## Built on by (most cited first, 30 of 86)

- [Leapfrog Diffusion Model for Stochastic Trajectory Prediction](https://mlsanity.com/paper/2303.10895/) (2023)
- [EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning](https://mlsanity.com/paper/2303.10876/) (2023)
- [Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction](https://mlsanity.com/paper/2303.16005/) (2023)
- [EigenTrajectory: Low-Rank Descriptors for Multi-Modal Trajectory Forecasting](https://mlsanity.com/paper/2307.09306/) (2023)
- [SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model](https://mlsanity.com/paper/2403.18452/) (2024)
- [Social-Transmotion: Promptable Human Trajectory Prediction](https://mlsanity.com/paper/2312.16168/) (2023)
- [FEND: A Future Enhanced Distribution-Aware Contrastive Learning Framework for Long-tail Trajectory Prediction](https://mlsanity.com/paper/2303.16574/) (2023)
- [Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction](https://mlsanity.com/paper/2403.18447/) (2024)
- [MoFlow: One-Step Flow Matching for Human Trajectory Forecasting via Implicit Maximum Likelihood Estimation based Distillation](https://mlsanity.com/paper/2503.09950/) (2025)
- [Behavioral Intention Prediction in Driving Scenes: A Survey](https://mlsanity.com/paper/2211.00385/) (2022)
- [Multi-agent Long-term 3D Human Pose Forecasting via Interaction-aware Trajectory Conditioning](https://mlsanity.com/paper/2404.05218/) (2024)
- [Dynamic-Group-Aware Networks for Multi-Agent Trajectory Prediction with Relational Reasoning](https://mlsanity.com/paper/2206.13114/) (2022)
- [SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2310.05370/) (2023)
- [SMEMO: Social Memory for Trajectory Forecasting](https://mlsanity.com/paper/2203.12446/) (2022)
- [GATraj: A Graph- and Attention-based Multi-Agent Trajectory Prediction Model](https://mlsanity.com/paper/2209.07857/) (2022)
- [Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning](https://mlsanity.com/paper/2209.11820/) (2022)
- [Adapting to Length Shift: FlexiLength Network for Trajectory Prediction](https://mlsanity.com/paper/2404.00742/) (2024)
- [MART: MultiscAle Relational Transformer Networks for Multi-agent Trajectory Prediction](https://mlsanity.com/paper/2407.21635/) (2024)
- [Sports-Traj: A Unified Trajectory Generation Model for Multi-Agent Movement in Sports](https://mlsanity.com/paper/2405.17680/) (2024)
- [DICE: Diverse Diffusion Model with Scoring for Trajectory Prediction](https://mlsanity.com/paper/2310.14570/) (2023)
- [Unsupervised Sampling Promoting for Stochastic Human Trajectory Prediction](https://mlsanity.com/paper/2304.04298/) (2023)
- [Joint Metrics Matter: A Better Standard for Trajectory Forecasting](https://mlsanity.com/paper/2305.06292/) (2023)
- [Joint-Relation Transformer for Multi-Person Motion Prediction](https://mlsanity.com/paper/2308.04808/) (2023)
- [Progressive Pretext Task Learning for Human Trajectory Prediction](https://mlsanity.com/paper/2407.11588/) (2024)
- [Unified Uncertainty-Aware Diffusion for Multi-Agent Trajectory Modeling](https://mlsanity.com/paper/2503.18589/) (2025)
- [Forecasting Human Trajectory from Scene History](https://mlsanity.com/paper/2210.08732/) (2022)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Collaborative Uncertainty Benefits Multi-Agent Multi-Modal Trajectory Forecasting](https://mlsanity.com/paper/2207.05195/) (2022)
- [Uncertainty-Aware Pedestrian Trajectory Prediction via Distributional Diffusion](https://mlsanity.com/paper/2303.08367/) (2023)
- [Unified Spatial-Temporal Edge-Enhanced Graph Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2502.02504/) (2025)

## Abstract (arXiv)

To realize trajectory prediction, most previous methods adopt the parameter-based approach, which encodes all the seen past-future instance pairs into model parameters. However, in this way, the model parameters come from all seen instances, which means a huge amount of irrelevant seen instances might also involve in predicting the current situation, disturbing the performance. To provide a more explicit link between the current situation and the seen instances, we imitate the mechanism of retrospective memory in neuropsychology and propose MemoNet, an instance-based approach that predicts the movement intentions of agents by looking for similar scenarios in the training data. In MemoNet, we design a pair of memory banks to explicitly store representative instances in the training set, acting as prefrontal cortex in the neural system, and a trainable memory addresser to adaptively search a current situation with similar instances in the memory bank, acting like basal ganglia. During prediction, MemoNet recalls previous memory by using the memory addresser to index related instances in the memory bank. We further propose a two-step trajectory prediction system, where the first step is to leverage MemoNet to predict the destination and the second step is to fulfill the whole trajectory according to the predicted destinations. Experiments show that the proposed MemoNet improves the FDE by 20.3%/10.2%/28.3% from the previous best method on SDD/ETH-UCY/NBA datasets. Experiments also show that our MemoNet has the ability to trace back to specific instances during prediction, promoting more interpretability.