# SMEMO: Social Memory for Trajectory Forecasting

arXiv: https://arxiv.org/abs/2203.12446 · HTML: https://mlsanity.com/paper/2203.12446/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 80 references that are in the map.
- First posted on arXiv 23 Mar 2022.

## Builds on (most cited first, 30 of 50)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Convolutional Social Pooling for Vehicle Trajectory Prediction](https://mlsanity.com/paper/1805.06771/) (2018)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [Multiple Futures Prediction](https://mlsanity.com/paper/1911.00997/) (2019)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [Memory-augmented Dense Predictive Coding for Video Representation Learning](https://mlsanity.com/paper/2008.01065/) (2020)
- [Multi-Agent Tensor Fusion for Contextual Trajectory Prediction](https://mlsanity.com/paper/1904.04776/) (2019)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Ask Me Anything: Dynamic Memory Networks for Natural Language Processing](https://mlsanity.com/paper/1506.07285/) (2015)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents](https://mlsanity.com/paper/1811.02146/) (2018)
- [SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction](https://mlsanity.com/paper/1903.02793/) (2019)
- [MAST: A Memory-Augmented Self-supervised Tracker](https://mlsanity.com/paper/2002.07793/) (2020)
- [SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2104.01528/) (2021)
- [Remember Intentions: Retrospective-Memory-based Trajectory Prediction](https://mlsanity.com/paper/2203.11474/) (2022)

## Built on by (most cited first, 26 of 26)

- [EigenTrajectory: Low-Rank Descriptors for Multi-Modal Trajectory Forecasting](https://mlsanity.com/paper/2307.09306/) (2023)
- [SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model](https://mlsanity.com/paper/2403.18452/) (2024)
- [Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction](https://mlsanity.com/paper/2403.18447/) (2024)
- [LG-Traj: LLM Guided Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2403.08032/) (2024)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Unified Spatial-Temporal Edge-Enhanced Graph Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2502.02504/) (2025)
- [BP-SGCN: Behavioral Pseudo-Label Informed Sparse Graph Convolution Network for Pedestrian and Heterogeneous Trajectory Prediction](https://mlsanity.com/paper/2502.14676/) (2025)
- [Continuous Locomotive Crowd Behavior Generation](https://mlsanity.com/paper/2504.04756/) (2025)
- [Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds](https://mlsanity.com/paper/2405.16439/) (2024)
- [Resonance: Learning to Predict Social-Aware Pedestrian Trajectories as Co-Vibrations](https://mlsanity.com/paper/2412.02447/) (2024)
- [Fashion Recommendation Based on Style and Social Events](https://mlsanity.com/paper/2208.00725/) (2022)
- [G2LTraj: A Global-to-Local Generation Approach for Trajectory Prediction](https://mlsanity.com/paper/2404.19330/) (2024)
- [FLODCAST: Flow and Depth Forecasting via Multimodal Recurrent Architectures](https://mlsanity.com/paper/2310.20593/) (2023)
- [Recent Advances in Multi-Agent Human Trajectory Prediction: A Comprehensive Review](https://mlsanity.com/paper/2506.14831/) (2025)
- [Intention-Aware Diffusion Model for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2508.07146/) (2025)
- [PMM-Net: Single-stage Multi-agent Trajectory Prediction with Patching-based Embedding and Explicit Modal Modulation](https://mlsanity.com/paper/2410.19544/) (2024)
- [Offline Reinforcement Learning using Human-Aligned Reward Labeling for Autonomous Emergency Braking in Occluded Pedestrian Crossing](https://mlsanity.com/paper/2504.08704/) (2025)
- [Reverberation: Learning the Latencies Before Forecasting Trajectories](https://mlsanity.com/paper/2511.11164/) (2025)
- [Diagnosing Capability Preservation and Task Sensitivity in Memory Augmented Document Classifiers](https://mlsanity.com/paper/2512.06582/) (2025)
- [SAIL: Scene-aware Adaptive Iterative Learning for Long-Tail Trajectory Prediction in Autonomous Vehicles](https://mlsanity.com/paper/2604.04573/) (2026)
- [Encore: Conditioning Trajectory Forecasting via Biased Ego Rehearsals](https://mlsanity.com/paper/2605.11463/) (2026)
- [SWIFT: A Small-World Interaction Framework for Flow-Aware Trajectory Prediction in Autonomous Driving](https://mlsanity.com/paper/2607.09741/) (2026)
- [TSCA-Net: Temporal-Spatial Clique Attention for Interpretable Multimodal Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2607.11939/) (2026)
- [A Unified Framework for Trajectory Prediction with Explicit Planning and Reaction Decomposition](https://mlsanity.com/paper/2608.05673/) (2026)
- [GEAR: From Dynamic Encoding to Dynamic Activation in Social Trajectory Prediction](https://mlsanity.com/paper/2609.13778/) (2026)
- [HAP: A Hand-Driven Active Perception Framework for Egocentric Head Motion Prediction](https://mlsanity.com/paper/2609.18548/) (2026)

## Abstract (arXiv)

Effective modeling of human interactions is of utmost importance when forecasting behaviors such as future trajectories. Each individual, with its motion, influences surrounding agents since everyone obeys to social non-written rules such as collision avoidance or group following. In this paper we model such interactions, which constantly evolve through time, by looking at the problem from an algorithmic point of view, i.e. as a data manipulation task. We present a neural network based on an end-to-end trainable working memory, which acts as an external storage where information about each agent can be continuously written, updated and recalled. We show that our method is capable of learning explainable cause-effect relationships between motions of different agents, obtaining state-of-the-art results on multiple trajectory forecasting datasets.