# Scene-LSTM: A Model for Human Trajectory Prediction

arXiv: https://arxiv.org/abs/1808.04018 · HTML: https://mlsanity.com/paper/1808.04018/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2018.

## Built on by (most cited first, 25 of 25)

- [On Exposing the Challenging Long Tail in Future Prediction of Traffic Actors](https://mlsanity.com/paper/2103.12474/) (2021)
- [CANet: A Context-Aware Network for Shadow Removal](https://mlsanity.com/paper/2108.09894/) (2021)
- [View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums](https://mlsanity.com/paper/2110.07288/) (2021)
- [EigenTrajectory: Low-Rank Descriptors for Multi-Modal Trajectory Forecasting](https://mlsanity.com/paper/2307.09306/) (2023)
- [SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model](https://mlsanity.com/paper/2403.18452/) (2024)
- [Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction](https://mlsanity.com/paper/2112.15012/) (2021)
- [Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction](https://mlsanity.com/paper/2403.18447/) (2024)
- [Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-based Approaches](https://mlsanity.com/paper/2111.06740/) (2021)
- [MSN: Multi-Style Network for Trajectory Prediction](https://mlsanity.com/paper/2107.00932/) (2021)
- [SSAGCN: Social Soft Attention Graph Convolution Network for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2112.02459/) (2021)
- [Unlimited Neighborhood Interaction for Heterogeneous Trajectory Prediction](https://mlsanity.com/paper/2108.00238/) (2021)
- [Another Vertical View: A Hierarchical Network for Heterogeneous Trajectory Prediction via Spectrums](https://mlsanity.com/paper/2304.05106/) (2023)
- [Uncertainty estimation of pedestrian future trajectory using Bayesian approximation](https://mlsanity.com/paper/2205.01887/) (2022)
- [Predicting Vehicles Trajectories in Urban Scenarios with Transformer Networks and Augmented Information](https://mlsanity.com/paper/2106.00559/) (2021)
- [Learning the Pedestrian-Vehicle Interaction for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2202.05334/) (2022)
- [Continuous Locomotive Crowd Behavior Generation](https://mlsanity.com/paper/2504.04756/) (2025)
- [Adaptive Human Trajectory Prediction via Latent Corridors](https://mlsanity.com/paper/2312.06653/) (2023)
- [Vulnerable Road User Detection and Safety Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2405.19202/) (2024)
- [TrajPRed: Trajectory Prediction with Region-based Relation Learning](https://mlsanity.com/paper/2404.06971/) (2024)
- [Visual-information-driven model for crowd simulation using temporal convolutional network](https://mlsanity.com/paper/2311.02996/) (2023)
- [Neural Interaction Energy for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2404.16579/) (2024)
- [Reverberation: Learning the Latencies Before Forecasting Trajectories](https://mlsanity.com/paper/2511.11164/) (2025)
- [EMT: A Visual Multi-Task Benchmark Dataset for Autonomous Driving](https://mlsanity.com/paper/2502.19260/) (2025)
- [KoopCast: Trajectory Forecasting via Koopman Operators](https://mlsanity.com/paper/2509.15513/) (2025)
- [EnvSocial-Diff: A Diffusion-Based Crowd Simulation Model with Environmental Conditioning and Individual-Group Interaction](https://mlsanity.com/paper/2603.23874/) (2026)

## Abstract (arXiv)

We develop a human movement trajectory prediction system that incorporates the scene information (Scene-LSTM) as well as human movement trajectories (Pedestrian movement LSTM) in the prediction process within static crowded scenes. We superimpose a two-level grid structure (scene is divided into grid cells each modeled by a scene-LSTM, which are further divided into smaller sub-grids for finer spatial granularity) and explore common human trajectories occurring in the grid cell (e.g., making a right or left turn onto sidewalks coming out of an alley; or standing still at bus/train stops). Two coupled LSTM networks, Pedestrian movement LSTMs (one per target) and the corresponding Scene-LSTMs (one per grid-cell) are trained simultaneously to predict the next movements. We show that such common path information greatly influences prediction of future movement. We further design a scene data filter that holds important non-linear movement information. The scene data filter allows us to select the relevant parts of the information from the grid cell's memory relative to a target's state. We evaluate and compare two versions of our method with the Linear and several existing LSTM-based methods on five crowded video sequences from the UCY [1] and ETH [2] datasets. The results show that our method reduces the location displacement errors compared to related methods and specifically about 80% reduction compared to social interaction methods.