# Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion

arXiv: https://arxiv.org/abs/2111.00993 · HTML: https://mlsanity.com/paper/2111.00993/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 1 Nov 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks](https://mlsanity.com/paper/1907.03395/) (2019)
- [Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2005.08514/) (2020)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [Motion Planning Among Dynamic, Decision-Making Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1805.01956/) (2018)
- [TraPHic: Trajectory Prediction in Dense and Heterogeneous Traffic Using Weighted Interactions](https://mlsanity.com/paper/1812.04767/) (2018)
- [Pose Flow: Efficient Online Pose Tracking](https://mlsanity.com/paper/1802.00977/) (2018)
- [The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction](https://mlsanity.com/paper/1912.06445/) (2019)
- [Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction](https://mlsanity.com/paper/2002.08945/) (2020)
- [Unsupervised Traffic Accident Detection in First-Person Videos](https://mlsanity.com/paper/1903.00618/) (2019)
- [Long-Term On-Board Prediction of People in Traffic Scenes under Uncertainty](https://mlsanity.com/paper/1711.09026/) (2017)
- [Future Person Localization in First-Person Videos](https://mlsanity.com/paper/1711.11217/) (2017)
- [Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision](https://mlsanity.com/paper/1911.01138/) (2019)
- [Multimodal Future Localization and Emergence Prediction for Objects in Egocentric View with a Reachability Prior](https://mlsanity.com/paper/2006.04700/) (2020)
- [Personalized Dynamics Models for Adaptive Assistive Navigation Systems](https://mlsanity.com/paper/1804.04118/) (2018)
- [Multiple Object Forecasting: Predicting Future Object Locations in Diverse Environments](https://mlsanity.com/paper/1909.11944/) (2019)
- [Egocentric Basketball Motion Planning from a Single First-Person Image](https://mlsanity.com/paper/1803.01413/) (2018)
- [Forecasting Pedestrian Trajectory with Machine-Annotated Training Data](https://mlsanity.com/paper/1905.03681/) (2019)
- [Indoor Future Person Localization from an Egocentric Wearable Camera](https://mlsanity.com/paper/2103.04019/) (2021)

## Built on by (most cited first, 17 of 17)

- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting](https://mlsanity.com/paper/2307.08243/) (2023)
- [EgoCast: Forecasting Egocentric Human Pose in the Wild](https://mlsanity.com/paper/2412.02903/) (2024)
- [Survey on video anomaly detection in dynamic scenes with moving cameras](https://mlsanity.com/paper/2308.07050/) (2023)
- [EgoNav: Egocentric Scene-aware Human Trajectory Prediction](https://mlsanity.com/paper/2403.19026/) (2024)
- [LocATe: End-to-end Localization of Actions in 3D with Transformers](https://mlsanity.com/paper/2203.10719/) (2022)
- [T2FPV: Dataset and Method for Correcting First-Person View Errors in Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2209.11294/) (2022)
- [Aria-NeRF: Multimodal Egocentric View Synthesis](https://mlsanity.com/paper/2311.06455/) (2023)
- [EgoCogNav: Cognition-aware Human Egocentric Navigation](https://mlsanity.com/paper/2511.17581/) (2025)
- [EgoSpeed-Net: Forecasting Speed-Control in Driver Behavior from Egocentric Video Data](https://mlsanity.com/paper/2209.13459/) (2022)
- [SLAM for Visually Impaired People: a Survey](https://mlsanity.com/paper/2212.04745/) (2022)
- [SceneAware: Scene-Constrained Pedestrian Trajectory Prediction with LLM-Guided Walkability](https://mlsanity.com/paper/2506.14144/) (2025)
- [EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction](https://mlsanity.com/paper/2605.19004/) (2026)
- [Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models](https://mlsanity.com/paper/2606.09142/) (2026)
- [Simultaneous Localization and Affordance Prediction of Tasks from Egocentric Video](https://mlsanity.com/paper/2407.13856/) (2024)
- [Learning Humanoid Navigation from Human Data](https://mlsanity.com/paper/2604.00416/) (2026)
- [An Attention-based Model for Robust Forecasting with Missing Modality](https://mlsanity.com/paper/2606.13970/) (2026)

## Abstract (arXiv)

In this paper, we address the problem of forecasting the trajectory of an egocentric camera wearer (ego-person) in crowded spaces. The trajectory forecasting ability learned from the data of different camera wearers walking around in the real world can be transferred to assist visually impaired people in navigation, as well as to instill human navigation behaviours in mobile robots, enabling better human-robot interactions. To this end, a novel egocentric human trajectory forecasting dataset was constructed, containing real trajectories of people navigating in crowded spaces wearing a camera, as well as extracted rich contextual data. We extract and utilize three different modalities to forecast the trajectory of the camera wearer, i.e., his/her past trajectory, the past trajectories of nearby people, and the environment such as the scene semantics or the depth of the scene. A Transformer-based encoder-decoder neural network model, integrated with a novel cascaded cross-attention mechanism that fuses multiple modalities, has been designed to predict the future trajectory of the camera wearer. Extensive experiments have been conducted, with results showing that our model outperforms the state-of-the-art methods in egocentric human trajectory forecasting.