# InCrowdFormer: On-Ground Pedestrian World Model From Egocentric Views

arXiv: https://arxiv.org/abs/2303.09534 · HTML: https://mlsanity.com/paper/2303.09534/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [Cross-view Transformers for real-time Map-view Semantic Segmentation](https://mlsanity.com/paper/2205.02833/) (2022)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [TransDreamer: Reinforcement Learning with Transformer World Models](https://mlsanity.com/paper/2202.09481/) (2022)
- [DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes](https://mlsanity.com/paper/1806.05620/) (2018)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [Improving Generative Imagination in Object-Centric World Models](https://mlsanity.com/paper/2010.02054/) (2020)
- [Relational Graph Learning for Crowd Navigation](https://mlsanity.com/paper/1909.13165/) (2019)
- [Body Meshes as Points](https://mlsanity.com/paper/2105.02467/) (2021)
- [MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation](https://mlsanity.com/paper/1906.06059/) (2019)
- [Robot Navigation in Crowds by Graph Convolutional Networks with Attention Learned from Human Gaze](https://mlsanity.com/paper/1909.10400/) (2019)
- [NavDreams: Towards Camera-Only RL Navigation Among Humans](https://mlsanity.com/paper/2203.12299/) (2022)
- [View Birdification in the Crowd: Ground-Plane Localization from Perceived Movements](https://mlsanity.com/paper/2111.05060/) (2021)

## Built on by (most cited first, 1 of 1)

- [Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision](https://mlsanity.com/paper/2506.06253/) (2025)

## Abstract (arXiv)

We introduce an on-ground Pedestrian World Model, a computational model that can predict how pedestrians move around an observer in the crowd on the ground plane, but from just the egocentric-views of the observer. Our model, InCrowdFormer, fully leverages the Transformer architecture by modeling pedestrian interaction and egocentric to top-down view transformation with attention, and autoregressively predicts on-ground positions of a variable number of people with an encoder-decoder architecture. We encode the uncertainties arising from unknown pedestrian heights with latent codes to predict the posterior distributions of pedestrian positions. We validate the effectiveness of InCrowdFormer on a novel prediction benchmark of real movements. The results show that InCrowdFormer accurately predicts the future coordination of pedestrians. To the best of our knowledge, InCrowdFormer is the first-of-its-kind pedestrian world model which we believe will benefit a wide range of egocentric-view applications including crowd navigation, tracking, and synthesis.