# "If you could see me through my eyes": Predicting Pedestrian Perception

arXiv: https://arxiv.org/abs/2202.13981 · HTML: https://mlsanity.com/paper/2202.13981/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 51 references that are in the map.
- First posted on arXiv 28 Feb 2022.

## Builds on (most cited first, 20 of 20)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [Autonomous Vehicles that Interact with Pedestrians: A Survey of Theory and Practice](https://mlsanity.com/paper/1805.11773/) (2018)
- [Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approach](https://mlsanity.com/paper/2008.03880/) (2020)
- [Joint Attention in Autonomous Driving (JAAD)](https://mlsanity.com/paper/1609.04741/) (2016)
- [ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis](https://mlsanity.com/paper/2011.10185/) (2020)
- [Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision](https://mlsanity.com/paper/1911.01138/) (2019)
- [Multimodal Future Localization and Emergence Prediction for Objects in Egocentric View with a Reachability Prior](https://mlsanity.com/paper/2006.04700/) (2020)
- [Predicting Motion of Vulnerable Road Users using High-Definition Maps and Efficient ConvNets](https://mlsanity.com/paper/1906.08469/) (2019)
- [3DOF Pedestrian Trajectory Prediction Learned from Long-Term Autonomous Mobile Robot Deployment Data](https://mlsanity.com/paper/1710.00126/) (2017)
- [Unsupervised Multiple Person Tracking using AutoEncoder-Based Lifted Multicuts](https://mlsanity.com/paper/2002.01192/) (2020)
- [Vulnerable road user detection: state-of-the-art and open challenges](https://mlsanity.com/paper/1902.03601/) (2019)
- [Dropout's Dream Land: Generalization from Learned Simulators to Reality](https://mlsanity.com/paper/2109.08342/) (2021)

## Built on by (most cited first, 1 of 1)

- [On the Road to Clarity: Exploring Explainable AI for World Models in a Driver Assistance System](https://mlsanity.com/paper/2404.17350/) (2024)

## Abstract (arXiv)

Pedestrians are particularly vulnerable road users in urban traffic. With the arrival of autonomous driving, novel technologies can be developed specifically to protect pedestrians. We propose a machine learning toolchain to train artificial neural networks as models of pedestrian behavior. In a preliminary study, we use synthetic data from simulations of a specific pedestrian crossing scenario to train a variational autoencoder and a long short-term memory network to predict a pedestrian's future visual perception. We can accurately predict a pedestrian's future perceptions within relevant time horizons. By iteratively feeding these predicted frames into these networks, they can be used as simulations of pedestrians as indicated by our results. Such trained networks can later be used to predict pedestrian behaviors even from the perspective of the autonomous car. Another future extension will be to re-train these networks with real-world video data.