# Pedestrian Environment Model for Automated Driving

arXiv: https://arxiv.org/abs/2308.09080 · HTML: https://mlsanity.com/paper/2308.09080/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 17 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Multiple Object Tracking: A Literature Review](https://mlsanity.com/paper/1409.7618/) (2014)
- [EagerMOT: 3D Multi-Object Tracking via Sensor Fusion](https://mlsanity.com/paper/2104.14682/) (2021)
- [MotionMixer: MLP-based 3D Human Body Pose Forecasting](https://mlsanity.com/paper/2207.00499/) (2022)
- [Traffic Control Gesture Recognition for Autonomous Vehicles](https://mlsanity.com/paper/2007.16072/) (2020)
- [PillarFlow: End-to-end Birds-eye-view Flow Estimation for Autonomous Driving](https://mlsanity.com/paper/2008.01179/) (2020)
- [Dynamic Occupancy Grid Mapping with Recurrent Neural Networks](https://mlsanity.com/paper/2011.08659/) (2020)
- [Pedestrian Behavior Prediction for Automated Driving: Requirements, Metrics, and Relevant Features](https://mlsanity.com/paper/2012.08418/) (2020)
- [Neural Enhanced Belief Propagation for Data Association in Multiobject Tracking](https://mlsanity.com/paper/2203.09948/) (2022)
- [A Multi-Task Recurrent Neural Network for End-to-End Dynamic Occupancy Grid Mapping](https://mlsanity.com/paper/2202.04461/) (2022)
- [A Spatio-Temporal Multilayer Perceptron for Gesture Recognition](https://mlsanity.com/paper/2204.11511/) (2022)

## Abstract (arXiv)

Besides interacting correctly with other vehicles, automated vehicles should also be able to react in a safe manner to vulnerable road users like pedestrians or cyclists. For a safe interaction between pedestrians and automated vehicles, the vehicle must be able to interpret the pedestrian's behavior. Common environment models do not contain information like body poses used to understand the pedestrian's intent. In this work, we propose an environment model that includes the position of the pedestrians as well as their pose information. We only use images from a monocular camera and the vehicle's localization data as input to our pedestrian environment model. We extract the skeletal information with a neural network human pose estimator from the image. Furthermore, we track the skeletons with a simple tracking algorithm based on the Hungarian algorithm and an ego-motion compensation. To obtain the 3D information of the position, we aggregate the data from consecutive frames in conjunction with the vehicle position. We demonstrate our pedestrian environment model on data generated with the CARLA simulator and the nuScenes dataset. Overall, we reach a relative position error of around 16% on both datasets.