# A Pedestrian Detection and Tracking Framework for Autonomous Cars: Efficient Fusion of Camera and LiDAR Data

arXiv: https://arxiv.org/abs/2108.12375 · HTML: https://mlsanity.com/paper/2108.12375/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 39 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 12 of 12)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [Fast Online Object Tracking and Segmentation: A Unifying Approach](https://mlsanity.com/paper/1812.05050/) (2018)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [Online Multi-Target Tracking Using Recurrent Neural Networks](https://mlsanity.com/paper/1604.03635/) (2016)
- [End-to-end Learning of Multi-sensor 3D Tracking by Detection](https://mlsanity.com/paper/1806.11534/) (2018)
- [Spatially Supervised Recurrent Convolutional Neural Networks for Visual Object Tracking](https://mlsanity.com/paper/1607.05781/) (2016)
- [Combining LiDAR Space Clustering and Convolutional Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1710.06160/) (2017)

## Abstract (arXiv)

This paper presents a novel method for pedestrian detection and tracking by fusing camera and LiDAR sensor data. To deal with the challenges associated with the autonomous driving scenarios, an integrated tracking and detection framework is proposed. The detection phase is performed by converting LiDAR streams to computationally tractable depth images, and then, a deep neural network is developed to identify pedestrian candidates both in RGB and depth images. To provide accurate information, the detection phase is further enhanced by fusing multi-modal sensor information using the Kalman filter. The tracking phase is a combination of the Kalman filter prediction and an optical flow algorithm to track multiple pedestrians in a scene. We evaluate our framework on a real public driving dataset. Experimental results demonstrate that the proposed method achieves significant performance improvement over a baseline method that solely uses image-based pedestrian detection.