# Real-time Full-stack Traffic Scene Perception for Autonomous Driving with Roadside Cameras

arXiv: https://arxiv.org/abs/2206.09770 · HTML: https://mlsanity.com/paper/2206.09770/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in VLMs for autonomous driving, one of 9 directions in End-to-end autonomous driving.
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 20 Jun 2022.

## Placed in

- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 19 of 19)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Tracking Objects as Points](https://mlsanity.com/paper/2004.01177/) (2020)
- [Disentangling Monocular 3D Object Detection](https://mlsanity.com/paper/1905.12365/) (2019)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [Stereo R-CNN based 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1902.09738/) (2019)
- [ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape](https://mlsanity.com/paper/1812.02781/) (2018)
- [Infrastructure-Based Object Detection and Tracking for Cooperative Driving Automation: A Survey](https://mlsanity.com/paper/2201.11871/) (2022)

## Built on by (most cited first, 13 of 13)

- [Diffusion Models for Imperceptible and Transferable Adversarial Attack](https://mlsanity.com/paper/2305.08192/) (2023)
- [In Pursuit of Many: A Review of Modern Multiple Object Tracking Systems](https://mlsanity.com/paper/2209.04796/) (2022)
- [Video Waterdrop Removal via Spatio-Temporal Fusion in Driving Scenes](https://mlsanity.com/paper/2302.05916/) (2023)
- [CAROM Air -- Vehicle Localization and Traffic Scene Reconstruction from Aerial Videos](https://mlsanity.com/paper/2306.00075/) (2023)
- [Robust Roadside Perception: an Automated Data Synthesis Pipeline Minimizing Human Annotation](https://mlsanity.com/paper/2306.17302/) (2023)
- [Evaluating Roadside Perception for Autonomous Vehicles: Insights from Field Testing](https://mlsanity.com/paper/2401.12392/) (2024)
- [Mcity Data Engine: Iterative Model Improvement Through Open-Vocabulary Data Selection](https://mlsanity.com/paper/2504.21614/) (2025)
- [ROCO: A Roundabout Traffic Conflict Dataset](https://mlsanity.com/paper/2303.00563/) (2023)
- [MSight: An Edge-Cloud Infrastructure-based Perception System for Connected Automated Vehicles](https://mlsanity.com/paper/2310.05290/) (2023)
- [Evaluation of Infrastructure-based Warning System on Driving Behaviors-A Roundabout Study](https://mlsanity.com/paper/2312.03891/) (2023)
- [HeightFormer: A Semantic Alignment Monocular 3D Object Detection Method from Roadside Perspective](https://mlsanity.com/paper/2410.07758/) (2024)
- [Label Correction for Road Segmentation Using Road-side Cameras](https://mlsanity.com/paper/2502.01281/) (2025)
- [LoRFT: Benchmarking Long-Range Vehicle Trajectory Reconstruction from Fixed Highway Cameras](https://mlsanity.com/paper/2607.19911/) (2026)

## Abstract (arXiv)

We propose a novel and pragmatic framework for traffic scene perception with roadside cameras. The proposed framework covers a full-stack of roadside perception pipeline for infrastructure-assisted autonomous driving, including object detection, object localization, object tracking, and multi-camera information fusion. Unlike previous vision-based perception frameworks rely upon depth offset or 3D annotation at training, we adopt a modular decoupling design and introduce a landmark-based 3D localization method, where the detection and localization can be well decoupled so that the model can be easily trained based on only 2D annotations. The proposed framework applies to either optical or thermal cameras with pinhole or fish-eye lenses. Our framework is deployed at a two-lane roundabout located at Ellsworth Rd. and State St., Ann Arbor, MI, USA, providing 7x24 real-time traffic flow monitoring and high-precision vehicle trajectory extraction. The whole system runs efficiently on a low-power edge computing device with all-component end-to-end delay of less than 20ms.