# Robust Roadside Perception: an Automated Data Synthesis Pipeline Minimizing Human Annotation

arXiv: https://arxiv.org/abs/2306.17302 · HTML: https://mlsanity.com/paper/2306.17302/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 58 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection](https://mlsanity.com/paper/2204.05575/) (2022)
- [Playing for Benchmarks](https://mlsanity.com/paper/1709.07322/) (2017)
- [GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving](https://mlsanity.com/paper/2101.06543/) (2021)
- [SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving](https://mlsanity.com/paper/2005.03844/) (2020)
- [TRACER: Extreme Attention Guided Salient Object Tracing Network](https://mlsanity.com/paper/2112.07380/) (2021)
- [A Vision-based System for Traffic Anomaly Detection using Deep Learning and Decision Trees](https://mlsanity.com/paper/2104.06856/) (2021)
- [Adversarial Image Composition with Auxiliary Illumination](https://mlsanity.com/paper/2009.08255/) (2020)
- [BAAI-VANJEE Roadside Dataset: Towards the Connected Automated Vehicle Highway technologies in Challenging Environments of China](https://mlsanity.com/paper/2105.14370/) (2021)
- [Mixing Real and Synthetic Data to Enhance Neural Network Training -- A Review of Current Approaches](https://mlsanity.com/paper/2007.08781/) (2020)
- [Real-time Full-stack Traffic Scene Perception for Autonomous Driving with Roadside Cameras](https://mlsanity.com/paper/2206.09770/) (2022)

## Built on by (most cited first, 6 of 6)

- [Vehicle-to-Everything Cooperative Perception for Autonomous Driving](https://mlsanity.com/paper/2310.03525/) (2023)
- [Evaluating Roadside Perception for Autonomous Vehicles: Insights from Field Testing](https://mlsanity.com/paper/2401.12392/) (2024)
- [FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units](https://mlsanity.com/paper/2401.12862/) (2024)
- [InScope: A New Real-world 3D Infrastructure-side Collaborative Perception Dataset for Open Traffic Scenarios](https://mlsanity.com/paper/2407.21581/) (2024)
- [MSight: An Edge-Cloud Infrastructure-based Perception System for Connected Automated Vehicles](https://mlsanity.com/paper/2310.05290/) (2023)
- [Joint Task Offloading, Inference Optimization and UAV Trajectory Planning for Generative AI Empowered Intelligent Transportation Digital Twin](https://mlsanity.com/paper/2604.07687/) (2026)

## Abstract (arXiv)

Recently, advancements in vehicle-to-infrastructure communication technologies have elevated the significance of infrastructure-based roadside perception systems for cooperative driving. This paper delves into one of its most pivotal challenges: data insufficiency. The lacking of high-quality labeled roadside sensor data with high diversity leads to low robustness, and low transfer-ability of current roadside perception systems. In this paper, a novel solution is proposed to address this problem that creates synthesized training data using Augmented Reality. A Generative Adversarial Network is then applied to enhance the reality further, that produces a photo-realistic synthesized dataset that is capable of training or fine-tuning a roadside perception detector which is robust to different weather and lighting conditions. Our approach was rigorously tested at two key intersections in Michigan, USA: the Mcity intersection and the State St./Ellsworth Rd roundabout. The Mcity intersection is located within the Mcity test field, a controlled testing environment. In contrast, the State St./Ellsworth Rd intersection is a bustling roundabout notorious for its high traffic flow and a significant number of accidents annually. Experimental results demonstrate that detectors trained solely on synthesized data exhibit commendable performance across all conditions. Furthermore, when integrated with labeled data, the synthesized data can notably bolster the performance of pre-existing detectors, especially in adverse conditions.