# InfraDet3D: Multi-Modal 3D Object Detection based on Roadside Infrastructure Camera and LiDAR Sensors

arXiv: https://arxiv.org/abs/2305.00314 · HTML: https://mlsanity.com/paper/2305.00314/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 29 Apr 2023.

## Builds on (most cited first, 21 of 21)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection](https://mlsanity.com/paper/2204.05575/) (2022)
- [Rope3D: TheRoadside Perception Dataset for Autonomous Driving and Monocular 3D Object Detection Task](https://mlsanity.com/paper/2203.13608/) (2022)
- [Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments](https://mlsanity.com/paper/2103.01627/) (2021)
- [A9-Dataset: Multi-Sensor Infrastructure-Based Dataset for Mobility Research](https://mlsanity.com/paper/2204.06527/) (2022)
- [3D BAT: A Semi-Automatic, Web-based 3D Annotation Toolbox for Full-Surround, Multi-Modal Data Streams](https://mlsanity.com/paper/1905.00525/) (2019)
- [IPS300+: a Challenging Multimodal Dataset for Intersection Perception System](https://mlsanity.com/paper/2106.02781/) (2021)
- [Providentia -- A Large-Scale Sensor System for the Assistance of Autonomous Vehicles and Its Evaluation](https://mlsanity.com/paper/1906.06789/) (2019)
- [A Survey of Robust 3D Object Detection Methods in Point Clouds](https://mlsanity.com/paper/2204.00106/) (2022)
- [Real-Time and Robust 3D Object Detection Within Road-Side LiDARs Using Domain Adaptation](https://mlsanity.com/paper/2204.00132/) (2022)
- [Targetless Extrinsic Calibration of Multiple Small FoV LiDARs and Cameras using Adaptive Voxelization](https://mlsanity.com/paper/2109.06550/) (2021)
- [Real-Time And Robust 3D Object Detection with Roadside LiDARs](https://mlsanity.com/paper/2207.05200/) (2022)
- [Roadside Lidar Vehicle Detection and Tracking Using Range And Intensity Background Subtraction](https://mlsanity.com/paper/2201.04756/) (2022)
- [UrbanNet: Leveraging Urban Maps for Long Range 3D Object Detection](https://mlsanity.com/paper/2110.05561/) (2021)
- [Traffic-Net: 3D Traffic Monitoring Using a Single Camera](https://mlsanity.com/paper/2109.09165/) (2021)
- [Cyber Mobility Mirror for Enabling Cooperative Driving Automation in Mixed Traffic: A Co-Simulation Platform](https://mlsanity.com/paper/2201.09463/) (2022)

## Built on by (most cited first, 25 of 25)

- [TUMTraf V2X Cooperative Perception Dataset](https://mlsanity.com/paper/2403.01316/) (2024)
- [Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perception](https://mlsanity.com/paper/2308.16714/) (2023)
- [RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception](https://mlsanity.com/paper/2403.10145/) (2024)
- [BEVHeight++: Toward Robust Visual Centric 3D Object Detection](https://mlsanity.com/paper/2309.16179/) (2023)
- [Towards Explainable, Safe Autonomous Driving with Language Embeddings for Novelty Identification and Active Learning: Framework and Experimental Analysis with Real-World Data Sets](https://mlsanity.com/paper/2402.07320/) (2024)
- [ActiveAnno3D -- An Active Learning Framework for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2402.03235/) (2024)
- [TUMTraf Event: Calibration and Fusion Resulting in a Dataset for Roadside Event-Based and RGB Cameras](https://mlsanity.com/paper/2401.08474/) (2024)
- [Collaborative Perception Datasets in Autonomous Driving: A Survey](https://mlsanity.com/paper/2404.14022/) (2024)
- [WARM-3D: A Weakly-Supervised Sim2Real Domain Adaptation Framework for Roadside Monocular 3D Object Detection](https://mlsanity.com/paper/2407.20818/) (2024)
- [RopeBEV: A Multi-Camera Roadside Perception Network in Bird's-Eye-View](https://mlsanity.com/paper/2409.11706/) (2024)
- [A9 Intersection Dataset: All You Need for Urban 3D Camera-LiDAR Roadside Perception](https://mlsanity.com/paper/2306.09266/) (2023)
- [MIC-BEV: Multi-Infrastructure Camera Bird's-Eye-View Transformer with Relation-Aware Fusion for 3D Object Detection](https://mlsanity.com/paper/2510.24688/) (2025)
- [Transformation Decoupling Strategy based on Screw Theory for Deterministic Point Cloud Registration with Gravity Prior](https://mlsanity.com/paper/2311.01432/) (2023)
- [FedRSU: Federated Learning for Scene Flow Estimation on Roadside Units](https://mlsanity.com/paper/2401.12862/) (2024)
- [PointCompress3D: A Point Cloud Compression Framework for Roadside LiDARs in Intelligent Transportation Systems](https://mlsanity.com/paper/2405.01750/) (2024)
- [Transfer Learning from Simulated to Real Scenes for Monocular 3D Object Detection](https://mlsanity.com/paper/2408.15637/) (2024)
- [Towards Vision Zero: The TUM Traffic Accid3nD Dataset](https://mlsanity.com/paper/2503.12095/) (2025)
- [Kaninfradet3D:A Road-side Camera-LiDAR Fusion 3D Perception Model based on Nonlinear Feature Extraction and Intrinsic Correlation](https://mlsanity.com/paper/2410.15814/) (2024)
- [Enhancing Highway Safety: Accident Detection on the A9 Test Stretch Using Roadside Sensors](https://mlsanity.com/paper/2502.00402/) (2025)
- [Safety-Critical Learning for Long-Tail Events: The TUM Traffic Accident Dataset](https://mlsanity.com/paper/2508.14567/) (2025)
- [LiGuard: A Streamlined Open-Source Framework for Rapid & Interactive Lidar Research](https://mlsanity.com/paper/2509.02902/) (2025)
- [UniDA3D: A Unified Domain-Adaptive Framework for Multi-View 3D Object Detection](https://mlsanity.com/paper/2603.27995/) (2026)
- [CLIFE: Camera-LiDAR Fusion Framework for Edge-Deployable Roadside VRU Perception](https://mlsanity.com/paper/2607.16154/) (2026)
- [IRGNN: Efficient Invariant Radar Graph Neural Network for Radar Point Cloud Object Detection](https://mlsanity.com/paper/2608.14394/) (2026)
- [Infrastructure-based Monocular 3D Vehicle Localization Framework with Experimental Validation](https://mlsanity.com/paper/2609.05523/) (2026)

## Abstract (arXiv)

Current multi-modal object detection approaches focus on the vehicle domain and are limited in the perception range and the processing capabilities. Roadside sensor units (RSUs) introduce a new domain for perception systems and leverage altitude to observe traffic. Cameras and LiDARs mounted on gantry bridges increase the perception range and produce a full digital twin of the traffic. In this work, we introduce InfraDet3D, a multi-modal 3D object detector for roadside infrastructure sensors. We fuse two LiDARs using early fusion and further incorporate detections from monocular cameras to increase the robustness and to detect small objects. Our monocular 3D detection module uses HD maps to ground object yaw hypotheses, improving the final perception results. The perception framework is deployed on a real-world intersection that is part of the A9 Test Stretch in Munich, Germany. We perform several ablation studies and experiments and show that fusing two LiDARs with two cameras leads to an improvement of +1.90 mAP compared to a camera-only solution. We evaluate our results on the A9 infrastructure dataset and achieve 68.48 mAP on the test set. The dataset and code will be available at https://a9-dataset.com to allow the research community to further improve the perception results and make autonomous driving safer.