# RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception

arXiv: https://arxiv.org/abs/2403.10145 · HTML: https://mlsanity.com/paper/2403.10145/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 57 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Builds on (most cited first, 30 of 43)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer](https://mlsanity.com/paper/2203.10638/) (2022)
- [OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication](https://mlsanity.com/paper/2109.07644/) (2021)
- [V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction](https://mlsanity.com/paper/2008.07519/) (2020)
- [DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection](https://mlsanity.com/paper/2204.05575/) (2022)
- [Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps](https://mlsanity.com/paper/2209.12836/) (2022)
- [CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers](https://mlsanity.com/paper/2207.02202/) (2022)
- [Learning Distilled Collaboration Graph for Multi-Agent Perception](https://mlsanity.com/paper/2111.00643/) (2021)
- [F-Cooper: Feature based Cooperative Perception for Autonomous Vehicle Edge Computing System Using 3D Point Clouds](https://mlsanity.com/paper/1909.06459/) (2019)
- [Cooper: Cooperative Perception for Connected Autonomous Vehicles based on 3D Point Clouds](https://mlsanity.com/paper/1905.05265/) (2019)
- [When2com: Multi-Agent Perception via Communication Graph Grouping](https://mlsanity.com/paper/2006.00176/) (2020)
- [V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving](https://mlsanity.com/paper/2202.08449/) (2022)
- [V2V4Real: A Real-world Large-scale Dataset for Vehicle-to-Vehicle Cooperative Perception](https://mlsanity.com/paper/2303.07601/) (2023)
- [Who2com: Collaborative Perception via Learnable Handshake Communication](https://mlsanity.com/paper/2003.09575/) (2020)
- [V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting](https://mlsanity.com/paper/2305.05938/) (2023)
- [Cooperative Perception for 3D Object Detection in Driving Scenarios using Infrastructure Sensors](https://mlsanity.com/paper/1912.12147/) (2019)
- [COOPERNAUT: End-to-End Driving with Cooperative Perception for Networked Vehicles](https://mlsanity.com/paper/2205.02222/) (2022)
- [Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges](https://mlsanity.com/paper/2301.06262/) (2023)
- [Latency-Aware Collaborative Perception](https://mlsanity.com/paper/2207.08560/) (2022)
- [Rope3D: TheRoadside Perception Dataset for Autonomous Driving and Monocular 3D Object Detection Task](https://mlsanity.com/paper/2203.13608/) (2022)
- [Collaboration Helps Camera Overtake LiDAR in 3D Detection](https://mlsanity.com/paper/2303.13560/) (2023)
- [BEVHeight: A Robust Framework for Vision-based Roadside 3D Object Detection](https://mlsanity.com/paper/2303.08498/) (2023)
- [Model-Agnostic Multi-Agent Perception Framework](https://mlsanity.com/paper/2203.13168/) (2022)
- [Keypoints-Based Deep Feature Fusion for Cooperative Vehicle Detection of Autonomous Driving](https://mlsanity.com/paper/2109.11615/) (2021)
- [A9-Dataset: Multi-Sensor Infrastructure-Based Dataset for Mobility Research](https://mlsanity.com/paper/2204.06527/) (2022)
- [Multi-Robot Collaborative Perception with Graph Neural Networks](https://mlsanity.com/paper/2201.01760/) (2022)
- [BEVHeight++: Toward Robust Visual Centric 3D Object Detection](https://mlsanity.com/paper/2309.16179/) (2023)
- [Adaptive Feature Fusion for Cooperative Perception using LiDAR Point Clouds](https://mlsanity.com/paper/2208.00116/) (2022)

## Built on by (most cited first, 30 of 38)

- [End-to-End Autonomous Driving through V2X Cooperation](https://mlsanity.com/paper/2404.00717/) (2024)
- [V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2412.01812/) (2024)
- [V2X-Radar: A Multi-modal Dataset with 4D Radar for Cooperative Perception](https://mlsanity.com/paper/2411.10962/) (2024)
- [UAV3D: A Large-scale 3D Perception Benchmark for Unmanned Aerial Vehicles](https://mlsanity.com/paper/2410.11125/) (2024)
- [Griffin: Aerial-Ground Cooperative Detection and Tracking Dataset and Benchmark](https://mlsanity.com/paper/2503.06983/) (2025)
- [SparseAlign: A Fully Sparse Framework for Cooperative Object Detection](https://mlsanity.com/paper/2503.12982/) (2025)
- [CoopTrack: Exploring End-to-End Learning for Efficient Cooperative Sequential Perception](https://mlsanity.com/paper/2507.19239/) (2025)
- [Collaborative Perception Datasets in Autonomous Driving: A Survey](https://mlsanity.com/paper/2404.14022/) (2024)
- [CooPre: Cooperative Pretraining for V2X Cooperative Perception](https://mlsanity.com/paper/2408.11241/) (2024)
- [AirV2X: Unified Air-Ground Vehicle-to-Everything Collaboration](https://mlsanity.com/paper/2506.19283/) (2025)
- [R-LiViT: A LiDAR-Visual-Thermal Dataset Enabling Vulnerable Road User Focused Roadside Perception](https://mlsanity.com/paper/2503.17122/) (2025)
- [Research Challenges and Progress in the End-to-End V2X Cooperative Autonomous Driving Competition](https://mlsanity.com/paper/2507.21610/) (2025)
- [V2X-ReaLO: An Open Online Framework and Dataset for Cooperative Perception in Reality](https://mlsanity.com/paper/2503.10034/) (2025)
- [Collaborative Perception Datasets for Autonomous Driving: A Review](https://mlsanity.com/paper/2504.12696/) (2025)
- [RopeBEV: A Multi-Camera Roadside Perception Network in Bird's-Eye-View](https://mlsanity.com/paper/2409.11706/) (2024)
- [Mixed Signals: A Diverse Point Cloud Dataset for Heterogeneous LiDAR V2X Collaboration](https://mlsanity.com/paper/2502.14156/) (2025)
- [DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation](https://mlsanity.com/paper/2509.23922/) (2025)
- [Multi-V2X: A Large Scale Multi-modal Multi-penetration-rate Dataset for Cooperative Perception](https://mlsanity.com/paper/2409.04980/) (2024)
- [MIC-BEV: Multi-Infrastructure Camera Bird's-Eye-View Transformer with Relation-Aware Fusion for 3D Object Detection](https://mlsanity.com/paper/2510.24688/) (2025)
- [InScope: A New Real-world 3D Infrastructure-side Collaborative Perception Dataset for Open Traffic Scenarios](https://mlsanity.com/paper/2407.21581/) (2024)
- [UrbanTwin: Synthetic Roadside LiDAR Datasets](https://mlsanity.com/paper/2509.06781/) (2025)
- [Linking Modality Isolation in Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2603.00609/) (2026)
- [V2U4Real: A Real-world Large-scale Dataset for Vehicle-to-UAV Cooperative Perception](https://mlsanity.com/paper/2603.25275/) (2026)
- [InfraOcc: An Infrastructure Occupancy Benchmark with Static-to-Dynamic Reasoning](https://mlsanity.com/paper/2608.30657/) (2026)
- [HeightFormer: A Semantic Alignment Monocular 3D Object Detection Method from Roadside Perspective](https://mlsanity.com/paper/2410.07758/) (2024)
- [A Synthetic Benchmark for Collaborative 3D Semantic Occupancy Prediction in V2X-Enabled Autonomous Driving](https://mlsanity.com/paper/2506.17004/) (2025)
- [A Fully Interpretable Statistical Approach for Roadside LiDAR Background Subtraction](https://mlsanity.com/paper/2510.22390/) (2025)
- [V2VLoc: Robust GNSS-Free Collaborative Perception via LiDAR Localization](https://mlsanity.com/paper/2511.14247/) (2025)
- [UPDA: Unsupervised Progressive Domain Adaptation for No-Reference Point Cloud Quality Assessment](https://mlsanity.com/paper/2602.11969/) (2026)
- [GT-Space: Enhancing Heterogeneous Collaborative Perception with Ground Truth Feature Space](https://mlsanity.com/paper/2603.19308/) (2026)

## Abstract (arXiv)

The value of roadside perception, which could extend the boundaries of autonomous driving and traffic management, has gradually become more prominent and acknowledged in recent years. However, existing roadside perception approaches only focus on the single-infrastructure sensor system, which cannot realize a comprehensive understanding of a traffic area because of the limited sensing range and blind spots. Orienting high-quality roadside perception, we need Roadside Cooperative Perception (RCooper) to achieve practical area-coverage roadside perception for restricted traffic areas. Rcooper has its own domain-specific challenges, but further exploration is hindered due to the lack of datasets. We hence release the first real-world, large-scale RCooper dataset to bloom the research on practical roadside cooperative perception, including detection and tracking. The manually annotated dataset comprises 50k images and 30k point clouds, including two representative traffic scenes (i.e., intersection and corridor). The constructed benchmarks prove the effectiveness of roadside cooperation perception and demonstrate the direction of further research. Codes and dataset can be accessed at: https://github.com/AIR-THU/DAIR-RCooper.