# UrbanTwin: Synthetic Roadside LiDAR Datasets arXiv: https://arxiv.org/abs/2509.06781 ยท HTML: https://mlsanity.com/paper/2509.06781/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 49 references that are in the map. - First posted on arXiv 8 Sep 2025. ## Builds on (most cited first, 25 of 25) - [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017) - [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019) - [DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection](https://mlsanity.com/paper/2204.05575/) (2022) - [V2X-Sim: Multi-Agent Collaborative Perception Dataset and Benchmark for Autonomous Driving](https://mlsanity.com/paper/2202.08449/) (2022) - [LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World](https://mlsanity.com/paper/2006.09348/) (2020) - [VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles](https://mlsanity.com/paper/2111.12083/) (2021) - [Rope3D: TheRoadside Perception Dataset for Autonomous Driving and Monocular 3D Object Detection Task](https://mlsanity.com/paper/2203.13608/) (2022) - [Transfer Learning from Synthetic to Real LiDAR Point Cloud for Semantic Segmentation](https://mlsanity.com/paper/2107.05399/) (2021) - [DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving](https://mlsanity.com/paper/2304.01168/) (2023) - [V2X-Real: a Large-Scale Dataset for Vehicle-to-Everything Cooperative Perception](https://mlsanity.com/paper/2403.16034/) (2024) - [A9-Dataset: Multi-Sensor Infrastructure-Based Dataset for Mobility Research](https://mlsanity.com/paper/2204.06527/) (2022) - [RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception](https://mlsanity.com/paper/2403.10145/) (2024) - [V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction](https://mlsanity.com/paper/2412.01812/) (2024) - [IPS300+: a Challenging Multimodal Dataset for Intersection Perception System](https://mlsanity.com/paper/2106.02781/) (2021) - [Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving](https://mlsanity.com/paper/2311.11056/) (2023) - [DOLPHINS: Dataset for Collaborative Perception enabled Harmonious and Interconnected Self-driving](https://mlsanity.com/paper/2207.07609/) (2022) - [Compositional Semantic Mix for Domain Adaptation in Point Cloud Segmentation](https://mlsanity.com/paper/2308.14619/) (2023) - [BAAI-VANJEE Roadside Dataset: Towards the Connected Automated Vehicle Highway technologies in Challenging Environments of China](https://mlsanity.com/paper/2105.14370/) (2021) - [SEED: A Simple and Effective 3D DETR in Point Clouds](https://mlsanity.com/paper/2407.10749/) (2024) - [HoloVIC: Large-scale Dataset and Benchmark for Multi-Sensor Holographic Intersection and Vehicle-Infrastructure Cooperative](https://mlsanity.com/paper/2403.02640/) (2024) - [WARM-3D: A Weakly-Supervised Sim2Real Domain Adaptation Framework for Roadside Monocular 3D Object Detection](https://mlsanity.com/paper/2407.20818/) (2024) - [R-LiViT: A LiDAR-Visual-Thermal Dataset Enabling Vulnerable Road User Focused Roadside Perception](https://mlsanity.com/paper/2503.17122/) (2025) - [SCOPE: A Synthetic Multi-Modal Dataset for Collective Perception Including Physical-Correct Weather Conditions](https://mlsanity.com/paper/2408.03065/) (2024) - [SynthmanticLiDAR: A Synthetic Dataset for Semantic Segmentation on LiDAR Imaging](https://mlsanity.com/paper/2501.19035/) (2025) - [InSPE: Rapid Evaluation of Heterogeneous Multi-Modal Infrastructure Sensor Placement](https://mlsanity.com/paper/2504.08240/) (2025) ## Built on by (most cited first, 2 of 2) - [Solution for UCF UrbanTwin LUMPI Track: Sim-to-Real Urban LiDAR 3D Object Detection](https://mlsanity.com/paper/2609.07590/) (2026) - [Solution for UCF UrbanTwin V2X-Real Track: Sim-to-Real Urban LiDAR 3D Object Detection](https://mlsanity.com/paper/2609.07608/) (2026) ## Abstract (arXiv) This article presents UrbanTwin datasets, high-fidelity, realistic replicas of three public roadside lidar datasets: LUMPI, V2X-Real-IC, and TUMTraf-I. Each UrbanTwin dataset contains 10K annotated frames corresponding to one of the public datasets. Annotations include 3D bounding boxes, instance segmentation labels, and tracking IDs for six object classes, along with semantic segmentation labels for nine classes. These datasets are synthesized using emulated lidar sensors within realistic digital twins, modeled based on surrounding geometry, road alignment at lane level, and the lane topology and vehicle movement patterns at intersections of the actual locations corresponding to each real dataset. Due to the precise digital twin modeling, the synthetic datasets are well aligned with their real counterparts, offering strong standalone and augmentative value for training deep learning models on tasks such as 3D object detection, tracking, and semantic and instance segmentation. We evaluate the alignment of the synthetic replicas through statistical and structural similarity analysis with real data, and further demonstrate their utility by training 3D object detection models solely on synthetic data and testing them on real, unseen data. The high similarity scores and improved detection performance, compared to models trained on real data, indicate that the UrbanTwin datasets effectively enhance existing benchmark datasets by increasing sample size and scene diversity. In addition, the digital twins can be adapted to test custom scenarios by modifying the design and dynamics of the simulations. To our knowledge, these are the first digitally synthesized datasets that can replace in-domain real-world datasets for lidar perception tasks. UrbanTwin datasets are publicly available at https://dataverse.harvard.edu/dataverse/ucf-ut.