# Digital Twin-Driven Real2Sim2Real: Simulator-Conditioned Generation via Paired Driving-Scene Reconstruction

arXiv: https://arxiv.org/abs/2610.08339 · HTML: https://mlsanity.com/paper/2610.08339/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 25 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 3 of 3)

- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention](https://mlsanity.com/paper/2412.03520/) (2024)
- [SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model](https://mlsanity.com/paper/2503.13952/) (2025)

## Abstract (arXiv)

Camera-based 3D perception for autonomous driving relies heavily on large annotated datasets, and deploying such a system to a new target region typically requires data collection and annotation. Generative augmentation has been proposed to reduce this cost, but existing approaches face a fundamental trade-off: label-conditioned methods consume the very annotations they aim to replace, while simulator-conditioned methods offer free annotations but lack visual grounding to specific real environments. This work investigates the extent to which a digital-twin-driven Real2Sim2Real pipeline (DT-R2S2R) can substitute for target-region real data. By reconstructing recorded driving clips inside a georeferenced digital twin (DT-R2S), we condition a diffusion model on geometrically aligned simulator renderings, establishing a digital twin-grounded Sim2Real model (DT-S2R). As a result, DT-S2R synthesizes photorealistic driving images given low-cost yet georeferenced simulator data across both reconstructed and novel simulator scenes within digital-twin coverage. The efficacy of generated data is verified on diverse 3D detectors. DETR3D, especially, reports 93.18% of mAP obtained by a target-region real-data oracle, without employing target images for detector training. Furthermore, simple co-training with existing out-of-target real data outperforms the oracle. Thus, DT-R2S2R can substantially reduce the cost of manual on-site data collection and annotation in digital twin-available districts, providing a practical foundation for scaling 3D perception.