# RALAD: Bridging the Real-to-Sim Domain Gap in Autonomous Driving with Retrieval-Augmented Learning

arXiv: https://arxiv.org/abs/2501.12296 · HTML: https://mlsanity.com/paper/2501.12296/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 21 Jan 2025.

## Builds on (most cited first, 18 of 18)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2203.07724/) (2022)
- [Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions](https://mlsanity.com/paper/2112.11561/) (2021)
- [How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence](https://mlsanity.com/paper/2305.01263/) (2023)
- [Transformation-Equivariant 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/2211.11962/) (2022)
- [MonoLayout: Amodal scene layout from a single image](https://mlsanity.com/paper/2002.08394/) (2020)
- [Towards Corner Case Detection for Autonomous Driving](https://mlsanity.com/paper/1902.09184/) (2019)
- [Corner Cases for Visual Perception in Automated Driving: Some Guidance on Detection Approaches](https://mlsanity.com/paper/2102.05897/) (2021)
- [MRFP: Learning Generalizable Semantic Segmentation from Sim-2-Real with Multi-Resolution Feature Perturbation](https://mlsanity.com/paper/2311.18331/) (2023)
- [Development and testing of an image transformer for explainable autonomous driving systems](https://mlsanity.com/paper/2110.05559/) (2021)
- [SynFog: A Photo-realistic Synthetic Fog Dataset based on End-to-end Imaging Simulation for Advancing Real-World Defogging in Autonomous Driving](https://mlsanity.com/paper/2403.17094/) (2024)
- [Hierarchical Optimal Transport for Comparing Histopathology Datasets](https://mlsanity.com/paper/2204.08324/) (2022)
- [A Dual-Cycled Cross-View Transformer Network for Unified Road Layout Estimation and 3D Object Detection in the Bird's-Eye-View](https://mlsanity.com/paper/2209.08844/) (2022)

## Built on by (most cited first, 2 of 2)

- [RealDrive: Retrieval-Augmented Driving with Diffusion Models](https://mlsanity.com/paper/2505.24808/) (2025)
- [CARD: A Multi-Modal Automotive Dataset for Dense 3D Reconstruction in Challenging Road Topography](https://mlsanity.com/paper/2605.05014/) (2026)

## Abstract (arXiv)

In the pursuit of robust autonomous driving systems, models trained on real-world datasets often struggle to adapt to new environments, particularly when confronted with corner cases such as extreme weather conditions. Collecting these corner cases in the real world is non-trivial, which necessitates the use of simulators for validation. However,the high computational cost and the domain gap in data distribution have hindered the seamless transition between real and simulated driving scenarios. To tackle this challenge, we propose Retrieval-Augmented Learning for Autonomous Driving (RALAD), a novel framework designed to bridge the real-to-sim gap at a low cost. RALAD features three primary designs, including (1) domain adaptation via an enhanced Optimal Transport (OT) method that accounts for both individual and grouped image distances, (2) a simple and unified framework that can be applied to various models, and (3) efficient fine-tuning techniques that freeze the computationally expensive layers while maintaining robustness. Experimental results demonstrate that RALAD compensates for the performance degradation in simulated environments while maintaining accuracy in real-world scenarios across three different models. Taking Cross View as an example, the mIOU and mAP metrics in real-world scenarios remain stable before and after RALAD fine-tuning, while in simulated environments,the mIOU and mAP metrics are improved by 10.30% and 12.29%, respectively. Moreover, the re-training cost of our approach is reduced by approximately 88.1%. Our code is available at https://github.com/JiachengZuo/RALAD.git.