# Image-to-Image Translation for Autonomous Driving from Coarsely-Aligned Image Pairs

arXiv: https://arxiv.org/abs/2209.11673 · HTML: https://mlsanity.com/paper/2209.11673/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Unsupervised Image-to-Image Translation Networks](https://mlsanity.com/paper/1703.00848/) (2017)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Learning to Discover Cross-Domain Relations with Generative Adversarial Networks](https://mlsanity.com/paper/1703.05192/) (2017)
- [DRIT++: Diverse Image-to-Image Translation via Disentangled Representations](https://mlsanity.com/paper/1905.01270/) (2019)
- [Canadian Adverse Driving Conditions Dataset](https://mlsanity.com/paper/2001.10117/) (2020)
- [Night-to-Day Image Translation for Retrieval-based Localization](https://mlsanity.com/paper/1809.09767/) (2018)
- [Ithaca365: Dataset and Driving Perception under Repeated and Challenging Weather Conditions](https://mlsanity.com/paper/2208.01166/) (2022)
- [Unpaired Image-to-Image Translation using Adversarial Consistency Loss](https://mlsanity.com/paper/2003.04858/) (2020)
- [Breaking the cycle -- Colleagues are all you need](https://mlsanity.com/paper/1911.10538/) (2019)
- [Adversarial Training for Adverse Conditions: Robust Metric Localisation using Appearance Transfer](https://mlsanity.com/paper/1803.03341/) (2018)
- [Robust Semantic Segmentation in Adverse Weather Conditions by means of Sensor Data Fusion](https://mlsanity.com/paper/1905.10117/) (2019)
- [PixelNN: Example-based Image Synthesis](https://mlsanity.com/paper/1708.05349/) (2017)
- [ZeroScatter: Domain Transfer for Long Distance Imaging and Vision through Scattering Media](https://mlsanity.com/paper/2102.05847/) (2021)

## Built on by (most cited first, 8 of 8)

- [ControlNet-XS: Rethinking the Control of Text-to-Image Diffusion Models as Feedback-Control Systems](https://mlsanity.com/paper/2312.06573/) (2023)
- [HawkDrive: A Transformer-driven Visual Perception System for Autonomous Driving in Night Scene](https://mlsanity.com/paper/2404.04653/) (2024)
- [Learning Real-World Image De-Weathering with Imperfect Supervision](https://mlsanity.com/paper/2310.14958/) (2023)
- [RainSD: Rain Style Diversification Module for Image Synthesis Enhancement using Feature-Level Style Distribution](https://mlsanity.com/paper/2401.00460/) (2023)
- [The Quest for Early Detection of Retinal Disease: 3D CycleGAN-based Translation of Optical Coherence Tomography into Confocal Microscopy](https://mlsanity.com/paper/2408.04091/) (2024)
- [LapLoss: Laplacian Pyramid-based Multiscale loss for Image Translation](https://mlsanity.com/paper/2503.05974/) (2025)
- [Pseudo-Label Guided Real-World Image De-weathering: A Learning Framework with Imperfect Supervision](https://mlsanity.com/paper/2504.09949/) (2025)
- [A2BM: Alignment-Aware Bridge Matching for Image-to-Image Translation](https://mlsanity.com/paper/2607.16294/) (2026)

## Abstract (arXiv)

A self-driving car must be able to reliably handle adverse weather conditions (e.g., snowy) to operate safely. In this paper, we investigate the idea of turning sensor inputs (i.e., images) captured in an adverse condition into a benign one (i.e., sunny), upon which the downstream tasks (e.g., semantic segmentation) can attain high accuracy. Prior work primarily formulates this as an unpaired image-to-image translation problem due to the lack of paired images captured under the exact same camera poses and semantic layouts. While perfectly-aligned images are not available, one can easily obtain coarsely-paired images. For instance, many people drive the same routes daily in both good and adverse weather; thus, images captured at close-by GPS locations can form a pair. Though data from repeated traversals are unlikely to capture the same foreground objects, we posit that they provide rich contextual information to supervise the image translation model. To this end, we propose a novel training objective leveraging coarsely-aligned image pairs. We show that our coarsely-aligned training scheme leads to a better image translation quality and improved downstream tasks, such as semantic segmentation, monocular depth estimation, and visual localization.