# HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images

arXiv: https://arxiv.org/abs/2003.04645 · HTML: https://mlsanity.com/paper/2003.04645/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2020.

## Built on by (most cited first, 30 of 41)

- [DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation](https://mlsanity.com/paper/2104.10834/) (2021)
- [Zero-Shot Day-Night Domain Adaptation with a Physics Prior](https://mlsanity.com/paper/2108.05137/) (2021)
- [There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledge](https://mlsanity.com/paper/2103.01353/) (2021)
- [Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks](https://mlsanity.com/paper/2303.15710/) (2023)
- [Cross-Domain Correlation Distillation for Unsupervised Domain Adaptation in Nighttime Semantic Segmentation](https://mlsanity.com/paper/2205.00858/) (2022)
- [MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty](https://mlsanity.com/paper/2401.12761/) (2024)
- [Complementary Random Masking for RGB-Thermal Semantic Segmentation](https://mlsanity.com/paper/2303.17386/) (2023)
- [Caltech Aerial RGB-Thermal Dataset in the Wild](https://mlsanity.com/paper/2403.08997/) (2024)
- [Uncertainty-aware Panoptic Segmentation](https://mlsanity.com/paper/2206.14554/) (2022)
- [Semantic Segmentation for Thermal Images: A Comparative Survey](https://mlsanity.com/paper/2205.13278/) (2022)
- [Dataset and Benchmark: Novel Sensors for Autonomous Vehicle Perception](https://mlsanity.com/paper/2401.13853/) (2024)
- [ThermalGen: Style-Disentangled Flow-Based Generative Models for RGB-to-Thermal Image Translation](https://mlsanity.com/paper/2509.24878/) (2025)
- [Unsupervised RGB-to-Thermal Domain Adaptation via Multi-Domain Attention Network](https://mlsanity.com/paper/2210.04367/) (2022)
- [RGB-D And Thermal Sensor Fusion: A Systematic Literature Review](https://mlsanity.com/paper/2305.11427/) (2023)
- [InfraParis: A multi-modal and multi-task autonomous driving dataset](https://mlsanity.com/paper/2309.15751/) (2023)
- [Segment Anything with Multiple Modalities](https://mlsanity.com/paper/2408.09085/) (2024)
- [Self-Supervised Moving Vehicle Detection from Audio-Visual Cues](https://mlsanity.com/paper/2201.12771/) (2022)
- [What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging](https://mlsanity.com/paper/2304.13651/) (2023)
- [CFMW: Cross-modality Fusion Mamba for Robust Object Detection under Adverse Weather](https://mlsanity.com/paper/2404.16302/) (2024)
- [M-SpecGene: Generalized Foundation Model for RGBT Multispectral Vision](https://mlsanity.com/paper/2507.16318/) (2025)
- [An Unsupervised Domain Adaptive Approach for Multimodal 2D Object Detection in Adverse Weather Conditions](https://mlsanity.com/paper/2203.03568/) (2022)
- [Modality-Incremental Learning with Disjoint Relevance Mapping Networks for Image-based Semantic Segmentation](https://mlsanity.com/paper/2411.17610/) (2024)
- [AnyThermal: Towards Learning Universal Representations for Thermal Perception](https://mlsanity.com/paper/2602.06203/) (2026)
- [Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation](https://mlsanity.com/paper/2307.04470/) (2023)
- [Understanding Dark Scenes by Contrasting Multi-Modal Observations](https://mlsanity.com/paper/2308.12320/) (2023)
- [ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Images](https://mlsanity.com/paper/2403.09871/) (2024)
- [Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation](https://mlsanity.com/paper/2410.21708/) (2024)
- [Towards Dynamic and Small Objects Refinement for Unsupervised Domain Adaptative Nighttime Semantic Segmentation](https://mlsanity.com/paper/2310.04747/) (2023)
- [Semantics from Space: Satellite-Guided Thermal Semantic Segmentation Annotation for Aerial Field Robots](https://mlsanity.com/paper/2403.14056/) (2024)
- [RHRSegNet: Relighting High-Resolution Night-Time Semantic Segmentation](https://mlsanity.com/paper/2407.06016/) (2024)

## Abstract (arXiv)

The majority of learning-based semantic segmentation methods are optimized for daytime scenarios and favorable lighting conditions. Real-world driving scenarios, however, entail adverse environmental conditions such as nighttime illumination or glare which remain a challenge for existing approaches. In this work, we propose a multimodal semantic segmentation model that can be applied during daytime and nighttime. To this end, besides RGB images, we leverage thermal images, making our network significantly more robust. We avoid the expensive annotation of nighttime images by leveraging an existing daytime RGB-dataset and propose a teacher-student training approach that transfers the dataset's knowledge to the nighttime domain. We further employ a domain adaptation method to align the learned feature spaces across the domains and propose a novel two-stage training scheme. Furthermore, due to a lack of thermal data for autonomous driving, we present a new dataset comprising over 20,000 time-synchronized and aligned RGB-thermal image pairs. In this context, we also present a novel target-less calibration method that allows for automatic robust extrinsic and intrinsic thermal camera calibration. Among others, we employ our new dataset to show state-of-the-art results for nighttime semantic segmentation.