# Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation

arXiv: https://arxiv.org/abs/2307.04470 · HTML: https://mlsanity.com/paper/2307.04470/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 60 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 30 of 32)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Improving robustness against common corruptions by covariate shift adaptation](https://mlsanity.com/paper/2006.16971/) (2020)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Parameter-free Online Test-time Adaptation](https://mlsanity.com/paper/2201.05718/) (2022)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [Pyramid Attention Network for Semantic Segmentation](https://mlsanity.com/paper/1805.10180/) (2018)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/1911.12676/) (2019)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [RGB-Infrared Cross-Modality Person Re-Identification via Joint Pixel and Feature Alignment](https://mlsanity.com/paper/1910.05839/) (2019)
- [Test-Time Adaptable Neural Networks for Robust Medical Image Segmentation](https://mlsanity.com/paper/2004.04668/) (2020)
- [PST900: RGB-Thermal Calibration, Dataset and Segmentation Network](https://mlsanity.com/paper/1909.10980/) (2019)
- [Unpaired Multi-modal Segmentation via Knowledge Distillation](https://mlsanity.com/paper/2001.03111/) (2020)
- [DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation](https://mlsanity.com/paper/2104.10834/) (2021)
- [MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2204.12667/) (2022)
- [Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification](https://mlsanity.com/paper/2002.04114/) (2020)
- [Collaborative Learning for Deep Neural Networks](https://mlsanity.com/paper/1805.11761/) (2018)
- [Peer Collaborative Learning for Online Knowledge Distillation](https://mlsanity.com/paper/2006.04147/) (2020)
- [FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation](https://mlsanity.com/paper/2110.08988/) (2021)
- [HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images](https://mlsanity.com/paper/2003.04645/) (2020)
- [Segmenting Objects in Day and Night:Edge-Conditioned CNN for Thermal Image Semantic Segmentation](https://mlsanity.com/paper/1907.10303/) (2019)
- [On-the-Fly Test-time Adaptation for Medical Image Segmentation](https://mlsanity.com/paper/2203.05574/) (2022)
- [AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video Semantic Segmentation](https://mlsanity.com/paper/2110.12369/) (2021)
- [Knowledge Distillation in Deep Learning and its Applications](https://mlsanity.com/paper/2007.09029/) (2020)

## Built on by (most cited first, 4 of 4)

- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [Variational Continual Test-Time Adaptation](https://mlsanity.com/paper/2402.08182/) (2024)
- [TSCLIP: Robust CLIP Fine-Tuning for Worldwide Cross-Regional Traffic Sign Recognition](https://mlsanity.com/paper/2409.15077/) (2024)
- [OAFuser: Towards Omni-Aperture Fusion for Light Field Semantic Segmentation](https://mlsanity.com/paper/2307.15588/) (2023)

## Abstract (arXiv)

The ability to scene understanding in adverse visual conditions, e.g., nighttime, has sparked active research for RGB-Thermal (RGB-T) semantic segmentation. However, it is essentially hampered by two critical problems: 1) the day-night gap of RGB images is larger than that of thermal images, and 2) the class-wise performance of RGB images at night is not consistently higher or lower than that of thermal images. we propose the first test-time adaptation (TTA) framework, dubbed Night-TTA, to address the problems for nighttime RGBT semantic segmentation without access to the source (daytime) data during adaptation. Our method enjoys three key technical parts. Firstly, as one modality (e.g., RGB) suffers from a larger domain gap than that of the other (e.g., thermal), Imaging Heterogeneity Refinement (IHR) employs an interaction branch on the basis of RGB and thermal branches to prevent cross-modal discrepancy and performance degradation. Then, Class Aware Refinement (CAR) is introduced to obtain reliable ensemble logits based on pixel-level distribution aggregation of the three branches. In addition, we also design a specific learning scheme for our TTA framework, which enables the ensemble logits and three student logits to collaboratively learn to improve the quality of predictions during the testing phase of our Night TTA. Extensive experiments show that our method achieves state-of-the-art (SoTA) performance with a 13.07% boost in mIoU.