mlsanityData to 14 Oct 2026

Complementary Random Masking for RGB-Thermal Semantic Segmentation

arXiv 2303.17386 ↗First posted 30 Mar 2023cs.CV cs.AI cs.ROcited by 27

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

RGB-thermal semantic segmentation is one potential solution to achieve reliable semantic scene understanding in adverse weather and lighting conditions. However, the previous studies mostly focus on designing a multi-modal fusion module without consideration of the nature of multi-modality inputs. Therefore, the networks easily become over-reliant on a single modality, making it difficult to learn complementary and meaningful representations for each modality. This paper proposes 1) a complementary random masking strategy of RGB-T images and 2) self-distillation loss between clean and masked input modalities. The proposed masking strategy prevents over-reliance on a single modality. It also improves the accuracy and robustness of the neural network by forcing the network to segment and classify objects even when one modality is partially available. Also, the proposed self-distillation loss encourages the network to extract complementary and meaningful representations from a single modality or complementary masked modalities. Based on the proposed method, we achieve state-of-the-art performance over three RGB-T semantic segmentation benchmarks. Our source code is available at https://github.com/UkcheolShin/CRM_RGBTSeg.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 32 of its 51 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

32 of 51 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2017Mask R-CNNcited by 7,277
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  12. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
20 more
  1. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  4. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  5. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  6. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  7. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  8. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  9. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  10. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  11. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
  12. 2018RGB-T Object Tracking:Benchmark and Baselinecited by 121
  13. 2019PST900: RGB-Thermal Calibration, Dataset and Segmentation Networkcited by 85
  14. 2021FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentationcited by 45
  15. 2020HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Imagescited by 41
  16. 2023Edge-guided Multi-domain RGB-to-TIR image Translation for Training Vision Tasks with Challenging Labelscited by 30
  17. 2019Sparse Depth Enhanced Direct Thermal-infrared SLAM Beyond the Visible Spectrumcited by 22
  18. 2021Self-Supervised Depth and Ego-Motion Estimation for Monocular Thermal Video Using Multi-Spectral Consistency Losscited by 16
  19. 2022Maximizing Self-supervision from Thermal Image for Effective Self-supervised Learning of Depth and Ego-motioncited by 14
  20. 2021BAANet: Learning Bi-directional Adaptive Attention Gates for Multispectral Pedestrian Detectioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 12; 2026: 4.

27 papers.

Sort:

By area:

  1. 2024XoFTR: Cross-modal Feature Matching Transformercited by 48
  2. 2024SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Trackingcited by 44
  3. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29
  4. 2024Caltech Aerial RGB-Thermal Dataset in the Wildcited by 19
  5. 2025ThermalGen: Style-Disentangled Flow-Based Generative Models for RGB-to-Thermal Image Translationcited by 14
  6. 2024HAPNet: Toward Superior RGB-Thermal Scene Parsing via Hybrid, Asymmetric, and Progressive Heterogeneous Feature Fusioncited by 10
  7. 2024RoadFormer+: Delivering RGB-X Scene Parsing through Scale-Aware Information Decoupling and Advanced Heterogeneous Feature Fusioncited by 10
  8. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  9. 2025Unveiling the Potential of Segment Anything Model 2 for RGB-Thermal Semantic Segmentation with Language Guidancecited by 9
  10. 2025Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detectioncited by 6
  11. 2026TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translationcited by 4
  12. 2025Robust Multimodal Learning via Cross-Modal Proxy Tokenscited by 2
15 more
  1. 2025Cross-modal State Space Modeling for Real-time RGB-thermal Wild Scene Semantic Segmentationcited by 2
  2. 2024CSFNet: A Cosine Similarity Fusion Network for Real-Time RGB-X Semantic Segmentation of Driving Scenescited by 1
  3. 2025Dual form Complementary Masking for Domain-Adaptive Image Segmentationcited by 1
  4. 2025On Modality Incomplete Infrared-Visible Object Detection: An Architecture Compatibility Perspectivecited by 1
  5. 2025MULTIAQUA: A multimodal maritime dataset and robust training strategies for multimodal semantic segmentationcited by 1
  6. 2026RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentationcited by 1
  7. 2023Dissecting RGB-D Learning for Improved Multi-modal Fusioncited by 0
  8. 2024IVGF: The Fusion-Guided Infrared and Visible General Frameworkcited by 0
  9. 2024IRFusionFormer: Enhancing Pavement Crack Segmentation with RGB-T Fusion and Topological-Based Losscited by 0
  10. 2025SARTM: Segment Any RGB Thermal Model with Language aided Distillationcited by 0
  11. 2025Boosting Cross-spectral Unsupervised Domain Adaptation for Thermal Semantic Segmentationcited by 0
  12. 2025RMMSS: Towards Advanced Robust Multi-Modal Semantic Segmentation with Hybrid Prototype Distillation and Feature Selectioncited by 0
  13. 2025DepthMatch: Semi-Supervised RGB-D Scene Parsing through Depth-Guided Regularizationcited by 0
  14. 2026RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentationcited by 0
  15. 2026Graph-based Semantic Calibration Network for Unaligned UAV RGBT Image Semantic Segmentation and A Large-scale Benchmarkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation
  2. 2026Seeing Roads Through Words: A Language-Guided Framework for RGB-T Driving Scene Segmentation
  3. 2025Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic Segmentation
  4. 2025Lightweight Facial Landmark Detection in Thermal Images via Multi-Level Cross-Modal Knowledge Transfer
  5. 2024Open-RGBT: Open-vocabulary RGB-T Zero-shot Semantic Segmentation in Open-world Environments
  6. 2023Learning to "Segment Anything" in Thermal Infrared Images through Knowledge Distillation with a Large Scale Dataset SATIR
  7. 2026Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detection
  8. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities