# Learning A Robust RGB-Thermal Detector for Extreme Modality Imbalance

arXiv: https://arxiv.org/abs/2505.22154 · HTML: https://mlsanity.com/paper/2505.22154/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 20 of 20)

- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [VarifocalNet: An IoU-aware Dense Object Detector](https://mlsanity.com/paper/2008.13367/) (2020)
- [LLVIP: A Visible-infrared Paired Dataset for Low-light Vision](https://mlsanity.com/paper/2108.10831/) (2021)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks](https://mlsanity.com/paper/2009.12664/) (2020)
- [Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems](https://mlsanity.com/paper/2008.03043/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection](https://mlsanity.com/paper/1803.05347/) (2018)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery](https://mlsanity.com/paper/2112.02991/) (2021)
- [SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities](https://mlsanity.com/paper/2204.02961/) (2022)
- [Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models](https://mlsanity.com/paper/2007.01179/) (2020)
- [Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2304.10756/) (2023)
- [Cross-Modality Proposal-guided Feature Mining for Unregistered RGB-Thermal Pedestrian Detection](https://mlsanity.com/paper/2308.12111/) (2023)
- [Multi-Modal Pedestrian Detection with Large Misalignment Based on Modal-Wise Regression and Multi-Modal IoU](https://mlsanity.com/paper/2107.11196/) (2021)

## Built on by (most cited first, 5 of 5)

- [RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation](https://mlsanity.com/paper/2609.12622/) (2026)
- [Representation Space Constrained Learning with Modality Decoupling for Multimodal Object Detection](https://mlsanity.com/paper/2511.15433/) (2025)
- [Evaluating an Adaptive Multispectral Turret System for Autonomous Tracking Across Variable Illumination Conditions](https://mlsanity.com/paper/2512.22263/) (2025)
- [Modality-Decoupled RGB-Thermal Object Detector via Query Fusion](https://mlsanity.com/paper/2601.08458/) (2026)
- [Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion](https://mlsanity.com/paper/2606.30215/) (2026)

## Abstract (arXiv)

RGB-Thermal (RGB-T) object detection utilizes thermal infrared (TIR) images to complement RGB data, improving robustness in challenging conditions. Traditional RGB-T detectors assume balanced training data, where both modalities contribute equally. However, in real-world scenarios, modality degradation-due to environmental factors or technical issues-can lead to extreme modality imbalance, causing out-of-distribution (OOD) issues during testing and disrupting model convergence during training. This paper addresses these challenges by proposing a novel base-and-auxiliary detector architecture. We introduce a modality interaction module to adaptively weigh modalities based on their quality and handle imbalanced samples effectively. Additionally, we leverage modality pseudo-degradation to simulate real-world imbalances in training data. The base detector, trained on high-quality pairs, provides a consistency constraint for the auxiliary detector, which receives degraded samples. This framework enhances model robustness, ensuring reliable performance even under severe modality degradation. Experimental results demonstrate the effectiveness of our method in handling extreme modality imbalances~(decreasing the Missing Rate by 55%) and improving performance across various baseline detectors.