# Representation Space Constrained Learning with Modality Decoupling for Multimodal Object Detection

arXiv: https://arxiv.org/abs/2511.15433 · HTML: https://mlsanity.com/paper/2511.15433/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 64 references that are in the map.
- First posted on arXiv 19 Nov 2025.

## Builds on (most cited first, 22 of 22)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [LLVIP: A Visible-infrared Paired Dataset for Low-light Vision](https://mlsanity.com/paper/2108.10831/) (2021)
- [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020)
- [Cross-Modality Fusion Transformer for Multispectral Object Detection](https://mlsanity.com/paper/2111.00273/) (2021)
- [Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks](https://mlsanity.com/paper/2009.12664/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection](https://mlsanity.com/paper/2308.07504/) (2023)
- [ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions](https://mlsanity.com/paper/2403.07392/) (2024)
- [Prime Sample Attention in Object Detection](https://mlsanity.com/paper/1904.04821/) (2019)
- [Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1808.04818/) (2018)
- [SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery](https://mlsanity.com/paper/2209.13351/) (2022)
- [Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery](https://mlsanity.com/paper/2112.02991/) (2021)
- [DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensing](https://mlsanity.com/paper/2407.08132/) (2024)
- [COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection](https://mlsanity.com/paper/2412.18076/) (2024)
- [UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning](https://mlsanity.com/paper/2404.17360/) (2024)
- [AMFD: Distillation via Adaptive Multimodal Fusion for Multispectral Pedestrian Detection](https://mlsanity.com/paper/2405.12944/) (2024)
- [WaveMamba: Wavelet-Driven Mamba Fusion for RGB-Infrared Object Detection](https://mlsanity.com/paper/2507.18173/) (2025)
- [Rethinking Multi-Modal Object Detection from the Perspective of Mono-Modality Feature Learning](https://mlsanity.com/paper/2503.11780/) (2025)
- [Anchor-free Small-scale Multispectral Pedestrian Detection](https://mlsanity.com/paper/2008.08418/) (2020)
- [MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection](https://mlsanity.com/paper/2404.18849/) (2024)
- [Learning A Robust RGB-Thermal Detector for Extreme Modality Imbalance](https://mlsanity.com/paper/2505.22154/) (2025)

## Built on by (most cited first, 1 of 1)

- [CFGPNet: Cross-Attention-Based Fused Gradient Programmed Network Framework for Multispectral Object Detection](https://mlsanity.com/paper/2608.06205/) (2026)

## Abstract (arXiv)

Multimodal object detection has attracted significant attention in both academia and industry for its enhanced robustness. Although numerous studies have focused on improving modality fusion strategies, most neglect fusion degradation, and none provide a theoretical analysis of its underlying causes. To fill this gap, this paper presents a systematic theoretical investigation of fusion degradation in multimodal detection and identifies two key optimization deficiencies: (1) the gradients of unimodal branch backbones are severely suppressed under multimodal architectures, resulting in under-optimization of the unimodal branches; (2) disparities in modality quality cause weaker modalities to experience stronger gradient suppression, which in turn results in imbalanced modality learning. To address these issues, this paper proposes a Representation Space Constrained Learning with Modality Decoupling (RSC-MD) method, which consists of two modules. The RSC module and the MD module are designed to respectively amplify the suppressed gradients and eliminate inter-modality coupling interference as well as modality imbalance, thereby enabling the comprehensive optimization of each modality-specific backbone. Extensive experiments conducted on the FLIR, LLVIP, M3FD, and MFAD datasets demonstrate that the proposed method effectively alleviates fusion degradation and achieves state-of-the-art performance across multiple benchmarks. The code and training procedures will be released at https://github.com/yikangshao/RSC-MD.