# Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection

arXiv: https://arxiv.org/abs/2405.16038 · HTML: https://mlsanity.com/paper/2405.16038/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 71 references that are in the map.
- First posted on arXiv 25 May 2024.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks](https://mlsanity.com/paper/2303.03667/) (2023)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks](https://mlsanity.com/paper/2009.12664/) (2020)
- [Weakly Supervised Object Localization and Detection: A Survey](https://mlsanity.com/paper/2104.07918/) (2021)
- [Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion](https://mlsanity.com/paper/2403.16387/) (2024)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [Detecting Everything in the Open World: Towards Universal Object Detection](https://mlsanity.com/paper/2303.11749/) (2023)
- [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://mlsanity.com/paper/2309.09668/) (2023)
- [CDUL: CLIP-Driven Unsupervised Learning for Multi-Label Image Classification](https://mlsanity.com/paper/2307.16634/) (2023)
- [Borrow from Anywhere: Pseudo Multi-modal Object Detection in Thermal Imagery](https://mlsanity.com/paper/1905.08789/) (2019)
- [CLIP-Driven Semantic Discovery Network for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2401.05806/) (2024)
- [TFDet: Target-Aware Fusion for RGB-T Pedestrian Detection](https://mlsanity.com/paper/2305.16580/) (2023)
- [Evaluation of Thermal Imaging on Embedded GPU Platforms for Application in Vehicular Assistance Systems](https://mlsanity.com/paper/2201.01661/) (2022)
- [Robust Environment Perception for Automated Driving: A Unified Learning Pipeline for Visual-Infrared Object Detection](https://mlsanity.com/paper/2206.03943/) (2022)

## Built on by (most cited first, 9 of 9)

- [RASMD: RGB And SWIR Multispectral Driving Dataset for Robust Perception in Adverse Conditions](https://mlsanity.com/paper/2504.07603/) (2025)
- [YOLOv11-RGBT: Towards a Comprehensive Single-Stage Multispectral Object Detection Framework](https://mlsanity.com/paper/2506.14696/) (2025)
- [On Modality Incomplete Infrared-Visible Object Detection: An Architecture Compatibility Perspective](https://mlsanity.com/paper/2511.06406/) (2025)
- [MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters](https://mlsanity.com/paper/2512.00363/) (2025)
- [Modality-Decoupled RGB-Thermal Object Detector via Query Fusion](https://mlsanity.com/paper/2601.08458/) (2026)
- [DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding](https://mlsanity.com/paper/2603.11380/) (2026)
- [UAV-CB: A Complex-Background RGB-T Dataset and Local Frequency Bridge Network for UAV Detection](https://mlsanity.com/paper/2603.17492/) (2026)
- [Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion](https://mlsanity.com/paper/2606.30215/) (2026)
- [Aligned Radiometric RGB-Thermal Fusion for UAV Facade Anomaly Screening](https://mlsanity.com/paper/2609.12521/) (2026)

## Abstract (arXiv)

Most recent multispectral object detectors employ a two-branch structure to extract features from RGB and thermal images. While the two-branch structure achieves better performance than a single-branch structure, it overlooks inference efficiency. This conflict is increasingly aggressive, as recent works solely pursue higher performance rather than both performance and efficiency. In this paper, we address this issue by improving the performance of efficient single-branch structures. We revisit the reasons causing the performance gap between these structures. For the first time, we reveal the information interference problem in the naive early-fusion strategy adopted by previous single-branch structures. Besides, we find that the domain gap between multispectral images, and weak feature representation of the single-branch structure are also key obstacles for performance. Focusing on these three problems, we propose corresponding solutions, including a novel shape-priority early-fusion strategy, a weakly supervised learning method, and a core knowledge distillation technique. Experiments demonstrate that single-branch networks equipped with these three contributions achieve significant performance enhancements while retaining high efficiency. Our code will be available at urlhttps://github.com/XueZ-phd/Efficient-RGB-T-Early-Fusion-Detection.