# Cross-Modality Fusion Transformer for Multispectral Object Detection

arXiv: https://arxiv.org/abs/2111.00273 · HTML: https://mlsanity.com/paper/2111.00273/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 30 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Multi-modal Transformer for Video Retrieval](https://mlsanity.com/paper/2007.10639/) (2020)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [LLVIP: A Visible-infrared Paired Dataset for Low-light Vision](https://mlsanity.com/paper/2108.10831/) (2021)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Learning to Generate Diverse Dance Motions with Transformer](https://mlsanity.com/paper/2008.08171/) (2020)
- [Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks](https://mlsanity.com/paper/2009.12664/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [MDMMT: Multidomain Multimodal Transformer for Video Retrieval](https://mlsanity.com/paper/2103.10699/) (2021)
- [Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection](https://mlsanity.com/paper/1803.05347/) (2018)
- [Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds](https://mlsanity.com/paper/2011.01143/) (2020)
- [Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1808.04818/) (2018)
- [GFD-SSD: Gated Fusion Double SSD for Multispectral Pedestrian Detection](https://mlsanity.com/paper/1903.06999/) (2019)
- [SSSDET: Simple Short and Shallow Network for Resource Efficient Vehicle Detection in Aerial Scenes](https://mlsanity.com/paper/1909.00292/) (2019)
- [FFAVOD: Feature Fusion Architecture for Video Object Detection](https://mlsanity.com/paper/2109.07298/) (2021)

## Built on by (most cited first, 30 of 86)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection](https://mlsanity.com/paper/2308.07504/) (2023)
- [Fusion-Mamba for Cross-modality Object Detection](https://mlsanity.com/paper/2404.09146/) (2024)
- [DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion](https://mlsanity.com/paper/2403.00326/) (2024)
- [E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection](https://mlsanity.com/paper/2403.09323/) (2024)
- [Cross-Modality High-Frequency Transformer for MR Image Super-Resolution](https://mlsanity.com/paper/2203.15314/) (2022)
- [Emergent Visual Sensors for Autonomous Vehicles](https://mlsanity.com/paper/2205.09383/) (2022)
- [COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection](https://mlsanity.com/paper/2412.18076/) (2024)
- [Removal then Selection: A Coarse-to-Fine Fusion Perspective for RGB-Infrared Object Detection](https://mlsanity.com/paper/2401.10731/) (2024)
- [RGB-X Object Detection via Scene-Specific Fusion Modules](https://mlsanity.com/paper/2310.19372/) (2023)
- [MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization](https://mlsanity.com/paper/2302.00290/) (2023)
- [WaveMamba: Wavelet-Driven Mamba Fusion for RGB-Infrared Object Detection](https://mlsanity.com/paper/2507.18173/) (2025)
- [Unsupervised RGB-to-Thermal Domain Adaptation via Multi-Domain Attention Network](https://mlsanity.com/paper/2210.04367/) (2022)
- [Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detection](https://mlsanity.com/paper/2403.01300/) (2024)
- [Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching](https://mlsanity.com/paper/2301.00131/) (2022)
- [Weakly Supervised Video Representation Learning with Unaligned Text for Sequential Videos](https://mlsanity.com/paper/2303.12370/) (2023)
- [When Pedestrian Detection Meets Multi-Modal Learning: Generalist Model and Benchmark Dataset](https://mlsanity.com/paper/2407.10125/) (2024)
- [Cross-Modal Fusion and Attention Mechanism for Weakly Supervised Video Anomaly Detection](https://mlsanity.com/paper/2412.20455/) (2024)
- [Pedestrian Detection in Low-Light Conditions: A Comprehensive Survey](https://mlsanity.com/paper/2401.07801/) (2024)
- [CFMW: Cross-modality Fusion Mamba for Robust Object Detection under Adverse Weather](https://mlsanity.com/paper/2404.16302/) (2024)
- [M-SpecGene: Generalized Foundation Model for RGBT Multispectral Vision](https://mlsanity.com/paper/2507.16318/) (2025)
- [MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields](https://mlsanity.com/paper/2302.02978/) (2023)
- [MambaST: A Plug-and-Play Cross-Spectral Spatial-Temporal Fuser for Efficient Pedestrian Detection](https://mlsanity.com/paper/2408.01037/) (2024)
- [Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detection](https://mlsanity.com/paper/2507.14643/) (2025)
- [MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection](https://mlsanity.com/paper/2404.18849/) (2024)
- [DPDETR: Decoupled Position Detection Transformer for Infrared-Visible Object Detection](https://mlsanity.com/paper/2408.06123/) (2024)
- [C3-OWD: A Curriculum Cross-modal Contrastive Learning Framework for Open-World Detection](https://mlsanity.com/paper/2509.23316/) (2025)
- [SeaDATE: Remedy Dual-Attention Transformer with Semantic Alignment via Contrast Learning for Multimodal Object Detection](https://mlsanity.com/paper/2410.11358/) (2024)
- [Mixture of Scale Experts for Alignment-free RGBT Video Object Detection and A Unified Benchmark](https://mlsanity.com/paper/2410.12143/) (2024)
- [Optimizing Multispectral Object Detection: A Bag of Tricks and Comprehensive Benchmarks](https://mlsanity.com/paper/2411.18288/) (2024)

## Abstract (arXiv)

Multispectral image pairs can provide the combined information, making object detection applications more reliable and robust in the open world. To fully exploit the different modalities, we present a simple yet effective cross-modality feature fusion approach, named Cross-Modality Fusion Transformer (CFT) in this paper. Unlike prior CNNs-based works, guided by the transformer scheme, our network learns long-range dependencies and integrates global contextual information in the feature extraction stage. More importantly, by leveraging the self attention of the transformer, the network can naturally carry out simultaneous intra-modality and inter-modality fusion, and robustly capture the latent interactions between RGB and Thermal domains, thereby significantly improving the performance of multispectral object detection. Extensive experiments and ablation studies on multiple datasets demonstrate that our approach is effective and achieves state-of-the-art detection performance. Our code and models are available at https://github.com/DocF/multispectral-object-detection.