# Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images

arXiv: https://arxiv.org/abs/2310.13876 · HTML: https://mlsanity.com/paper/2310.13876/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 21 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Domain Separation Networks](https://mlsanity.com/paper/1608.06019/) (2016)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges](https://mlsanity.com/paper/2102.12219/) (2021)
- [Cross-Modality Fusion Transformer for Multispectral Object Detection](https://mlsanity.com/paper/2111.00273/) (2021)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion](https://mlsanity.com/paper/2103.11832/) (2021)
- [Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark](https://mlsanity.com/paper/2206.13996/) (2022)
- [Hyperspectral Image Classification-Traditional to Deep Models: A Survey for Future Prospects](https://mlsanity.com/paper/2101.06116/) (2021)
- [SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery](https://mlsanity.com/paper/2209.13351/) (2022)
- [Modality Compensation Network: Cross-Modal Adaptation for Action Recognition](https://mlsanity.com/paper/2001.11657/) (2020)
- [ViTransPAD: Video Transformer using convolution and self-attention for Face Presentation Attack Detection](https://mlsanity.com/paper/2203.01562/) (2022)
- [Incomplete Multimodal Learning for Remote Sensing Data Fusion](https://mlsanity.com/paper/2304.11381/) (2023)

## Built on by (most cited first, 2 of 2)

- [Indirect Attention: Turning Context Misalignment into a Feature](https://mlsanity.com/paper/2509.26015/) (2025)
- [Towards a Generalizable Fusion Architecture for Multimodal Object Detection](https://mlsanity.com/paper/2510.17078/) (2025)

## Abstract (arXiv)

Object detection in Remote Sensing Images (RSI) is a critical task for numerous applications in Earth Observation (EO). Differing from object detection in natural images, object detection in remote sensing images faces challenges of scarcity of annotated data and the presence of small objects represented by only a few pixels. Multi-modal fusion has been determined to enhance the accuracy by fusing data from multiple modalities such as RGB, infrared (IR), lidar, and synthetic aperture radar (SAR). To this end, the fusion of representations at the mid or late stage, produced by parallel subnetworks, is dominant, with the disadvantages of increasing computational complexity in the order of the number of modalities and the creation of additional engineering obstacles. Using the cross-attention mechanism, we propose a novel multi-modal fusion strategy for mapping relationships between different channels at the early stage, enabling the construction of a coherent input by aligning the different modalities. By addressing fusion in the early stage, as opposed to mid or late-stage methods, our method achieves competitive and even superior performance compared to existing techniques. Additionally, we enhance the SWIN transformer by integrating convolution layers into the feed-forward of non-shifting blocks. This augmentation strengthens the model's capacity to merge separated windows through local attention, thereby improving small object detection. Extensive experiments prove the effectiveness of the proposed multimodal fusion module and the architecture, demonstrating their applicability to object detection in multimodal aerial imagery.