# Weakly Aligned Feature Fusion for Multimodal Object Detection

arXiv: https://arxiv.org/abs/2204.09848 · HTML: https://mlsanity.com/paper/2204.09848/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 79 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 30 of 32)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [CityPersons: A Diverse Dataset for Pedestrian Detection](https://mlsanity.com/paper/1702.05693/) (2017)
- [Learning Rich Features from RGB-D Images for Object Detection and Segmentation](https://mlsanity.com/paper/1407.5736/) (2014)
- [Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images](https://mlsanity.com/paper/1511.02300/) (2015)
- [Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks](https://mlsanity.com/paper/2009.12664/) (2020)
- [Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems](https://mlsanity.com/paper/2008.03043/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection](https://mlsanity.com/paper/1803.05347/) (2018)
- [Repulsion Loss: Detecting Pedestrians in a Crowd](https://mlsanity.com/paper/1711.07752/) (2017)
- [Occlusion-aware R-CNN: Detecting Pedestrians in a Crowd](https://mlsanity.com/paper/1807.08407/) (2018)
- [Multimodal Deep Learning for Robust RGB-D Object Recognition](https://mlsanity.com/paper/1507.06821/) (2015)
- [Fusion of Multispectral Data Through Illumination-aware Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1802.09972/) (2018)
- [Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1808.04818/) (2018)
- [Learning Cross-Modal Deep Representations for Robust Pedestrian Detection](https://mlsanity.com/paper/1704.02431/) (2017)
- [TJU-DHD: A Diverse High-Resolution Dataset for Object Detection](https://mlsanity.com/paper/2011.09170/) (2020)
- [What Can Help Pedestrian Detection?](https://mlsanity.com/paper/1705.02757/) (2017)
- [Amodal Completion and Size Constancy in Natural Scenes](https://mlsanity.com/paper/1509.08147/) (2015)
- [Taking a Deeper Look at Pedestrians](https://mlsanity.com/paper/1501.05790/) (2015)
- [Filtered Channel Features for Pedestrian Detection](https://mlsanity.com/paper/1501.05759/) (2015)
- [Box-level Segmentation Supervised Deep Neural Networks for Accurate and Real-time Multispectral Pedestrian Detection](https://mlsanity.com/paper/1902.05291/) (2019)

## Built on by (most cited first, 26 of 26)

- [C2Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection](https://mlsanity.com/paper/2306.16175/) (2023)
- [DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion](https://mlsanity.com/paper/2403.00326/) (2024)
- [DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensing](https://mlsanity.com/paper/2407.08132/) (2024)
- [TFDet: Target-Aware Fusion for RGB-T Pedestrian Detection](https://mlsanity.com/paper/2305.16580/) (2023)
- [UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning](https://mlsanity.com/paper/2404.17360/) (2024)
- [PedDet: Adaptive Spectral Optimization for Multimodal Pedestrian Detection](https://mlsanity.com/paper/2502.14063/) (2025)
- [Fusion Meets Diverse Conditions: A High-diversity Benchmark and Baseline for UAV-based Multimodal Object Detection with Condition Cues](https://mlsanity.com/paper/2510.13620/) (2025)
- [DPDETR: Decoupled Position Detection Transformer for Infrared-Visible Object Detection](https://mlsanity.com/paper/2408.06123/) (2024)
- [Mixture of Scale Experts for Alignment-free RGBT Video Object Detection and A Unified Benchmark](https://mlsanity.com/paper/2410.12143/) (2024)
- [Cross-modal Offset-guided Dynamic Alignment and Fusion for Weakly Aligned UAV Object Detection](https://mlsanity.com/paper/2506.16737/) (2025)
- [YOLOv11-RGBT: Towards a Comprehensive Single-Stage Multispectral Object Detection Framework](https://mlsanity.com/paper/2506.14696/) (2025)
- [COXNet: Cross-Layer Fusion with Adaptive Alignment and Scale Integration for RGBT Tiny Object Detection](https://mlsanity.com/paper/2508.09533/) (2025)
- [DEPFusion: Dual-Domain Enhancement and Priority-Guided Mamba Fusion for UAV Multispectral Object Detection](https://mlsanity.com/paper/2509.07327/) (2025)
- [FoRA: Low-Rank Adaptation Model beyond Multimodal Siamese Network](https://mlsanity.com/paper/2407.16129/) (2024)
- [DOD-SA: Infrared-Visible Decoupled Object Detection with Single-Modality Annotations](https://mlsanity.com/paper/2508.10445/) (2025)
- [Unifying Heterogeneous Multi-Modal Remote Sensing Detection Via Language-Pivoted Pretraining](https://mlsanity.com/paper/2603.01758/) (2026)
- [DLRMamba: Distilling Low-Rank Mamba for Edge Multispectral Fusion Object Detection](https://mlsanity.com/paper/2603.06920/) (2026)
- [Multimodal Object Detection via Probabilistic a priori Information Integration](https://mlsanity.com/paper/2405.15596/) (2024)
- [Large Language Model Guided Progressive Feature Alignment for Multimodal UAV Object Detection](https://mlsanity.com/paper/2503.06948/) (2025)
- [Transformer-Based Dual-Optical Attention Fusion Crowd Head Point Counting and Localization Network](https://mlsanity.com/paper/2505.06937/) (2025)
- [LASFNet: A Lightweight Attention-Guided Self-Modulation Feature Fusion Network for Multimodal Object Detection](https://mlsanity.com/paper/2506.21018/) (2025)
- [DGE-YOLO: Dual-Branch Gathering and Attention for Accurate UAV Object Detection](https://mlsanity.com/paper/2506.23252/) (2025)
- [HiddenObject: Modality-Agnostic Fusion for Multimodal Hidden Object Detection](https://mlsanity.com/paper/2508.21135/) (2025)
- [MO R-CNN: Multispectral Oriented R-CNN for Object Detection in Remote Sensing Image](https://mlsanity.com/paper/2509.16957/) (2025)
- [Dual Sparse Aggregation Transformer for Multispectral Object Detection](https://mlsanity.com/paper/2606.31015/) (2026)
- [Bridging Severe Cross-Modal Misalignment: End-to-End Visible-Infrared Object Detection via Explicit Feature-Domain Affine Registration](https://mlsanity.com/paper/2608.10680/) (2026)

## Abstract (arXiv)

To achieve accurate and robust object detection in the real-world scenario, various forms of images are incorporated, such as color, thermal, and depth. However, multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned, making one object has different positions in different modalities. For the deep learning method, this problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training. In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the position shift problem. First, a region feature (RF) alignment module with adjacent similarity constraint is designed to consistently predict the position shift between two modalities and adaptively align the cross-modal RFs. Second, we propose a novel region of interest (RoI) jitter strategy to improve the robustness to unexpected shift patterns. Third, we present a new multimodal feature fusion method that selects the more reliable feature and suppresses the less useful one via feature reweighting. In addition, by locating bounding boxes in both modalities and building their relationships, we provide novel multimodal labeling named KAIST-Paired. Extensive experiments on 2-D and 3-D object detection, RGB-T, and RGB-D datasets demonstrate the effectiveness and robustness of our method.