# RGB-X Object Detection via Scene-Specific Fusion Modules

arXiv: https://arxiv.org/abs/2310.19372 · HTML: https://mlsanity.com/paper/2310.19372/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 30 Oct 2023.

## Builds on (most cited first, 29 of 29)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges](https://mlsanity.com/paper/1902.07830/) (2019)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation](https://mlsanity.com/paper/1711.10871/) (2017)
- [Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning](https://mlsanity.com/paper/2003.02437/) (2020)
- [Cross-Modality Fusion Transformer for Multispectral Object Detection](https://mlsanity.com/paper/2111.00273/) (2021)
- [Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks](https://mlsanity.com/paper/2009.12664/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [SA-Net: Shuffle Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2102.00240/) (2021)
- [Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks](https://mlsanity.com/paper/2303.15710/) (2023)
- [BM-NAS: Bilevel Multimodal Neural Architecture Search](https://mlsanity.com/paper/2104.09379/) (2021)
- [Unsupervised RGB-to-Thermal Domain Adaptation via Multi-Domain Attention Network](https://mlsanity.com/paper/2210.04367/) (2022)
- [Online Self-Supervised Thermal Water Segmentation for Aerial Vehicles](https://mlsanity.com/paper/2307.09027/) (2023)

## Built on by (most cited first, 15 of 15)

- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [Caltech Aerial RGB-Thermal Dataset in the Wild](https://mlsanity.com/paper/2403.08997/) (2024)
- [Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision](https://mlsanity.com/paper/2411.18025/) (2024)
- [MonoTher-Depth: Enhancing Thermal Depth Estimation via Confidence-Aware Distillation](https://mlsanity.com/paper/2504.16127/) (2025)
- [IV-tuning: Parameter-Efficient Transfer Learning for Infrared-Visible Tasks](https://mlsanity.com/paper/2412.16654/) (2024)
- [IVGF: The Fusion-Guided Infrared and Visible General Framework](https://mlsanity.com/paper/2409.00973/) (2024)
- [OSSA: Unsupervised One-Shot Style Adaptation](https://mlsanity.com/paper/2410.00900/) (2024)
- [VLC Fusion: Vision-Language Conditioned Sensor Fusion for Robust Object Detection](https://mlsanity.com/paper/2505.12715/) (2025)
- [No Calibration, No Depth, No Problem: Cross-Sensor View Synthesis with 3D Consistency](https://mlsanity.com/paper/2602.23559/) (2026)
- [SpectraDINO: Modality-Conditioned Adaptation of RGB Vision Foundation Models Across Infrared Bands](https://mlsanity.com/paper/2605.02258/) (2026)
- [Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion](https://mlsanity.com/paper/2606.30215/) (2026)
- [Descriptor: LYNRED Mobility Dataset Multimodal Detection Subset (LYNRED-MDS)](https://mlsanity.com/paper/2607.01871/) (2026)
- [InfraNet: Quality-Aware RGB Guidance for Efficient Infrared Object Detection](https://mlsanity.com/paper/2607.03795/) (2026)
- [LDFE: Laplacian Decoupled Feature Enhancement Block for Dual-Stream CNN-based RGB-IR Object Detection](https://mlsanity.com/paper/2607.08076/) (2026)
- [RSFusionDet: Underwater RGB-Sonar Multimodal Object Detection](https://mlsanity.com/paper/2608.25367/) (2026)

## Abstract (arXiv)

Multimodal deep sensor fusion has the potential to enable autonomous vehicles to visually understand their surrounding environments in all weather conditions. However, existing deep sensor fusion methods usually employ convoluted architectures with intermingled multimodal features, requiring large coregistered multimodal datasets for training. In this work, we present an efficient and modular RGB-X fusion network that can leverage and fuse pretrained single-modal models via scene-specific fusion modules, thereby enabling joint input-adaptive network architectures to be created using small, coregistered multimodal datasets. Our experiments demonstrate the superiority of our method compared to existing works on RGB-thermal and RGB-gated datasets, performing fusion using only a small amount of additional parameters. Our code is available at https://github.com/dsriaditya999/RGBXFusion.