# UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning

arXiv: https://arxiv.org/abs/2404.17360 · HTML: https://mlsanity.com/paper/2404.17360/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 87 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [LLVIP: A Visible-infrared Paired Dataset for Low-light Vision](https://mlsanity.com/paper/2108.10831/) (2021)
- [A Survey of the Recent Architectures of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1901.06032/) (2019)
- [ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation](https://mlsanity.com/paper/1905.10089/) (2019)
- [Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks](https://mlsanity.com/paper/2009.12664/) (2020)
- [Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems](https://mlsanity.com/paper/2008.03043/) (2020)
- [Dense Distinct Query for End-to-End Object Detection](https://mlsanity.com/paper/2303.12776/) (2023)
- [ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection](https://mlsanity.com/paper/2308.07504/) (2023)
- [Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection](https://mlsanity.com/paper/1803.05347/) (2018)
- [RGBT Salient Object Detection: A Large-scale Dataset and Benchmark](https://mlsanity.com/paper/2007.03262/) (2020)
- [Siamese Network for RGB-D Salient Object Detection and Beyond](https://mlsanity.com/paper/2008.12134/) (2020)
- [ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions](https://mlsanity.com/paper/2403.07392/) (2024)
- [RGB-T Image Saliency Detection via Collaborative Graph Learning](https://mlsanity.com/paper/1905.06741/) (2019)
- [Fusion of Multispectral Data Through Illumination-aware Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1802.09972/) (2018)
- [Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting](https://mlsanity.com/paper/2012.04529/) (2020)

## Built on by (most cited first, 17 of 17)

- [LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs](https://mlsanity.com/paper/2507.00505/) (2025)
- [Seeing Through the Noise: Improving Infrared Small Target Detection and Segmentation from Noise Suppression Perspective](https://mlsanity.com/paper/2508.06878/) (2025)
- [Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2509.20107/) (2025)
- [Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detection](https://mlsanity.com/paper/2507.14643/) (2025)
- [SLGNet: Synergizing Structural Priors and Language-Guided Modulation for Multimodal Object Detection](https://mlsanity.com/paper/2601.02249/) (2026)
- [RGBT-GroundBench: Visual Grounding Beyond RGB in Complex Real-World Scenarios](https://mlsanity.com/paper/2512.24561/) (2025)
- [IRDFusion: Iterative Relation-Map Difference guided Feature Fusion for Multispectral Object Detection](https://mlsanity.com/paper/2509.09085/) (2025)
- [Representation Space Constrained Learning with Modality Decoupling for Multimodal Object Detection](https://mlsanity.com/paper/2511.15433/) (2025)
- [Modality Dominance-Aware Optimization for Embodied RGB-Infrared Perception](https://mlsanity.com/paper/2601.00598/) (2026)
- [Breaking Self-Attention Failure: Rethinking Query Initialization for Infrared Small Target Detection](https://mlsanity.com/paper/2601.02837/) (2026)
- [Unified Restoration-Perception Learning: Maritime Infrared-Visible Image Fusion and Segmentation](https://mlsanity.com/paper/2603.28414/) (2026)
- [Towards All-Day Perception for Off-Road Driving: A Large-Scale Multispectral Dataset and Comprehensive Benchmark](https://mlsanity.com/paper/2604.27499/) (2026)
- [SpectraDINO: Modality-Conditioned Adaptation of RGB Vision Foundation Models Across Infrared Bands](https://mlsanity.com/paper/2605.02258/) (2026)
- [Dual Sparse Aggregation Transformer for Multispectral Object Detection](https://mlsanity.com/paper/2606.31015/) (2026)
- [RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection](https://mlsanity.com/paper/2608.04833/) (2026)
- [InterPruner: Interactive Structured Pruning via Taylor-Implicit Criterion and Language-Prior Modulator for Multimodal Object Detection](https://mlsanity.com/paper/2608.10724/) (2026)
- [IRSTD-Agent: Agentic Infrared Small Target Detection via Zoom-Guided Interaction Learning](https://mlsanity.com/paper/2610.05342/) (2026)

## Abstract (arXiv)

Semantic analysis on visible (RGB) and infrared (IR) images has gained significant attention due to their enhanced accuracy and robustness under challenging conditions including low-illumination and adverse weather. However, due to the lack of pre-trained foundation models on the large-scale infrared image datasets, existing methods prefer to design task-specific frameworks and directly fine-tune them with pre-trained foundation models on their RGB-IR semantic relevance datasets, which results in poor scalability and limited generalization. To address these limitations, we propose UniRGB-IR, a scalable and efficient framework for RGB-IR semantic tasks that introduces a novel adapter mechanism to effectively incorporate rich multi-modal features into pre-trained RGB-based foundation models. Our framework comprises three key components: a vision transformer (ViT) foundation model, a Multi-modal Feature Pool (MFP) module, and a Supplementary Feature Injector (SFI) module. The MFP and SFI modules cooperate with each other as an adpater to effectively complement the ViT features with the contextual multi-scale features. During training process, we freeze the entire foundation model to inherit prior knowledge and only optimize the MFP and SFI modules. Furthermore, to verify the effectiveness of our framework, we utilize the ViT-Base as the pre-trained foundation model to perform extensive experiments. Experimental results on various RGB-IR semantic tasks demonstrate that our method can achieve state-of-the-art performance.