# MambaTrans: Multimodal Fusion Image Translation via Large Language Model Priors for Downstream Visual Tasks

arXiv: https://arxiv.org/abs/2508.07803 · HTML: https://mlsanity.com/paper/2508.07803/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 28 of 28)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [YOLOv11: An Overview of the Key Architectural Enhancements](https://mlsanity.com/paper/2410.17725/) (2024)
- [MambaIR: A Simple Baseline for Image Restoration with State-Space Model](https://mlsanity.com/paper/2402.15648/) (2024)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion](https://mlsanity.com/paper/2211.10960/) (2022)
- [A Task-guided, Implicitly-searched and Meta-initialized Deep Model for Image Fusion](https://mlsanity.com/paper/2305.15862/) (2023)
- [Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption](https://mlsanity.com/paper/2501.10761/) (2025)
- [Source-Free Open Compound Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2106.03422/) (2021)
- [Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond](https://mlsanity.com/paper/2503.01210/) (2025)
- [DCEvo: Discriminative Cross-Dimensional Evolutionary Learning for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2503.17673/) (2025)
- [FastInst: A Simple Query-Based Model for Real-Time Instance Segmentation](https://mlsanity.com/paper/2303.08594/) (2023)
- [All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark](https://mlsanity.com/paper/2402.02090/) (2024)
- [Simultaneous Tri-Modal Medical Image Fusion and Super-Resolution using Conditional Diffusion Model](https://mlsanity.com/paper/2404.17357/) (2024)
- [TSJNet: A Multi-modality Target and Semantic Awareness Joint-driven Image Fusion Network](https://mlsanity.com/paper/2402.01212/) (2024)
- [HalluciDet: Hallucinating RGB Modality for Person Detection Through Privileged Information](https://mlsanity.com/paper/2310.04662/) (2023)
- [Modality Translation for Object Detection Adaptation Without Forgetting Prior Knowledge](https://mlsanity.com/paper/2404.01492/) (2024)

## Built on by (most cited first, 1 of 1)

- [InfraNet: Quality-Aware RGB Guidance for Efficient Infrared Object Detection](https://mlsanity.com/paper/2607.03795/) (2026)

## Abstract (arXiv)

The goal of multimodal image fusion is to integrate complementary information from infrared and visible images, generating multimodal fused images for downstream tasks. Existing downstream pre-training models are typically trained on visible images. However, the significant pixel distribution differences between visible and multimodal fusion images can degrade downstream task performance, sometimes even below that of using only visible images. This paper explores adapting multimodal fused images with significant modality differences to object detection and semantic segmentation models trained on visible images. To address this, we propose MambaTrans, a novel multimodal fusion image modality translator. MambaTrans uses descriptions from a multimodal large language model and masks from semantic segmentation models as input. Its core component, the Multi-Model State Space Block, combines mask-image-text cross-attention and a 3D-Selective Scan Module, enhancing pure visual capabilities. By leveraging object detection prior knowledge, MambaTrans minimizes detection loss during training and captures long-term dependencies among text, masks, and images. This enables favorable results in pre-trained models without adjusting their parameters. Experiments on public datasets show that MambaTrans effectively improves multimodal image performance in downstream tasks.