# MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters

arXiv: https://arxiv.org/abs/2512.00363 · HTML: https://mlsanity.com/paper/2512.00363/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Point clouds, detection and remote sensing, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 29 Nov 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Point clouds, detection and remote sensing](https://mlsanity.com/ssm/point-clouds-detection-and-remote-sensing/)

## Builds on (most cited first, 30 of 32)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning](https://mlsanity.com/paper/2003.02437/) (2020)
- [Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks](https://mlsanity.com/paper/2009.12664/) (2020)
- [DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2304.06648/) (2023)
- [Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems](https://mlsanity.com/paper/2008.03043/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration](https://mlsanity.com/paper/2205.11876/) (2022)
- [ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection](https://mlsanity.com/paper/2308.07504/) (2023)
- [C2Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection](https://mlsanity.com/paper/2306.16175/) (2023)
- [SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery](https://mlsanity.com/paper/2209.13351/) (2022)
- [Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery](https://mlsanity.com/paper/2112.02991/) (2021)
- [5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks](https://mlsanity.com/paper/2408.08345/) (2024)
- [DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion](https://mlsanity.com/paper/2403.00326/) (2024)
- [Mixture of Nested Experts: Adaptive Processing of Visual Tokens](https://mlsanity.com/paper/2407.19985/) (2024)
- [SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection](https://mlsanity.com/paper/2412.20665/) (2024)
- [MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization](https://mlsanity.com/paper/2302.00290/) (2023)
- [Vision Transformer Adapters for Generalizable Multitask Learning](https://mlsanity.com/paper/2308.12372/) (2023)
- [WaveMamba: Wavelet-Driven Mamba Fusion for RGB-Infrared Object Detection](https://mlsanity.com/paper/2507.18173/) (2025)
- [Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection](https://mlsanity.com/paper/2405.16038/) (2024)

## Abstract (arXiv)

Multimodal remote sensing object detection aims to achieve more accurate and robust perception under challenging conditions by fusing complementary information from different modalities. However, existing approaches that rely on attention-based or deformable convolution fusion blocks still struggle to balance performance and lightweight design. Beyond fusion complexity, extracting modality features with shared backbones yields suboptimal representations due to insufficient modality-specific modeling, whereas dual-stream architectures nearly double the parameter count, ultimately limiting practical deployment. To this end, we propose MM-DETR, a lightweight and efficient framework for multimodal object detection. Specifically, we propose a Mamba-based dual granularity fusion encoder that reformulates global interaction as channel-wise dynamic gating and leverages a 1D selective scan for efficient cross-modal modeling with linear complexity. Following this design, we further reinterpret multimodal fusion as a modality completion problem. A region-aware 2D selective scanning completion branch is introduced to recover modality-specific cues, supporting fine-grained fusion along a bidirectional pyramid pathway with minimal overhead. To further reduce parameter redundancy while retaining strong feature extraction capability, a lightweight frequency-aware modality adapter is inserted into the shared backbone. This adapter employs a spatial-frequency co-expert structure to capture modality-specific cues, while a pixel-wise router dynamically balances expert contributions for efficient spatial-frequency fusion. Extensive experiments conducted on four multimodal benchmark datasets demonstrate the effectiveness and generalization capability of the proposed method.