mlsanityData to 7 Oct 2026

M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension

arXiv 2407.01131 ↗First posted 1 Jul 2024cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Referring expression comprehension (REC) is a vision-language task to locate a target object in an image based on a language expression. Fully fine-tuning general-purpose pre-trained vision-language foundation models for REC yields impressive performance but becomes increasingly costly. Parameter-efficient transfer learning (PETL) methods have shown strong performance with fewer tunable parameters. However, directly applying PETL to REC faces two challenges: (1) insufficient multi-modal interaction between pre-trained vision-language foundation models, and (2) high GPU memory usage due to gradients passing through the heavy vision-language foundation models. To this end, we present M2IST: Multi-Modal Interactive Side-Tuning with M3ISAs: Mixture of Multi-Modal Interactive Side-Adapters. During fine-tuning, we fix the pre-trained uni-modal encoders and update M3ISAs to enable efficient vision-language alignment for REC. Empirical results reveal that M2IST achieves better performance-efficiency trade-off than full fine-tuning and other PETL methods, requiring only 2.11% tunable parameters, 39.61% GPU memory, and 63.46% training time while maintaining competitive performance. Our code is released at https://github.com/xuyang-liu16/M2IST.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 74 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

53 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2018YOLOv3: An Incremental Improvementcited by 2,652
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2016Modeling Context in Referring Expressionscited by 1,054
38 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  3. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  4. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  5. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  6. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  7. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  8. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 194
  9. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  10. 2018Learning to Assemble Neural Module Tree Networks for Visual Groundingcited by 139
  11. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  12. 2020Improving One-stage Visual Grounding by Recursive Sub-query Constructioncited by 137
  13. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  14. 2019A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehensioncited by 111
  15. 2019Learning to Compose and Reason with Language Tree Structures for Visual Groundingcited by 98
  16. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  17. 2017Grounding Referring Expressions in Images by Variational Contextcited by 82
  18. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  19. 2023Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentationcited by 61
  20. 2017Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queriescited by 56
  21. 2024Explicit Visual Prompts for Visual Object Trackingcited by 50
  22. 2019Referring Expression Object Segmentation with Caption-Aware Consistencycited by 46
  23. 2023Prompt-based Distribution Alignment for Unsupervised Domain Adaptationcited by 44
  24. 2021Visual Grounding with Transformerscited by 42
  25. 2022VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrievalcited by 36
  26. 2022Cross-Modal Adapter for Vision-Language Retrievalcited by 35
  27. 2019A Real-time Global Inference Network for One-stage Referring Expression Comprehensioncited by 32
  28. 2022Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Groundingcited by 29
  29. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  30. 2023Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learningcited by 26
  31. 2023Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictionscited by 24
  32. 2023Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrievalcited by 24
  33. 2023DTL: Disentangled Transfer Learning for Visual Recognitioncited by 22
  34. 2023VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounderscited by 18
  35. 2024RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adaptercited by 16
  36. 2024Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inferencecited by 12
  37. 2024Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Modelscited by 9
  38. 2024MEFT: Memory-Efficient Fine-Tuning through Sparse Adaptercited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 1.

11 papers.

  1. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Taskscited by 27
  2. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 25
  3. 2025Twin Co-Adaptive Dialogue for Progressive Image Generationcited by 11
  4. 2024ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentationcited by 8
  5. 2025SwimVG: Step-wise Multimodal Fusion and Adaption for Visual Groundingcited by 8
  6. 2025TR-PTS: Task-Relevant Parameter and Token Selection for Efficient Tuningcited by 6
  7. 2025Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Modelscited by 4
  8. 2025DMPT: Decoupled Modality-aware Prompt Tuning for Multi-modal Object Re-identificationcited by 4
  9. 2025TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generationcited by 1
  10. 2026BARE: Towards Bias-Aware and Reasoning-Enhanced One-Tower Visual Groundingcited by 1
  11. 2025Robust RGB-T Tracking via Learnable Visual Fourier Prompt Fine-tuning and Modality Fusion Prompt Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension
  2. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
  3. 2024LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension
  4. 2026Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks
  5. 2025KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge
  6. 2026Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker
  7. 2026Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehension
  8. 2023Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Models