mlsanityData to 7 Oct 2026

ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models

arXiv 2407.21534 ↗First posted 31 Jul 2024cs.CVcited by 33

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this work, we propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs) through test-time optimization of a learnable latent variable. We observe that attention, as the core module of MLLMs, connects text prompt tokens and visual tokens, ultimately determining the final results. Our approach involves adjusting visual tokens from the MLP output at test time, controlling the attention response to ensure text prompt tokens attend to visual tokens in referring regions. We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map. This enables detailed region description and reasoning without the need for substantial training costs or model retraining. Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point. The results demonstrate that our method exhibits out-of-domain generalization and interpretability.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 82 references that are in the map. First posted on arXiv 31 Jul 2024.

Builds on

48 of 82 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models17 Apr
  2. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  3. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  4. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models5 Mar
  5. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception5 Mar
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  7. 2023Osprey: Pixel Understanding with Visual Instruction Tuning15 Dec
  8. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMs7 Dec
  9. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts1 Dec
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  11. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V17 Oct
  12. 2023Improved Baselines with Visual Instruction Tuning5 Oct
6 more
  1. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs1 Oct
  2. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models25 Aug
  3. 2023RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension3 Aug
  4. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  9. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  10. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  11. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  12. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
18 more
  1. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  2. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  3. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  4. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  5. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  6. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  7. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  8. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  9. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  10. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  11. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  12. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  13. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  14. 2024ChatterBox: Multi-round Multimodal Referring and Groundingcited by 24
  15. 2024LVLM-Interpret: An Interpretability Tool for Large Vision-Language Modelscited by 23
  16. 2023Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objectscited by 17
  17. 2023Pixel Aligned Language Modelscited by 17
  18. 2024SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 12.

33 papers.

Sort:

By area:

  1. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  2. 2025Kwai Keye-VL Technical Reportcited by 47
  3. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  4. 2026Flow caching for autoregressive video generationcited by 19
  5. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  6. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  7. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  8. 2025NeSyGeo: A Neuro-Symbolic Framework for Multimodal Geometric Reasoning Data Generationcited by 10
  9. 2025Dense360: Dense Understanding from Omnidirectional Panoramascited by 9
  10. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  11. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  12. 2025CyberV: Cybernetics for Test-time Scaling in Video Understandingcited by 8
21 more
  1. 2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 8
  2. 2024ImageRAG: Enhancing Ultra High Resolution Remote Sensing Imagery Analysis with ImageRAGcited by 6
  3. 2026LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Modelscited by 6
  4. 2025Exploring Multimodal Prompt for Visualization Authoring with Large Language Modelscited by 5
  5. 2025SAMPO:Scale-wise Autoregression with Motion PrOmpt for generative world modelscited by 4
  6. 2025Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Modelscited by 4
  7. 2026Do MLLMs Really See It: Reinforcing Visual Attention in Multimodal LLMscited by 4
  8. 2025HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learningcited by 3
  9. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  10. 2024Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Imagecited by 1
  11. 2025Adapting Vision-Language Models for Evaluating World Modelscited by 1
  12. 2025VideoAnchor: Reinforcing Subspace-Structured Visual Cues for Coherent Visual-Spatial Reasoningcited by 1
  13. 2026Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindnesscited by 1
  14. 2026T2SGrid: Temporal-to-Spatial Gridification for Video Temporal Groundingcited by 1
  15. 2026P2-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimizationcited by 1
  16. 2026Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigationcited by 1
  17. 2026HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documentscited by 0
  18. 2026Holistic Optimal Label Selection for Robust Prompt Learning under Partial Labelscited by 0
  19. 2026Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learningcited by 0
  20. 2026Probabilistic Concept-Aware Steering for Trustworthy LLM Inferencecited by 0
  21. 2026Test-Time Hallucination Control in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Test-Time Computing for Referring Multimodal Large Language Models
  2. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  3. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  4. 2024Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models
  5. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge
  6. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs
  7. 2025Zoomer: Adaptive Image Focus Optimization for Black-box MLLM
  8. 2024Wings: Learning Multimodal LLMs without Text-only Forgetting