mlsanityData to 7 Oct 2026

Test-Time Computing for Referring Multimodal Large Language Models

arXiv 2602.19505 ↗First posted 23 Feb 2026cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

We propose ControlMLLM++, a novel test-time adaptation framework that injects learnable visual prompts into frozen multimodal large language models (MLLMs) to enable fine-grained region-based visual reasoning without any model retraining or fine-tuning. Leveraging the insight that cross-modal attention maps intrinsically encode semantic correspondences between textual tokens and visual regions, ControlMLLM++ optimizes a latent visual token modifier during inference via a task-specific energy function to steer model attention towards user-specified areas. To enhance optimization stability and mitigate language prompt biases, ControlMLLM++ incorporates an improved optimization strategy (Optim++) and a prompt debiasing mechanism (PromptDebias). Supporting diverse visual prompt types including bounding boxes, masks, scribbles, and points, our method demonstrates strong out-of-domain generalization and interpretability. The code is available at https://github.com/mrwu-mac/ControlMLLM.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 56 of its 79 references that are in the map. First posted on arXiv 23 Feb 2026.

Builds on

56 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
38 more
  1. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  4. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  10. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  11. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  12. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  13. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  14. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  15. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 280
  16. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  17. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  18. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  19. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  20. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 172
  21. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  22. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  23. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  24. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  25. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  26. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  27. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  28. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  29. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  30. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  31. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuningcited by 65
  32. 2021Person Re-identification Method Based on Color Attack and Joint Defencecited by 56
  33. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  34. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  35. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  36. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  37. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMscited by 29
  38. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
  2. 2024Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models
  3. 2025Learning to Inference Adaptively for Multimodal Large Language Models
  4. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
  5. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  6. 2025Test-Time Warmup for Multimodal Large Language Models
  7. 2025Zoomer: Adaptive Image Focus Optimization for Black-box MLLM
  8. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings