mlsanityData to 7 Oct 2026

Attention Prompting on Image for Large Vision-Language Models

arXiv 2409.17143 ↗First posted 25 Sep 2024cs.CV cs.AIcited by 23

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Compared with Large Language Models (LLMs), Large Vision-Language Models (LVLMs) can also accept images as input, thus showcasing more interesting emergent capabilities and demonstrating impressive performance on various vision-language tasks. Motivated by text prompting in LLMs, visual prompting has been explored to enhance LVLMs' capabilities of perceiving visual information. However, previous visual prompting techniques solely process visual inputs without considering text queries, limiting the models' ability to follow text instructions to complete tasks. To fill this gap, in this work, we propose a new prompting technique named Attention Prompting on Image, which just simply overlays a text-query-guided attention heatmap on the original input image and effectively enhances LVLM on various tasks. Specifically, we generate an attention heatmap for the input image dependent on the text query with an auxiliary model like CLIP. Then the heatmap simply multiplies the pixel values of the original image to obtain the actual input image for the LVLM. Extensive experiments on various vison-language benchmarks verify the effectiveness of our technique. For example, Attention Prompting on Image improves LLaVA-1.5 by 3.8% and 2.9% on MM-Vet and LLaVA-Wild benchmarks, respectively.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 69 of its 75 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

69 of 75 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  6. 2023Fine-Grained Visual Prompting7 Jun
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2021Learning to Prompt for Continual Learningcited by 938
  13. 2023Vision Transformers Need Registerscited by 844
  14. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  15. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  16. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  17. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  18. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  19. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  20. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  21. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  22. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  23. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  24. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  25. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  26. 2021Ensemble deep learning: A reviewcited by 283
  27. 2022S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learningcited by 246
  28. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  29. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  30. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  31. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  32. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  33. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  34. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  35. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  36. 2023Caption Anything: Interactive Image Description with Diverse Multimodal Controlscited by 98
  37. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  38. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  2. 2025ADVEDM:Fine-grained Adversarial Attack against VLM-based Embodied Agentscited by 19
  3. 2024VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoningcited by 13
  4. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Modelcited by 13
  5. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  6. 2025Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMscited by 5
  7. 2025TRAP: Targeted Redirecting of Agentic Preferencescited by 4
  8. 2025Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigationcited by 3
  9. 2026VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Promptingcited by 3
  10. 2025AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMscited by 2
  11. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  12. 2026LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videoscited by 2
11 more
  1. 2025Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcarecited by 1
  2. 2025MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioningcited by 1
  3. 2026SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMscited by 1
  4. 2025The Role of Background Information in Reducing Object Hallucination in Vision-Language Models: Insights from Cutoff API Promptingcited by 0
  5. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thoughtcited by 0
  6. 2025Multi-Sequence Parotid Gland Lesion Segmentation via Expert Text-Guided Segment Anything Modelcited by 0
  7. 2025Towards Open World Detection: A Surveycited by 0
  8. 2025SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generationcited by 0
  9. 2025TRANSPORTER: Transferring Visual Semantics from VLM Manifoldscited by 0
  10. 2026ATA: Bridging Implicit Reasoning with Attention-Guided and Action-Guided Inference for Vision-Language Action Modelscited by 0
  11. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Visual Prompting in Multimodal Large Language Models: A Survey
  2. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models
  3. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
  4. 2024A-VL: Adaptive Attention for Large Vision-Language Models
  5. 2025Where do Large Vision-Language Models Look at when Answering Questions?
  6. 2024Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models
  7. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
  8. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features