mlsanityData to 14 Oct 2026

Prompt Highlighter: Interactive Control for Multi-Modal LLMs

arXiv 2312.04302 ↗First posted 7 Dec 2023cs.CV cs.CLcited by 29

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. Multi-modal LLMs empower multi-modality understanding with the capability of semantic generation yet bring less explainability and heavier reliance on prompt contents due to their autoregressive generative nature. While manipulating prompt formats could improve outputs, designing specific and precise prompts per task can be challenging and ineffective. To tackle this issue, we introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. Motivated by the classifier-free diffusion guidance, we form regular and unconditional context pairs based on highlighted tokens, demonstrating that the autoregressive generation in models can be guided in a classifier-free way. Notably, we find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs. Our approach is compatible with current LLMs and VLMs, achieving impressive customized generation results without training. Experiments confirm its effectiveness in focusing on input contexts and generating reliable content. Without tuning on LLaVA-v1.5, our method secured 70.7 in the MMBench test and 1552.5 in MME-perception. The code is available at: https://github.com/dvlab-research/Prompt-Highlighter/

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 53 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

39 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
21 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  13. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  14. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  15. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  16. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  17. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 151
  18. 2023Stay on topic with Classifier-Free Guidancecited by 91
  19. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  20. 2023Real-World Image Variation by Aligning Diffusion Inversion Chaincited by 37
  21. 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 10; 2026: 7.

29 papers.

Sort:

By area:

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  2. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  4. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 159
  5. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  6. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 49
  7. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 46
  8. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  9. 2024Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrievalcited by 32
  10. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibrationcited by 23
  11. 2025Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMscited by 14
  12. 2025Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Modelscited by 10
17 more
  1. 2024Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Modelscited by 5
  2. 2025SDRT: Enhance Vision-Language Models by Self-Distillation with Diverse Reasoning Tracescited by 5
  3. 2025Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Modelscited by 5
  4. 2025Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Modelscited by 4
  5. 2026Video-CoE: Reinforcing Video Event Prediction via Chain of Eventscited by 4
  6. 2024Detailed Object Description with Controllable Dimensionscited by 3
  7. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Modelscited by 3
  8. 2025VizDefender: Unmasking Visualization Tampering through Proactive Localization and Intent Inferencecited by 3
  9. 2025ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinementcited by 2
  10. 2025Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMscited by 2
  11. 2024Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIPcited by 1
  12. 2026Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMscited by 1
  13. 2026ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruningcited by 1
  14. 2026Test-Time Computing for Referring Multimodal Large Language Modelscited by 1
  15. 2026Learning Consistent Temporal Grounding between Related Tasks in Sports Coachingcited by 1
  16. 2026Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answeringcited by 0
  17. 2026Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models
  2. 2024LLMs Meet Multimodal Generation and Editing: A Survey
  3. 2025LLMs can see and hear without any training
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
  5. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
  6. 2024Multi-modal Generation via Cross-Modal In-Context Learning
  7. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception
  8. 2024Visual Prompting in Multimodal Large Language Models: A Survey