mlsanityData to 14 Oct 2026

Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

arXiv 2403.02325 ↗First posted 4 Mar 2024cs.CV cs.AI cs.CLcited by 64

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Highlighting particularly relevant regions of an image can improve the performance of vision-language models (VLMs) on various vision-language (VL) tasks by guiding the model to attend more closely to these regions of interest. For example, VLMs can be given a "visual prompt", where visual markers such as bounding boxes delineate key image regions. However, current VLMs that can incorporate visual guidance are either proprietary and expensive or require costly training on curated data that includes visual prompts. We introduce Contrastive Region Guidance (CRG), a training-free guidance method that enables open-source VLMs to respond to visual prompts. CRG contrasts model outputs produced with and without visual prompts, factoring out biases revealed by the model when answering without the information required to produce a correct answer (i.e., the model's prior). CRG achieves substantial improvements in a wide variety of VL tasks: When region annotations are provided, CRG increases absolute accuracy by up to 11.1% on ViP-Bench, a collection of six diverse region-based tasks such as recognition, math, and object relationship reasoning. We also show CRG's applicability to spatial reasoning, with 10% improvement on What'sUp, as well as to compositional generalization -- improving accuracy by 11.5% and 7.5% on two challenging splits from SugarCrepe -- and to image-text alignment for generated images, where we improve by up to 8.4 AUROC and 6.8 F1 points on SeeTRUE. When reference regions are absent, CRG allows us to re-rank proposed regions in referring expression comprehension and phrase grounding benchmarks like RefCOCO/+/g and Flickr30K Entities, with an average gain of 3.2% in accuracy. Our analysis explores alternative masking strategies for CRG, quantifies CRG's probability shift, and evaluates the role of region guidance strength, empirically validating CRG's design choices.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 63 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

17 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 16

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023Segment Anythingcited by 8,234
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  6. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  8. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  9. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  10. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  11. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  12. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
4 more
  1. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  2. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  3. 2023Stay on topic with Classifier-Free Guidancecited by 91
  4. 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 24; 2026: 21.

64 papers.

Sort:

By area:

  1. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  2. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 59
  3. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  4. 2024Toward Visual Grounding: A Surveycited by 52
  5. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 47
  6. 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 43
  7. 2024AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledgecited by 37
  8. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decodingcited by 22
  9. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignmentcited by 20
  10. 2024Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentationcited by 20
  11. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  12. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
38 more
  1. 2025Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMscited by 14
  2. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decodingcited by 12
  3. 2024Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LMcited by 11
  4. 2025TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Interventioncited by 11
  5. 2025Towards Understanding Visual Grounding in Visual Language Modelscited by 10
  6. 2024Contextual Emotion Recognition using Large Vision Language Modelscited by 9
  7. 2024RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in Large Vision Language Modelscited by 8
  8. 2024VACoDe: Visual Augmented Contrastive Decodingcited by 8
  9. 2024Natural Language Inference Improves Compositionality in Vision-Language Modelscited by 8
  10. 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Modelscited by 8
  11. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
  12. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 8
  13. 2025Policy Contrastive Decoding for Robotic Foundation Modelscited by 8
  14. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Modelscited by 7
  15. 2025ConText-CIR: Learning from Concepts in Text for Composed Image Retrievalcited by 7
  16. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  17. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  18. 2026Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Trainingcited by 6
  19. 2024Transparent and Coherent Procedural Mistake Detectioncited by 4
  20. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generationcited by 4
  21. 2024LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehensioncited by 3
  22. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gaincited by 3
  23. 2026Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Stylecited by 3
  24. 2026CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generationcited by 3
  25. 2024ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialoguecited by 2
  26. 2025URECA: Unique Region Caption Anythingcited by 2
  27. 2025Controlling Multimodal LLMs via Reward-guided Decodingcited by 2
  28. 2025Guideline-Consistent Segmentation via Multi-Agent Refinementcited by 2
  29. 2026Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Modelscited by 2
  30. 2025Feedback-Driven Vision-Language Alignment with Minimal Human Supervisioncited by 1
  31. 2025MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoningcited by 1
  32. 2025LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQAcited by 1
  33. 2026Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMscited by 1
  34. 2026ScribbleSense: Generative Scribble-Based Texture Editing with Intent Predictioncited by 1
  35. 2026RAWDet-7: A Multi-Scenario Benchmark for Object Detection and Description on Quantized RAW Imagescited by 1
  36. 20263D-DRES: Detailed 3D Referring Expression Segmentationcited by 1
  37. 2025Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgettingcited by 0
  38. 2025Zero-Shot Referring Expression Comprehension via Vison-Language True/False Verificationcited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  2. 2024RegionGPT: Towards Region Understanding Vision Language Model
  3. 2024POINTS: Improving Your Vision-language Model with Affordable Strategies
  4. 2023Language Models as Black-Box Optimizers for Vision-Language Models
  5. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  6. 2023VIGC: Visual Instruction Generation and Correction
  7. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  8. 2024Targeted Visual Prompting for Medical Visual Question Answering