# Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

arXiv: https://arxiv.org/abs/2403.02325 · HTML: https://mlsanity.com/paper/2403.02325/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 63 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 17 of 17)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [PaLI-X: On Scaling up a Multilingual Vision and Language Model](https://mlsanity.com/paper/2305.18565/) (2023)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts](https://mlsanity.com/paper/2312.00784/) (2023)
- [Stay on topic with Classifier-Free Guidance](https://mlsanity.com/paper/2306.17806/) (2023)
- [Guiding Image Captioning Models Toward More Specific Captions](https://mlsanity.com/paper/2307.16686/) (2023)

## Built on by (most cited first, 30 of 64)

- [Multi-Object Hallucination in Vision-Language Models](https://mlsanity.com/paper/2407.06192/) (2024)
- [Reducing Hallucinations in Vision-Language Models via Latent Space Steering](https://mlsanity.com/paper/2410.15778/) (2024)
- [Visual Prompting in Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2409.15310/) (2024)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models](https://mlsanity.com/paper/2405.17820/) (2024)
- [ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learning](https://mlsanity.com/paper/2505.21457/) (2025)
- [AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge](https://mlsanity.com/paper/2409.07394/) (2024)
- [Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding](https://mlsanity.com/paper/2410.13321/) (2024)
- [Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment](https://mlsanity.com/paper/2405.17871/) (2024)
- [Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation](https://mlsanity.com/paper/2408.15205/) (2024)
- [Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks](https://mlsanity.com/paper/2501.08326/) (2025)
- [ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing](https://mlsanity.com/paper/2506.19848/) (2025)
- [Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMs](https://mlsanity.com/paper/2505.00744/) (2025)
- [VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding](https://mlsanity.com/paper/2411.15839/) (2024)
- [Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM](https://mlsanity.com/paper/2411.01610/) (2024)
- [TruthPrInt: Mitigating Large Vision-Language Models Object Hallucination Via Latent Truthful-Guided Pre-Intervention](https://mlsanity.com/paper/2503.10602/) (2025)
- [Towards Understanding Visual Grounding in Visual Language Models](https://mlsanity.com/paper/2509.10345/) (2025)
- [Contextual Emotion Recognition using Large Vision Language Models](https://mlsanity.com/paper/2405.08992/) (2024)
- [RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in Large Vision Language Models](https://mlsanity.com/paper/2405.17821/) (2024)
- [VACoDe: Visual Augmented Contrastive Decoding](https://mlsanity.com/paper/2408.05337/) (2024)
- [Natural Language Inference Improves Compositionality in Vision-Language Models](https://mlsanity.com/paper/2410.22315/) (2024)
- [Personalization Toolkit: Training Free Personalization of Large Vision Language Models](https://mlsanity.com/paper/2502.02452/) (2025)
- [Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation](https://mlsanity.com/paper/2504.16060/) (2025)
- [Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering](https://mlsanity.com/paper/2505.12826/) (2025)
- [Policy Contrastive Decoding for Robotic Foundation Models](https://mlsanity.com/paper/2505.13255/) (2025)
- [Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models](https://mlsanity.com/paper/2505.12343/) (2025)
- [ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval](https://mlsanity.com/paper/2505.20764/) (2025)
- [Causal Graphical Models for Vision-Language Compositional Understanding](https://mlsanity.com/paper/2412.09353/) (2024)
- [GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs](https://mlsanity.com/paper/2507.18043/) (2025)
- [Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Training](https://mlsanity.com/paper/2604.23121/) (2026)

## Abstract (arXiv)

Highlighting particularly relevant regions of an image can improve the performance of vision-language models (VLMs) on various vision-language (VL) tasks by guiding the model to attend more closely to these regions of interest. For example, VLMs can be given a "visual prompt", where visual markers such as bounding boxes delineate key image regions. However, current VLMs that can incorporate visual guidance are either proprietary and expensive or require costly training on curated data that includes visual prompts. We introduce Contrastive Region Guidance (CRG), a training-free guidance method that enables open-source VLMs to respond to visual prompts. CRG contrasts model outputs produced with and without visual prompts, factoring out biases revealed by the model when answering without the information required to produce a correct answer (i.e., the model's prior). CRG achieves substantial improvements in a wide variety of VL tasks: When region annotations are provided, CRG increases absolute accuracy by up to 11.1% on ViP-Bench, a collection of six diverse region-based tasks such as recognition, math, and object relationship reasoning. We also show CRG's applicability to spatial reasoning, with 10% improvement on What'sUp, as well as to compositional generalization -- improving accuracy by 11.5% and 7.5% on two challenging splits from SugarCrepe -- and to image-text alignment for generated images, where we improve by up to 8.4 AUROC and 6.8 F1 points on SeeTRUE. When reference regions are absent, CRG allows us to re-rank proposed regions in referring expression comprehension and phrase grounding benchmarks like RefCOCO/+/g and Flickr30K Entities, with an average gain of 3.2% in accuracy. Our analysis explores alternative masking strategies for CRG, quantifies CRG's probability shift, and evaluates the role of region guidance strength, empirically validating CRG's design choices.