# RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models

arXiv: https://arxiv.org/abs/2411.04097 · HTML: https://mlsanity.com/paper/2411.04097/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 1 of its 63 references that are in the map.
- First posted on arXiv 6 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 1 of 1)

- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)

## Built on by (most cited first, 20 of 20)

- [Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Method](https://mlsanity.com/paper/2412.09082/) (2024)
- [Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning](https://mlsanity.com/paper/2503.11832/) (2025)
- [Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?](https://mlsanity.com/paper/2504.00509/) (2025)
- [Cross-modal Causal Relation Alignment for Video Question Grounding](https://mlsanity.com/paper/2503.07635/) (2025)
- [Policy Contrastive Decoding for Robotic Foundation Models](https://mlsanity.com/paper/2505.13255/) (2025)
- [SpurLens: Automatic Detection of Spurious Cues in Multimodal LLMs](https://mlsanity.com/paper/2503.08884/) (2025)
- [The Philosophical Foundations of Growing AI Like A Child](https://mlsanity.com/paper/2502.10742/) (2025)
- [GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs](https://mlsanity.com/paper/2509.25178/) (2025)
- [Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach](https://mlsanity.com/paper/2511.12978/) (2025)
- [Web Artifact Attacks Disrupt Vision Language Models](https://mlsanity.com/paper/2503.13652/) (2025)
- [LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Model](https://mlsanity.com/paper/2506.11402/) (2025)
- [TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models](https://mlsanity.com/paper/2512.01048/) (2025)
- [Unbiased Visual Reasoning with Controlled Visual Inputs](https://mlsanity.com/paper/2512.22183/) (2025)
- [Birds of a Feather Flock Together: Background-Invariant Representations via Linear Structure in VLMs](https://mlsanity.com/paper/2605.11107/) (2026)
- [Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates](https://mlsanity.com/paper/2606.07596/) (2026)
- [Counterfactual Reasoning for Fine-Grained Evidence Disentanglement in VideoQA](https://mlsanity.com/paper/2606.09181/) (2026)
- [Symbal: Detecting Systematic Misalignments in Model-Generated Captions](https://mlsanity.com/paper/2607.15216/) (2026)
- [Invisible Shortcuts: Why Vision Encoders Know Your Camera](https://mlsanity.com/paper/2608.05424/) (2026)
- [Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models](https://mlsanity.com/paper/2608.29996/) (2026)
- [Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Models](https://mlsanity.com/paper/2609.17790/) (2026)

## Abstract (arXiv)

Fine-tuned vision-language models (VLMs) often capture spurious correlations between image features and textual attributes, resulting in degraded zero-shot performance at test time. Existing approaches for addressing spurious correlations (i) primarily operate at the global image-level rather than intervening directly on fine-grained image features and (ii) are predominantly designed for unimodal settings. In this work, we present RaVL, which takes a fine-grained perspective on VLM robustness by discovering and mitigating spurious correlations using local image features rather than operating at the global image level. Given a fine-tuned VLM, RaVL first discovers spurious correlations by leveraging a region-level clustering approach to identify precise image features contributing to zero-shot classification errors. Then, RaVL mitigates the identified spurious correlation with a novel region-aware loss function that enables the VLM to focus on relevant regions and ignore spurious relationships during fine-tuning. We evaluate RaVL on 654 VLMs with various model architectures, data domains, and learned spurious correlations. Our results show that RaVL accurately discovers (191% improvement over the closest baseline) and mitigates (8.2% improvement on worst-group image classification accuracy) spurious correlations. Qualitative evaluations on general-domain and medical-domain VLMs confirm our findings.