# Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images

arXiv: https://arxiv.org/abs/2502.13928 · HTML: https://mlsanity.com/paper/2502.13928/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback](https://mlsanity.com/paper/2312.00849/) (2023)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Aligning Modalities in Vision Large Language Models via Preference Fine-tuning](https://mlsanity.com/paper/2402.11411/) (2024)
- [A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting](https://mlsanity.com/paper/2312.03594/) (2023)
- [Silkie: Preference Distillation for Large Visual Language Models](https://mlsanity.com/paper/2312.10665/) (2023)
- [NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples](https://mlsanity.com/paper/2410.14669/) (2024)
- [The effectiveness of MAE pre-pretraining for billion-scale pretraining](https://mlsanity.com/paper/2303.13496/) (2023)
- [V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization](https://mlsanity.com/paper/2411.02712/) (2024)
- [Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding](https://mlsanity.com/paper/2312.00081/) (2023)
- [Probing Visual Language Priors in VLMs](https://mlsanity.com/paper/2501.00569/) (2024)
- [Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective](https://mlsanity.com/paper/2403.18346/) (2024)
- [CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples](https://mlsanity.com/paper/2402.13254/) (2024)
- [CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs](https://mlsanity.com/paper/2501.16629/) (2025)
- [Modality-Fair Preference Optimization for Trustworthy MLLM Alignment](https://mlsanity.com/paper/2410.15334/) (2024)
- [FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension](https://mlsanity.com/paper/2409.14750/) (2024)
- [Supervised Fine-tuning in turn Improves Visual Foundation Models](https://mlsanity.com/paper/2401.10222/) (2024)
- [Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation](https://mlsanity.com/paper/2409.15125/) (2024)

## Built on by (most cited first, 14 of 14)

- [VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions](https://mlsanity.com/paper/2508.02329/) (2025)
- [Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization](https://mlsanity.com/paper/2509.25717/) (2025)
- [Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks](https://mlsanity.com/paper/2604.19697/) (2026)
- [OViP: Online Vision-Language Preference Learning for VLM Hallucination](https://mlsanity.com/paper/2505.15963/) (2025)
- [Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs](https://mlsanity.com/paper/2505.22396/) (2025)
- [No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs](https://mlsanity.com/paper/2606.31933/) (2026)
- [A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models](https://mlsanity.com/paper/2505.20901/) (2025)
- [Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs](https://mlsanity.com/paper/2509.24491/) (2025)
- [TraversalBench: Challenging Paths to Follow for Vision Language Models](https://mlsanity.com/paper/2604.10999/) (2026)
- [Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding](https://mlsanity.com/paper/2605.10855/) (2026)
- [Beyond the Linear Separability Ceiling: Aligning Representations in VLMs](https://mlsanity.com/paper/2507.07574/) (2025)
- [Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervision](https://mlsanity.com/paper/2509.12771/) (2025)
- [Overconfident and Blind to Details: Fixing Prompt Insensitivity with Abductive Preference Learning](https://mlsanity.com/paper/2510.09887/) (2025)
- [Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization](https://mlsanity.com/paper/2605.31312/) (2026)

## Abstract (arXiv)

Recent studies have shown that Large Vision-Language Models (VLMs) tend to neglect image content and over-rely on language-model priors, resulting in errors in visually grounded tasks and hallucinations. We hypothesize that this issue arises because existing VLMs are not explicitly trained to generate texts that are accurately grounded in fine-grained image details. To enhance visual feedback during VLM training, we propose S-VCO (Symmetrical Visual Contrastive Optimization), a novel finetuning objective that steers the model toward capturing important visual details and aligning them with corresponding text tokens. To further facilitate this detailed alignment, we introduce MVC, a paired image-text dataset built by automatically filtering and augmenting visual counterfactual data to challenge the model with hard contrastive cases involving Minimal Visual Contrasts. Experiments show that our method consistently improves VLM performance across diverse benchmarks covering various abilities and domains, achieving up to a 22% reduction in hallucinations, and significant gains in vision-centric and general tasks. Notably, these improvements become increasingly pronounced in benchmarks with higher visual dependency. In short, S-VCO offers a significant enhancement of VLM's visually-dependent task performance while retaining or even improving the model's general abilities. We opensource our code at https://s-vco.github.io/