# Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs

arXiv: https://arxiv.org/abs/2603.02556 · HTML: https://mlsanity.com/paper/2603.02556/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 12 of its 100 references that are in the map.
- First posted on arXiv 3 Mar 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Synthetic data and self-improvement for LLMs / [Self-improving MLLMs and synthetic data](https://mlsanity.com/synth/self-improving-mllms-and-synthetic-data/)

## Builds on (most cited first, 12 of 12)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023)
- [To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning](https://mlsanity.com/paper/2311.07574/) (2023)
- [Virgo: A Preliminary Exploration on Reproducing o1-like MLLM](https://mlsanity.com/paper/2501.01904/) (2025)
- [Incentivizing LLMs to Self-Verify Their Answers](https://mlsanity.com/paper/2506.01369/) (2025)

## Built on by (most cited first, 10 of 10)

- [Douyin Multimodal Embedding Model Technical Report](https://mlsanity.com/paper/2608.02148/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [Self-Evolving Visual Questioner](https://mlsanity.com/paper/2606.13929/) (2026)
- [ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capability](https://mlsanity.com/paper/2607.14145/) (2026)
- [VD-DeepStack: Bridging Visual Comparison and Language Reasoning for Few-Shot Anomaly Detection](https://mlsanity.com/paper/2609.34949/) (2026)
- [Context-Aware RL for Agentic and Multimodal LLMs](https://mlsanity.com/paper/2606.17053/) (2026)
- [AgentFinVQA: A Deployable Multi-Agent Pipeline for Auditable Financial Chart QA](https://mlsanity.com/paper/2606.19782/) (2026)
- [VC-Tooler: Learning Compositional and Adaptive Visual Tool Use](https://mlsanity.com/paper/2608.02217/) (2026)
- [Physico-Geospatial Grounded Scene Interpretation for Mobile Robotics](https://mlsanity.com/paper/2609.06629/) (2026)
- [BIRD: Distilling Decision Boundaries into Rationales for MLLM Adaptation](https://mlsanity.com/paper/2609.33713/) (2026)

## Abstract (arXiv)

Reasoning has emerged as a key capability of large language models. In linguistic tasks, this capability can be enhanced by self-improving techniques that refine reasoning paths for subsequent finetuning. However, extending these language-based self-improving approaches to vision language models (VLMs) presents a unique challenge:~visual hallucinations in reasoning paths cannot be effectively verified or rectified. Our solution starts with a key observation about visual contrast: when presented with a contrastive VQA pair, i.e., two visually similar images with synonymous questions, VLMs identify relevant visual cues more precisely. Motivated by this observation, we propose Visual Contrastive Self-Taught Reasoner (VC-STaR), a novel self-improving framework that leverages visual contrast to mitigate hallucinations in model-generated rationales. We collect a diverse suite of VQA datasets, curate contrastive pairs according to multi-modal similarity, and generate rationales using VC-STaR. Consequently, we obtain a new visual reasoning dataset, VisCoR-55K, which is then used to boost the reasoning capability of various VLMs through supervised finetuning. Extensive experiments show that VC-STaR not only outperforms existing self-improving approaches but also surpasses models finetuned on the SoTA visual reasoning datasets, demonstrating that the inherent contrastive ability of VLMs can bootstrap their own visual reasoning. Project at: https://github.com/zhiyupan42/VC-STaR.