# Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization

arXiv: https://arxiv.org/abs/2504.18397 · HTML: https://mlsanity.com/paper/2504.18397/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 56 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://mlsanity.com/paper/2406.18629/) (2024)
- [KTO: Model Alignment as Prospect Theoretic Optimization](https://mlsanity.com/paper/2402.01306/) (2024)
- [RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness](https://mlsanity.com/paper/2405.17220/) (2024)
- [Direct Preference Optimization with an Offset](https://mlsanity.com/paper/2402.10571/) (2024)
- [NoteLLM-2: Multimodal Large Representation Models for Recommendation](https://mlsanity.com/paper/2405.16789/) (2024)
- [Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid](https://mlsanity.com/paper/2408.02034/) (2024)
- [Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants](https://mlsanity.com/paper/2310.00653/) (2023)

## Built on by (most cited first, 22 of 22)

- [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification](https://mlsanity.com/paper/2508.05629/) (2025)
- [GRIT: Teaching MLLMs to Think with Images](https://mlsanity.com/paper/2505.15879/) (2025)
- [From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2509.25373/) (2025)
- [Latent Implicit Visual Reasoning](https://mlsanity.com/paper/2512.21218/) (2025)
- [HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling](https://mlsanity.com/paper/2510.00054/) (2025)
- [DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking](https://mlsanity.com/paper/2507.20879/) (2025)
- [Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning](https://mlsanity.com/paper/2509.19090/) (2025)
- [CropVLM: Learning to Zoom for Fine-Grained Vision-Language Perception](https://mlsanity.com/paper/2511.19820/) (2025)
- [Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation](https://mlsanity.com/paper/2602.24041/) (2026)
- [DEPO: Dual-Efficiency Preference Optimization for LLM Agents](https://mlsanity.com/paper/2511.15392/) (2025)
- [Parallel Diffusion Solver via Residual Dirichlet Policy Optimization](https://mlsanity.com/paper/2512.22796/) (2025)
- [HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework](https://mlsanity.com/paper/2602.23615/) (2026)
- [Foveated Reasoning: Stateful, Action-based Visual Focusing for Vision-Language Models](https://mlsanity.com/paper/2604.21079/) (2026)
- [LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA](https://mlsanity.com/paper/2509.10026/) (2025)
- [GuardAlign: Test-time Safety Alignment in Multimodal Large Language Models](https://mlsanity.com/paper/2602.24027/) (2026)
- [ActiveScope: Actively Seeking and Correcting Perception for MLLMs](https://mlsanity.com/paper/2606.24292/) (2026)
- [When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities](https://mlsanity.com/paper/2607.08605/) (2026)
- [Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Drift](https://mlsanity.com/paper/2609.39183/) (2026)
- [Watch Wider and Think Deeper: Collaborative Cross-modal Chain-of-Thought for Complex Visual Reasoning](https://mlsanity.com/paper/2601.02422/) (2026)
- [Scalable Multi-Task Low-Rank Model Adaptation](https://mlsanity.com/paper/2603.01526/) (2026)
- [Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning](https://mlsanity.com/paper/2603.01759/) (2026)
- [Towards High-Resolution Visual Perception via Hierarchical Entity Exploration](https://mlsanity.com/paper/2607.00816/) (2026)

## Abstract (arXiv)

Chain-of-thought (CoT) reasoning greatly improves the interpretability and problem-solving abilities of multimodal large language models (MLLMs). However, existing approaches are focused on text CoT, limiting their ability to leverage visual cues. Visual CoT remains underexplored, and the only work is based on supervised fine-tuning (SFT) that relies on extensive labeled bounding-box data and is hard to generalize to unseen cases. In this paper, we introduce Unsupervised Visual CoT (UV-CoT), a novel framework for image-level CoT reasoning via preference optimization. UV-CoT performs preference comparisons between model-generated bounding boxes (one is preferred and the other is dis-preferred), eliminating the need for bounding-box annotations. We get such preference data by introducing an automatic data generation pipeline. Given an image, our target MLLM (e.g., LLaVA-1.5-7B) generates seed bounding boxes using a template prompt and then answers the question using each bounded region as input. An evaluator MLLM (e.g., OmniLLM-12B) ranks the responses, and these rankings serve as supervision to train the target MLLM with UV-CoT by minimizing negative log-likelihood losses. By emulating human perception--identifying key regions and reasoning based on them--UV-CoT can improve visual comprehension, particularly in spatial reasoning tasks where textual descriptions alone fall short. Our experiments on six datasets demonstrate the superiority of UV-CoT, compared to the state-of-the-art textual and visual CoT methods. Our zero-shot testing on four unseen datasets shows the strong generalization of UV-CoT. The code is available in https://github.com/kesenzhao/UV-CoT.