mlsanityData to 7 Oct 2026

S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models

arXiv 2604.18512 ↗First posted 20 Apr 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have demonstrated remarkable progress in single-image understanding, yet effective reasoning across multiple images remains challenging. We identify a critical capability gap in existing multi-image alignment approaches: current methods focus primarily on localized reasoning with pre-specified image indices (``Look at Image 3 and...''), bypassing the essential skills of global visual search and autonomous cross-image comparison. To address this limitation, we introduce a Simple-to-Hard (S2H) learning framework that systematically constructs multi-image preference data across three hierarchical reasoning levels requiring an increasing level of capabilities: (1) single-image localized reasoning, (2) multi-image localized comparison, and (3) global visual search. Unlike prior work that relies on model-specific attributes, such as hallucinations or attention heuristics, to generate preference pairs, our approach leverages prompt-driven complexity to create chosen/rejected pairs that are applicable across different models. Through extensive evaluations on LLaVA and Qwen-VL models, we show that our diverse multi-image reasoning data significantly enhances multi-image reasoning performance, yielding significant improvements over baseline methods across benchmarks. Importantly, our approach maintains strong single-image reasoning performance while simultaneously strengthening multi-image understanding capabilities, thus advancing the state of the art for holistic visual preference alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 31 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

17 of 31 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  5. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  7. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  8. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  9. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  10. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
  11. 2024Thinking Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Modelscited by 44
  12. 2016Families in the Wild (FIW): Large-Scale Kinship Image Database and Benchmarkscited by 13
1 more
  1. 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modescited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models
  2. 2024Self-Supervised Visual Preference Alignment
  3. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
  4. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
  5. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
  6. 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation
  7. 2025Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains
  8. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?