# OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

arXiv: https://arxiv.org/abs/2502.18411 · HTML: https://mlsanity.com/paper/2502.18411/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 37 of its 69 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)

## Built on by (most cited first, 28 of 28)

- [VisualPRM: An Effective Process Reward Model for Multimodal Reasoning](https://mlsanity.com/paper/2503.10291/) (2025)
- [SIM-CoT: Supervised Implicit Chain-of-Thought](https://mlsanity.com/paper/2509.20317/) (2025)
- [MM-IFEngine: Towards Multimodal Instruction Following](https://mlsanity.com/paper/2504.07957/) (2025)
- [Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback](https://mlsanity.com/paper/2505.20075/) (2025)
- [LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding](https://mlsanity.com/paper/2605.27365/) (2026)
- [Advancing Complex Video Object Segmentation via Progressive Concept Construction](https://mlsanity.com/paper/2507.15852/) (2025)
- [ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing](https://mlsanity.com/paper/2506.19848/) (2025)
- [BaseReward: A Strong Baseline for Multimodal Reward Model](https://mlsanity.com/paper/2509.16127/) (2025)
- [Aligning Multimodal LLM with Human Preference: A Survey](https://mlsanity.com/paper/2503.14504/) (2025)
- [Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences](https://mlsanity.com/paper/2510.23451/) (2025)
- [ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch](https://mlsanity.com/paper/2601.13606/) (2026)
- [Self-Improvement in Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2510.02665/) (2025)
- [MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization](https://mlsanity.com/paper/2603.03192/) (2026)
- [Visual Self-Refine: A Pixel-Guided Paradigm for Accurate Chart Parsing](https://mlsanity.com/paper/2602.16455/) (2026)
- [Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Models](https://mlsanity.com/paper/2508.01741/) (2025)
- [AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model](https://mlsanity.com/paper/2510.11496/) (2025)
- [GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devices](https://mlsanity.com/paper/2503.06019/) (2025)
- [Building a Precise Video Language with Human-AI Oversight](https://mlsanity.com/paper/2604.21718/) (2026)
- [Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training](https://mlsanity.com/paper/2505.08971/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models](https://mlsanity.com/paper/2512.03463/) (2025)
- [CoT4Det: A Chain-of-Thought Framework for Perception-Oriented Vision-Language Tasks](https://mlsanity.com/paper/2512.06663/) (2025)
- [ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs](https://mlsanity.com/paper/2604.06484/) (2026)
- [Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understanding](https://mlsanity.com/paper/2604.14692/) (2026)
- [Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games](https://mlsanity.com/paper/2606.19338/) (2026)
- [The Boy Who Cried Wolf: Adversarial Misclassification of Safe Inputs as Unsafe in Multimodal Guardrails](https://mlsanity.com/paper/2608.01373/) (2026)
- [Text-Anchored Semantic Perturbations for Transferable Jailbreak Attacks on Multimodal Large Language Models](https://mlsanity.com/paper/2608.22312/) (2026)
- [Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation](https://mlsanity.com/paper/2609.16059/) (2026)

## Abstract (arXiv)

Recent advancements in open-source multi-modal large language models (MLLMs) have primarily focused on enhancing foundational capabilities, leaving a significant gap in human preference alignment. This paper introduces OmniAlign-V, a comprehensive dataset of 200K high-quality training samples featuring diverse images, complex questions, and varied response formats to improve MLLMs' alignment with human preferences. We also present MM-AlignBench, a human-annotated benchmark specifically designed to evaluate MLLMs' alignment with human values. Experimental results show that finetuning MLLMs with OmniAlign-V, using Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO), significantly enhances human preference alignment while maintaining or enhancing performance on standard VQA benchmarks, preserving their fundamental capabilities. Our datasets, benchmark, code and checkpoints have been released at https://github.com/PhoenixZ810/OmniAlign-V.