mlsanityData to 14 Oct 2026

Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

arXiv 2411.10442 ↗First posted 15 Nov 2024cs.CL cs.CVcited by 239

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

Existing open-source multimodal large language models (MLLMs) generally follow a training process involving pre-training and supervised fine-tuning. However, these models suffer from distribution shifts, which limit their multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance. To address this, we introduce a preference optimization (PO) process to enhance the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data side, we design an automated preference data construction pipeline to create MMPR, a high-quality, large-scale multimodal reasoning preference dataset; and (2) on the model side, we explore integrating PO with MLLMs, developing a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance. Our approach enhances the multimodal reasoning abilities of both InternVL2-8B and InternVL2-76B. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance comparable to the 10× larger InternVL2-76B. We hope this study could inspire further advancements in MLLMs. Code, data, and model are released.

Key facts. Cited by 239 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 104 of its 120 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

104 of 120 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024MathGLM-Vision: Solving Mathematical Problems with Multi-Modal Large Language Model10 Sep
  2. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
  3. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?1 Jul
  4. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought26 May
  5. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  8. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoning10 Aug
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 95

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  12. 2023Visual Instruction Tuningcited by 7,869
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  13. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  14. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  15. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  16. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  17. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  18. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  19. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  20. 2019Towards VQA Models That Can Readcited by 1,718
  21. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  22. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  23. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  24. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  25. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  26. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  27. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  28. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  29. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  30. 2016A Diagram Is Worth A Dozen Imagescited by 804
  31. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  32. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  33. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  34. 2018Object Hallucination in Image Captioningcited by 669
  35. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  36. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  37. 2017Learning multiple visual domains with residual adapterscited by 557
  38. 2021InfographicVQAcited by 437

50 most cited of 95 All 95 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 171; 2026: 62.

239 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  5. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 303
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  7. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 139
  8. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 125
  9. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  10. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  11. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  12. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 81
38 more
  1. 2025VGR: Visual Grounded Reasoningcited by 73
  2. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  3. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 69
  4. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 69
  5. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
  6. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 65
  7. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  8. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
  9. 2025XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery?cited by 51
  10. 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 49
  11. 2025MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoningcited by 49
  12. 2025Kwai Keye-VL Technical Reportcited by 47
  13. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  14. 2025GeoLLaVA-8K: Scaling Remote-Sensing Multimodal Large Language Models to 8K Resolutioncited by 44
  15. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  16. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 41
  17. 2025Spotlight on Token Perception for Multimodal Reinforcement Learningcited by 39
  18. 2025Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learningcited by 38
  19. 2025ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoningcited by 37
  20. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  21. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  22. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  23. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  24. 2026MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verificationcited by 32
  25. 2025MM-PRM: Enhancing Multimodal Mathematical Reasoning with Scalable Step-Level Supervisioncited by 31
  26. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 30
  27. 2026Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligencecited by 30
  28. 2025OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preferencecited by 28
  29. 2025Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Informationcited by 28
  30. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  31. 2025AStar: Boosting Multimodal Reasoning with Automated Structured Thinkingcited by 25
  32. 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoningcited by 25
  33. 2025TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understandingcited by 25
  34. 2025Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understandingcited by 24
  35. 2025VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuningcited by 24
  36. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  37. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  38. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuitioncited by 22

50 most cited of 239 All 239 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale
  2. 2025Towards Efficient Multimodal Unified Reasoning Model via Model Merging
  3. 2025DRIFT: Transferring Reasoning Priors for Efficient MLLM Fine-Tuning
  4. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
  5. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
  6. 2026Efficient Data Selection for Multimodal Models via Incremental Optimization Utility
  7. 2024EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation
  8. 2025Towards Harmless Multimodal Assistants with Blind Preference Optimization