mlsanityData to 14 Oct 2026

RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness

arXiv 2405.17220 ↗First posted 27 May 2024cs.CLcited by 131

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Traditional feedback learning for hallucination reduction relies on labor-intensive manual labeling or expensive proprietary models. This leaves the community without foundational knowledge about how to build high-quality feedback with open-source MLLMs. In this work, we introduce RLAIF-V, a novel framework that aligns MLLMs in a fully open-source paradigm. RLAIF-V maximally explores open-source MLLMs from two perspectives, including high-quality feedback data generation for preference learning and self-feedback guidance for inference-time scaling. Extensive experiments on six benchmarks in both automatic and human evaluation show that RLAIF-V substantially enhances the trustworthiness of models at both preference learning and inference time. RLAIF-V 7B reduces object hallucination by 80.7% and overall hallucination by 33.7%. Remarkably, RLAIF-V 12B further reveals the self-alignment potential of open-source MLLMs, where the model can learn from feedback of itself to achieve super GPT-4V trustworthiness.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 60 of its 77 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

60 of 77 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Automated Multi-level Preference for MLLMs18 May
  2. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback22 Apr
  3. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback7 Apr
  4. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  5. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization13 Mar
  6. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuning18 Feb
  7. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  8. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  11. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug

Other references · 49

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
37 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  12. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  13. 2016A Diagram Is Worth A Dozen Imagescited by 804
  14. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  15. 2024Self-Rewarding Language Modelscited by 681
  16. 2018Object Hallucination in Image Captioningcited by 669
  17. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  18. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  21. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  22. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  23. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  24. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  25. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  26. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  27. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  28. 2023Fine-tuning Language Models for Factualitycited by 221
  29. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  30. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  31. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  32. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  33. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewritescited by 61
  34. 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Modelscited by 47
  35. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistantscited by 25
  36. 2024Visual Hallucination: Definition, Quantification, and Prescriptive Remediationscited by 11
  37. 2024Skip n: A Simple Method to Reduce Hallucination in Large Vision-Language Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 55; 2026: 50.

131 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  3. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  4. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 111
  5. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
  6. 2024GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AIcited by 96
  7. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  8. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 90
  9. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  10. 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
  11. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  12. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 58
38 more
  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  2. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  3. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 45
  4. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  5. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  6. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  7. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  8. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  9. 2025DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimizationcited by 27
  10. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  11. 2024Understanding Alignment in Multimodal LLMs: A Comprehensive Studycited by 23
  12. 2024Can LVLMs Obtain a Driver's License? A Benchmark Towards Reliable AGI for Autonomous Drivingcited by 22
  13. 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimizationcited by 22
  14. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  15. 2025Improving Medical Reasoning with Curriculum-Aware Reinforcement Learningcited by 16
  16. 2026ARIS: Autonomous Research via Adversarial Multi-Agent Collaborationcited by 16
  17. 2024VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Textcited by 15
  18. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  19. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Datacited by 15
  20. 2025DreamPRM: Domain-Reweighted Process Reward Model for Multimodal Reasoningcited by 15
  21. 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 15
  22. 2025ReAL-AD: Towards Human-Like Reasoning in End-to-End Autonomous Drivingcited by 15
  23. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 15
  24. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 15
  25. 2025DAMA: Data- and Model-aware Alignment of Multi-modal LLMscited by 14
  26. 2025Q-Ponder: A Unified Training Pipeline for Reasoning-based Visual Quality Assessmentcited by 14
  27. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 13
  28. 2024Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolutioncited by 11
  29. 2025Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Studycited by 11
  30. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Modelscited by 11
  31. 2025ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Modelcited by 11
  32. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 10
  33. 2024Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approachcited by 9
  34. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
  35. 2026CreatiParser: Generative Image Parsing of Raster Graphic Designs into Editable Layerscited by 9
  36. 2024EACO: Enhancing Alignment in Multimodal LLMs via Critical Observationcited by 8
  37. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  38. 2025Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoningcited by 8

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs
  2. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling
  3. 2026Online Self-Calibration Against Hallucination in Vision-Language Models
  4. 2025Feedback-Enhanced Hallucination-Resistant Vision-Language Model for Real-Time Scene Understanding
  5. 2026Score-Control for Hallucination Reduction in Diffusion Models
  6. 2026Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methods
  7. 2026Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation
  8. 2026IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination