mlsanityData to 14 Oct 2026

Aligning Large Multimodal Models with Factually Augmented RLHF

arXiv 2309.14525 ↗First posted 25 Sep 2023cs.CV cs.CLcited by 712

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in "hallucination", generating textual outputs that are not grounded by the multimodal information in context. To address the multimodal misalignment issue, we adapt the Reinforcement Learning from Human Feedback (RLHF) from the text domain to the task of vision-language alignment, where human annotators are asked to compare two responses and pinpoint the more hallucinated one, and the vision-language model is trained to maximize the simulated human rewards. We propose a new alignment algorithm called Factually Augmented RLHF that augments the reward model with additional factual information such as image captions and ground-truth multi-choice options, which alleviates the reward hacking phenomenon in RLHF and further improves the performance. We also enhance the GPT-4-generated training data (for vision instruction tuning) with previously available human-written image-text pairs to improve the general capabilities of our model. To evaluate the proposed approach in real-world scenarios, we develop a new evaluation benchmark MMHAL-BENCH with a special focus on penalizing hallucinations. As the first LMM trained with RLHF, our approach achieves remarkable improvement on the LLaVA-Bench dataset with the 94% performance level of the text-only GPT-4 (while previous best methods can only achieve the 87% level), and an improvement by 60% on MMHAL-BENCH over other baselines. We opensource our code, model, data at https://llava-rlhf.github.io.

Key facts. Cited by 712 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 81 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

57 of 81 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models18 Sep
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  4. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  6. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  13. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  15. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  16. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  17. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  18. 2023PaLM 2 Technical Reportcited by 1,162
  19. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  20. 2023LIMA: Less Is More for Alignmentcited by 1,100
  21. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  22. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  23. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  24. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  25. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  26. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  27. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  28. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  29. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  30. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  31. 2018Object Hallucination in Image Captioningcited by 669
  32. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  33. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  34. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  35. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  36. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  37. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  38. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 27; 2024: 210; 2025: 287; 2026: 188.

712 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  5. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  6. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  7. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  8. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  9. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  10. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  11. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  12. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
38 more
  1. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  2. 2024What matters when building vision-language models?cited by 326
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  4. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  5. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  6. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  7. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  8. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  9. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  10. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  11. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  12. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  13. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
  14. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  15. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  16. 2024An Introduction to Vision-Language Modelingcited by 178
  17. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  18. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  19. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  20. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  21. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  22. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 136
  23. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  24. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  25. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  26. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  27. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  28. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  29. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  30. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 125
  31. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 125
  32. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  33. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  34. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  35. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 109
  36. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  37. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  38. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100

50 most cited of 712 All 712 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
  2. 2024FLAME: Factuality-Aware Alignment for Large Language Models
  3. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  4. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  5. 2025Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM
  6. 2025Aligning to What? Limits to RLHF Based Alignment
  7. 2025Robust Multimodal Large Language Models Against Modality Conflict
  8. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability