mlsanityData to 14 Oct 2026

Calibrated Self-Rewarding Vision Language Models

arXiv 2405.14622 ↗First posted 23 May 2024cs.LG cs.CL cs.CVcited by 78

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) have made substantial progress by integrating pre-trained large language models (LLMs) and vision models through instruction tuning. Despite these advancements, LVLMs often exhibit the hallucination phenomenon, where generated text responses appear linguistically plausible but contradict the input image, indicating a misalignment between image and text pairs. This misalignment arises because the model tends to prioritize textual information over visual input, even when both the language model and visual representations are of high quality. Existing methods leverage additional models or human annotations to curate preference data and enhance modality alignment through preference optimization. These approaches may not effectively reflect the target LVLM's preferences, making the curated preferences easily distinguishable. Our work addresses these challenges by proposing the Calibrated Self-Rewarding (CSR) approach, which enables the model to self-improve by iteratively generating candidate responses, evaluating the reward for each response, and curating preference data for fine-tuning. In the reward modeling, we employ a step-wise strategy and incorporate visual constraints into the self-rewarding process to place greater emphasis on visual input. Empirical results demonstrate that CSR enhances performance and reduces hallucinations across ten benchmarks and tasks, achieving substantial improvements over existing methods by 7.62%. Our empirical results are further supported by rigorous theoretical analysis, under mild assumptions, verifying the effectiveness of introducing visual constraints into the self-rewarding paradigm. Additionally, CSR shows compatibility with different vision-language models and the ability to incrementally improve performance through iterative fine-tuning. Our data and code are available at https://github.com/YiyangZhou/CSR.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 69 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

45 of 69 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  2. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  3. 2023VILA: On Pre-training for Visual Language Models12 Dec
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  5. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
24 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  6. 2024Self-Rewarding Language Modelscited by 681
  7. 2018Object Hallucination in Image Captioningcited by 669
  8. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  9. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  11. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  13. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  14. 2018Multiaccuracy: Black-Box Post-Processing for Fairness in Classificationcited by 222
  15. 2023Fine-tuning Language Models for Factualitycited by 221
  16. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  17. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  18. 2019Linearized two-layers neural networks in high dimensioncited by 155
  19. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  20. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 126
  21. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  22. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  23. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  24. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 47; 2026: 9.

78 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  3. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 145
  4. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  5. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 109
  6. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
  7. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
  8. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 85
  9. 2025MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understandingcited by 67
  10. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  11. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  12. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
38 more
  1. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 50
  2. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 46
  3. 2024Probing Visual Language Priors in VLMscited by 45
  4. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  5. 2025First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Trainingcited by 34
  6. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  7. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 31
  8. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  9. 2025textttComplex-Edit: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmarkcited by 27
  10. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 27
  11. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learningcited by 25
  12. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  13. 2024Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehensioncited by 23
  14. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 22
  15. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  16. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 20
  17. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 19
  18. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  19. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 18
  20. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  21. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 16
  22. 2025Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Datacited by 12
  23. 2024Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolutioncited by 11
  24. 2025ForgetMe: Evaluating Selective Forgetting in Generative Modelscited by 11
  25. 2025MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibrationcited by 11
  26. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 9
  27. 2024EACO: Enhancing Alignment in Multimodal LLMs via Critical Observationcited by 8
  28. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  29. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  30. 2025GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?cited by 8
  31. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 7
  32. 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generationcited by 6
  33. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
  34. 2024CSRec: Rethinking Sequential Recommendation from A Causal Perspectivecited by 4
  35. 2024Continual SFT Matches Multimodal RLHF with Negative Supervisioncited by 4
  36. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  37. 2025CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoningcited by 4
  38. 2025Personalized Reward Modeling for Text-to-Image Generationcited by 4

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data
  2. 2026Online Self-Calibration Against Hallucination in Vision-Language Models
  3. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback
  4. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
  5. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
  6. 2025Improving Alignment in LVLMs with Debiased Self-Judgment
  7. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
  8. 2025HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models