mlsanityData to 7 Oct 2026

VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment

arXiv 2410.09421 ↗First posted 12 Oct 2024cs.CV cs.CLcited by 56

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

As large vision-language models (LVLMs) evolve rapidly, the demand for high-quality and diverse data to align these models becomes increasingly crucial. However, the creation of such data with human supervision proves costly and time-intensive. In this paper, we investigate the efficacy of AI feedback to scale supervision for aligning LVLMs. We introduce VLFeedback, the first large-scale vision-language feedback dataset, comprising over 82K multi-modal instructions and comprehensive rationales generated by off-the-shelf models without human annotations. To evaluate the effectiveness of AI feedback for vision-language alignment, we train Silkie, an LVLM fine-tuned via direct preference optimization on VLFeedback. Silkie showcases exceptional performance regarding helpfulness, visual faithfulness, and safety metrics. It outperforms its base model by 6.9% and 9.5% in perception and cognition tasks, reduces hallucination issues on MMHal-Bench, and exhibits enhanced resilience against red-teaming attacks. Furthermore, our analysis underscores the advantage of AI feedback, particularly in fostering preference diversity to deliver more comprehensive improvements. Our dataset, training code and models are available at https://vlf-silkie.github.io.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 55 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

33 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback16 Nov

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
20 more
  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  4. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  6. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  7. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  8. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  9. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 197
  10. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  11. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  12. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  13. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  14. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedbackcited by 80
  15. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  16. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  17. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 50
  18. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuningcited by 28
  19. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteriacited by 27
  20. 2024Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 39; 2026: 10.

56 papers.

Sort:

By area:

  1. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  2. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  3. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Keycited by 64
  4. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  5. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  6. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 28
  7. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  8. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  9. 2025Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferencescited by 22
  10. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
  11. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  12. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
38 more
  1. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  2. 2025DAMA: Data- and Model-aware Alignment of Multi-modal LLMscited by 14
  3. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  4. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesiscited by 12
  5. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  6. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Modelscited by 10
  7. 2025Boosting Chart-to-Code Generation in MLLM via Dual Preference-Guided Refinementcited by 10
  8. 2025Anyprefer: An Agentic Framework for Preference Data Synthesiscited by 10
  9. 2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questionscited by 10
  10. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  11. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lenscited by 8
  12. 2025MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understandingcited by 8
  13. 2024Systematic Reward Gap Optimization for Mitigating VLM Hallucinationscited by 7
  14. 2025Self-Improving VLM Judges Without Human Annotationscited by 7
  15. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
  16. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Modelscited by 5
  17. 2025Flash-VL 2B: Optimizing Vision-Language Model Performance for Ultra-Low Latency and High Throughputcited by 4
  18. 2025Weak-to-Strong Generalization under Distribution Shiftscited by 4
  19. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4
  20. 2024Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Modelscited by 3
  21. 2025Transferring Textual Preferences to Vision-Language Understanding through Model Mergingcited by 3
  22. 2025TEMPLE: Incentivizing Temporal Understanding of Video Large Language Models via Progressive Pre-SFT Alignmentcited by 3
  23. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 3
  24. 2025Personalized Reward Modeling for Text-to-Image Generationcited by 3
  25. 2025Towards Harmless Multimodal Assistants with Blind Preference Optimizationcited by 2
  26. 2025Alignment Data Map for Efficient Preference Data Selection and Diagnosiscited by 2
  27. 2025Controlling Multimodal LLMs via Reward-guided Decodingcited by 2
  28. 2025Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflectioncited by 2
  29. 2026Yuan3.0 Flash: An Open Multimodal Large Language Model for Enterprise Applicationscited by 2
  30. 2025Instruction-Oriented Preference Alignment for Enhancing Multi-Modal Comprehension Capability of MLLMscited by 1
  31. 2025A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Modelscited by 1
  32. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimizationcited by 1
  33. 2025Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMscited by 1
  34. 2025Multimodal Large Language Models with Adaptive Preference Optimization for Sequential Recommendationcited by 1
  35. 2026Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Modelscited by 1
  36. 2026DeDPO: Debiased Direct Preference Optimization for Diffusion Modelscited by 1
  37. 2026Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguitycited by 1
  38. 2025MINOS: A Multimodal Evaluation Model for Bidirectional Generation Between Image and Textcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Silkie: Preference Distillation for Large Visual Language Models
  2. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback
  3. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Models
  4. 2025Token-Level Inference-Time Alignment for Vision-Language Models
  5. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
  6. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
  7. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  8. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment