mlsanityData to 14 Oct 2026

Silkie: Preference Distillation for Large Visual Language Models

arXiv 2312.10665 ↗First posted 17 Dec 2023cs.CV cs.CLcited by 130

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

This paper explores preference distillation for large vision language models (LVLMs), improving their ability to generate helpful and faithful responses anchoring the visual context. We first build a vision-language feedback (VLFeedback) dataset utilizing AI annotation. Specifically, responses are generated by models sampled from 12 LVLMs, conditioned on multi-modal instructions sourced from various datasets. We adopt GPT-4V to assess the generated outputs regarding helpfulness, visual faithfulness, and ethical considerations. Furthermore, the preference supervision is distilled into Qwen-VL-Chat through the direct preference optimization (DPO) method. The resulting model Silkie, achieves 6.9% and 9.5% relative improvement on the MME benchmark regarding the perception and cognition capabilities, respectively. Silkie also demonstrates reduced hallucination by setting a new state-of-the-art score of 3.02 on the MMHal-Bench benchmark. Further analysis shows that DPO with our VLFeedback dataset mainly boosts the fine-grained perception and complex cognition abilities of LVLMs, leading to more comprehensive improvements compared to human-annotated preference datasets.

Key facts. Cited by 130 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 42 references that are in the map. First posted on arXiv 17 Dec 2023.

Builds on

34 of 42 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  2. 2023What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning2 Nov
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  6. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  7. 2023A Survey on Multimodal Large Language Models23 Jun
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,085
13 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  2. 2022Red Teaming Language Models with Language Modelscited by 1,026
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  4. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  7. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  8. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  9. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  10. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  11. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  12. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  13. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 54; 2025: 54; 2026: 22.

130 papers.

Sort:

By area:

  1. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  3. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  5. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  6. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  7. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  8. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  9. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  10. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  11. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  12. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 136
38 more
  1. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  2. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  3. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  4. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  5. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
  6. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  7. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedbackcited by 80
  8. 2024Calibrated Self-Rewarding Vision Language Modelscited by 78
  9. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Modelscited by 75
  10. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  11. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  12. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 68
  13. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  14. 2024Red Teaming Visual Language Modelscited by 65
  15. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 65
  16. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  17. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  18. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  19. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  20. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  21. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 44
  22. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
  23. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 37
  24. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  25. 2024Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Taskscited by 32
  26. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  27. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMscited by 31
  28. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  29. 2024Automated Multi-level Preference for MLLMscited by 29
  30. 2024Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Modelscited by 29
  31. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 29
  32. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  33. 2025The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representationscited by 29
  34. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  35. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28
  36. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  37. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learningcited by 25
  38. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 24

50 most cited of 130 All 130 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment
  2. 2026A Dataset for Dynamic Human Preferences for Vision Language Models
  3. 2026Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs
  4. 2025Token-Level Inference-Time Alignment for Vision-Language Models
  5. 2025Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards
  6. 2024Self-Supervised Visual Preference Alignment
  7. 2026Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation
  8. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization