mlsanityData to 7 Oct 2026

Self-Supervised Visual Preference Alignment

arXiv 2404.10501 ↗First posted 16 Apr 2024cs.CV cs.AI cs.CLcited by 31

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

This paper makes the first attempt towards unsupervised preference alignment in Vision-Language Models (VLMs). We generate chosen and rejected responses with regard to the original and augmented image pairs, and conduct preference alignment with direct preference optimization. It is based on a core idea: properly designed augmentation to the image input will induce VLM to generate false but hard negative responses, which helps the model to learn from and produce more robust and powerful answers. The whole pipeline no longer hinges on supervision from GPT-4 or human involvement during alignment, and is highly efficient with few lines of code. With only 8k randomly sampled unsupervised data, it achieves 90% relative score to GPT-4 on complex reasoning in LLaVA-Bench, and improves LLaVA-7B/13B by 6.7%/5.6% score on complex multi-modal benchmark MM-Vet. Visualizations shows its improved ability to align with user-intentions. A series of ablations are firmly conducted to reveal the latent mechanism of the approach, which also indicates its potential towards further scaling. Code are available in https://github.com/Kevinz-code/SeVa.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 55 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

45 of 55 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward1 Apr
  2. 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models16 Feb
  3. 2023VILA: On Pre-training for Visual Language Models12 Dec
  4. 2023Merlin:Empowering Multimodal LLMs with Foresight Minds30 Nov
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  8. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  9. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data20 Aug
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  12. 2023ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning18 Jul
2 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
19 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  8. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  10. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  11. 2023SVIT: Scaling up Visual Instruction Tuningcited by 161
  12. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  13. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  14. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  15. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  16. 2023DTL: Disentangled Transfer Learning for Visual Recognitioncited by 22
  17. 2023Multi-Label Self-Supervised Learning with Scene Imagescited by 17
  18. 2023Good Questions Help Zero-Shot Image Reasoningcited by 8
  19. 2023Coarse Is Better? A New Pipeline Towards Self-Supervised Learning with Uncurated Imagescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 19; 2026: 4.

31 papers.

Sort:

By area:

  1. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  2. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  3. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  4. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  5. 2024Probing Visual Language Priors in VLMscited by 44
  6. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  7. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
  8. 2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learningcited by 15
  9. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  10. 2025Perception in Reflectioncited by 13
  11. 2024Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolutioncited by 11
  12. 2025Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perceptioncited by 10
19 more
  1. 2024Enhancing Descriptive Captions with Visual Attributes for Multimodal Perceptioncited by 9
  2. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewardingcited by 9
  3. 2025Configurable Preference Tuning with Rubric-Guided Synthetic Datacited by 7
  4. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understandingcited by 7
  5. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  6. 2024Continual SFT Matches Multimodal RLHF with Negative Supervisioncited by 4
  7. 2025On Data Synthesis and Post-training for Visual Abstract Reasoningcited by 3
  8. 2025SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winnercited by 2
  9. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  10. 2025Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibrationcited by 2
  11. 2025Direct Diffusion Score Preference Optimization via Stepwise Contrastive Policy-Pair Supervisioncited by 2
  12. 2024Enhancing Audiovisual Speech Recognition through Bifocal Preference Optimizationcited by 1
  13. 2025Feedback-Driven Vision-Language Alignment with Minimal Human Supervisioncited by 1
  14. 2025LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMscited by 1
  15. 2025Improving Large Vision and Language Models by Learning from a Panel of Peerscited by 1
  16. 2025Token-Level Inference-Time Alignment for Vision-Language Modelscited by 0
  17. 2026CPPO: Contrastive Perception Policy Optimization for VLM Agentscited by 0
  18. 2026S2COPE: Self-Supervised Concept Discovery via Preference Learningcited by 0
  19. 2026Flow-of-Thought: A Framework for Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
  2. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
  3. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
  4. 2025VaPR -- Vision-language Preference alignment for Reasoning
  5. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment
  6. 2026S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models
  7. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization
  8. 2026A Dataset for Dynamic Human Preferences for Vision Language Models