mlsanityData to 14 Oct 2026

Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment

arXiv 2410.14148 ↗First posted 18 Oct 2024cs.CV cs.CLcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

The recent advancements in large language models (LLMs) and pre-trained vision models have accelerated the development of vision-language large models (VLLMs), enhancing the interaction between visual and linguistic modalities. Despite their notable success across various domains, VLLMs face challenges in modality alignment, which can lead to issues like hallucinations and unsafe content generation. Current alignment techniques often rely on coarse feedback and external datasets, limiting scalability and performance. In this paper, we propose FiSAO (Fine-Grained Self-Alignment Optimization), a novel self-alignment method that utilizes the model's own visual encoder as a fine-grained verifier to improve vision-language alignment without the need for additional data. By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data. Through both theoretical analysis and experimental validation, we demonstrate that FiSAO effectively addresses the misalignment problem in VLLMs, marking the first instance of token-level rewards being applied to such models.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 66 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

53 of 66 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?5 Jul
  2. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuning18 Feb
  3. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  4. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  5. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  6. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback16 Nov
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
34 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2024Qwen2 Technical Reportcited by 2,120
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  12. 2018Object Hallucination in Image Captioningcited by 669
  13. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  14. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  15. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  16. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 316
  17. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 226
  18. 2018Multiaccuracy: Black-Box Post-Processing for Fairness in Classificationcited by 222
  19. 2019Linearized two-layers neural networks in high dimensioncited by 155
  20. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 150
  21. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  22. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  23. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  24. 2024Calibrated Self-Rewarding Vision Language Modelscited by 78
  25. 2023Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcarecited by 62
  26. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 47
  27. 2023Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluationcited by 46
  28. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  29. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  30. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  31. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22
  32. 2022Unifying Vision-Language Representation Space with Single-tower Transformercited by 22
  33. 2024Language Representations Can be What Recommenders Need: Findings and Potentialscited by 18
  34. 2024Beyond DAGs: A Latent Partial Causal Model for Multimodal Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 5.

19 papers.

Sort:

By area:

  1. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  2. 2025textttComplex-Edit: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmarkcited by 27
  3. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  4. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 16
  5. 2025DAMA: Data- and Model-aware Alignment of Multi-modal LLMscited by 14
  6. 2025MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Modelscited by 12
  7. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  8. 2025Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Miningcited by 8
  9. 2026Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?cited by 8
  10. 2024Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigationcited by 7
  11. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimizationcited by 6
  12. 2025Self-Improvement in Multimodal Large Language Models: A Surveycited by 6
7 more
  1. 2025From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualizationcited by 5
  2. 2025ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoningcited by 2
  3. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  4. 2026ClinCoT: Clinical-Aware Visual Chain-of-Thought for Medical Vision Language Modelscited by 1
  5. 2026Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMscited by 0
  6. 2026BARQ: Balanced Codebook Refinement for Low-Bit LLM Quantizationcited by 0
  7. 2026Do correct answers have stable latent knowledge support?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
  2. 2025Token-Level Inference-Time Alignment for Vision-Language Models
  3. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
  4. 2026ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints
  5. 2024Self-Supervised Visual Preference Alignment
  6. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks
  7. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
  8. 2026Dive Into the Implicit Biases of Low-rank Vision-language Alignment