mlsanityData to 14 Oct 2026

Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation

arXiv 2412.14487 ↗First posted 19 Dec 2024cs.CVcited by 7

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Direct Preference Optimization (DPO) has been demonstrated to be highly effective in mitigating hallucinations in Large Vision Language Models (LVLMs) by aligning their outputs more closely with human preferences. Despite the recent progress, existing methods suffer from two drawbacks: 1) Lack of scalable token-level rewards; and 2) Neglect of visual-anchored tokens. To this end, we propose a novel Token Preference Optimization model with self-calibrated rewards (dubbed as TPO), which adaptively attends to visual-correlated tokens without fine-grained annotations. Specifically, we introduce a token-level visual-anchored reward as the difference of the logistic distributions of generated tokens conditioned on the raw image and the corrupted one. In addition, to highlight the informative visual-anchored tokens, a visual-aware training objective is proposed to enhance more accurate token-level optimization. Extensive experimental results have manifested the state-of-the-art performance of the proposed TPO. For example, by building on top of LLAVA-1.5-7B, our TPO boosts the performance absolute improvement for hallucination benchmarks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 38 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

25 of 38 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization5 Nov
  2. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMs25 Sep
  3. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  4. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback22 Apr
  5. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  10. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  11. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  12. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision13 Nov
1 more
  1. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  10. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  11. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  12. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignmentcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 24
  2. 2025DAMA: Data- and Model-aware Alignment of Multi-modal LLMscited by 14
  3. 2025AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimizationcited by 6
  4. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  5. 2025ReLoop: "Seeing Twice and Thinking Backwards" via Closed-loop Training to Mitigate Hallucinations in Multimodal understandingcited by 1
  6. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  7. 2026CARL: Constraint-Aware Reinforcement Learning for Planning with LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
  2. 2026P2-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
  3. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs
  4. 2026Online Self-Calibration Against Hallucination in Vision-Language Models
  5. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key
  6. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucination
  7. 2026Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs
  8. 2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation