mlsanityData to 14 Oct 2026

TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMs

arXiv 2507.21584 ↗First posted 29 Jul 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) are prone to hallucinations, generating plausible but visually ungrounded outputs, partly because direct preference optimization (DPO) overfits to superficial linguistic cues under static preference supervision. We propose TARS, a token-adaptive preference strategy that reformulates DPO as a principled min-max optimization problem. The inner maximization selectively perturbs visual-agnostic tokens to induce worst-case distributional shifts, while the outer minimization enforces alignment with causal visual signals rather than surface-level patterns. A novel spectral alignment loss further regularizes hidden representations in the frequency domain via the Fast Fourier Transform (FFT), preserving global semantic structure without rigid token-level correspondence. We evaluate TARS across multiple hallucination benchmarks. Using only 4.8k preference samples without expert feedback, TARS reduces hallucination rates from 26.4% to 13.2% and cognition scores from 2.5 to 0.4, outperforming standard DPO by a large margin. Notably, TARS surpasses 5× LLM-based data augmentation trained on 28.8k samples (Hal-Rate: 16.0% vs. 13.2%), demonstrating that reshaping the optimization landscape via adversarial token perturbation is fundamentally more effective than scaling training data. TARS further narrows the gap with GPT-4o on key metrics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 84 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

73 of 84 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training9 Mar
  2. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs28 Jan
  3. 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key16 Jan
  4. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization5 Nov
  5. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs19 Aug
  6. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  7. 2024Multi-Object Hallucination in Vision-Language Models8 Jul
  8. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models4 Jun
  9. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment28 May
  10. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  11. 2024Unified Hallucination Detection for Multimodal Large Language Models5 Feb
  12. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
6 more
  1. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  3. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  4. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  5. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May
  6. 2022MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model11 Oct

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
38 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2017Constrained Policy Optimizationcited by 983
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  8. 2018Object Hallucination in Image Captioningcited by 669
  9. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  10. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  11. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  12. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  13. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  14. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  15. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  16. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  17. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 172
  18. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  19. 2021Causal Attention for Vision-Language Taskscited by 111
  20. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  21. 2025Efficient Reasoning Models: A Surveycited by 95
  22. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  23. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 90
  24. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  25. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  26. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 61
  27. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  28. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  29. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  30. 2023Explore Spurious Correlations at the Concept Level in Language Models for Text Classificationcited by 48
  31. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  32. 2024A Concept-Based Explainability Framework for Large Multimodal Modelscited by 39
  33. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  34. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learningcited by 30
  35. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
  36. 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Modelscited by 26
  37. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistantscited by 25
  38. 2025Reasoning of Large Language Models over Knowledge Graphs with Super-Relationscited by 21

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
  2. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
  3. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization
  4. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
  5. 2026Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs
  6. 2026Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models
  7. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation
  8. 2025Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization