mlsanityData to 7 Oct 2026

Token-Level Inference-Time Alignment for Vision-Language Models

arXiv 2510.21794 ↗First posted 20 Oct 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) have become essential backbones of modern multimodal intelligence, yet their outputs remain prone to hallucination-plausible text misaligned with visual inputs. Existing alignment approaches often rely on expensive fine-tuning with annotated preference data or sequence-level inference strategies that provide only coarse, delayed feedback. To overcome these limitations, we present TITA (Token-level Inference-Time Alignment), a lightweight framework that freezes the base VLM and instead trains a reward model to approximate its distribution. During inference, implicit preference signals are extracted as log-probability ratios between the reward model and the target VLM, yielding dense autoregressive feedback. This formulation can be viewed as an inference-time variant of Direct Preference Optimization (DPO), providing token-level corrective signals without retraining the backbone. Extensive evaluations on LLaVA-1.5-7B and 13B show consistent gains across 12 benchmarks, with improvements of 8.6% on MMVet and 6.7% on POPE, indicating stronger general understanding and reduced hallucinations. Additional experiments on Qwen2.5-VL-7B and DeepSeek-VL2-27.5B show comparable gains, especially in hallucination reduction and VQA accuracy, while incurring negligible inference overhead.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 59 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

53 of 59 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  2. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Models7 Oct
  3. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  5. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks12 Jun
  6. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models7 Jun
  7. 2024Calibrated Self-Rewarding Vision Language Models23 May
  8. 2024Self-Supervised Visual Preference Alignment16 Apr
  9. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  11. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
4 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  4. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 37

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2024GPT-4o System Cardcited by 4,467
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
25 more
  1. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  4. 2018Object Hallucination in Image Captioningcited by 661
  5. 2024Self-Rewarding Language Modelscited by 654
  6. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  7. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  8. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 223
  10. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  11. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  12. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  13. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attackscited by 89
  14. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  15. 2024Visual Hallucinations of Multi-modal Large Language Modelscited by 68
  16. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 54
  17. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  18. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  19. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  20. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  21. 2024ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modelingcited by 34
  22. 2024Hallucination Benchmark in Medical Visual Question Answeringcited by 18
  23. 2023Multi-Label Self-Supervised Learning with Scene Imagescited by 17
  24. 2024Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Groundingcited by 10
  25. 2024MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answeringcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints
  2. 2024Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment
  3. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuning
  4. 2025VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
  5. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
  6. 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
  7. 2026Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Models
  8. 2025Test-Time Consistency in Vision Language Models