mlsanityData to 7 Oct 2026

TLDR: Token-Level Detective Reward Model for Large Vision Language Models

arXiv 2410.04734 ↗First posted 7 Oct 2024cs.LG cs.CL cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Although reward models have been successful in improving multimodal large language models, the reward models themselves remain brutal and contain minimal information. Notably, existing reward models only mimic human annotations by assigning only one binary feedback to any text, no matter how long the text is. In the realm of multimodal language models, where models are required to process both images and texts, a naive reward model may learn implicit biases toward texts and become less grounded in images. In this paper, we propose a Token-Level Detective Reward Model (TLDR) to provide fine-grained annotations to each text token. We first introduce a perturbation-based method to generate synthetic hard negatives and their token-level labels to train TLDR models. Then we show the rich usefulness of TLDR models both in assisting off-the-shelf models to self-correct their generations, and in serving as a hallucination evaluation tool. We show that TLDR automatically trains a token-level likelihood optimization, and can improve the base model's performance significantly. Finally, we show that TLDR models can significantly speed up human annotation by 3 times to acquire a broader range of high-quality vision language data.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 54 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

42 of 54 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  3. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models3 May
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  6. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization13 Mar
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  8. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning5 Sep

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
22 more
  1. 2018Object Hallucination in Image Captioningcited by 661
  2. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  4. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  5. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  8. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  9. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  10. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  11. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  12. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  13. 2020Learning Dense Representations of Phrases at Scalecited by 91
  14. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  15. 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
  16. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  17. 2023FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Modelscited by 44
  18. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  19. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  20. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  21. 2024Selective Preference Optimization via Token-Level Reward Function Estimationcited by 25
  22. 2023SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examplescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 3.

15 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  3. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  4. 2025Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositionscited by 15
  5. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 12
  6. 2025Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learningcited by 11
  7. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  8. 2025Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Modelscited by 8
  9. 2025Vevo2: A Unified and Controllable Framework for Speech and Singing Voice Generationcited by 5
  10. 2025ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoningcited by 2
  11. 2024VisualLens: Personalization through Task-Agnostic Visual Historycited by 0
  12. 2025Token-Level Inference-Time Alignment for Vision-Language Modelscited by 0
3 more
  1. 2026Real-Time Visual Attribution Streaming in Thinking Modelcited by 0
  2. 2026Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Updatecited by 0
  3. 2026Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shiftcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image
  2. 2024Calibrated Self-Rewarding Vision Language Models
  3. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models
  4. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models
  5. 2025Self-Rewarding Large Vision-Language Models for Optimizing Prompts in Text-to-Image Generation
  6. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
  7. 2025Controlling Multimodal LLMs via Reward-guided Decoding
  8. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model