mlsanityData to 14 Oct 2026

The Devil is in the EOS: Sequence Training for Detailed Image Captioning

arXiv 2507.20077 ↗First posted 26 Jul 2025cs.CV cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Despite significant advances in vision-language models (VLMs), image captioning often suffers from a lack of detail, with base models producing short, generic captions. This limitation persists even though VLMs are equipped with strong vision and language backbones. While supervised data and complex reward functions have been proposed to improve detailed image captioning, we identify a simpler underlying issue: a bias towards the end-of-sequence (EOS) token, which is introduced during cross-entropy training. We propose an unsupervised method to debias the model's tendency to predict the EOS token prematurely. By reducing this bias, we encourage the generation of longer, more detailed captions without the need for intricate reward functions or supervision. Our approach is straightforward, effective, and easily applicable to any pretrained model. We demonstrate its effectiveness through experiments with three VLMs and on three detailed captioning benchmarks. Our results show a substantial increase in caption length and relevant details, albeit with an expected increase in the rate of hallucinations.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 26 Jul 2025.

Builds on

32 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  2. 2024From Pixels to Prose: A Large Dataset of Dense Image Captions14 Jun
  3. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun

Other references · 29

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  10. 2018Object Hallucination in Image Captioningcited by 669
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  12. 2016Self-critical Sequence Training for Image Captioningcited by 547
17 more
  1. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  2. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  3. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  4. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  5. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 86
  6. 2024Benchmarking and Improving Detail Image Captioncited by 81
  7. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  8. 2018Discriminability objective for training descriptive captionscited by 67
  9. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 59
  10. 2020The EOS Decision and Length Extrapolationcited by 46
  11. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 45
  12. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
  13. 2021Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modelingcited by 10
  14. 2024From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichmentcited by 8
  15. 2020CapWAP: Captioning with a Purposecited by 7
  16. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioningcited by 5
  17. 2023Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training
  2. 2024DreamLIP: Language-Image Pre-training with Long Captions
  3. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training
  4. 2025Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training
  5. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions
  6. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  7. 2025Generating Accurate and Detailed Captions for High-Resolution Images
  8. 2026CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning