mlsanityData to 14 Oct 2026

LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences

arXiv 2507.19362 ↗First posted 25 Jul 2025cs.CV cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. LOTUS comprehensively evaluates various aspects, including caption quality (e.g., alignment, descriptiveness), risks (eg, hallucination), and societal biases (e.g., gender bias) while enabling preference-oriented evaluations by tailoring criteria to diverse user preferences. Our analysis of recent LVLMs reveals no single model excels across all criteria, while correlations emerge between caption detail and bias risks. Preference-oriented evaluations demonstrate that optimal model selection depends on user priorities.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 43 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

30 of 43 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
18 more
  1. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  3. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  4. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  5. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  6. 2019Connecting Vision and Language with Localized Narrativescited by 226
  7. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  8. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  9. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  10. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captionscited by 70
  11. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 52
  12. 2020Mitigating Gender Bias in Captioning Systemscited by 50
  13. 2022Men Also Do Laundry: Multi-Attribute Bias Amplificationcited by 21
  14. 2023MLLMs-Augmented Visual-Language Representation Learningcited by 21
  15. 2023IC3: Image Captioning by Committee Consensuscited by 20
  16. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIPcited by 19
  17. 2023Model-Agnostic Gender Debiased Image Captioningcited by 17
  18. 2024Resampled Datasets Are Not Enough: Mitigating Societal Bias Beyond Single Attributescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 5
  2. 2026HalDec-Bench: Benchmarking Hallucination Detector in Image Captioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  2. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives
  3. 2024Benchmarking and Improving Detail Image Caption
  4. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
  5. 2026ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs
  6. 2025Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
  7. 2023CLAIR: Evaluating Image Captions with Large Language Models
  8. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?