mlsanityData to 14 Oct 2026

From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment

arXiv 2406.13912 ↗First posted 20 Jun 2024cs.CVcited by 8

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Large language models (LLMs) have enhanced the capacity of vision-language models to caption visual text. This generative approach to image caption enrichment further makes textual captions more descriptive, improving alignment with the visual context. However, while many studies focus on benefits of generative caption enrichment (GCE), are there any negative side effects? We compare standard-format captions and recent GCE processes from the perspectives of "gender bias" and "hallucination", showing that enriched captions suffer from increased gender bias and hallucination. Furthermore, models trained on these enriched captions amplify gender bias by an average of 30.9% and increase hallucination by 59.5%. This study serves as a caution against the trend of making captions more descriptive.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 40 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

30 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities26 Jan

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  10. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  11. 2021Image Segmentation Using Text and Image Promptscited by 512
  12. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
17 more
  1. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  2. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 410
  3. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  4. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  5. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  6. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  7. 2015Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labelscited by 64
  8. 2021Directional Bias Amplificationcited by 54
  9. 2023MISGENDERED: Limits of Large Language Models in Understanding Pronounscited by 43
  10. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  11. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  12. 2024Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generationcited by 28
  13. 2022Men Also Do Laundry: Multi-Attribute Bias Amplificationcited by 21
  14. 2023IC3: Image Captioning by Committee Consensuscited by 20
  15. 2015Language Models for Image Captioning: The Quirks and What Workscited by 19
  16. 2023Model-Agnostic Gender Debiased Image Captioningcited by 17
  17. 2021Training a Resilient Q-Network against Observational Interferencecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 2.

8 papers.

  1. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIPcited by 19
  2. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  3. 2024Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognitioncited by 11
  4. 2025CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioningcited by 5
  5. 2025Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluationcited by 5
  6. 2025The Devil is in the EOS: Sequence Training for Detailed Image Captioningcited by 1
  7. 2026ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Modelscited by 0
  8. 2026A Matched-Budget Audit Framework for Recaptioned Image-Text Supervision Distributionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
  2. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives
  3. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals
  4. 2024Is Your Text-to-Image Model Robust to Caption Noise?
  5. 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions
  6. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution
  7. 2025Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training
  8. 2026Gender Bias in Vision-Language In-Context Learning