mlsanityData to 7 Oct 2026

"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models

arXiv 2511.08917 ↗First posted 12 Nov 2025cs.HC cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) are increasingly used by blind and low-vision (BLV) people to identify and understand products in their everyday lives, such as food, personal care items, and household goods. Despite their prevalence, we lack an empirical understanding of how common image quality issues--such as blur, misframing, and rotation--affect the accuracy of VLM-generated captions and whether the resulting captions meet BLV people's information needs. Based on a survey of 86 BLV participants, we develop an annotated dataset of 1,859 product images from BLV people to systematically evaluate how image quality issues affect VLM-generated captions. While the best VLM achieves 98% accuracy on images with no quality issues, accuracy drops to 75% overall when quality issues are present, worsening considerably as issues compound. We discuss the need for model evaluations that center on disabled people's experiences throughout the process and offer concrete recommendations for HCI and ML researchers to make VLMs more reliable for BLV people.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 145 references that are in the map. First posted on arXiv 12 Nov 2025.

Builds on

73 of 145 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions21 Jul
  2. 2025Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward23 Apr
  3. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar
  4. 2025Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models22 Feb
  5. 2024Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities21 Dec
  6. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  7. 2024Long-Form Answers to Visual Questions from Blind and Low Vision People12 Aug
  8. 2023Improving Multimodal Datasets with Image Captioning19 Jul

Other references · 65

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  10. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  12. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
38 more
  1. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  3. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  4. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  5. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  6. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  7. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  8. 2017Arbitrary-Oriented Scene Text Detection via Rotation Proposalscited by 176
  9. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  10. 2021No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistencycited by 165
  11. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  12. 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
  13. 2015Learning a Convolutional Neural Network for Non-uniform Motion Blur Removalcited by 148
  14. 2024What Large Language Models Know and What People Think They Knowcited by 109
  15. 2020Captioning Images Taken by People Who Are Blindcited by 108
  16. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 101
  17. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 85
  18. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  19. 2023Selectively Answering Ambiguous Questionscited by 63
  20. 2019TIGEr: Text-to-Image Grounding for Image Caption Evaluationcited by 58
  21. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  22. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  23. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 44
  24. 2019Why Does a Visual Question Have Different Answers?cited by 34
  25. 2023ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognitioncited by 34
  26. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  27. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 32
  28. 2020Exposure Trajectory Recovery from Motion Blurcited by 31
  29. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Modelscited by 31
  30. 2020Assessing Image Quality Issues for Real-World Problemscited by 28
  31. 2024WorldScribe: Towards Context-Aware Live Visual Descriptionscited by 26
  32. 2024Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision Peoplecited by 24
  33. 2024VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Modelscited by 23
  34. 2022A Tri-Layer Plugin to Improve Occluded Detectioncited by 21
  35. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  36. 2025Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Taskscited by 21
  37. 2023IC3: Image Captioning by Committee Consensuscited by 20
  38. 2023GenAssist: Making Image Generation Accessiblecited by 18

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Toward Scalable Audio Description Quality Control: A Workflow for Evaluating Human and VLM Raterscited by 0
  2. 2026Cripping AI: Reimagining AI Through Lived Disability Experiencescited by 0
  3. 2026Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Vision-Language Models under Cultural and Inclusive Considerations
  2. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  3. 2024Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction
  4. 2023Explaining CLIP's performance disparities on data from blind/low vision users
  5. 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models
  6. 2025Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives
  7. 2024Is Your Text-to-Image Model Robust to Caption Noise?
  8. 2025How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptions