# Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions

arXiv: https://arxiv.org/abs/2507.15692 · HTML: https://mlsanity.com/paper/2507.15692/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 43 of its 75 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 43)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)
- [Personalized Language Modeling from Personalized Human Feedback](https://mlsanity.com/paper/2402.05133/) (2024)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [Logic-Guided Data Augmentation and Regularization for Consistent Question Answering](https://mlsanity.com/paper/2004.10157/) (2020)
- [Graphologue: Exploring Large Language Model Responses with Interactive Diagrams](https://mlsanity.com/paper/2305.11473/) (2023)
- [Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation](https://mlsanity.com/paper/2310.12953/) (2023)
- [Design Principles for Generative AI Applications](https://mlsanity.com/paper/2401.14484/) (2024)
- [Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models](https://mlsanity.com/paper/2305.11483/) (2023)
- [Writer-Defined AI Personas for On-Demand Feedback Generation](https://mlsanity.com/paper/2309.10433/) (2023)
- [Supporting Sensemaking of Large Language Model Outputs at Scale](https://mlsanity.com/paper/2401.13726/) (2024)
- [WorldScribe: Towards Context-Aware Live Visual Descriptions](https://mlsanity.com/paper/2408.06627/) (2024)
- [Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People](https://mlsanity.com/paper/2403.15604/) (2024)
- [RELIC: Investigating Large Language Model Responses using Self-Consistency](https://mlsanity.com/paper/2311.16842/) (2023)
- [Rescribe: Authoring and Automatically Editing Audio Descriptions](https://mlsanity.com/paper/2010.03667/) (2020)
- [Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions](https://mlsanity.com/paper/2302.07248/) (2023)
- [AVscript: Accessible Video Editing with Audio-Visual Scripts](https://mlsanity.com/paper/2302.14117/) (2023)
- [Proxona: Supporting Creators' Sensemaking and Ideation with LLM-Powered Audience Personas](https://mlsanity.com/paper/2408.10937/) (2024)
- [GenAssist: Making Image Generation Accessible](https://mlsanity.com/paper/2307.07589/) (2023)
- [Long-Form Answers to Visual Questions from Blind and Low Vision People](https://mlsanity.com/paper/2408.06303/) (2024)
- [Quality Estimation for Image Captions Based on Large-scale Human Evaluations](https://mlsanity.com/paper/1909.03396/) (2019)
- [Misfitting With AI: How Blind People Verify and Contest AI Errors](https://mlsanity.com/paper/2408.06546/) (2024)
- [Making Short-Form Videos Accessible with Hierarchical Video Summaries](https://mlsanity.com/paper/2402.10382/) (2024)

## Built on by (most cited first, 6 of 6)

- ["It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models](https://mlsanity.com/paper/2511.08917/) (2025)
- [How Multimodal Large Language Models Support Access to Visual Information: A Diary Study With Blind and Low Vision People](https://mlsanity.com/paper/2602.13469/) (2026)
- [Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users](https://mlsanity.com/paper/2602.16930/) (2026)
- [Finding the Signal in the Noise: An Exploratory Study on Assessing the Effectiveness of AI and Accessibility Forums for Blind Users' Support Needs](https://mlsanity.com/paper/2602.18623/) (2026)
- [Cripping AI: Reimagining AI Through Lived Disability Experiences](https://mlsanity.com/paper/2605.02080/) (2026)
- [Chart-Supported or Model-Supplied? Examining MLLM-Generated Claims for Accessible Visualization](https://mlsanity.com/paper/2607.25021/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) provide new opportunities for blind and low vision (BLV) people to access visual information in their daily lives. However, these models often produce errors that are difficult to detect without sight, posing safety and social risks in scenarios from medication identification to outfit selection. While BLV MLLM users use creative workarounds such as cross-checking between tools and consulting sighted individuals, these approaches are often time-consuming and impractical. We explore how systematically surfacing variations across multiple MLLM responses can support BLV users to detect unreliable information without visually inspecting the image. We contribute a design space for eliciting and presenting variations in MLLM descriptions, a prototype system implementing three variation presentation styles, and findings from a user study with 15 BLV participants. Our results demonstrate that presenting variations significantly increases users' ability to identify unreliable claims (by 4.9x using our approach compared to single descriptions) and significantly decreases perceived reliability of MLLM responses. 14 of 15 participants preferred seeing variations of MLLM responses over a single description, and all expressed interest in using our system for tasks from understanding a tornado's path to posting an image on social media.