# Chart-Supported or Model-Supplied? Examining MLLM-Generated Claims for Accessible Visualization

arXiv: https://arxiv.org/abs/2607.25021 · HTML: https://mlsanity.com/paper/2607.25021/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 17 of 17)

- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning](https://mlsanity.com/paper/2312.10160/) (2023)
- [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021)
- [Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People](https://mlsanity.com/paper/2403.15604/) (2024)
- [Misfitting With AI: How Blind People Verify and Contest AI Errors](https://mlsanity.com/paper/2408.06546/) (2024)
- [Rich Screen Reader Experiences for Accessible Data Visualization](https://mlsanity.com/paper/2205.04917/) (2022)
- [VizAbility: Enhancing Chart Accessibility with LLM-based Conversational Interaction](https://mlsanity.com/paper/2310.09611/) (2023)
- [A Dataset of Alt Texts from HCI Publications: Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in Scientific Papers](https://mlsanity.com/paper/2209.13718/) (2022)
- ["Before, I Asked My Mom, Now I Ask ChatGPT": Visual Privacy Management with Generative AI for Blind and Low-Vision People](https://mlsanity.com/paper/2507.00286/) (2025)
- [Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions](https://mlsanity.com/paper/2507.15692/) (2025)
- [Umwelt: Accessible Structured Editing of Multimodal Data Representations](https://mlsanity.com/paper/2403.00106/) (2024)
- [MAIDR: Making Statistical Visualizations Accessible with Multimodal Data Representation](https://mlsanity.com/paper/2403.00717/) (2024)
- [Faithful Chart Summarization with ChaTS-Pi](https://mlsanity.com/paper/2405.19094/) (2024)
- [Grounded Intuition of GPT-Vision's Abilities with Scientific Images](https://mlsanity.com/paper/2311.02069/) (2023)
- [Playing telephone with generative models: "verification disability," "compelled reliance," and accessibility in data visualization](https://mlsanity.com/paper/2508.12192/) (2025)
- [ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models](https://mlsanity.com/paper/2605.23694/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) can connect visualization patterns to external causes, consequences, and domain knowledge, but the evidential basis of these interpretations is often unclear. We present an exploratory study of 102 visualizations from four sources, three MLLMs, and four input conditions that vary access to the image, accessible chart context (non-image artifacts such as data tables, captions, alt text, and screen-reader structures), and withheld-context framing. Across 1,224 descriptions, we analyze model-attributed DIRECT, DERIVED, and SPECULATIVE labels and conduct an automated audit of numeric agreement. Accessible chart context shifted Gemini and GPT toward DIRECT claims and improved numeric agreement for some models. Adding the image to the full context did not yield a consistent numeric benefit, and the withheld-context prompt did not reliably increase cautious language. The prompt-defined Real-World Significance section remained predominantly SPECULATIVE. These results motivate accessible description systems that distinguish claims supported by supplied evidence from model-supplied interpretation.