# Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content

arXiv: https://arxiv.org/abs/2110.04406 · HTML: https://mlsanity.com/paper/2110.04406/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 108 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 20 of 20)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model](https://mlsanity.com/paper/2010.09142/) (2020)
- [NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries](https://mlsanity.com/paper/2008.10723/) (2020)
- [Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations](https://mlsanity.com/paper/2110.00680/) (2021)
- [Figure Captioning with Reasoning and Sequence-Level Training](https://mlsanity.com/paper/1906.02850/) (2019)
- [Chart-Text: A Fully Automated Chart Image Descriptor](https://mlsanity.com/paper/1812.10636/) (2018)
- [Computer Vision and Conflicting Values: Describing People with Automated Alt Text](https://mlsanity.com/paper/2105.12754/) (2021)
- [Characterizing Automated Data Insights](https://mlsanity.com/paper/2008.13060/) (2020)
- [Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Charts](https://mlsanity.com/paper/2101.08235/) (2021)
- [Viral Visualizations: How Coronavirus Skeptics Use Orthodox Data Practices to Promote Unorthodox Science Online](https://mlsanity.com/paper/2101.07993/) (2021)
- [Optimizing colormaps with consideration for color vision deficiency to enable accurate interpretation of scientific data](https://mlsanity.com/paper/1712.01662/) (2017)
- [Sociotechnical Considerations for Accessible Visualization Design](https://mlsanity.com/paper/1909.05118/) (2019)
- [What are Data Insights to Professional Visualization Users?](https://mlsanity.com/paper/2008.13057/) (2020)
- [Communicative Visualizations as a Learning Problem](https://mlsanity.com/paper/2009.07095/) (2020)
- [Understanding Visualization: A Formal Approach using Category Theory and Semiotics](https://mlsanity.com/paper/1311.4376/) (2013)

## Built on by (most cited first, 30 of 39)

- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [Chart Question Answering: State of the Art and Future Directions](https://mlsanity.com/paper/2205.03966/) (2022)
- [DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articles](https://mlsanity.com/paper/2308.04076/) (2023)
- [LineCap: Line Charts for Data Visualization Captioning Models](https://mlsanity.com/paper/2207.07243/) (2022)
- [Summaries as Captions: Generating Figure Captions for Scientific Documents with Automated Text Summarization](https://mlsanity.com/paper/2302.12324/) (2023)
- [GPT-Based Models Meet Simulation: How to Efficiently Use Large-Scale Pre-Trained Language Models Across Simulation Tasks](https://mlsanity.com/paper/2306.13679/) (2023)
- [ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset](https://mlsanity.com/paper/2501.09349/) (2025)
- [ChartCap: Mitigating Hallucination of Dense Chart Captioning](https://mlsanity.com/paper/2508.03164/) (2025)
- [Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions](https://mlsanity.com/paper/2409.19747/) (2024)
- [CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Models](https://mlsanity.com/paper/2505.17202/) (2025)
- [DATAWEAVER: Authoring Data-Driven Narratives through the Integrated Composition of Visualization and Text](https://mlsanity.com/paper/2503.22946/) (2025)
- [Grounded Intuition of GPT-Vision's Abilities with Scientific Images](https://mlsanity.com/paper/2311.02069/) (2023)
- [AltChart: Enhancing VLM-based Chart Summarization Through Multi-Pretext Tasks](https://mlsanity.com/paper/2405.13580/) (2024)
- [Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptions](https://mlsanity.com/paper/2503.13369/) (2025)
- [Text2GraphQuery-Bench: A Text to Graph Query Benchmark](https://mlsanity.com/paper/2602.11745/) (2026)
- [Alt4Blind: A User Interface to Simplify Charts Alt-Text Creation](https://mlsanity.com/paper/2405.19111/) (2024)
- [MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features](https://mlsanity.com/paper/2409.16765/) (2024)
- [ChartKG: A Knowledge-Graph-Based Representation for Chart Images](https://mlsanity.com/paper/2410.09761/) (2024)
- [Aligning Instance-Semantic Sparse Representation towards Unsupervised Object Segmentation and Shape Abstraction with Repeatable Primitives](https://mlsanity.com/paper/2503.06947/) (2025)
- [ChartGaze: Enhancing Chart Understanding in LVLMs with Eye-Tracking Guided Attention Refinement](https://mlsanity.com/paper/2509.13282/) (2025)
- [ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models](https://mlsanity.com/paper/2605.23694/) (2026)
- [Leveraging Author-Specific Context for Scientific Figure Caption Generation: 3rd SciCap Challenge](https://mlsanity.com/paper/2510.07993/) (2025)
- [Safire: Similarity Framework for Visualization Retrieval](https://mlsanity.com/paper/2510.16662/) (2025)
- [Three Modalities, Two Design Probes, One Prototype, and No Vision: Experience-Based Co-Design of a Multi-modal 3D Data Visualization Tool](https://mlsanity.com/paper/2604.09426/) (2026)
- [Querying Multimodal Scientific Papers with AI: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientists](https://mlsanity.com/paper/2607.18514/) (2026)
- [AnnoBench: A Benchmark for Visualization Annotation Generation](https://mlsanity.com/paper/2607.25911/) (2026)
- [Captioning Visualizations with Large Language Models (CVLLM): A Tutorial](https://mlsanity.com/paper/2406.19512/) (2024)
- [Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks](https://mlsanity.com/paper/2510.06071/) (2025)

## Abstract (arXiv)

Natural language descriptions sometimes accompany visualizations to better communicate and contextualize their insights, and to improve their accessibility for readers with disabilities. However, it is difficult to evaluate the usefulness of these descriptions, and how effectively they improve access to meaningful information, because we have little understanding of the semantic content they convey, and how different readers receive this content. In response, we introduce a conceptual model for the semantic content conveyed by natural language descriptions of visualizations. Developed through a grounded theory analysis of 2,147 sentences, our model spans four levels of semantic content: enumerating visualization construction properties (e.g., marks and encodings); reporting statistical concepts and relations (e.g., extrema and correlations); identifying perceptual and cognitive phenomena (e.g., complex trends and patterns); and elucidating domain-specific insights (e.g., social and political context). To demonstrate how our model can be applied to evaluate the effectiveness of visualization descriptions, we conduct a mixed-methods evaluation with 30 blind and 90 sighted readers, and find that these reader groups differ significantly on which semantic content they rank as most useful. Together, our model and findings suggest that access to meaningful information is strongly reader-specific, and that research in automatic visualization captioning should orient toward descriptions that more richly communicate overall trends and statistics, sensitive to reader preferences. Our work further opens a space of research on natural language as a data interface coequal with visualization.