# A Systematic Survey on Image Description Techniques for STEM Domains

arXiv: https://arxiv.org/abs/2607.21611 · HTML: https://mlsanity.com/paper/2607.21611/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 59 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SciCap: Generating Captions for Scientific Figures](https://mlsanity.com/paper/2110.11624/) (2021)
- [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021)
- [Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures](https://mlsanity.com/paper/1601.03896/) (2016)
- [A Prompt Log Analysis of Text-to-Image Generation Systems](https://mlsanity.com/paper/2303.04587/) (2023)
- [SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning](https://mlsanity.com/paper/2306.03491/) (2023)
- [Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning](https://mlsanity.com/paper/2412.13540/) (2024)
- [Rich Screen Reader Experiences for Accessible Data Visualization](https://mlsanity.com/paper/2205.04917/) (2022)
- [VizAbility: Enhancing Chart Accessibility with LLM-based Conversational Interaction](https://mlsanity.com/paper/2310.09611/) (2023)
- [Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers](https://mlsanity.com/paper/2507.10787/) (2025)
- [MatplotAlt: A Python Library for Adding Alt Text to Matplotlib Figures in Computational Notebooks](https://mlsanity.com/paper/2503.20089/) (2025)
- [Textual Description for Mathematical Equations](https://mlsanity.com/paper/2008.02980/) (2020)
- [Alt4Blind: A User Interface to Simplify Charts Alt-Text Creation](https://mlsanity.com/paper/2405.19111/) (2024)
- [Evaluation of Automated Image Descriptions for Visually Impaired Students](https://mlsanity.com/paper/2106.15553/) (2021)
- [A framework for improving the accessibility of research papers on arXiv.org](https://mlsanity.com/paper/2212.07286/) (2022)

## Abstract (arXiv)

The proliferation of visual data in Science, Technology, Engineering, and Mathematics (STEM) fields presents accessibility barrier for individuals with blindness or visual impairments. While recent advances in Artificial Intelligence (AI) offer new opportunities to generate textual descriptions of STEM images, the research landscape is fragmented and its impact on real users remains limited. This systematic survey examines 20 peer-reviewed studies on AI-based techniques for describing STEM visuals, with a specific focus on accessibility and human-computer interaction. Following the PRISMA methodology and a ROBIS-based risk-of-bias assessment, the review analyzes (i) the types of STEM visuals targeted, (ii) the AI and machine learning architectures employed, (iii) the datasets and evaluation metrics adopted, and (iv) the interaction modalities through which descriptions are delivered. The analysis reveals a shift from static, one-shot alt text toward interactive and multimodal systems that integrate conversational interfaces, keyboard navigation, and audio or haptic feedback. However, critical challenges persist, including factual inaccuracies and hallucinations, the scarcity of accessibility-first datasets co-designed with blind and low-vision users, and a heavy reliance on automatic text-overlap metrics that poorly capture perceived usefulness and trust. The survey concludes by outlining key research directions for HCI, emphasizing user-controlled verbosity, explainable and verifiable AI pipelines, and the integration of accessible description tools into mainstream STEM authoring and learning environments.