A Systematic Survey on Image Description Techniques for STEM Domains
Not on the map. Why?
Abstract (arXiv)
The proliferation of visual data in Science, Technology, Engineering, and Mathematics (STEM) fields presents accessibility barrier for individuals with blindness or visual impairments. While recent advances in Artificial Intelligence (AI) offer new opportunities to generate textual descriptions of STEM images, the research landscape is fragmented and its impact on real users remains limited. This systematic survey examines 20 peer-reviewed studies on AI-based techniques for describing STEM visuals, with a specific focus on accessibility and human-computer interaction. Following the PRISMA methodology and a ROBIS-based risk-of-bias assessment, the review analyzes (i) the types of STEM visuals targeted, (ii) the AI and machine learning architectures employed, (iii) the datasets and evaluation metrics adopted, and (iv) the interaction modalities through which descriptions are delivered. The analysis reveals a shift from static, one-shot alt text toward interactive and multimodal systems that integrate conversational interfaces, keyboard navigation, and audio or haptic feedback. However, critical challenges persist, including factual inaccuracies and hallucinations, the scarcity of accessibility-first datasets co-designed with blind and low-vision users, and a heavy reliance on automatic text-overlap metrics that poorly capture perceived usefulness and trust. The survey concludes by outlining key research directions for HCI, emphasizing user-controlled verbosity, explainable and verifiable AI pipelines, and the integration of accessible description tools into mainstream STEM authoring and learning environments.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 59 references that are in the map. First posted on arXiv 22 May 2026.
Builds on
15 of 59 references.
References · 15
- 2017Attention Is All You Needcited by 60,361
- 2021SciCap: Generating Captions for Scientific Figurescited by 77
- 2021Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Contentcited by 39
- 2016Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measurescited by 36
- 2023A Prompt Log Analysis of Text-to-Image Generation Systemscited by 24
- 2023SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioningcited by 24
- 2024Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoningcited by 19
- 2022Rich Screen Reader Experiences for Accessible Data Visualizationcited by 12
- 2023VizAbility: Enhancing Chart Accessibility with LLM-based Conversational Interactioncited by 12
- 2025Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Paperscited by 10
- 2025MatplotAlt: A Python Library for Adding Alt Text to Matplotlib Figures in Computational Notebookscited by 3
- 2020Textual Description for Mathematical Equationscited by 2
Built on by
Not cited yet.