# AI2D-RST: A multimodal corpus of 1000 primary school science diagrams

arXiv: https://arxiv.org/abs/1912.03879 · HTML: https://mlsanity.com/paper/1912.03879/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2019.

## Built on by (most cited first, 30 of 60)

- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction](https://mlsanity.com/paper/2501.01957/) (2025)
- [VITA: Towards Open-Source Interactive Omni Multimodal LLM](https://mlsanity.com/paper/2408.05211/) (2024)
- [Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning](https://mlsanity.com/paper/2401.06805/) (2024)
- [Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want](https://mlsanity.com/paper/2403.20271/) (2024)
- [Scaling Spatial Intelligence with Multimodal Foundation Models](https://mlsanity.com/paper/2511.13719/) (2025)
- [Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning](https://mlsanity.com/paper/2503.07065/) (2025)
- [Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology](https://mlsanity.com/paper/2507.07999/) (2025)
- [Scaling Language-Free Visual Representation Learning](https://mlsanity.com/paper/2504.01017/) (2025)
- [Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models](https://mlsanity.com/paper/2504.15271/) (2025)
- [Reconstructive Visual Instruction Tuning](https://mlsanity.com/paper/2410.09575/) (2024)
- [Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2601.16208/) (2026)
- [Beyond Language Modeling: An Exploration of Multimodal Pretraining](https://mlsanity.com/paper/2603.03276/) (2026)
- [Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM](https://mlsanity.com/paper/2412.09530/) (2024)
- [Boosting MLLM Reasoning with Text-Debiased Hint-GRPO](https://mlsanity.com/paper/2503.23905/) (2025)
- [MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA](https://mlsanity.com/paper/2410.07177/) (2024)
- [GlitchBench: Can large multimodal models detect video game glitches?](https://mlsanity.com/paper/2312.05291/) (2023)
- [Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training](https://mlsanity.com/paper/2509.26625/) (2025)
- [Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects](https://mlsanity.com/paper/2312.05278/) (2023)
- [MMPB: It's Time for Multi-Modal Personalization](https://mlsanity.com/paper/2509.22820/) (2025)
- [PPTBench: Towards Holistic Evaluation of Large Language Models for PowerPoint Layout and Design Understanding](https://mlsanity.com/paper/2512.02624/) (2025)
- [Gatekeeper: Improving Model Cascades Through Confidence Tuning](https://mlsanity.com/paper/2502.19335/) (2025)
- [COSMic: A Coherence-Aware Generation Metric for Image Descriptions](https://mlsanity.com/paper/2109.05281/) (2021)
- [AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models](https://mlsanity.com/paper/2406.09295/) (2024)
- [TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings](https://mlsanity.com/paper/2409.09564/) (2024)
- [WorldVQA: Measuring Atomic World Knowledge in Multimodal Large Language Models](https://mlsanity.com/paper/2602.02537/) (2026)
- [Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding](https://mlsanity.com/paper/2603.18472/) (2026)
- [Multi-modal Summarization in Model-Based Engineering: Automotive Software Development Case Study](https://mlsanity.com/paper/2503.04506/) (2025)
- [Ranked from Within: Ranking Large Multimodal Models Without Labels](https://mlsanity.com/paper/2412.06461/) (2024)
- [Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task](https://mlsanity.com/paper/2505.21850/) (2025)

## Abstract (arXiv)

This article introduces AI2D-RST, a multimodal corpus of 1000 English-language diagrams that represent topics in primary school natural sciences, such as food webs, life cycles, moon phases and human physiology. The corpus is based on the Allen Institute for Artificial Intelligence Diagrams (AI2D) dataset, a collection of diagrams with crowd-sourced descriptions, which was originally developed to support research on automatic diagram understanding and visual question answering. Building on the segmentation of diagram layouts in AI2D, the AI2D-RST corpus presents a new multi-layer annotation schema that provides a rich description of their multimodal structure. Annotated by trained experts, the layers describe (1) the grouping of diagram elements into perceptual units, (2) the connections set up by diagrammatic elements such as arrows and lines, and (3) the discourse relations between diagram elements, which are described using Rhetorical Structure Theory (RST). Each annotation layer in AI2D-RST is represented using a graph. The corpus is freely available for research and teaching.