# Semiotically-grounded distant viewing of diagrams: insights from two multimodal corpora

arXiv: https://arxiv.org/abs/2103.04692 · HTML: https://mlsanity.com/paper/2103.04692/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 71 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 4 of 4)

- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [AI2D-RST: A multimodal corpus of 1000 primary school science diagrams](https://mlsanity.com/paper/1912.03879/) (2019)
- [Crowdsourcing in Computer Vision](https://mlsanity.com/paper/1611.02145/) (2016)
- [Multimodal Machine Translation through Visuals and Speech](https://mlsanity.com/paper/1911.12798/) (2019)

## Built on by (most cited first, 1 of 1)

- [Explainable Search and Discovery of Visual Cultural Heritage Collections with Multimodal Large Language Models](https://mlsanity.com/paper/2411.04663/) (2024)

## Abstract (arXiv)

In this article, we bring together theories of multimodal communication and computational methods to study how primary school science diagrams combine multiple expressive resources. We position our work within the field of digital humanities, and show how annotations informed by multimodality research, which target expressive resources and discourse structure, allow imposing structure on the output of computational methods. We illustrate our approach by analysing two multimodal diagram corpora: the first corpus is intended to support research on automatic diagram processing, whereas the second is oriented towards studying diagrams as a mode of communication. Our results show that multimodally-informed annotations can bring out structural patterns in the diagrams, which also extend across diagrams that deal with different topics.