# Meaning to Form: Measuring Systematicity as Information

arXiv: https://arxiv.org/abs/1906.05906 · HTML: https://mlsanity.com/paper/1906.05906/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2019.

## Built on by (most cited first, 16 of 16)

- [What do tokens know about their characters and how do they know it?](https://mlsanity.com/paper/2206.02608/) (2022)
- [A Bayesian Framework for Information-Theoretic Probing](https://mlsanity.com/paper/2109.03853/) (2021)
- [Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddings](https://mlsanity.com/paper/2205.13858/) (2022)
- [How (Non-)Optimal is the Lexicon?](https://mlsanity.com/paper/2104.14279/) (2021)
- [Modeling the Unigram Distribution](https://mlsanity.com/paper/2106.02289/) (2021)
- [Semantics of European poetry is shaped by conservative forces: The relationship between poetic meter and meaning in accentual-syllabic verse](https://mlsanity.com/paper/2109.07148/) (2021)
- [Morphological Processing of Low-Resource Languages: Where We Are and What's Next](https://mlsanity.com/paper/2203.08909/) (2022)
- [The time scale of redundancy between prosody and linguistic context](https://mlsanity.com/paper/2503.11630/) (2025)
- [Finding Concept-specific Biases in Form--Meaning Associations](https://mlsanity.com/paper/2104.06325/) (2021)
- [What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models](https://mlsanity.com/paper/2407.17974/) (2024)
- [Using Information Theory to Characterize Prosodic Typology: The Case of Tone, Pitch-Accent and Stress-Accent](https://mlsanity.com/paper/2505.07659/) (2025)
- [On Homophony and R'enyi Entropy](https://mlsanity.com/paper/2109.13766/) (2021)
- [Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect](https://mlsanity.com/paper/2507.10013/) (2025)
- [Systematicity between Forms and Meanings across Languages Supports Efficient Communication](https://mlsanity.com/paper/2601.17181/) (2026)
- [Adversarially Probing Cross-Family Sound Symbolism in 27 Languages](https://mlsanity.com/paper/2512.12245/) (2025)
- [Using Prosody to Predict Syntactic Structure](https://mlsanity.com/paper/2608.30260/) (2026)

## Abstract (arXiv)

A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between a word form and its meaning, or does some systematic phenomenon pervade? For instance, does the character bigram textitgl have any systematic relationship to the meaning of words like textitglisten, textitgleam and textitglow? In this work, we offer a holistic quantification of the systematicity of the sign using mutual information and recurrent neural networks. We employ these in a data-driven and massively multilingual approach to the question, examining 106 languages. We find a statistically significant reduction in entropy when modeling a word form conditioned on its semantic representation. Encouragingly, we also recover well-attested English examples of systematic affixes. We conclude with the meta-point: Our approximate effect size (measured in bits) is quite small---despite some amount of systematicity between form and meaning, an arbitrary relationship and its resulting benefits dominate human language.