# With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2409.14917 · HTML: https://mlsanity.com/paper/2409.14917/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 10 of 10)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [What do tokens know about their characters and how do they know it?](https://mlsanity.com/paper/2206.02608/) (2022)
- [TwistList: Resources and Baselines for Tongue Twister Generation](https://mlsanity.com/paper/2306.03457/) (2023)
- [Kiki or Bouba? Sound Symbolism in Vision-and-Language Models](https://mlsanity.com/paper/2310.16781/) (2023)
- [Train & Constrain: Phonologically Informed Tongue-Twister Generation from Topics and Paraphrases](https://mlsanity.com/paper/2403.13901/) (2024)
- ["splink" is happy and "phrouth" is scary: Emotion Intensity Analysis for Nonsense Words](https://mlsanity.com/paper/2202.12132/) (2022)

## Built on by (most cited first, 4 of 4)

- [Iconicity in Large Language Models](https://mlsanity.com/paper/2501.05643/) (2025)
- [Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect](https://mlsanity.com/paper/2507.10013/) (2025)
- [Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Models](https://mlsanity.com/paper/2607.10162/) (2026)
- [Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Models](https://mlsanity.com/paper/2609.36475/) (2026)

## Abstract (arXiv)

Recently, Large Language Models (LLMs) and Vision Language Models (VLMs) have demonstrated aptitude as potential substitutes for human participants in experiments testing psycholinguistic phenomena. However, an understudied question is to what extent models that only have access to vision and text modalities are able to implicitly understand sound-based phenomena via abstract reasoning from orthography and imagery alone. To investigate this, we analyse the ability of VLMs and LLMs to demonstrate sound symbolism (i.e., to recognise a non-arbitrary link between sounds and concepts) as well as their ability to "hear" via the interplay of the language and vision modules of open and closed-source multimodal models. We perform multiple experiments, including replicating the classic Kiki-Bouba and Mil-Mal shape and magnitude symbolism tasks, and comparing human judgements of linguistic iconicity with that of LLMs. Our results show that VLMs demonstrate varying levels of agreement with human labels, and more task information may be required for VLMs versus their human counterparts for in silico experimentation. We additionally see through higher maximum agreement levels that Magnitude Symbolism is an easier pattern for VLMs to identify than Shape Symbolism, and that an understanding of linguistic iconicity is highly dependent on model size.