# Kiki or Bouba? Sound Symbolism in Vision-and-Language Models

arXiv: https://arxiv.org/abs/2310.16781 · HTML: https://mlsanity.com/paper/2310.16781/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 77 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [Editing Implicit Assumptions in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.08084/) (2023)
- [Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion](https://mlsanity.com/paper/2310.03502/) (2023)
- [Crawling the Internal Knowledge-Base of Language Models](https://mlsanity.com/paper/2301.12810/) (2023)
- [GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis](https://mlsanity.com/paper/2301.12959/) (2023)
- [Attenuating Bias in Word Vectors](https://mlsanity.com/paper/1901.07656/) (2019)
- [Disentangling visual and written concepts in CLIP](https://mlsanity.com/paper/2206.07835/) (2022)
- [Visual Commonsense in Pretrained Unimodal and Multimodal Models](https://mlsanity.com/paper/2205.01850/) (2022)
- [Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding](https://mlsanity.com/paper/2303.12513/) (2023)

## Built on by (most cited first, 14 of 14)

- [Emergent Visual-Semantic Hierarchies in Image-Text Representations](https://mlsanity.com/paper/2407.08521/) (2024)
- [Evolving Interpretable Visual Classifiers with Large Language Models](https://mlsanity.com/paper/2404.09941/) (2024)
- [With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models](https://mlsanity.com/paper/2409.14917/) (2024)
- [What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models](https://mlsanity.com/paper/2407.17974/) (2024)
- [Measuring Sound Symbolism in Audio-visual Models](https://mlsanity.com/paper/2409.12306/) (2024)
- [Verbalized Representation Learning for Interpretable Few-Shot Generalization](https://mlsanity.com/paper/2411.18651/) (2024)
- [Iconicity in Large Language Models](https://mlsanity.com/paper/2501.05643/) (2025)
- [Teaching Humans Subtle Differences with DIFFusion](https://mlsanity.com/paper/2504.08046/) (2025)
- [Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effect](https://mlsanity.com/paper/2507.10013/) (2025)
- [Seeing isn't Hearing: Benchmarking Vision Language Models at Interpreting Spectrograms](https://mlsanity.com/paper/2511.13225/) (2025)
- [Adversarially Probing Cross-Family Sound Symbolism in 27 Languages](https://mlsanity.com/paper/2512.12245/) (2025)
- [Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Models](https://mlsanity.com/paper/2607.10162/) (2026)
- [An evolutionary model of animats with VLM-based subjective evaluation](https://mlsanity.com/paper/2608.07537/) (2026)
- [Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Models](https://mlsanity.com/paper/2609.36475/) (2026)

## Abstract (arXiv)

Although the mapping between sound and meaning in human language is assumed to be largely arbitrary, research in cognitive science has shown that there are non-trivial correlations between particular sounds and meanings across languages and demographic groups, a phenomenon known as sound symbolism. Among the many dimensions of meaning, sound symbolism is particularly salient and well-demonstrated with regards to cross-modal associations between language and the visual domain. In this work, we address the question of whether sound symbolism is reflected in vision-and-language models such as CLIP and Stable Diffusion. Using zero-shot knowledge probing to investigate the inherent knowledge of these models, we find strong evidence that they do show this pattern, paralleling the well-known kiki-bouba effect in psycholinguistics. Our work provides a novel method for demonstrating sound symbolism and understanding its nature using computational tools. Our code will be made publicly available.