mlsanityData to 14 Oct 2026

Kiki or Bouba? Sound Symbolism in Vision-and-Language Models

arXiv 2310.16781 ↗First posted 25 Oct 2023cs.CV cs.CL cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Although the mapping between sound and meaning in human language is assumed to be largely arbitrary, research in cognitive science has shown that there are non-trivial correlations between particular sounds and meanings across languages and demographic groups, a phenomenon known as sound symbolism. Among the many dimensions of meaning, sound symbolism is particularly salient and well-demonstrated with regards to cross-modal associations between language and the visual domain. In this work, we address the question of whether sound symbolism is reflected in vision-and-language models such as CLIP and Stable Diffusion. Using zero-shot knowledge probing to investigate the inherent knowledge of these models, we find strong evidence that they do show this pattern, paralleling the well-known kiki-bouba effect in psycholinguistics. Our work provides a novel method for demonstrating sound symbolism and understanding its nature using computational tools. Our code will be made publicly available.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 33 of its 77 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

33 of 77 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
21 more
  1. 2019Language Models as Knowledge Bases?cited by 1,936
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  3. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  4. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  5. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  6. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  7. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  9. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  10. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  11. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 130
  12. 2023Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusioncited by 114
  13. 2023Crawling the Internal Knowledge-Base of Language Modelscited by 85
  14. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 85
  15. 2019Attenuating Bias in Word Vectorscited by 64
  16. 2022Disentangling visual and written concepts in CLIPcited by 58
  17. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  18. 2023Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understandingcited by 20
  19. 2020What is Learned in Visually Grounded Neural Syntax Acquisitioncited by 15
  20. 2015Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenescited by 9
  21. 2015Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Taskscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 5; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  2. 2024Evolving Interpretable Visual Classifiers with Large Language Modelscited by 18
  3. 2024With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Modelscited by 4
  4. 2024What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language modelscited by 3
  5. 2024Measuring Sound Symbolism in Audio-visual Modelscited by 3
  6. 2024Verbalized Representation Learning for Interpretable Few-Shot Generalizationcited by 2
  7. 2025Iconicity in Large Language Modelscited by 1
  8. 2025Teaching Humans Subtle Differences with DIFFusioncited by 1
  9. 2025Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effectcited by 1
  10. 2025Seeing isn't Hearing: Benchmarking Vision Language Models at Interpreting Spectrogramscited by 0
  11. 2025Adversarially Probing Cross-Family Sound Symbolism in 27 Languagescited by 0
  12. 2026Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Modelscited by 0
2 more
  1. 2026An evolutionary model of animats with VLM-based subjective evaluationcited by 0
  2. 2026Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism
  2. 2024Language Models Don't Learn the Physical Manifestation of Language
  3. 2025The Mechanistic Emergence of Symbol Grounding in Language Models
  4. 2024Constructive Approach to Bidirectional Influence between Qualia Structure and Language Emergence
  5. 2023Visual cognition in multimodal large language models
  6. 2025Contrasting Cognitive Styles in Vision-Language Models: Holistic Attention in Japanese Versus Analytical Focus in English
  7. 2025The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping
  8. 2025From Monolingual to Bilingual: Investigating Language Conditioning in Large Language Models for Psycholinguistic Tasks