mlsanityData to 7 Oct 2026

Measuring Sound Symbolism in Audio-visual Models

arXiv 2409.12306 ↗First posted 18 Sep 2024cs.CL cs.CV cs.SDcited by 3

Not on the map. Why?

Abstract (arXiv)

Audio-visual pre-trained models have gained substantial attention recently and demonstrated superior performance on various audio-visual tasks. This study investigates whether pre-trained audio-visual models demonstrate non-arbitrary associations between sounds and visual representationsunicodex2013known as sound symbolismunicodex2013which is also observed in humans. We developed a specialized dataset with synthesized images and audio samples and assessed these models using a non-parametric approach in a zero-shot setting. Our findings reveal a significant correlation between the models' outputs and established patterns of sound symbolism, particularly in models trained on speech data. These results suggest that such models can capture sound-meaning connections akin to human language processing, providing insights into both cognitive architectures and machine learning strategies.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 33 of its 66 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

33 of 66 references.

References · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  10. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  11. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  12. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
21 more
  1. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  2. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  3. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  4. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  5. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 229
  6. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  7. 2018Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Inputcited by 81
  8. 2020Deep Audio-Visual Learning: A Surveycited by 68
  9. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
  10. 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
  11. 2022MAViL: Masked Audio-Video Learnerscited by 56
  12. 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
  13. 2022Abstract Visual Reasoning with Tangram Shapescited by 43
  14. 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
  15. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  16. 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
  17. 2022Learning State-Aware Visual Representations from Audible Interactionscited by 23
  18. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  19. 2023Weakly-Supervised Audio-Visual Segmentationcited by 19
  20. 2023Kiki or Bouba? Sound Symbolism in Vision-and-Language Modelscited by 14
  21. 2023Audio-Visual Neural Syntax Acquisitioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolismcited by 3
  2. 2025Cross-modal Associations in Vision and Language Models: Revisiting the Bouba-Kiki Effectcited by 1
  3. 2026Similar Choices, Different Attention: Cross-Modal Associations in Humans and Vision-Language Modelscited by 0