mlsanityData to 7 Oct 2026

What Do Language Models Hear? Probing for Auditory Representations in Language Models

arXiv 2402.16998 ↗First posted 26 Feb 2024cs.CL cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

This work explores whether language models encode meaningfully grounded representations of sounds of objects. We learn a linear probe that retrieves the correct text representation of an object given a snippet of audio related to that object, where the sound representation is given by a pretrained audio model. This probe is trained via a contrastive loss that pushes the language representations and sound representations of an object to be close to one another. After training, the probe is tested on its ability to generalize to objects that were not seen during training. Across different language models and audio models, we find that the probe generalization is above chance in many cases, indicating that despite being trained only on raw text, language models encode grounded knowledge of sounds for some objects.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

28 of 44 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  8. 2021AST: Audio Spectrogram Transformercited by 634
  9. 2019Designing and Interpreting Probes with Control Taskscited by 608
  10. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  11. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  12. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
16 more
  1. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  2. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  3. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  4. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  5. 2022Linearly Mapping from Image to Text Spacecited by 139
  6. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  7. 2020Do Language Embeddings Capture Scales?cited by 63
  8. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  9. 2023Large language models predict human sensory judgments across six modalitiescited by 47
  10. 2022What do tokens know about their characters and how do they know it?cited by 36
  11. 2022Geographic and Geopolitical Biases of Language Modelscited by 29
  12. 2023Emergent Representations of Program Semantics in Language Models Trained on Programscited by 28
  13. 2021Do Language Models Know the Way to Rome?cited by 18
  14. 2022Words are all you need? Language as an approximation for human similarity judgmentscited by 18
  15. 2023Emergence of Abstract State Representations in Embodied Sequence Modelingcited by 9
  16. 2023More than Correlation: Do Large Language Models Learn Causal Representations of Space?cited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  2. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  3. 2024Do Music Generation Models Encode Music Theory?cited by 12
  4. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 10
  5. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  6. 2025The Zero Body Problem: Probing LLM Use of Sensory Languagecited by 5
  7. 2026The Indra Representation Hypothesis for Multimodal Alignmentcited by 4
  8. 2024Aligning Large Language Models and Geometric Deep Models for Protein Representationcited by 3
  9. 2025Probing Neural Combinatorial Optimization Modelscited by 3
  10. 2024AudioBERT: Audio Knowledge Augmented Language Modelcited by 2
  11. 2025Imagine to Hear: Auditory Knowledge Generation can be an Effective Assistant for Language Modelscited by 2
  12. 2026How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluationcited by 2
3 more
  1. 2024Prevailing Research Areas for Music AI in the Era of Foundation Modelscited by 1
  2. 2026How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Themcited by 1
  3. 2025What if Othello-Playing Language Models Could See?cited by 0