mlsanityData to 14 Oct 2026

From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Models

arXiv 2506.01133 ↗First posted 1 Jun 2025cs.CL cs.AI cs.SDcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The emergence of large language models (LLMs) has demonstrated that systems trained solely on text can acquire extensive world knowledge, develop reasoning capabilities, and internalize abstract semantic concepts--showcasing properties that can be associated with general intelligence. This raises an intriguing question: Do such concepts emerge in models trained on other modalities, such as speech? Furthermore, when models are trained jointly on multiple modalities: Do they develop a richer, more structured semantic understanding? To explore this, we analyze the conceptual structures learned by speech and textual models both individually and jointly. We employ Latent Concept Analysis, an unsupervised method for uncovering and interpreting latent representations in neural networks, to examine how semantic abstractions form across modalities. For reproducibility we made scripts and other resources available to the community.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 32 of its 40 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

32 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  6. 2024Qwen2-Audio Technical Reportcited by 560
  7. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
  8. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  9. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  10. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 101
  11. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  12. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
19 more
  1. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
  2. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 60
  3. 2022Discovering Latent Concepts Learned in BERTcited by 56
  4. 2017The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representationscited by 42
  5. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 38
  6. 2023Probing self-supervised speech models for phonetic and phonemic information: a case study in aspirationcited by 36
  7. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  8. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  9. 2022On the Transformation of Latent Space in Fine-Tuned NLP Modelscited by 19
  10. 2024Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0cited by 19
  11. 2023Can LLMs facilitate interpretation of pre-trained language models?cited by 16
  12. 2023SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERTcited by 13
  13. 2023NeuroX Library for Neuron Analysis of Deep NLP Modelscited by 11
  14. 2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9
  15. 2023Scaling up Discovery of Latent Concepts in Deep NLP Modelscited by 9
  16. 2024How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representationscited by 9
  17. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  18. 2024Speech Representation Analysis based on Inter- and Intra-Model Similaritiescited by 4
  19. 2023Acoustic Word Embeddings for Untranscribed Target Languages with Continued Pretraining and Learned Poolingcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025The Curious Case of Visual Grounding: Different Effects for Speech- and Text-based Language Encoderscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?
  2. 2025The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models
  3. 2025Interpreting Language Models Through Concept Descriptions: A Survey
  4. 2025Latent Concept Disentanglement in Transformer-based Language Models
  5. 2025Mechanistic Indicators of Understanding in Large Language Models
  6. 2024A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debates
  7. 2026Are You Thinking What I am Thinking? : Examining Conceptual Separation in Neural Architectures
  8. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesis