mlsanityData to 14 Oct 2026

Self-supervised models of audio effectively explain human cortical responses to speech

arXiv 2205.14252 ↗First posted 27 May 2022cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Self-supervised language models are very effective at predicting high-level cortical responses during language comprehension. However, the best current models of lower-level auditory processing in the human brain rely on either hand-constructed acoustic filters or representations from supervised audio neural networks. In this work, we capitalize on the progress of self-supervised speech representation learning (SSL) to create new state-of-the-art models of the human auditory system. Compared against acoustic baselines, phonemic features, and supervised models, representations from the middle layers of self-supervised models (APC, wav2vec, wav2vec 2.0, and HuBERT) consistently yield the best prediction performance for fMRI recordings within the auditory cortex (AC). Brain areas involved in low-level auditory processing exhibit a preference for earlier SSL model layers, whereas higher-level semantic areas prefer later layers. We show that these trends are due to the models' ability to encode information at multiple linguistic levels (acoustic, phonetic, and lexical) along their representation depth. Overall, these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 14 of its 51 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

14 of 51 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2019Designing and Interpreting Probes with Control Taskscited by 640
  8. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  10. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  11. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  12. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
2 more
  1. 2021Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responsescited by 34
  2. 2021Inductive biases, pretraining and fine-tuning jointly account for brain responses to speechcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 7; 2025: 11; 2026: 9.

37 papers.

Sort:

By area:

  1. 2022Decoding speech perception from non-invasive brain recordingscited by 130
  2. 2023Scaling laws for language encoding models in fMRIcited by 68
  3. 2022Toward a realistic model of speech processing in the brain with self-supervised learningcited by 63
  4. 2023Probing self-supervised speech models for phonetic and phonemic information: a case study in aspirationcited by 36
  5. 2023Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)cited by 21
  6. 2024Improving Semantic Understanding in Speech Language Models via Brain-tuningcited by 20
  7. 2023Speech language models lack important brain-relevant semanticscited by 19
  8. 2023A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker Onecited by 13
  9. 2024Evidence from fMRI Supports a Two-Phase Abstraction Process in Language Modelscited by 13
  10. 2023Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Braincited by 9
  11. 2023Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decodingcited by 9
  12. 2023Mapping Brains with Language Models: A Surveycited by 8
25 more
  1. 2025Multi-modal brain encoding models for multi-modal stimulicited by 8
  2. 2025BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Languagecited by 7
  3. 2024Generative causal testing to bridge data-driven models and scientific theories in language neurosciencecited by 5
  4. 2025Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)cited by 4
  5. 2025Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Modelscited by 4
  6. 2025Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglementcited by 4
  7. 2024Predicting Artificial Neural Network Representations to Learn Recognition Model for Music Identification from Brain Recordingscited by 3
  8. 2025Brain-tuned Speech Models Better Reflect Speech Processing Stages in the Braincited by 3
  9. 2024Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRIcited by 2
  10. 2025A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimulicited by 2
  11. 2026Abstraction Induces the Brain Alignment of Language and Speech Modelscited by 2
  12. 2024Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disordercited by 1
  13. 2025Scaling and context steer LLMs along the same computational path as the human braincited by 1
  14. 2025From Minutes to Days: Scaling Intracranial Speech Decoding with Supervised Pretrainingcited by 1
  15. 2024LinBridge: A Learnable Framework for Interpreting Nonlinear Neural Encoding Modelscited by 0
  16. 2025Toward a Realistic Encoding Model of Auditory Affective Understanding in the Braincited by 0
  17. 2025Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory taskscited by 0
  18. 2026Do Models Hear Like Us? Probing the Representational Alignment of Audio LLMs and Naturalistic EEGcited by 0
  19. 2026Linguistic properties and model scale in brain encoding: from small to compressed language modelscited by 0
  20. 2026How does longer temporal context enhance multimodal narrative video processing in the brain?cited by 0
  21. 2026voice2mode: Phonation Mode Classification in Singing using Self-Supervised Speech Modelscited by 0
  22. 2026Expectation and Acoustic Neural Network Representations Enhance Music Identification from Brain Activitycited by 0
  23. 2026Platonic Projection Structures: Operator-Induced Observability in Representation Learningcited by 0
  24. 2026Interpretable MEG Decoding of Perceived Speech: Cortical Sources and the Stimulus Features That Drive Retrievalcited by 0
  25. 2026Integrating Language Models into Listened and Imagined Speech Decoding from MEGcited by 0