# Self-supervised models of audio effectively explain human cortical responses to speech

arXiv: https://arxiv.org/abs/2205.14252 · HTML: https://mlsanity.com/paper/2205.14252/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 51 references that are in the map.
- First posted on arXiv 27 May 2022.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Generative Pre-Training for Speech with Autoregressive Predictive Coding](https://mlsanity.com/paper/1910.12607/) (2019)
- [Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses](https://mlsanity.com/paper/2106.05426/) (2021)
- [Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech](https://mlsanity.com/paper/2103.01032/) (2021)

## Built on by (most cited first, 30 of 37)

- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Toward a realistic model of speech processing in the brain with self-supervised learning](https://mlsanity.com/paper/2206.01685/) (2022)
- [Probing self-supervised speech models for phonetic and phonemic information: a case study in aspiration](https://mlsanity.com/paper/2306.06232/) (2023)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Improving Semantic Understanding in Speech Language Models via Brain-tuning](https://mlsanity.com/paper/2410.09230/) (2024)
- [Speech language models lack important brain-relevant semantics](https://mlsanity.com/paper/2311.04664/) (2023)
- [A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker One](https://mlsanity.com/paper/2302.09908/) (2023)
- [Evidence from fMRI Supports a Two-Phase Abstraction Process in Language Models](https://mlsanity.com/paper/2409.05771/) (2024)
- [Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain](https://mlsanity.com/paper/2311.07766/) (2023)
- [Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding](https://mlsanity.com/paper/2312.09430/) (2023)
- [Mapping Brains with Language Models: A Survey](https://mlsanity.com/paper/2306.05126/) (2023)
- [Multi-modal brain encoding models for multi-modal stimuli](https://mlsanity.com/paper/2505.20027/) (2025)
- [BrainWavLM: Fine-tuning Speech Representations with Brain Responses to Language](https://mlsanity.com/paper/2502.08866/) (2025)
- [Generative causal testing to bridge data-driven models and scientific theories in language neuroscience](https://mlsanity.com/paper/2410.00812/) (2024)
- [Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)](https://mlsanity.com/paper/2505.20029/) (2025)
- [Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models](https://mlsanity.com/paper/2510.21520/) (2025)
- [Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement](https://mlsanity.com/paper/2510.22860/) (2025)
- [Predicting Artificial Neural Network Representations to Learn Recognition Model for Music Identification from Brain Recordings](https://mlsanity.com/paper/2412.15560/) (2024)
- [Brain-tuned Speech Models Better Reflect Speech Processing Stages in the Brain](https://mlsanity.com/paper/2506.03832/) (2025)
- [Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI](https://mlsanity.com/paper/2405.18726/) (2024)
- [A Multimodal Sequence-to-Sequence Model for Cross-Subject Prediction of Brain Responses to Naturalistic Stimuli](https://mlsanity.com/paper/2507.18104/) (2025)
- [Abstraction Induces the Brain Alignment of Language and Speech Models](https://mlsanity.com/paper/2602.04081/) (2026)
- [Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disorder](https://mlsanity.com/paper/2407.10376/) (2024)
- [Scaling and context steer LLMs along the same computational path as the human brain](https://mlsanity.com/paper/2512.01591/) (2025)
- [From Minutes to Days: Scaling Intracranial Speech Decoding with Supervised Pretraining](https://mlsanity.com/paper/2512.15830/) (2025)
- [LinBridge: A Learnable Framework for Interpreting Nonlinear Neural Encoding Models](https://mlsanity.com/paper/2410.20053/) (2024)
- [Toward a Realistic Encoding Model of Auditory Affective Understanding in the Brain](https://mlsanity.com/paper/2509.21381/) (2025)
- [Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory tasks](https://mlsanity.com/paper/2511.16849/) (2025)
- [Do Models Hear Like Us? Probing the Representational Alignment of Audio LLMs and Naturalistic EEG](https://mlsanity.com/paper/2601.16540/) (2026)

## Abstract (arXiv)

Self-supervised language models are very effective at predicting high-level cortical responses during language comprehension. However, the best current models of lower-level auditory processing in the human brain rely on either hand-constructed acoustic filters or representations from supervised audio neural networks. In this work, we capitalize on the progress of self-supervised speech representation learning (SSL) to create new state-of-the-art models of the human auditory system. Compared against acoustic baselines, phonemic features, and supervised models, representations from the middle layers of self-supervised models (APC, wav2vec, wav2vec 2.0, and HuBERT) consistently yield the best prediction performance for fMRI recordings within the auditory cortex (AC). Brain areas involved in low-level auditory processing exhibit a preference for earlier SSL model layers, whereas higher-level semantic areas prefer later layers. We show that these trends are due to the models' ability to encode information at multiple linguistic levels (acoustic, phonetic, and lexical) along their representation depth. Overall, these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.