# Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech

arXiv: https://arxiv.org/abs/2103.01032 · HTML: https://mlsanity.com/paper/2103.01032/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 48 references that are in the map.
- First posted on arXiv 25 Feb 2021.

## Builds on (most cited first, 7 of 7)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Towards an integration of deep learning and neuroscience](https://mlsanity.com/paper/1606.03813/) (2016)
- [Machine Learning for Neuroimaging with Scikit-Learn](https://mlsanity.com/paper/1412.3919/) (2014)
- [Wav2Letter: an End-to-End ConvNet-based Speech Recognition System](https://mlsanity.com/paper/1609.03193/) (2016)
- [Audio tagging with noisy labels and minimal supervision](https://mlsanity.com/paper/1906.02975/) (2019)
- [Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq](https://mlsanity.com/paper/1805.10387/) (2018)

## Built on by (most cited first, 14 of 14)

- [Scaling laws for language encoding models in fMRI](https://mlsanity.com/paper/2305.11863/) (2023)
- [Toward a realistic model of speech processing in the brain with self-supervised learning](https://mlsanity.com/paper/2206.01685/) (2022)
- [Self-supervised models of audio effectively explain human cortical responses to speech](https://mlsanity.com/paper/2205.14252/) (2022)
- [From Language to Cognition: How LLMs Outgrow the Human Language Network](https://mlsanity.com/paper/2503.01830/) (2025)
- [Successes and critical failures of neural networks in capturing human-like speech recognition](https://mlsanity.com/paper/2204.03740/) (2022)
- [Deep Recurrent Encoder: A scalable end-to-end network to model brain signals](https://mlsanity.com/paper/2103.02339/) (2021)
- [Interpreting intermediate convolutional layers of generative CNNs trained on waveforms](https://mlsanity.com/paper/2104.09489/) (2021)
- [Interpreting intermediate convolutional layers in unsupervised acoustic word classification](https://mlsanity.com/paper/2110.02375/) (2021)
- [Deep Learning Models to Study Sentence Comprehension in the Human Brain](https://mlsanity.com/paper/2301.06340/) (2023)
- [Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network](https://mlsanity.com/paper/2406.15109/) (2024)
- [MEG-MASC: a high-quality magneto-encephalography dataset for evaluating natural speech processing](https://mlsanity.com/paper/2208.11488/) (2022)
- [Don't stop the training: continuously-updating self-supervised algorithms best account for auditory responses in the cortex](https://mlsanity.com/paper/2202.07290/) (2022)
- [Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks](https://mlsanity.com/paper/2404.14024/) (2024)
- [Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory tasks](https://mlsanity.com/paper/2511.16849/) (2025)

## Abstract (arXiv)

Our ability to comprehend speech remains, to date, unrivaled by deep learning models. This feat could result from the brain's ability to fine-tune generic sound representations for speech-specific processes. To test this hypothesis, we compare i) five types of deep neural networks to ii) human brain responses elicited by spoken sentences and recorded in 102 Dutch subjects using functional Magnetic Resonance Imaging (fMRI). Each network was either trained on an acoustics scene classification, a speech-to-text task (based on Bengali, English, or Dutch), or not trained. The similarity between each model and the brain is assessed by correlating their respective activations after an optimal linear projection. The differences in brain-similarity across networks revealed three main results. First, speech representations in the brain can be accounted for by random deep networks. Second, learning to classify acoustic scenes leads deep nets to increase their brain similarity. Third, learning to process phonetically-related speech inputs (i.e., Dutch vs English) leads deep nets to reach higher levels of brain-similarity than learning to process phonetically-distant speech inputs (i.e. Dutch vs Bengali). Together, these results suggest that the human brain fine-tunes its heavily-trained auditory hierarchy to learn to process speech.