# Successes and critical failures of neural networks in capturing human-like speech recognition

arXiv: https://arxiv.org/abs/2204.03740 · HTML: https://mlsanity.com/paper/2204.03740/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 65 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 10 of 10)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Are Convolutional Neural Networks or Transformers more like human vision?](https://mlsanity.com/paper/2105.07197/) (2021)
- [Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech](https://mlsanity.com/paper/2103.01032/) (2021)
- [A neural network walks into a lab: towards using deep nets as models for human behavior](https://mlsanity.com/paper/2005.02181/) (2020)
- [The effect of task and training on intermediate representations in convolutional neural networks revealed with modified RV similarity analysis](https://mlsanity.com/paper/1912.02260/) (2019)
- [Computational Complexity of Segmentation](https://mlsanity.com/paper/2201.13106/) (2022)

## Built on by (most cited first, 7 of 7)

- [Decoding speech perception from non-invasive brain recordings](https://mlsanity.com/paper/2208.12266/) (2022)
- [Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0](https://mlsanity.com/paper/2407.03005/) (2024)
- [Intuitionistic Fuzzy Broad Learning System: Enhancing Robustness Against Noise and Outliers](https://mlsanity.com/paper/2307.08713/) (2023)
- [Optimal Input Gain: All You Need to Supercharge a Feed-Forward Neural Network](https://mlsanity.com/paper/2303.17732/) (2023)
- [Long-term Conversation Analysis: Exploring Utility and Privacy](https://mlsanity.com/paper/2306.16071/) (2023)
- [Domain Invariant Learning for Gaussian Processes and Bayesian Exploration](https://mlsanity.com/paper/2312.11318/) (2023)
- [Graph-Embedded Intuitionistic Fuzzy Broad Learning System: A Multi-view Framework](https://mlsanity.com/paper/2607.16728/) (2026)

## Abstract (arXiv)

Natural and artificial audition can in principle acquire different solutions to a given problem. The constraints of the task, however, can nudge the cognitive science and engineering of audition to qualitatively converge, suggesting that a closer mutual examination would potentially enrich artificial hearing systems and process models of the mind and brain. Speech recognition - an area ripe for such exploration - is inherently robust in humans to a number transformations at various spectrotemporal granularities. To what extent are these robustness profiles accounted for by high-performing neural network systems? We bring together experiments in speech recognition under a single synthesis framework to evaluate state-of-the-art neural networks as stimulus-computable, optimized observers. In a series of experiments, we (1) clarify how influential speech manipulations in the literature relate to each other and to natural speech, (2) show the granularities at which machines exhibit out-of-distribution robustness, reproducing classical perceptual phenomena in humans, (3) identify the specific conditions where model predictions of human performance differ, and (4) demonstrate a crucial failure of all artificial systems to perceptually recover where humans do, suggesting alternative directions for theory and model building. These findings encourage a tighter synergy between the cognitive science and engineering of audition.