# Feature Learning in Deep Neural Networks - Studies on Speech Recognition Tasks

arXiv: https://arxiv.org/abs/1301.3605 · HTML: https://mlsanity.com/paper/1301.3605/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2013.

## Built on by (most cited first, 14 of 14)

- [A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs](https://mlsanity.com/paper/2106.04110/) (2021)
- [Transformer Based Deliberation for Two-Pass Speech Recognition](https://mlsanity.com/paper/2101.11577/) (2021)
- [JOIST: A Joint Speech and Text Streaming Model For ASR](https://mlsanity.com/paper/2210.07353/) (2022)
- [Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks](https://mlsanity.com/paper/2307.11807/) (2023)
- [Improving Deliberation by Text-Only and Semi-Supervised Training](https://mlsanity.com/paper/2206.14716/) (2022)
- [Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloads](https://mlsanity.com/paper/2104.05755/) (2021)
- [Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures](https://mlsanity.com/paper/2304.12576/) (2023)
- [Accented Speech Recognition Inspired by Human Perception](https://mlsanity.com/paper/2104.04627/) (2021)
- [Chain-based Discriminative Autoencoders for Speech Recognition](https://mlsanity.com/paper/2203.13687/) (2022)
- [Sampling-Frequency-Independent Audio Source Separation Using Convolution Layer Based on Impulse Invariant Method](https://mlsanity.com/paper/2105.04079/) (2021)
- [A Novel Semi-supervised Framework for Call Center Agent Malpractice Detection via Neural Feature Learning](https://mlsanity.com/paper/2106.02433/) (2021)
- [A contextual analysis of multi-layer perceptron models in classifying hand-written digits and letters: limited resources](https://mlsanity.com/paper/2107.01782/) (2021)
- [Domain Adapting Deep Reinforcement Learning for Real-world Speech Emotion Recognition](https://mlsanity.com/paper/2207.12248/) (2022)
- [Space Filling Curves is All You Need: Communication-Avoiding Matrix Multiplication Made Simple](https://mlsanity.com/paper/2601.16294/) (2026)

## Abstract (arXiv)

Recent studies have shown that deep neural networks (DNNs) perform significantly better than shallow networks and Gaussian mixture models (GMMs) on large vocabulary speech recognition tasks. In this paper, we argue that the improved accuracy achieved by the DNNs is the result of their ability to extract discriminative internal representations that are robust to the many sources of variability in speech signals. We show that these representations become increasingly insensitive to small perturbations in the input with increasing network depth, which leads to better speech recognition performance with deeper networks. We also show that DNNs cannot extrapolate to test samples that are substantially different from the training examples. If the training data are sufficiently representative, however, internal features learned by the DNN are relatively stable with respect to speaker differences, bandwidth differences, and environment distortion. This enables DNN-based recognizers to perform as well or better than state-of-the-art systems based on GMMs or shallow networks without the need for explicit model adaptation or feature normalization.