# Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems

arXiv: https://arxiv.org/abs/1709.04482 · HTML: https://mlsanity.com/paper/1709.04482/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2017.

## Built on by (most cited first, 30 of 32)

- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure](https://mlsanity.com/paper/2101.00387/) (2021)
- [Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques](https://mlsanity.com/paper/2104.13225/) (2021)
- [Wave to Syntax: Probing spoken language models for syntax](https://mlsanity.com/paper/2305.18957/) (2023)
- [What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model](https://mlsanity.com/paper/2306.06524/) (2023)
- [Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models](https://mlsanity.com/paper/2104.07578/) (2021)
- [POWSM: A Phonetic Open Whisper-Style Speech Foundation Model](https://mlsanity.com/paper/2510.24992/) (2025)
- [Representation Potentials of Foundation Models for Multimodal Alignment: A Survey](https://mlsanity.com/paper/2510.05184/) (2025)
- [What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysis](https://mlsanity.com/paper/2107.00439/) (2021)
- [How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations](https://mlsanity.com/paper/2411.17666/) (2024)
- [Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations](https://mlsanity.com/paper/2509.15655/) (2025)
- [Causal Tracing of Audio-Text Fusion in Large Audio Language Models](https://mlsanity.com/paper/2603.13768/) (2026)
- [Effects of Layer Freezing on Transferring a Speech Recognition System to Under-resourced Languages](https://mlsanity.com/paper/2102.04097/) (2021)
- [Interpreting intermediate convolutional layers in unsupervised acoustic word classification](https://mlsanity.com/paper/2110.02375/) (2021)
- [Retrieving Speaker Information from Personalized Acoustic Models for Speech Recognition](https://mlsanity.com/paper/2111.04194/) (2021)
- [MixRep: Hidden Representation Mixup for Low-Resource Speech Recognition](https://mlsanity.com/paper/2310.18450/) (2023)
- [A predictive learning model can simulate temporal dynamics and context effects found in neural representations of continuous speech](https://mlsanity.com/paper/2405.08237/) (2024)
- [A framework for analyzing concept representations in neural models](https://mlsanity.com/paper/2605.01381/) (2026)
- [SCaLa: Supervised Contrastive Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04187/) (2021)
- [Implications of Annotation Artifacts in Edge Probing Test Datasets](https://mlsanity.com/paper/2310.13856/) (2023)
- [Probing the Information Encoded in Neural-based Acoustic Models of Automatic Speech Recognition Systems](https://mlsanity.com/paper/2402.19443/) (2024)
- [Non-verbal information in spontaneous speech -- towards a new framework of analysis](https://mlsanity.com/paper/2403.03522/) (2024)
- [Establishing degrees of closeness between audio recordings along different dimensions using large-scale cross-lingual models](https://mlsanity.com/paper/2402.05581/) (2024)
- [On the Effectiveness of Incremental Training of Large Language Models](https://mlsanity.com/paper/2411.18700/) (2024)
- [Leveraging Zipformer Model for Effective Language Identification in Code-Switched Child-Directed Speech](https://mlsanity.com/paper/2508.09430/) (2025)
- [Evaluating the Representation of Vowels in Wav2Vec Feature Extractor: A Layer-Wise Analysis Using MFCCs](https://mlsanity.com/paper/2508.17914/) (2025)
- [Categorize Early, Integrate Late: Divergent Processing Strategies in Automatic Speech Recognition](https://mlsanity.com/paper/2601.06972/) (2026)
- [Probing in the Wild: A Case Study of Self-Supervised Speech Representations on Mandarin Sub-dialects with Unsupervised Articulatory Analysis](https://mlsanity.com/paper/2606.25459/) (2026)
- [Phonological Perception of Sign Language Models](https://mlsanity.com/paper/2606.28667/) (2026)

## Abstract (arXiv)

Neural models have become ubiquitous in automatic speech recognition systems. While neural networks are typically used as acoustic models in more complex systems, recent studies have explored end-to-end speech recognition systems based on neural networks, which can be trained to directly predict text from input acoustic features. Although such systems are conceptually elegant and simpler than traditional systems, it is less obvious how to interpret the trained models. In this work, we analyze the speech representations learned by a deep end-to-end model that is based on convolutional and recurrent layers, and trained with a connectionist temporal classification (CTC) loss. We use a pre-trained model to generate frame-level features which are given to a classifier that is trained on frame classification into phones. We evaluate representations from different layers of the deep model and compare their quality for predicting phone labels. Our experiments shed light on important aspects of the end-to-end model such as layer depth, model complexity, and other design choices.