# What does a network layer hear? Analyzing hidden representations of end-to-end ASR through speech synthesis

arXiv: https://arxiv.org/abs/1911.01102 · HTML: https://mlsanity.com/paper/1911.01102/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2019.

## Built on by (most cited first, 13 of 13)

- [Towards Relatable Explainable AI with the Perceptual Process](https://mlsanity.com/paper/2112.14005/) (2021)
- [A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker One](https://mlsanity.com/paper/2302.09908/) (2023)
- [Exploration of Adapter for Noise Robust Automatic Speech Recognition](https://mlsanity.com/paper/2402.18275/) (2024)
- [Reducing the gap between streaming and non-streaming Transducer-based ASR by adaptive two-stage knowledge distillation](https://mlsanity.com/paper/2306.15171/) (2023)
- [Bring the Noise: Introducing Noise Robustness to Pretrained Automatic Speech Recognition](https://mlsanity.com/paper/2309.02145/) (2023)
- [Retrieving Speaker Information from Personalized Acoustic Models for Speech Recognition](https://mlsanity.com/paper/2111.04194/) (2021)
- [MixRep: Hidden Representation Mixup for Low-Resource Speech Recognition](https://mlsanity.com/paper/2310.18450/) (2023)
- [Tailored Design of Audio-Visual Speech Recognition Models using Branchformers](https://mlsanity.com/paper/2407.06606/) (2024)
- [Probing the Information Encoded in Neural-based Acoustic Models of Automatic Speech Recognition Systems](https://mlsanity.com/paper/2402.19443/) (2024)
- [Privacy-Preserving Speech Representation Learning using Vector Quantization](https://mlsanity.com/paper/2203.09518/) (2022)
- [Hardware Acceleration of Explainable Artificial Intelligence](https://mlsanity.com/paper/2305.04887/) (2023)
- [A Dataset and Baselines for Measuring and Predicting the Music Piece Memorability](https://mlsanity.com/paper/2405.12847/) (2024)
- [MulCPred: Learning Multi-modal Concepts for Explainable Pedestrian Action Prediction](https://mlsanity.com/paper/2409.09446/) (2024)

## Abstract (arXiv)

End-to-end speech recognition systems have achieved competitive results compared to traditional systems. However, the complex transformations involved between layers given highly variable acoustic signals are hard to analyze. In this paper, we present our ASR probing model, which synthesizes speech from hidden representations of end-to-end ASR to examine the information maintain after each layer calculation. Listening to the synthesized speech, we observe gradual removal of speaker variability and noise as the layer goes deeper, which aligns with the previous studies on how deep network functions in speech recognition. This paper is the first study analyzing the end-to-end speech recognition model by demonstrating what each layer hears. Speaker verification and speech enhancement measurements on synthesized speech are also conducted to confirm our observation further.