# Probing the Information Encoded in Neural-based Acoustic Models of Automatic Speech Recognition Systems

arXiv: https://arxiv.org/abs/2402.19443 · HTML: https://mlsanity.com/paper/2402.19443/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 14 of 14)

- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Explainable deep learning models in medical image analysis](https://mlsanity.com/paper/2005.13799/) (2020)
- [Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification](https://mlsanity.com/paper/2110.05777/) (2021)
- [Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems](https://mlsanity.com/paper/1709.04482/) (2017)
- [DeepHateExplainer: Explainable Hate Speech Detection in Under-resourced Bengali Language](https://mlsanity.com/paper/2012.14353/) (2020)
- [The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verification](https://mlsanity.com/paper/2010.11255/) (2020)
- [Probing the Information Encoded in X-vectors](https://mlsanity.com/paper/1909.06351/) (2019)
- [Analyzing Phonetic and Graphemic Representations in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/1907.04224/) (2019)
- [What does a network layer hear? Analyzing hidden representations of end-to-end ASR through speech synthesis](https://mlsanity.com/paper/1911.01102/) (2019)
- [Retrieving Speaker Information from Personalized Acoustic Models for Speech Recognition](https://mlsanity.com/paper/2111.04194/) (2021)

## Built on by (most cited first, 1 of 1)

- [Beyond task performance: Decoding bioacoustic embeddings with speech features](https://mlsanity.com/paper/2606.14662/) (2026)

## Abstract (arXiv)

Deep learning architectures have made significant progress in terms of performance in many research areas. The automatic speech recognition (ASR) field has thus benefited from these scientific and technological advances, particularly for acoustic modeling, now integrating deep neural network architectures. However, these performance gains have translated into increased complexity regarding the information learned and conveyed through these black-box architectures. Following many researches in neural networks interpretability, we propose in this article a protocol that aims to determine which and where information is located in an ASR acoustic model (AM). To do so, we propose to evaluate AM performance on a determined set of tasks using intermediate representations (here, at different layer levels). Regarding the performance variation and targeted tasks, we can emit hypothesis about which information is enhanced or perturbed at different architecture steps. Experiments are performed on both speaker verification, acoustic environment classification, gender classification, tempo-distortion detection systems and speech sentiment/emotion identification. Analysis showed that neural-based AMs hold heterogeneous information that seems surprisingly uncorrelated with phoneme recognition, such as emotion, sentiment or speaker identity. The low-level hidden layers globally appears useful for the structuring of information while the upper ones would tend to delete useless information for phoneme recognition.