# Retrieving Speaker Information from Personalized Acoustic Models for Speech Recognition

arXiv: https://arxiv.org/abs/2111.04194 · HTML: https://mlsanity.com/paper/2111.04194/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 7 Nov 2021.

## Builds on (most cited first, 12 of 12)

- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework](https://mlsanity.com/paper/2010.15965/) (2020)
- [Training Keyword Spotting Models on Non-IID Data with Federated Learning](https://mlsanity.com/paper/2005.10406/) (2020)
- [Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems](https://mlsanity.com/paper/1709.04482/) (2017)
- [Federated Acoustic Modeling For Automatic Speech Recognition](https://mlsanity.com/paper/2102.04429/) (2021)
- [The GDPR & Speech Data: Reflections of Legal and Technology Communities, First Steps towards a Common Understanding](https://mlsanity.com/paper/1907.03458/) (2019)
- [Analyzing Phonetic and Graphemic Representations in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/1907.04224/) (2019)
- [Federated Learning in ASR: Not as Easy as You Think](https://mlsanity.com/paper/2109.15108/) (2021)
- [What does a network layer hear? Analyzing hidden representations of end-to-end ASR through speech synthesis](https://mlsanity.com/paper/1911.01102/) (2019)
- [Privacy attacks for automatic speech recognition acoustic models in a federated learning framework](https://mlsanity.com/paper/2111.03777/) (2021)

## Built on by (most cited first, 2 of 2)

- [Federated Learning for ASR based on Wav2vec 2.0](https://mlsanity.com/paper/2302.10790/) (2023)
- [Probing the Information Encoded in Neural-based Acoustic Models of Automatic Speech Recognition Systems](https://mlsanity.com/paper/2402.19443/) (2024)

## Abstract (arXiv)

The widespread of powerful personal devices capable of collecting voice of their users has opened the opportunity to build speaker adapted speech recognition system (ASR) or to participate to collaborative learning of ASR. In both cases, personalized acoustic models (AM), i.e. fine-tuned AM with specific speaker data, can be built. A question that naturally arises is whether the dissemination of personalized acoustic models can leak personal information. In this paper, we show that it is possible to retrieve the gender of the speaker, but also his identity, by just exploiting the weight matrix changes of a neural acoustic model locally adapted to this speaker. Incidentally we observe phenomena that may be useful towards explainability of deep neural networks in the context of speech processing. Gender can be identified almost surely using only the first layers and speaker verification performs well when using middle-up layers. Our experimental study on the TED-LIUM 3 dataset with HMM/TDNN models shows an accuracy of 95% for gender detection, and an Equal Error Rate of 9.07% for a speaker verification task by only exploiting the weights from personalized models that could be exchanged instead of user data.