# On the Encoding of Gender in Transformer-based ASR Representations

arXiv: https://arxiv.org/abs/2406.09855 · HTML: https://mlsanity.com/paper/2406.09855/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022)
- [Twists, Humps, and Pebbles: Multilingual Speech Recognition Models Exhibit Gender Performance Gaps](https://mlsanity.com/paper/2402.17954/) (2024)
- [Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance](https://mlsanity.com/paper/1908.08717/) (2019)
- [Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation](https://mlsanity.com/paper/2203.09866/) (2022)
- [Evaluating Gender Bias in Speech Translation](https://mlsanity.com/paper/2010.14465/) (2020)

## Built on by (most cited first, 4 of 4)

- [Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation](https://mlsanity.com/paper/2511.21517/) (2025)
- [What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-training](https://mlsanity.com/paper/2506.00981/) (2025)
- [Different Speech Translation Models Encode and Translate Speaker Gender Differently](https://mlsanity.com/paper/2506.02172/) (2025)
- [A framework for analyzing concept representations in neural models](https://mlsanity.com/paper/2605.01381/) (2026)

## Abstract (arXiv)

While existing literature relies on performance differences to uncover gender biases in ASR models, a deeper analysis is essential to understand how gender is encoded and utilized during transcript generation. This work investigates the encoding and utilization of gender in the latent representations of two transformer-based ASR models, Wav2Vec2 and HuBERT. Using linear erasure, we demonstrate the feasibility of removing gender information from each layer of an ASR model and show that such an intervention has minimal impacts on the ASR performance. Additionally, our analysis reveals a concentration of gender information within the first and last frames in the final layers, explaining the ease of erasing gender in these layers. Our findings suggest the prospect of creating gender-neutral embeddings that can be integrated into ASR frameworks without compromising their efficacy.