# Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation

arXiv: https://arxiv.org/abs/1601.02828 · HTML: https://mlsanity.com/paper/1601.02828/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jan 2016.

## Built on by (most cited first, 30 of 40)

- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech](https://mlsanity.com/paper/2109.06952/) (2021)
- [Investigation of Data Augmentation Techniques for Disordered Speech Recognition](https://mlsanity.com/paper/2201.05562/) (2022)
- [From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition](https://mlsanity.com/paper/2301.07851/) (2023)
- [Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2202.10290/) (2022)
- [Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2205.06445/) (2022)
- [Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition](https://mlsanity.com/paper/2201.05554/) (2022)
- [Conformer Based Elderly Speech Recognition System for Alzheimer's Disease Detection](https://mlsanity.com/paper/2206.13232/) (2022)
- [Text is All You Need: Personalizing ASR Models using Controllable Speech Synthesis](https://mlsanity.com/paper/2303.14885/) (2023)
- [Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition](https://mlsanity.com/paper/2203.10274/) (2022)
- [Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2407.13782/) (2024)
- [Use of Speech Impairment Severity for Dysarthric Speech Recognition](https://mlsanity.com/paper/2305.10659/) (2023)
- [TS-Net: OCR Trained to Switch Between Text Transcription Styles](https://mlsanity.com/paper/2103.05489/) (2021)
- [Exploiting Cross-domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognition](https://mlsanity.com/paper/2206.07327/) (2022)
- [Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactions](https://mlsanity.com/paper/2210.05047/) (2022)
- [Advancing Test-Time Adaptation in Wild Acoustic Test Settings](https://mlsanity.com/paper/2310.09505/) (2023)
- [Towards Writing Style Adaptation in Handwriting Recognition](https://mlsanity.com/paper/2302.06318/) (2023)
- [A Unified Speaker Adaptation Approach for ASR](https://mlsanity.com/paper/2110.08545/) (2021)
- [On-the-Fly Feature Based Rapid Speaker Adaptation for Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2203.14593/) (2022)
- [BASM: A Bottom-up Adaptive Spatiotemporal Model for Online Food Ordering Service](https://mlsanity.com/paper/2211.12033/) (2022)
- [Two-pass Decoding and Cross-adaptation Based System Combination of End-to-end Conformer and Hybrid TDNN ASR Systems](https://mlsanity.com/paper/2206.11596/) (2022)
- [Retrievable Domain-Sensitive Feature Memory for Multi-Domain Recommendation](https://mlsanity.com/paper/2405.12892/) (2024)
- [Perceiver-Prompt: Flexible Speaker Adaptation in Whisper for Chinese Disordered Speech Recognition](https://mlsanity.com/paper/2406.09873/) (2024)
- [Generalizing Speaker Verification for Spoof Awareness in the Embedding Space](https://mlsanity.com/paper/2401.11156/) (2024)
- [USM: Unbiased Survey Modeling for Limiting Negative User Experiences in Recommendation Systems](https://mlsanity.com/paper/2412.10674/) (2024)
- [Improving Robustness of Retrieval Augmented Translation via Shuffling of Suggestions](https://mlsanity.com/paper/2210.05059/) (2022)
- [Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation](https://mlsanity.com/paper/2407.06310/) (2024)
- [Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation](https://mlsanity.com/paper/2502.05758/) (2025)
- [Enhancing Quantised End-to-End ASR Models via Personalisation](https://mlsanity.com/paper/2309.09136/) (2023)
- [Scenario-Adaptive Fine-Grained Personalization Network: Tailoring User Behavior Representation to the Scenario Context](https://mlsanity.com/paper/2404.09709/) (2024)

## Abstract (arXiv)

This work presents a broad study on the adaptation of neural network acoustic models by means of learning hidden unit contributions (LHUC) -- a method that linearly re-combines hidden units in a speaker- or environment-dependent manner using small amounts of unsupervised adaptation data. We also extend LHUC to a speaker adaptive training (SAT) framework that leads to a more adaptable DNN acoustic model, working both in a speaker-dependent and a speaker-independent manner, without the requirements to maintain auxiliary speaker-dependent feature extractors or to introduce significant speaker-dependent changes to the DNN structure. Through a series of experiments on four different speech recognition benchmarks (TED talks, Switchboard, AMI meetings, and Aurora4) comprising 270 test speakers, we show that LHUC in both its test-only and SAT variants results in consistent word error rate reductions ranging from 5% to 23% relative depending on the task and the degree of mismatch between training and test data. In addition, we have investigated the effect of the amount of adaptation data per speaker, the quality of unsupervised adaptation targets, the complementarity to other adaptation techniques, one-shot adaptation, and an extension to adapting DNNs trained in a sequence discriminative manner.