# Data-Efficient ASR Personalization for Non-Normative Speech Using an Uncertainty-Based Phoneme Difficulty Score for Guided Sampling

arXiv: https://arxiv.org/abs/2509.20396 · HTML: https://mlsanity.com/paper/2509.20396/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Improved Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2005.09629/) (2020)
- [Personalizing ASR for Dysarthric and Accented Speech with Limited Data](https://mlsanity.com/paper/1907.13511/) (2019)
- [Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition](https://mlsanity.com/paper/2010.11428/) (2020)
- [Training Data Augmentation for Dysarthric Automatic Speech Recognition by Text-to-Dysarthric-Speech Synthesis](https://mlsanity.com/paper/2406.08568/) (2024)
- [Accurate synthesis of Dysarthric Speech for ASR data augmentation](https://mlsanity.com/paper/2308.08438/) (2023)
- [Parameter-efficient Dysarthric Speech Recognition Using Adapter Fusion and Householder Transformation](https://mlsanity.com/paper/2306.07090/) (2023)
- [Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces](https://mlsanity.com/paper/2503.15124/) (2025)
- [Adapting Foundation Speech Recognition Models to Impaired Speech: A Semantic Re-chaining Approach for Personalization of German Speech](https://mlsanity.com/paper/2506.21622/) (2025)
- [Fine-Tuning Automatic Speech Recognition for People with Parkinson's: An Effective Strategy for Enhancing Speech Technology Accessibility](https://mlsanity.com/paper/2409.19818/) (2024)
- [Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition](https://mlsanity.com/paper/2407.12817/) (2024)
- [DyPCL: Dynamic Phoneme-level Contrastive Learning for Dysarthric Speech Recognition](https://mlsanity.com/paper/2501.19010/) (2025)

## Built on by (most cited first, 1 of 1)

- [Demonstration of Adapt4Me: An Uncertainty-Aware Authoring Environment for Personalizing Automatic Speech Recognition to Non-normative Speech](https://mlsanity.com/paper/2603.20112/) (2026)

## Abstract (arXiv)

ASR systems struggle with non-normative speech due to high acoustic variability and data scarcity. We propose a data-efficient method using phoneme-level uncertainty to guide fine-tuning for personalization. Instead of computationally expensive ensembles, we leverage Variational Low-Rank Adaptation (VI LoRA) to estimate epistemic uncertainty in foundation models. These estimates form a composite Phoneme Difficulty Score (PhDScore) that drives a targeted oversampling strategy. Evaluated on English and German datasets, including a longitudinal analysis against two clinical reports taken one year apart, we demonstrate that: (1) VI LoRA-based uncertainty aligns better with expert clinical assessments than standard entropy; (2) PhDScore captures stable, persistent articulatory difficulties; and (3) uncertainty-guided sampling significantly improves ASR accuracy for impaired speech.