# Multimodal Consistency-Guided Reference-Free Data Selection for ASR Accent Adaptation

arXiv: https://arxiv.org/abs/2602.13263 · HTML: https://mlsanity.com/paper/2602.13263/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models](https://mlsanity.com/paper/2310.06313/) (2023)
- [IMAGDressing-v1: Customizable Virtual Dressing](https://mlsanity.com/paper/2407.12705/) (2024)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [PRISM: A Rich Class of Parameterized Submodular Information Measures for Guided Subset Selection](https://mlsanity.com/paper/2103.00128/) (2021)
- [Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model](https://mlsanity.com/paper/2502.09533/) (2025)
- [Personalizing ASR for Dysarthric and Accented Speech with Limited Data](https://mlsanity.com/paper/1907.13511/) (2019)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [IMAGGarment: Fine-Grained Garment Generation for Controllable Fashion Design](https://mlsanity.com/paper/2504.13176/) (2025)
- [An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models](https://mlsanity.com/paper/1909.06678/) (2019)
- [A Memoization Framework for Scaling Submodular Optimization to Large Scale Problems](https://mlsanity.com/paper/1902.10176/) (2019)
- [A Reference-less Quality Metric for Automatic Speech Recognition via Contrastive-Learning of a Multi-Language Model with Self-Supervision](https://mlsanity.com/paper/2306.13114/) (2023)
- [DITTO: Data-efficient and Fair Targeted Subset Selection for ASR Accent Adaptation](https://mlsanity.com/paper/2110.04908/) (2021)
- [Fast Word Error Rate Estimation Using Self-Supervised Representations for Speech and Text](https://mlsanity.com/paper/2310.08225/) (2023)
- [Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filtering](https://mlsanity.com/paper/2506.03681/) (2025)
- [Error-driven Fixed-Budget ASR Personalization for Accented Speakers](https://mlsanity.com/paper/2103.03142/) (2021)
- [Unsupervised Active Learning: Optimizing Labeling Cost-Effectiveness for Automatic Speech Recognition](https://mlsanity.com/paper/2308.14814/) (2023)
- [Automatic Speech Recognition System-Independent Word Error Rate Estimation](https://mlsanity.com/paper/2404.16743/) (2024)

## Abstract (arXiv)

Automatic speech recognition (ASR) systems often degrade on accented speech because acoustic-phonetic and prosodic shifts induce a mismatch to training data, making labeled accent adaptation costly. However, common pseudo-label selection heuristics are largely text-centric (e.g., perplexity (PPL) filtering) and can prefer fluent yet acoustically mismatched hypotheses, leading to error amplification when fine-tuning. To address this, we introduce a multimodal consistency-guided, reference-free data selection pipeline for ASR accent adaptation under a transductive, label-free protocol. The pipeline starts with a target-aware preselection step based on submodular mutual information to improve query relevance and reduce downstream computation. It then generates multiple pseudo-transcriptions per utterance via perturbation-based decoding and scores each hypothesis using two reference-free signals: speech--text alignment in a shared embedding space and predicted word error rate (WER). A simple percentile-based selection rule retains reliable pseudo-labels for fine-tuning while discarding noisy utterances. In an in-domain setting, selecting ~1.5k utterances from a 30k pool achieves 10.91% WER, close to 10.45% obtained using 30k supervised labels. In a cross-domain setting with a mismatched candidate pool, consistency-filtered subsets avoid the degradation caused by unfiltered pseudo-labels under strong accent shift, and matched-hour experiments on a stronger ASR backbone further confirm gains over random sampling and recent selection baselines.