# Automatic Speech Recognition System-Independent Word Error Rate Estimation

arXiv: https://arxiv.org/abs/2404.16743 · HTML: https://mlsanity.com/paper/2404.16743/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Builds on (most cited first, 15 of 15)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition](https://mlsanity.com/paper/2010.11428/) (2020)
- [LDNet: Unified Listener Dependent Modeling in MOS Prediction for Synthetic Speech](https://mlsanity.com/paper/2110.09103/) (2021)
- [Active and Semi-Supervised Learning in ASR: Benefits on the Acoustic and Language Models](https://mlsanity.com/paper/1903.02852/) (2019)
- [DNN No-Reference PSTN Speech Quality Prediction](https://mlsanity.com/paper/2007.14598/) (2020)
- [Word Error Rate Estimation Without ASR Output: e-WER2](https://mlsanity.com/paper/2008.03403/) (2020)
- [Multilingual Word Error Rate Estimation: e-WER3](https://mlsanity.com/paper/2304.00649/) (2023)
- [Fast Word Error Rate Estimation Using Self-Supervised Representations for Speech and Text](https://mlsanity.com/paper/2310.08225/) (2023)

## Built on by (most cited first, 2 of 2)

- [Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filtering](https://mlsanity.com/paper/2506.03681/) (2025)
- [Multimodal Consistency-Guided Reference-Free Data Selection for ASR Accent Adaptation](https://mlsanity.com/paper/2602.13263/) (2026)

## Abstract (arXiv)

Word error rate (WER) is a metric used to evaluate the quality of transcriptions produced by Automatic Speech Recognition (ASR) systems. In many applications, it is of interest to estimate WER given a pair of a speech utterance and a transcript. Previous work on WER estimation focused on building models that are trained with a specific ASR system in mind (referred to as ASR system-dependent). These are also domain-dependent and inflexible in real-world applications. In this paper, a hypothesis generation method for ASR System-Independent WER estimation (SIWE) is proposed. In contrast to prior work, the WER estimators are trained using data that simulates ASR system output. Hypotheses are generated using phonetically similar or linguistically more likely alternative words. In WER estimation experiments, the proposed method reaches a similar performance to ASR system-dependent WER estimators on in-domain data and achieves state-of-the-art performance on out-of-domain data. On the out-of-domain data, the SIWE model outperformed the baseline estimators in root mean square error and Pearson correlation coefficient by relative 17.58% and 18.21%, respectively, on Switchboard and CALLHOME. The performance was further improved when the WER of the training set was close to the WER of the evaluation dataset.