# Word-Level ASR Quality Estimation for Efficient Corpus Sampling and Post-Editing through Analyzing Attentions of a Reference-Free Metric

arXiv: https://arxiv.org/abs/2401.11268 · HTML: https://mlsanity.com/paper/2401.11268/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 20 Jan 2024.

## Builds on (most cited first, 16 of 16)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers](https://mlsanity.com/paper/2012.15828/) (2020)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives](https://mlsanity.com/paper/1909.01380/) (2019)
- [Unbabel's Participation in the WMT20 Metrics Shared Task](https://mlsanity.com/paper/2010.15535/) (2020)
- [The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results](https://mlsanity.com/paper/2110.04392/) (2021)
- [Translation Error Detection as Rationale Extraction](https://mlsanity.com/paper/2108.12197/) (2021)
- [A Reference-less Quality Metric for Automatic Speech Recognition via Contrastive-Learning of a Multi-Language Model with Self-Supervision](https://mlsanity.com/paper/2306.13114/) (2023)
- [Multilingual Word Error Rate Estimation: e-WER3](https://mlsanity.com/paper/2304.00649/) (2023)
- [NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning](https://mlsanity.com/paper/2306.12577/) (2023)
- [EvolveMT: an Ensemble MT Engine Improving Itself with Usage Only](https://mlsanity.com/paper/2306.11823/) (2023)
- [Efficient Machine Translation Corpus Generation](https://mlsanity.com/paper/2306.11838/) (2023)

## Built on by (most cited first, 4 of 4)

- [An Automated End-to-End Open-Source Software for High-Quality Text-to-Speech Dataset Generation](https://mlsanity.com/paper/2402.16380/) (2024)
- [AutoMode-ASR: Learning to Select ASR Systems for Better Quality and Cost](https://mlsanity.com/paper/2409.12476/) (2024)
- [Utterance-Level Methods for Identifying Reliable ASR-Output for Child Speech](https://mlsanity.com/paper/2604.19801/) (2026)
- [Transcribing Children's Speech: ASR Performance and Obtaining Reliable Orthographic Transcriptions](https://mlsanity.com/paper/2605.28833/) (2026)

## Abstract (arXiv)

In the realm of automatic speech recognition (ASR), the quest for models that not only perform with high accuracy but also offer transparency in their decision-making processes is crucial. The potential of quality estimation (QE) metrics is introduced and evaluated as a novel tool to enhance explainable artificial intelligence (XAI) in ASR systems. Through experiments and analyses, the capabilities of the NoRefER (No Reference Error Rate) metric are explored in identifying word-level errors to aid post-editors in refining ASR hypotheses. The investigation also extends to the utility of NoRefER in the corpus-building process, demonstrating its effectiveness in augmenting datasets with insightful annotations. The diagnostic aspects of NoRefER are examined, revealing its ability to provide valuable insights into model behaviors and decision patterns. This has proven beneficial for prioritizing hypotheses in post-editing workflows and fine-tuning ASR models. The findings suggest that NoRefER is not merely a tool for error detection but also a comprehensive framework for enhancing ASR systems' transparency, efficiency, and effectiveness. To ensure the reproducibility of the results, all source codes of this study are made publicly available.