# NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning

arXiv: https://arxiv.org/abs/2306.12577 · HTML: https://mlsanity.com/paper/2306.12577/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers](https://mlsanity.com/paper/2012.15828/) (2020)
- [Unbabel's Participation in the WMT20 Metrics Shared Task](https://mlsanity.com/paper/2010.15535/) (2020)
- [Learning Word-Level Confidence For Subword End-to-End ASR](https://mlsanity.com/paper/2103.06716/) (2021)
- [Word Error Rate Estimation Without ASR Output: e-WER2](https://mlsanity.com/paper/2008.03403/) (2020)
- [Correcting Automated and Manual Speech Transcription Errors using Warped Language Models](https://mlsanity.com/paper/2103.14580/) (2021)
- [WER-BERT: Automatic WER Estimation with BERT in a Balanced Ordinal Classification Paradigm](https://mlsanity.com/paper/2101.05478/) (2021)
- [Unsupervised Automatic Speech Recognition: A Review](https://mlsanity.com/paper/2106.04897/) (2021)
- [Neural Zero-Inflated Quality Estimation Model For Automatic Speech Recognition System](https://mlsanity.com/paper/1910.01289/) (2019)

## Built on by (most cited first, 4 of 4)

- [Word-Level ASR Quality Estimation for Efficient Corpus Sampling and Post-Editing through Analyzing Attentions of a Reference-Free Metric](https://mlsanity.com/paper/2401.11268/) (2024)
- [On the Robust Approximation of ASR Metrics](https://mlsanity.com/paper/2502.12408/) (2025)
- [An Automated End-to-End Open-Source Software for High-Quality Text-to-Speech Dataset Generation](https://mlsanity.com/paper/2402.16380/) (2024)
- [AutoMode-ASR: Learning to Select ASR Systems for Better Quality and Cost](https://mlsanity.com/paper/2409.12476/) (2024)

## Abstract (arXiv)

This paper introduces NoRefER, a novel referenceless quality metric for automatic speech recognition (ASR) systems. Traditional reference-based metrics for evaluating ASR systems require costly ground-truth transcripts. NoRefER overcomes this limitation by fine-tuning a multilingual language model for pair-wise ranking ASR hypotheses using contrastive learning with Siamese network architecture. The self-supervised NoRefER exploits the known quality relationships between hypotheses from multiple compression levels of an ASR for learning to rank intra-sample hypotheses by quality, which is essential for model comparisons. The semi-supervised version also uses a referenced dataset to improve its inter-sample quality ranking, which is crucial for selecting potentially erroneous samples. The results indicate that NoRefER correlates highly with reference-based metrics and their intra-sample ranks, indicating a high potential for referenceless ASR evaluation or a/b testing.