# WER-BERT: Automatic WER Estimation with BERT in a Balanced Ordinal Classification Paradigm

arXiv: https://arxiv.org/abs/2101.05478 · HTML: https://mlsanity.com/paper/2101.05478/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 14 Jan 2021.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Very Deep Convolutional Neural Networks for Raw Waveforms](https://mlsanity.com/paper/1610.00087/) (2016)
- [Sequential Domain Adaptation through Elastic Weight Consolidation for Sentiment Analysis](https://mlsanity.com/paper/2007.01189/) (2020)
- [A SentiWordNet Strategy for Curriculum Learning in Sentiment Analysis](https://mlsanity.com/paper/2005.04749/) (2020)
- [ASR Performance Prediction on Unseen Broadcast Programs using Convolutional Neural Networks](https://mlsanity.com/paper/1804.08477/) (2018)
- [Sequential Learning of Convolutional Features for Effective Text Classification](https://mlsanity.com/paper/1909.00080/) (2019)
- [A Position Aware Decay Weighted Network for Aspect based Sentiment Analysis](https://mlsanity.com/paper/2005.01027/) (2020)

## Built on by (most cited first, 5 of 5)

- [CORAA: a large corpus of spontaneous and prepared speech manually validated for speech recognition in Brazilian Portuguese](https://mlsanity.com/paper/2110.15731/) (2021)
- [A Reference-less Quality Metric for Automatic Speech Recognition via Contrastive-Learning of a Multi-Language Model with Self-Supervision](https://mlsanity.com/paper/2306.13114/) (2023)
- [NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning](https://mlsanity.com/paper/2306.12577/) (2023)
- [Big model only for hard audios: Sample dependent Whisper model selection for efficient inferences](https://mlsanity.com/paper/2309.12712/) (2023)
- [On the Robust Approximation of ASR Metrics](https://mlsanity.com/paper/2502.12408/) (2025)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) systems are evaluated using Word Error Rate (WER), which is calculated by comparing the number of errors between the ground truth and the transcription of the ASR system. This calculation, however, requires manual transcription of the speech signal to obtain the ground truth. Since transcribing audio signals is a costly process, Automatic WER Evaluation (e-WER) methods have been developed to automatically predict the WER of a speech system by only relying on the transcription and the speech signal features. While WER is a continuous variable, previous works have shown that positing e-WER as a classification problem is more effective than regression. However, while converting to a classification setting, these approaches suffer from heavy class imbalance. In this paper, we propose a new balanced paradigm for e-WER in a classification setting. Within this paradigm, we also propose WER-BERT, a BERT based architecture with speech features for e-WER. Furthermore, we introduce a distance loss function to tackle the ordinal nature of e-WER classification. The proposed approach and paradigm are evaluated on the Librispeech dataset and a commercial (black box) ASR system, Google Cloud's Speech-to-Text API. The results and experiments demonstrate that WER-BERT establishes a new state-of-the-art in automatic WER estimation.