# Scaling Laws for Discriminative Speech Recognition Rescoring Models

arXiv: https://arxiv.org/abs/2306.15815 · HTML: https://mlsanity.com/paper/2306.15815/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2023.

## Built on by (most cited first, 5 of 5)

- [The Larger the Better? Improved LLM Code-Generation via Budget Reallocation](https://mlsanity.com/paper/2404.00725/) (2024)
- [Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition](https://mlsanity.com/paper/2309.15223/) (2023)
- [Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection](https://mlsanity.com/paper/2403.04073/) (2024)
- [Speech Recognition Rescoring with Large Speech-Text Foundation Models](https://mlsanity.com/paper/2409.16654/) (2024)
- [Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition](https://mlsanity.com/paper/2401.10447/) (2024)

## Abstract (arXiv)

Recent studies have found that model performance has a smooth power-law relationship, or scaling laws, with training data and model size, for a wide range of problems. These scaling laws allow one to choose nearly optimal data and model sizes. We study whether this scaling property is also applicable to second-pass rescoring, which is an important component of speech recognition systems. We focus on RescoreBERT as the rescoring model, which uses a pre-trained Transformer-based architecture fined tuned with an ASR discriminative loss. Using such a rescoring model, we show that the word error rate (WER) follows a scaling law for over two orders of magnitude as training data and model size increase. In addition, it is found that a pre-trained model would require less data than a randomly initialized model of the same size, representing effective data transferred from pre-training step. This effective data transferred is found to also follow a scaling law with the data and model size.