# RAS: a Reliability Oriented Metric for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2604.24278 · HTML: https://mlsanity.com/paper/2604.24278/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback](https://mlsanity.com/paper/2403.18349/) (2024)
- [Reducing Tool Hallucination via Reliability Alignment](https://mlsanity.com/paper/2412.04141/) (2024)
- [Advocating Character Error Rate for Multilingual ASR Evaluation](https://mlsanity.com/paper/2410.07400/) (2024)
- [An evaluation of word-level confidence estimation for end-to-end automatic speech recognition](https://mlsanity.com/paper/2101.05525/) (2021)
- [TALCS: An Open-Source Mandarin-English Code-Switching Corpus and a Speech Recognition Baseline](https://mlsanity.com/paper/2206.13135/) (2022)
- [Alignment for Efficient Tool Calling of Large Language Models](https://mlsanity.com/paper/2503.06708/) (2025)
- [ASR Rescoring and Confidence Estimation with ELECTRA](https://mlsanity.com/paper/2110.01857/) (2021)
- [Enhancing LLM Reliability via Explicit Knowledge Boundary Modeling](https://mlsanity.com/paper/2503.02233/) (2025)
- [Identifying and Calibrating Overconfidence in Noisy Speech Recognition](https://mlsanity.com/paper/2509.07195/) (2025)

## Abstract (arXiv)

Automatic speech recognition systems often produce confident yet incorrect transcriptions under noisy or ambiguous conditions, which can be misleading for both users and downstream applications. Standard evaluation based on Word Error Rate focuses solely on accuracy and fails to capture transcription reliability. We introduce an abstention-aware transcription framework that enables ASR models to explicitly abstain from uncertain segments. To evaluate reliability under abstention, we propose RAS, a reliability-oriented metric that balances transcription informativeness and error aversion, with its trade-off parameter calibrated by human preference. We then train an abstention-aware ASR model through supervised bootstrapping followed by reinforcement learning. Our experiments demonstrate substantial improvements in transcription reliability while maintaining competitive accuracy.