# SeMaScore : a new evaluation metric for automatic speech recognition tasks

arXiv: https://arxiv.org/abs/2401.07506 · HTML: https://mlsanity.com/paper/2401.07506/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 15 Jan 2024.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric](https://mlsanity.com/paper/2110.05376/) (2021)
- [H_eval: A new hybrid evaluation metric for automatic speech recognition tasks](https://mlsanity.com/paper/2211.01722/) (2022)
- [Assessing ASR Model Quality on Disordered Speech using BERTScore](https://mlsanity.com/paper/2209.10591/) (2022)

## Built on by (most cited first, 12 of 12)

- [Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models](https://mlsanity.com/paper/2502.12414/) (2025)
- [WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue](https://mlsanity.com/paper/2511.16544/) (2025)
- [Evaluation of Automatic Speech Recognition Using Generative Large Language Models](https://mlsanity.com/paper/2604.21928/) (2026)
- [SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models](https://mlsanity.com/paper/2507.19361/) (2025)
- [From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection](https://mlsanity.com/paper/2606.23060/) (2026)
- [Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics](https://mlsanity.com/paper/2603.22709/) (2026)
- [HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systems](https://mlsanity.com/paper/2606.23048/) (2026)
- [Graph-Based Phonetic Error Correction of Noisy ASR](https://mlsanity.com/paper/2606.24889/) (2026)
- [What Counts as an Error? Dual-Reference Benchmarking for Atypical ASR](https://mlsanity.com/paper/2606.31112/) (2026)
- [Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study](https://mlsanity.com/paper/2608.25574/) (2026)
- [Talk2Agent: Benchmarking Voice Interfaces for Text Agents](https://mlsanity.com/paper/2609.38867/) (2026)
- [Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR](https://mlsanity.com/paper/2610.20461/) (2026)

## Abstract (arXiv)

In this study, we present SeMaScore, generated using a segment-wise mapping and scoring algorithm that serves as an evaluation metric for automatic speech recognition tasks. SeMaScore leverages both the error rate and a more robust similarity score. We show that our algorithm's score generation improves upon the state-of-the-art BERTScore. Our experimental results show that SeMaScore corresponds well with expert human assessments, signal-to-noise ratio levels, and other natural language metrics. We outperform BERTScore by 41x in metric computation speed. Overall, we demonstrate that SeMaScore serves as a more dependable evaluation metric, particularly in real-world situations involving atypical speech patterns.