# Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usability

arXiv: https://arxiv.org/abs/2106.02016 · HTML: https://mlsanity.com/paper/2106.02016/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Earnings-21: A Practical Benchmark for ASR in the Wild](https://mlsanity.com/paper/2104.11348/) (2021)
- [The PyTorch-Kaldi Speech Recognition Toolkit](https://mlsanity.com/paper/1811.07453/) (2018)
- [Espresso: A Fast End-to-end Neural Speech Recognition Toolkit](https://mlsanity.com/paper/1909.08723/) (2019)
- [PyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASR](https://mlsanity.com/paper/2005.09824/) (2020)
- [Domain-aware Neural Language Models for Speech Recognition](https://mlsanity.com/paper/2101.03229/) (2021)

## Built on by (most cited first, 14 of 14)

- [Quantifying the Dialect Gap and its Correlates Across Languages](https://mlsanity.com/paper/2310.15135/) (2023)
- [OpenOmni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agents](https://mlsanity.com/paper/2408.03047/) (2024)
- [WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue](https://mlsanity.com/paper/2511.16544/) (2025)
- [Evaluation of Automatic Speech Recognition Using Generative Large Language Models](https://mlsanity.com/paper/2604.21928/) (2026)
- [Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation](https://mlsanity.com/paper/2605.29430/) (2026)
- [Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition](https://mlsanity.com/paper/2604.09121/) (2026)
- [Framework for Curating Speech Datasets and Evaluating ASR Systems: A Case Study for Polish](https://mlsanity.com/paper/2408.00005/) (2024)
- [CallScreenBench: Benchmarking Small Language Models as Phone Secretaries](https://mlsanity.com/paper/2608.01033/) (2026)
- [Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study](https://mlsanity.com/paper/2608.25574/) (2026)
- [VoiceCodeBench: Evaluating Exact Structured-Token Recovery in Automatic Speech Recognition](https://mlsanity.com/paper/2608.28916/) (2026)
- [Context-Aware Interleaved Batching for WhisperX](https://mlsanity.com/paper/2608.31170/) (2026)
- [Talk2Agent: Benchmarking Voice Interfaces for Text Agents](https://mlsanity.com/paper/2609.38867/) (2026)
- [When Does an ASR Error Matter? Task-Role Diagnostics for Tool-Using Voice Agents](https://mlsanity.com/paper/2610.17619/) (2026)
- [Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR](https://mlsanity.com/paper/2610.20461/) (2026)

## Abstract (arXiv)

Recent advances in supervised, semi-supervised and self-supervised deep learning algorithms have shown significant improvement in the performance of automatic speech recognition(ASR) systems. The state-of-the-art systems have achieved a word error rate (WER) less than 5%. However, in the past, researchers have argued the non-suitability of the WER metric for the evaluation of ASR systems for downstream tasks such as spoken language understanding (SLU) and information retrieval. The reason is that the WER works at the surface level and does not include any syntactic and semantic knowledge.The current work proposes Semantic-WER (SWER), a metric to evaluate the ASR transcripts for downstream applications in general. The SWER can be easily customized for any down-stream task.