mlsanityData to 7 Oct 2026

Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usability

arXiv 2106.02016 ↗First posted 3 Jun 2021cs.CL cs.SD eess.AScited by 12

Not on the map. Why?

Abstract (arXiv)

Recent advances in supervised, semi-supervised and self-supervised deep learning algorithms have shown significant improvement in the performance of automatic speech recognition(ASR) systems. The state-of-the-art systems have achieved a word error rate (WER) less than 5%. However, in the past, researchers have argued the non-suitability of the WER metric for the evaluation of ASR systems for downstream tasks such as spoken language understanding (SLU) and information retrieval. The reason is that the WER works at the surface level and does not include any syntactic and semantic knowledge.The current work proposes Semantic-WER (SWER), a metric to evaluate the ASR transcripts for downstream applications in general. The SWER can be easily customized for any down-stream task.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 9 of its 25 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

9 of 25 references.

References · 9

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  5. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 60
  6. 2018The PyTorch-Kaldi Speech Recognition Toolkitcited by 38
  7. 2019Espresso: A Fast End-to-end Neural Speech Recognition Toolkitcited by 31
  8. 2020PyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASRcited by 16
  9. 2021Domain-aware Neural Language Models for Speech Recognitioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 8.

12 papers.

  1. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  2. 2024OpenOmni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agentscited by 3
  3. 2026Evaluation of Automatic Speech Recognition Using Generative Large Language Modelscited by 3
  4. 2025WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialoguecited by 2
  5. 2026Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluationcited by 2
  6. 2026Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognitioncited by 1
  7. 2024Framework for Curating Speech Datasets and Evaluating ASR Systems: A Case Study for Polishcited by 0
  8. 2026CallScreenBench: Benchmarking Small Language Models as Phone Secretariescited by 0
  9. 2026Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Studycited by 0
  10. 2026VoiceCodeBench: Evaluating Exact Structured-Token Recovery in Automatic Speech Recognitioncited by 0
  11. 2026Context-Aware Interleaved Batching for WhisperXcited by 0
  12. 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agentscited by 0