mlsanityData to 7 Oct 2026

Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding

arXiv 2104.02138 ↗First posted 5 Apr 2021cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

Word Error Rate (WER) has been the predominant metric used to evaluate the performance of automatic speech recognition (ASR) systems. However, WER is sometimes not a good indicator for downstream Natural Language Understanding (NLU) tasks, such as intent recognition, slot filling, and semantic parsing in task-oriented dialog systems. This is because WER takes into consideration only literal correctness instead of semantic correctness, the latter of which is typically more important for these downstream tasks. In this study, we propose a novel Semantic Distance (SemDist) measure as an alternative evaluation metric for ASR systems to address this issue. We define SemDist as the distance between a reference and hypothesis pair in a sentence-level embedding space. To represent the reference and hypothesis as a sentence embedding, we exploit RoBERTa, a state-of-the-art pre-trained deep contextualized language model based on the transformer architecture. We demonstrate the effectiveness of our proposed metric on various downstream tasks, including intent recognition, semantic parsing, and named entity recognition.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

19 of 32 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  11. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  12. 2018Semantic Parsing for Task Oriented Dialog using Hierarchical Representationscited by 101
7 more
  1. 2018Towards end-to-end spoken language understandingcited by 92
  2. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  3. 2020Deep Shallow Fusion for RNN-T Personalizationcited by 64
  4. 2020Alignment Restricted Streaming Recurrent Neural Network Transducercited by 42
  5. 2020Conversational Semantic Parsingcited by 39
  6. 2020Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducercited by 22
  7. 2019From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognitioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 3; 2024: 3; 2025: 10; 2026: 19.

40 papers.

Sort:

By area:

  1. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  2. 2024Advocating Character Error Rate for Multilingual ASR Evaluationcited by 22
  3. 2021Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metriccited by 21
  4. 2024SeMaScore : a new evaluation metric for automatic speech recognition taskscited by 11
  5. 2022Assessing ASR Model Quality on Disordered Speech using BERTScorecited by 8
  6. 2023Spaiche: Extending State-of-the-Art ASR Models to Swiss German Dialectscited by 8
  7. 2022H_eval: A new hybrid evaluation metric for automatic speech recognition taskscited by 7
  8. 2025STRICT: Stress Test of Rendering Images Containing Textcited by 7
  9. 2025Hallucination Benchmark for Speech Foundation Modelscited by 7
  10. 2025Identifying Reliable Evaluation Metrics for Scientific Text Revisioncited by 6
  11. 2022Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognitioncited by 5
  12. 2023A Deep Dive into the Disparity of Word Error Rates Across Thousands of NPTEL MOOC Videoscited by 5
28 more
  1. 2025LASER: An LLM-based ASR Scoring and Evaluation Rubriccited by 5
  2. 2022Toward Zero Oracle Word Error Rate on the Switchboard Benchmarkcited by 4
  3. 2026HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metricscited by 4
  4. 2025PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speechcited by 3
  5. 2026Evaluation of Automatic Speech Recognition Using Generative Large Language Modelscited by 3
  6. 2024Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learningcited by 2
  7. 2025Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challengescited by 2
  8. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Modelscited by 2
  9. 2025Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognitioncited by 2
  10. 2025WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialoguecited by 2
  11. 2026A Comprehensive Analysis of Tokenization and Self-Supervised Learning in End-to-End Automatic Speech Recognition applied on French Languagecited by 2
  12. 2026Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluationcited by 2
  13. 2026Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognitioncited by 1
  14. 2026A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognitioncited by 1
  15. 2026WASIL: In-the-Wild Arabic Spoken Interactions with LLMscited by 1
  16. 2025What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systemscited by 0
  17. 2026Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metricscited by 0
  18. 2026Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancycited by 0
  19. 2026Voice Memory for Agentic Speech Recognitioncited by 0
  20. 2026CallScreenBench: Benchmarking Small Language Models as Phone Secretariescited by 0
  21. 2026Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Studycited by 0
  22. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0
  23. 2026VoiceCodeBench: Evaluating Exact Structured-Token Recovery in Automatic Speech Recognitioncited by 0
  24. 2026SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognitioncited by 0
  25. 2026MedWER: A Reproducible, Model-Free Evaluation Protocol for Medical Speech Recognitioncited by 0
  26. 2026From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivitycited by 0
  27. 2026mu-bench: A Multilingual Utterance Transcription Benchmarkcited by 0
  28. 2026Talk2Agent: Benchmarking Voice Interfaces for Text Agentscited by 0