mlsanityData to 7 Oct 2026

Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric

arXiv 2110.05376 ↗First posted 11 Oct 2021cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Measuring automatic speech recognition (ASR) system quality is critical for creating user-satisfying voice-driven applications. Word Error Rate (WER) has been traditionally used to evaluate ASR system quality; however, it sometimes correlates poorly with user perception/judgement of transcription quality. This is because WER weighs every word equally and does not consider semantic correctness which has a higher impact on user perception. In this work, we propose evaluating ASR output hypotheses quality with SemDist that can measure semantic correctness by using the distance between the semantic vectors of the reference and hypothesis extracted from a pre-trained language model. Our experimental results of 71K and 36K user annotated ASR output quality show that SemDist achieves higher correlation with user perception than WER. We also show that SemDist has higher correlation with downstream Natural Language Understanding (NLU) tasks than WER.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 18 of its 23 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

18 of 23 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  12. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
6 more
  1. 2018Semantic Parsing for Task Oriented Dialog using Hierarchical Representationscited by 101
  2. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  3. 2020Alignment Restricted Streaming Recurrent Neural Network Transducercited by 42
  4. 2021Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understandingcited by 40
  5. 2020Conversational Semantic Parsingcited by 39
  6. 2020General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inferencecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 1; 2024: 3; 2025: 2; 2026: 11.

21 papers.

Sort:

By area:

  1. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  2. 2024SeMaScore : a new evaluation metric for automatic speech recognition taskscited by 11
  3. 2022H_eval: A new hybrid evaluation metric for automatic speech recognition taskscited by 7
  4. 2022Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognitioncited by 5
  5. 2023Solving Cosine Similarity Underestimation between High Frequency Words by L2 Norm Discountingcited by 5
  6. 2024Style-agnostic evaluation of ASR using multiple reference transcriptscited by 4
  7. 2025Towards Audio Token Compression in Large Audio Language Modelscited by 4
  8. 2026HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metricscited by 4
  9. 2026Evaluation of Automatic Speech Recognition Using Generative Large Language Modelscited by 3
  10. 2026Qualitative Evaluation of Language Model Rescoring in Automatic Speech Recognitioncited by 3
  11. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Modelscited by 2
  12. 2022TRScore: A Novel GPT-based Readability Scorer for ASR Segmentation and Punctuation model evaluation and selectioncited by 1
9 more
  1. 2026Beyond Word Error Rate: Auditing the Diversity Tax in Speech Recognition through Dataset Cartographycited by 1
  2. 2026A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognitioncited by 1
  3. 2024Decoupled Sequence and Structure Generation for Realistic Antibody Designcited by 0
  4. 2026Query Suggestion for Retrieval-Augmented Generation via Dynamic In-Context Learningcited by 0
  5. 2026Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Studycited by 0
  6. 2026No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpuscited by 0
  7. 2026Linguistic Distance Segregates Latent Representations in Automatic Speech Recognition Systemscited by 0
  8. 2026Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WERcited by 0
  9. 2026MICRO: Multi-Fidelity Active Search for Severe Error Discoverycited by 0