# Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding

arXiv: https://arxiv.org/abs/2104.02138 · HTML: https://mlsanity.com/paper/2104.02138/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 5 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [Semantic Parsing for Task Oriented Dialog using Hierarchical Representations](https://mlsanity.com/paper/1810.07942/) (2018)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Alignment Restricted Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2011.03072/) (2020)
- [Conversational Semantic Parsing](https://mlsanity.com/paper/2009.13655/) (2020)
- [Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2010.13878/) (2020)
- [From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.01493/) (2019)

## Built on by (most cited first, 30 of 42)

- [Quantifying the Dialect Gap and its Correlates Across Languages](https://mlsanity.com/paper/2310.15135/) (2023)
- [Advocating Character Error Rate for Multilingual ASR Evaluation](https://mlsanity.com/paper/2410.07400/) (2024)
- [Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric](https://mlsanity.com/paper/2110.05376/) (2021)
- [SeMaScore : a new evaluation metric for automatic speech recognition tasks](https://mlsanity.com/paper/2401.07506/) (2024)
- [H_eval: A new hybrid evaluation metric for automatic speech recognition tasks](https://mlsanity.com/paper/2211.01722/) (2022)
- [Assessing ASR Model Quality on Disordered Speech using BERTScore](https://mlsanity.com/paper/2209.10591/) (2022)
- [Spaiche: Extending State-of-the-Art ASR Models to Swiss German Dialects](https://mlsanity.com/paper/2304.11075/) (2023)
- [Hallucination Benchmark for Speech Foundation Models](https://mlsanity.com/paper/2510.16567/) (2025)
- [STRICT: Stress Test of Rendering Images Containing Text](https://mlsanity.com/paper/2505.18985/) (2025)
- [Identifying Reliable Evaluation Metrics for Scientific Text Revision](https://mlsanity.com/paper/2506.04772/) (2025)
- [Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition](https://mlsanity.com/paper/2211.16319/) (2022)
- [A Deep Dive into the Disparity of Word Error Rates Across Thousands of NPTEL MOOC Videos](https://mlsanity.com/paper/2307.10587/) (2023)
- [LASER: An LLM-based ASR Scoring and Evaluation Rubric](https://mlsanity.com/paper/2510.07437/) (2025)
- [Toward Zero Oracle Word Error Rate on the Switchboard Benchmark](https://mlsanity.com/paper/2206.06192/) (2022)
- [HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics](https://mlsanity.com/paper/2604.27542/) (2026)
- [WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue](https://mlsanity.com/paper/2511.16544/) (2025)
- [PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech](https://mlsanity.com/paper/2512.23686/) (2025)
- [Evaluation of Automatic Speech Recognition Using Generative Large Language Models](https://mlsanity.com/paper/2604.21928/) (2026)
- [Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learning](https://mlsanity.com/paper/2501.00039/) (2024)
- [Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges](https://mlsanity.com/paper/2507.18161/) (2025)
- [SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models](https://mlsanity.com/paper/2507.19361/) (2025)
- [Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognition](https://mlsanity.com/paper/2510.19471/) (2025)
- [A Comprehensive Analysis of Tokenization and Self-Supervised Learning in End-to-End Automatic Speech Recognition applied on French Language](https://mlsanity.com/paper/2605.03696/) (2026)
- [Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation](https://mlsanity.com/paper/2605.29430/) (2026)
- [Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition](https://mlsanity.com/paper/2604.09121/) (2026)
- [A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition](https://mlsanity.com/paper/2605.03671/) (2026)
- [WASIL: In-the-Wild Arabic Spoken Interactions with LLMs](https://mlsanity.com/paper/2605.16364/) (2026)
- [What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systems](https://mlsanity.com/paper/2508.04402/) (2025)
- [Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics](https://mlsanity.com/paper/2603.22709/) (2026)
- [Read What You Hear: Reference-Free Hypotheses Evaluation with Acoustic Discrepancy](https://mlsanity.com/paper/2606.04680/) (2026)

## Abstract (arXiv)

Word Error Rate (WER) has been the predominant metric used to evaluate the performance of automatic speech recognition (ASR) systems. However, WER is sometimes not a good indicator for downstream Natural Language Understanding (NLU) tasks, such as intent recognition, slot filling, and semantic parsing in task-oriented dialog systems. This is because WER takes into consideration only literal correctness instead of semantic correctness, the latter of which is typically more important for these downstream tasks. In this study, we propose a novel Semantic Distance (SemDist) measure as an alternative evaluation metric for ASR systems to address this issue. We define SemDist as the distance between a reference and hypothesis pair in a sentence-level embedding space. To represent the reference and hypothesis as a sentence embedding, we exploit RoBERTa, a state-of-the-art pre-trained deep contextualized language model based on the transformer architecture. We demonstrate the effectiveness of our proposed metric on various downstream tasks, including intent recognition, semantic parsing, and named entity recognition.