mlsanityData to 7 Oct 2026

H_eval: A new hybrid evaluation metric for automatic speech recognition tasks

arXiv 2211.01722 ↗First posted 3 Nov 2022cs.CL cs.SD eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

Many studies have examined the shortcomings of word error rate (WER) as an evaluation metric for automatic speech recognition (ASR) systems. Since WER considers only literal word-level correctness, new evaluation metrics based on semantic similarity such as semantic distance (SD) and BERTScore have been developed. However, we found that these metrics have their own limitations, such as a tendency to overly prioritise keywords. We propose H_eval, a new hybrid evaluation metric for ASR systems that considers both semantic correctness and error rate and performs significantly well in scenarios where WER and SD perform poorly. Due to lighter computation compared to BERTScore, it offers 49 times reduction in metric computation time. Furthermore, we show that H_eval correlates strongly with downstream NLP tasks. Also, to reduce the metric calculation time, we built multiple fast and lightweight models using distillation techniques

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 14 of its 31 references that are in the map. First posted on arXiv 3 Nov 2022.

Builds on

14 of 31 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  11. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  12. 2021Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understandingcited by 40
2 more
  1. 2021Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metriccited by 21
  2. 2022Assessing ASR Model Quality on Disordered Speech using BERTScorecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2024SeMaScore : a new evaluation metric for automatic speech recognition taskscited by 11
  2. 2025Aligning ASR Evaluation with Human and LLM Judgments: Intelligibility Metrics Using Phonetic, Semantic, and NLI Approachescited by 7
  3. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Modelscited by 2
  4. 2025WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialoguecited by 2
  5. 2026Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluationcited by 2
  6. 2026Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER)cited by 1
  7. 2025What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systemscited by 0