# Aligning ASR Evaluation with Human and LLM Judgments: Intelligibility Metrics Using Phonetic, Semantic, and NLI Approaches

arXiv: https://arxiv.org/abs/2506.16528 · HTML: https://mlsanity.com/paper/2506.16528/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Combining Fact Extraction and Verification with Neural Semantic Matching Networks](https://mlsanity.com/paper/1811.07039/) (2018)
- [WavLLM: Towards Robust and Adaptive Speech Large Language Model](https://mlsanity.com/paper/2404.00656/) (2024)
- [A spelling correction model for end-to-end speech recognition](https://mlsanity.com/paper/1902.07178/) (2019)
- [MENLI: Robust Evaluation Metrics from Natural Language Inference](https://mlsanity.com/paper/2208.07316/) (2022)
- [Learning from Past Mistakes: Improving Automatic Speech Recognition Output via Noisy-Clean Phrase Context Modeling](https://mlsanity.com/paper/1802.02607/) (2018)
- [H_eval: A new hybrid evaluation metric for automatic speech recognition tasks](https://mlsanity.com/paper/2211.01722/) (2022)
- [Revisiting ASR Error Correction with Specialized Models](https://mlsanity.com/paper/2405.15216/) (2024)
- [Fine-Tuning Automatic Speech Recognition for People with Parkinson's: An Effective Strategy for Enhancing Speech Technology Accessibility](https://mlsanity.com/paper/2409.19818/) (2024)

## Built on by (most cited first, 7 of 7)

- [The Interspeech 2025 Speech Accessibility Project Challenge](https://mlsanity.com/paper/2507.22047/) (2025)
- [LASER: An LLM-based ASR Scoring and Evaluation Rubric](https://mlsanity.com/paper/2510.07437/) (2025)
- [WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue](https://mlsanity.com/paper/2511.16544/) (2025)
- [Beyond Word Error Rate: Auditing the Diversity Tax in Speech Recognition through Dataset Cartography](https://mlsanity.com/paper/2603.05267/) (2026)
- [When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition](https://mlsanity.com/paper/2605.02782/) (2026)
- [Benchmarking Speech-to-Speech Translation Models](https://mlsanity.com/paper/2606.03241/) (2026)
- [A Cross Community Agenda for Speech AI](https://mlsanity.com/paper/2609.13168/) (2026)

## Abstract (arXiv)

Traditional ASR metrics like WER and CER fail to capture intelligibility, especially for dysarthric and dysphonic speech, where semantic alignment matters more than exact word matches. ASR systems struggle with these speech types, often producing errors like phoneme repetitions and imprecise consonants, yet the meaning remains clear to human listeners. We identify two key challenges: (1) Existing metrics do not adequately reflect intelligibility, and (2) while LLMs can refine ASR output, their effectiveness in correcting ASR transcripts of dysarthric speech remains underexplored. To address this, we propose a novel metric integrating Natural Language Inference (NLI) scores, semantic similarity, and phonetic similarity. Our ASR evaluation metric achieves a 0.890 correlation with human judgments on Speech Accessibility Project data, surpassing traditional methods and emphasizing the need to prioritize intelligibility over error-based measures.