# Evaluation of Automatic Speech Recognition Using Generative Large Language Models

arXiv: https://arxiv.org/abs/2604.21928 · HTML: https://mlsanity.com/paper/2604.21928/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [Advocating Character Error Rate for Multilingual ASR Evaluation](https://mlsanity.com/paper/2410.07400/) (2024)
- [Evaluating User Perception of Speech Recognition System Quality with Semantic Distance Metric](https://mlsanity.com/paper/2110.05376/) (2021)
- [Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usability](https://mlsanity.com/paper/2106.02016/) (2021)
- [SeMaScore : a new evaluation metric for automatic speech recognition tasks](https://mlsanity.com/paper/2401.07506/) (2024)
- [Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing](https://mlsanity.com/paper/1712.02033/) (2017)

## Built on by (most cited first, 3 of 3)

- [Context-Aware Interleaved Batching for WhisperX](https://mlsanity.com/paper/2608.31170/) (2026)
- [Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER](https://mlsanity.com/paper/2609.21663/) (2026)
- [Talk2Agent: Benchmarking Voice Interfaces for Text Agents](https://mlsanity.com/paper/2609.38867/) (2026)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLMs achieve 92--94% agreement with human annotators for hypothesis selection, compared to 63% for WER, also outperforming semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation.