# Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing

arXiv: https://arxiv.org/abs/1712.02033 · HTML: https://mlsanity.com/paper/1712.02033/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2017.

## Built on by (most cited first, 11 of 11)

- [Measuring the Accuracy of Automatic Speech Recognition Solutions](https://mlsanity.com/paper/2408.16287/) (2024)
- [HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics](https://mlsanity.com/paper/2604.27542/) (2026)
- [Evaluation of Automatic Speech Recognition Using Generative Large Language Models](https://mlsanity.com/paper/2604.21928/) (2026)
- [Beyond Levenshtein: Leveraging Multiple Algorithms for Robust Word Error Rate Computations And Granular Error Classifications](https://mlsanity.com/paper/2408.15616/) (2024)
- [A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition](https://mlsanity.com/paper/2605.03671/) (2026)
- [Empowering the Deaf and Hard of Hearing Community: Enhancing Video Captions Using Large Language Models](https://mlsanity.com/paper/2412.00342/) (2024)
- [What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systems](https://mlsanity.com/paper/2508.04402/) (2025)
- [EvolveCaptions: Empowering DHH Users Through Real-Time Collaborative Captioning](https://mlsanity.com/paper/2510.02181/) (2025)
- [OpenWER: Improving Cross-Lingual ASR Evaluation and Enabling Token-Based Accuracy Metrics](https://mlsanity.com/paper/2606.21237/) (2026)
- [Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study](https://mlsanity.com/paper/2608.25574/) (2026)
- [Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER](https://mlsanity.com/paper/2609.21663/) (2026)

## Abstract (arXiv)

The accuracy of Automated Speech Recognition (ASR) technology has improved, but it is still imperfect in many settings. Researchers who evaluate ASR performance often focus on improving the Word Error Rate (WER) metric, but WER has been found to have little correlation with human-subject performance on many applications. We propose a new captioning-focused evaluation metric that better predicts the impact of ASR recognition errors on the usability of automatically generated captions for people who are Deaf or Hard of Hearing (DHH). Through a user study with 30 DHH users, we compared our new metric with the traditional WER metric on a caption usability evaluation task. In a side-by-side comparison of pairs of ASR text output (with identical WER), the texts preferred by our new metric were preferred by DHH participants. Further, our metric had significantly higher correlation with DHH participants' subjective scores on the usability of a caption, as compared to the correlation between WER metric and participant subjective scores. This new metric could be used to select ASR systems for captioning applications, and it may be a better metric for ASR researchers to consider when optimizing ASR systems.