# EvolveCaptions: Empowering DHH Users Through Real-Time Collaborative Captioning

arXiv: https://arxiv.org/abs/2510.02181 · HTML: https://mlsanity.com/paper/2510.02181/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 66 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Builds on (most cited first, 7 of 7)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Speaker adaptation for Wav2vec2 based dysarthric ASR](https://mlsanity.com/paper/2204.00770/) (2022)
- [Personalized Automatic Speech Recognition Trained on Small Disordered Speech Datasets](https://mlsanity.com/paper/2110.04612/) (2021)
- [Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing](https://mlsanity.com/paper/1712.02033/) (2017)
- [Deaf, Hard of Hearing, and Hearing Perspectives on using Automatic Speech Recognition in Conversation](https://mlsanity.com/paper/1909.01176/) (2019)
- [Perceiver-Prompt: Flexible Speaker Adaptation in Whisper for Chinese Disordered Speech Recognition](https://mlsanity.com/paper/2406.09873/) (2024)
- [See-Through Captions: Real-Time Captioning on Transparent Display for Deaf and Hard-of-Hearing People](https://mlsanity.com/paper/2101.11326/) (2021)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) systems often fail to accurately transcribe speech from Deaf and Hard of Hearing (DHH) individuals, especially during real-time conversations. Existing personalization approaches typically require extensive pre-recorded data and place the burden of adaptation on the DHH speaker. We present EvolveCaptions, a real-time, collaborative ASR adaptation system that supports in-situ personalization with minimal effort. Hearing participants correct ASR errors during live conversations. Based on these corrections, the system generates short, phonetically targeted prompts for the DHH speaker to record, which are then used to fine-tune the ASR model. In a study with 12 DHH and six hearing participants, EvolveCaptions reduced Word Error Rate (WER) across all DHH users within one hour of use, using only five minutes of recording time on average. Participants described the system as intuitive, low-effort, and well-integrated into communication. These findings demonstrate the promise of collaborative, real-time ASR adaptation for more equitable communication.