# Prompting Whisper for Improved Verbatim Transcription and End-to-end Miscue Detection

arXiv: https://arxiv.org/abs/2505.23627 · HTML: https://mlsanity.com/paper/2505.23627/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 21 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 6 of 6)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Full Text-Dependent End to End Mispronunciation Detection and Diagnosis with Easy Data Augmentation Techniques](https://mlsanity.com/paper/2104.08428/) (2021)
- [From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition](https://mlsanity.com/paper/2302.09044/) (2023)
- [Sparsely Shared LoRA on Whisper for Child Speech Recognition](https://mlsanity.com/paper/2309.11756/) (2023)
- [Automatic Assessment of Oral Reading Accuracy for Reading Diagnostics](https://mlsanity.com/paper/2306.03444/) (2023)
- [Towards a Single ASR Model That Generalizes to Disordered Speech](https://mlsanity.com/paper/2412.19315/) (2024)

## Built on by (most cited first, 1 of 1)

- [Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing](https://mlsanity.com/paper/2607.18934/) (2026)

## Abstract (arXiv)

Identifying mistakes (i.e., miscues) made while reading aloud is commonly approached post-hoc by comparing automatic speech recognition (ASR) transcriptions to the target reading text. However, post-hoc methods perform poorly when ASR inaccurately transcribes verbatim speech. To improve on current methods for reading error annotation, we propose a novel end-to-end architecture that incorporates the target reading text via prompting and is trained for both improved verbatim transcription and direct miscue detection. Our contributions include: first, demonstrating that incorporating reading text through prompting benefits verbatim transcription performance over fine-tuning, and second, showing that it is feasible to augment speech recognition tasks for end-to-end miscue detection. We conducted two case studies -- children's read-aloud and adult atypical speech -- and found that our proposed strategies improve verbatim transcription and miscue detection compared to current state-of-the-art.