# Acoustically Precise Hesitation Tagging Is Essential for End-to-End Verbatim Transcription Systems

arXiv: https://arxiv.org/abs/2506.04076 · HTML: https://mlsanity.com/paper/2506.04076/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 9 of 9)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback](https://mlsanity.com/paper/2412.11986/) (2024)
- [Speak & Improve Challenge 2025: Tasks and Baseline Systems](https://mlsanity.com/paper/2412.11985/) (2024)
- [Towards End-to-End Spoken Grammatical Error Correction](https://mlsanity.com/paper/2311.05550/) (2023)
- [The NTNU System at the Interspeech 2020 Non-Native Children's Speech ASR Challenge](https://mlsanity.com/paper/2005.08433/) (2020)

## Built on by (most cited first, 2 of 2)

- [Session-Level Spoken Language Assessment with a Multimodal Foundation Model via Multi-Target Learning](https://mlsanity.com/paper/2509.16025/) (2025)
- [Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR](https://mlsanity.com/paper/2606.14391/) (2026)

## Abstract (arXiv)

Verbatim transcription for automatic speaking assessment demands accurate capture of disfluencies, crucial for downstream tasks like error analysis and feedback. However, many ASR systems discard or generalize hesitations, losing important acoustic details. We fine-tune Whisper models on the Speak & Improve 2025 corpus using low-rank adaptation (LoRA), without recourse to external audio training data. We compare three annotation schemes: removing hesitations (Pure), generic tags (Rich), and acoustically precise fillers inferred by Gemini 2.0 Flash from existing audio-transcript pairs (Extra). Our challenge system achieved 6.47% WER (Pure) and 5.81% WER (Extra). Post-challenge experiments reveal that fine-tuning Whisper Large V3 Turbo with the "Extra" scheme yielded a 5.5% WER, an 11.3% relative improvement over the "Pure" scheme (6.2% WER). This demonstrates that explicit, realistic filled-pause labeling significantly enhances ASR accuracy for verbatim L2 speech transcription.