mlsanityData to 14 Oct 2026

Whisper Has an Internal Word Aligner

arXiv 2509.09987 ↗First posted 12 Sep 2025eess.AS cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

There is an increasing interest in obtaining accurate word-level timestamps from strong automatic speech recognizers, in particular Whisper. Existing approaches either require additional training or are simply not competitive. The evaluation in prior work is also relatively loose, typically using a tolerance of more than 200 ms. In this work, we discover attention heads in Whisper that capture accurate word alignments and are distinctively different from those that do not. Moreover, we find that using characters produces finer and more accurate alignments than using wordpieces. Based on these findings, we propose an unsupervised approach to extracting word alignments by filtering attention heads while teacher forcing Whisper with characters. Our approach not only does not require training but also produces word alignments that are more accurate than prior work under a stricter tolerance between 20 ms and 100 ms.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 12 Sep 2025.

Builds on

29 of 43 references.

References · 29

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  4. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  5. 2019Attention is not not Explanationcited by 636
  6. 2015Listen, Attend and Spellcited by 543
  7. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  8. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  9. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  10. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  11. 2020CTC-Segmentation of Large Corpora for German End-to-end Speech Recognitioncited by 76
  12. 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
17 more
  1. 2024CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptionscited by 34
  2. 2018OpenNMT: Neural Machine Translation Toolkitcited by 27
  3. 2020On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformerscited by 24
  4. 2024Tradition or Innovation: A Comparison of Modern ASR Methods for Forced Alignmentcited by 21
  5. 2024YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detectioncited by 16
  6. 2020Understanding Self-Attention of Self-Supervised Audio Transformerscited by 15
  7. 2023Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignmentcited by 13
  8. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  9. 2024Less Peaky and More Accurate CTC Forced Alignment by Label Priorscited by 12
  10. 2022Phoneme Segmentation Using Self-Supervised Speech Modelscited by 11
  11. 2021On the Difficulty of Segmenting Words with Attentioncited by 6
  12. 2023Automatic Disfluency Detection from Untranscribed Speechcited by 6
  13. 2024Attention-Constrained Inference for Robust Decoder-Only Text-to-Speechcited by 6
  14. 2018Unsupervised Word Segmentation from Speech with Attentioncited by 4
  15. 2023Delay-penalized CTC implemented based on Finite State Transducercited by 4
  16. 2022Supervised Attention in Sequence-to-Sequence Models for Speech Recognitioncited by 3
  17. 2024The Conformer Encoder May Reverse the Time Dimensioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Modelscited by 1
  2. 2026Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMscited by 1
  3. 2026Quantifying the Generation Modality Gap in Speech-Text Language Modelscited by 1
  4. 2026AURA: Uncertainty-Routed Activation Editing for Acoustic Grounding in Speech Foundation Modelscited by 0
  5. 2026FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speechcited by 0
  6. 2026Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictionscited by 0
  7. 2026Where Words Begin and Endcited by 0