mlsanityData to 14 Oct 2026

CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions

arXiv 2408.16589 ↗First posted 29 Aug 2024cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

We demonstrate that carefully adjusting the tokenizer of the Whisper speech recognition model significantly improves the precision of word-level timestamps when applying dynamic time warping to the decoder's cross-attention scores. We fine-tune the model to produce more verbatim speech transcriptions and employ several techniques to increase robustness against multiple speakers and background noise. These adjustments achieve state-of-the-art performance on benchmarks for verbatim speech transcription, word segmentation, and the timed detection of filler events, and can further mitigate transcription hallucinations. The code is available open https://github.com/nyrahealth/CrisperWhisper.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 16 of its 35 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

16 of 35 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  8. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  9. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  10. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  11. 2024Careless Whisper: Speech-to-Text Hallucination Harmscited by 53
  12. 2019Learning Sound Event Classifiers from Web Audio with Noisy Labelscited by 22
4 more
  1. 2023From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognitioncited by 18
  2. 2022Filler Word Detection and Classification: A Dataset and Benchmarkcited by 7
  3. 2023Automatic Disfluency Detection from Untranscribed Speechcited by 6
  4. 2023Careful Whisper -- leveraging advances in automatic speech recognition for robust and interpretable aphasia subtype classificationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 14.

34 papers.

Sort:

By area:

  1. 2025Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilitiescited by 100
  2. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 27
  3. 2025Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluationcited by 23
  4. 2025Analysis and Evaluation of Synthetic Data Generation in Speech Dysfluency Detectioncited by 8
  5. 2025EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wildcited by 7
  6. 2025Whisper Has an Internal Word Alignercited by 7
  7. 2025KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AIcited by 7
  8. 2025Word Level Timestamp Generation for Automatic Speech Recognition and Translationcited by 6
  9. 2025Automatic Speech Recognition Biases in Newcastle English: an Error Analysiscited by 6
  10. 2025Exploring Generative Error Correction for Dysarthric Speech Recognitioncited by 3
  11. 2025Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasiacited by 3
  12. 2026Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decodingcited by 3
22 more
  1. 2026Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timingcited by 3
  2. 2025WhisperD: Dementia Speech Recognition and Filler Word Detection with Whispercited by 2
  3. 2025The Impact of Automatic Speech Transcription on Speaker Attributioncited by 2
  4. 2025Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHAcited by 1
  5. 2025Dysarthria Normalization via Local Lie Group Transformations for Robust ASRcited by 1
  6. 2025Smooth Operators: LLMs Translating Imperfect Hints into Disfluency-Rich Transcriptscited by 1
  7. 2025Talking to Robots: A Practical Examination of Speech Foundation Models for HRI Applicationscited by 1
  8. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  9. 2025Linguistic and Audio Embedding-Based Machine Learning for Alzheimer's Dementia and Mild Cognitive Impairment Detection: Insights from the PROCESS Challengecited by 1
  10. 2026ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognitioncited by 1
  11. 2026In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictionscited by 1
  12. 2026From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detectioncited by 1
  13. 2026SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Modelscited by 1
  14. 2026Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMscited by 1
  15. 2026Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialoguecited by 1
  16. 2025Revealing the Role of Audio Channels in ASR Performance Degradationcited by 0
  17. 2026PAREDA: A Multi-Accent Speech Dataset of Natural Language Processing Research Discussionscited by 0
  18. 2026Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASRcited by 0
  19. 2026Mitigating Scoring Errors and Compensating for Nonverbal Subtests in Speech-Based Dementia Assessmentcited by 0
  20. 2026HALAS: A Human-Annotated Dataset of Hallucinations of Modern ASR Systemscited by 0
  21. 2026FA-Bench: A Benchmark for Phone- and Word-Level Timestamp Accuracy in Forced Alignment and ASR on Clean and Noisy Speechcited by 0
  22. 2026From Detection To Anticipation: Continuous Onset Forecasting For Stuttered Speechcited by 0