mlsanityData to 7 Oct 2026

WhisperX: Time-Accurate Speech Transcription of Long-Form Audio

arXiv 2303.00747 ↗First posted 1 Mar 2023cs.SD eess.AScited by 320

Not on the map. Why?

Abstract (arXiv)

Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, their application to long audio transcription via buffered or sliding window approaches is prone to drifting, hallucination & repetition; and prohibits batched transcription due to their sequential nature. Further, timestamps corresponding each utterance are prone to inaccuracies and word-level timestamps are not available out-of-the-box. To overcome these challenges, we present WhisperX, a time-accurate speech recognition system with word-level timestamps utilising voice activity detection and forced phoneme alignment. In doing so, we demonstrate state-of-the-art performance on long-form transcription and word segmentation benchmarks. Additionally, we show that pre-segmenting audio with our proposed VAD Cut & Merge strategy improves transcription quality and enables a twelve-fold transcription speedup via batched inference.

Key facts. Cited by 320 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Mar 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 67; 2025: 111; 2026: 118.

320 papers.

Sort:

By area:

  1. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
  2. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  3. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
  4. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  5. 2023AutoAD: Movie Description in Contextcited by 50
  6. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 50
  7. 2023AutoAD II: The Sequel -- Who, When, and What in Movie Audio Descriptioncited by 48
  8. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  9. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  10. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  11. 2024MatchTime: Towards Automatic Soccer Game Commentary Generationcited by 37
  12. 2024AutoAD III: The Prequel -- Back to the Pixelscited by 35
38 more
  1. 2024LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learningcited by 34
  2. 2024CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptionscited by 33
  3. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 32
  4. 2024Empirical evidence of Large Language Model's influence on human spoken communicationcited by 27
  5. 2025LeVo: High-Quality Song Generation with Multi-Preference Alignmentcited by 23
  6. 2025Paper2Video: Automatic Video Generation from Scientific Paperscited by 23
  7. 2024SpoofCeleb: Speech Deepfake Detection and SASV In The Wildcited by 22
  8. 2023Audio-Visual Instance Segmentationcited by 21
  9. 2023EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Modelscited by 21
  10. 2024AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Descriptioncited by 21
  11. 2025SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinementcited by 21
  12. 2025Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generationcited by 19
  13. 2024Scaling Up Video Summarization Pretraining with Large Language Modelscited by 17
  14. 2024Moonshine: Speech Recognition for Live Transcription and Voice Commandscited by 17
  15. 2025FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systemscited by 17
  16. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  17. 2024YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detectioncited by 16
  18. 2025SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learningcited by 16
  19. 2024Towards Hierarchical Spoken Language Dysfluency Modelingcited by 15
  20. 2024Measuring the Accuracy of Automatic Speech Recognition Solutionscited by 15
  21. 2024Anatomy of Industrial Scale Multilingual ASRcited by 14
  22. 2025X-LeBench: A Benchmark for Extremely Long Egocentric Video Understandingcited by 14
  23. 2026VIBEVOICE-ASR Technical Reportcited by 14
  24. 2023Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuningcited by 13
  25. 2024Less Peaky and More Accurate CTC Forced Alignment by Label Priorscited by 12
  26. 2025Contextual Gesture: Co-Speech Gesture Video Generation through Context-aware Gesture Representationcited by 12
  27. 2025Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Downcited by 12
  28. 2024dMel: Speech Tokenization made Simplecited by 11
  29. 2024DistinctAD: Distinctive Audio Description Generation in Contextscited by 11
  30. 2025BLAB: Brutally Long Audio Benchcited by 11
  31. 2025MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mindcited by 11
  32. 2023Multi-Sentence Grounding for Long-term Instructional Videocited by 10
  33. 2025VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Modelscited by 10
  34. 2025Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signalscited by 10
  35. 2025PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AIcited by 10
  36. 2024FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wildcited by 9
  37. 2024Accompanied Singing Voice Synthesis with Fully Text-controlled Melodycited by 9
  38. 2024MMSummary: Multimodal Summary Generation for Fetal Ultrasound Videocited by 9

50 most cited of 320 All 320 (JSON)