# WhisperX: Time-Accurate Speech Transcription of Long-Form Audio

arXiv: https://arxiv.org/abs/2303.00747 · HTML: https://mlsanity.com/paper/2303.00747/

## Key facts

- Cited by 326 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2023.

## Built on by (most cited first, 30 of 326)

- [Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation](https://mlsanity.com/paper/2407.05361/) (2024)
- [CinePile: A Long Video Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2405.08813/) (2024)
- [Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding](https://mlsanity.com/paper/2505.18079/) (2025)
- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023)
- [PG-Video-LLaVA: Pixel Grounding Large Video-Language Models](https://mlsanity.com/paper/2311.13435/) (2023)
- [AutoAD: Movie Description in Context](https://mlsanity.com/paper/2303.16899/) (2023)
- [AutoAD II: The Sequel -- Who, When, and What in Movie Audio Description](https://mlsanity.com/paper/2310.06838/) (2023)
- [MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning](https://mlsanity.com/paper/2311.17435/) (2023)
- [Retrieval-Augmented Egocentric Video Captioning](https://mlsanity.com/paper/2401.00789/) (2024)
- [VidChapters-7M: Video Chapters at Scale](https://mlsanity.com/paper/2309.13952/) (2023)
- [MatchTime: Towards Automatic Soccer Game Commentary Generation](https://mlsanity.com/paper/2406.18530/) (2024)
- [AutoAD III: The Prequel -- Back to the Pixels](https://mlsanity.com/paper/2404.14412/) (2024)
- [LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning](https://mlsanity.com/paper/2408.07981/) (2024)
- [CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions](https://mlsanity.com/paper/2408.16589/) (2024)
- [Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning](https://mlsanity.com/paper/2309.11500/) (2023)
- [Empirical evidence of Large Language Model's influence on human spoken communication](https://mlsanity.com/paper/2409.01754/) (2024)
- [LeVo: High-Quality Song Generation with Multi-Preference Alignment](https://mlsanity.com/paper/2506.07520/) (2025)
- [Paper2Video: Automatic Video Generation from Scientific Papers](https://mlsanity.com/paper/2510.05096/) (2025)
- [AutoAD-Zero: A Training-Free Framework for Zero-Shot Audio Description](https://mlsanity.com/paper/2407.15850/) (2024)
- [SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement](https://mlsanity.com/paper/2506.07634/) (2025)
- [EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models](https://mlsanity.com/paper/2312.14069/) (2023)
- [SpoofCeleb: Speech Deepfake Detection and SASV In The Wild](https://mlsanity.com/paper/2409.17285/) (2024)
- [Audio-Visual Instance Segmentation](https://mlsanity.com/paper/2310.18709/) (2023)
- [Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation](https://mlsanity.com/paper/2501.15907/) (2025)
- [FD-Bench: A Full-Duplex Benchmarking Pipeline Designed for Full Duplex Spoken Dialogue Systems](https://mlsanity.com/paper/2507.19040/) (2025)
- [Moonshine: Speech Recognition for Live Transcription and Voice Commands](https://mlsanity.com/paper/2410.15608/) (2024)
- [Scaling Up Video Summarization Pretraining with Large Language Models](https://mlsanity.com/paper/2404.03398/) (2024)
- [Learning Video Context as Interleaved Multimodal Sequences](https://mlsanity.com/paper/2407.21757/) (2024)
- [YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection](https://mlsanity.com/paper/2408.15297/) (2024)
- [SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning](https://mlsanity.com/paper/2509.10555/) (2025)

## Abstract (arXiv)

Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, their application to long audio transcription via buffered or sliding window approaches is prone to drifting, hallucination & repetition; and prohibits batched transcription due to their sequential nature. Further, timestamps corresponding each utterance are prone to inaccuracies and word-level timestamps are not available out-of-the-box. To overcome these challenges, we present WhisperX, a time-accurate speech recognition system with word-level timestamps utilising voice activity detection and forced phoneme alignment. In doing so, we demonstrate state-of-the-art performance on long-form transcription and word segmentation benchmarks. Additionally, we show that pre-segmenting audio with our proposed VAD Cut & Merge strategy improves transcription quality and enables a twelve-fold transcription speedup via batched inference.