# WhisperKit: On-device Real-time ASR with Billion-Scale Transformers

arXiv: https://arxiv.org/abs/2507.10860 · HTML: https://mlsanity.com/paper/2507.10860/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 8 of 8)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Earnings-22: A Practical Benchmark for Accents in the Wild](https://mlsanity.com/paper/2203.15591/) (2022)
- [Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection](https://mlsanity.com/paper/2005.11185/) (2020)
- [Recurrent Drafter for Fast Speculative Decoding in Large Language Models](https://mlsanity.com/paper/2403.09919/) (2024)
- [BitsFusion: 1.99 bits Weight Quantization of Diffusion Model](https://mlsanity.com/paper/2406.04333/) (2024)
- [Turning Whisper into Real-Time Transcription System](https://mlsanity.com/paper/2307.14743/) (2023)
- [Conformer-Based Speech Recognition On Extreme Edge-Computing Devices](https://mlsanity.com/paper/2312.10359/) (2023)

## Built on by (most cited first, 4 of 4)

- [Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild](https://mlsanity.com/paper/2604.07354/) (2026)
- [Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper](https://mlsanity.com/paper/2608.30927/) (2026)
- [Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery](https://mlsanity.com/paper/2609.27980/) (2026)
- [AS2D: Accelerating On-Demand Audio Understanding on Mobile Devices](https://mlsanity.com/paper/2609.37617/) (2026)

## Abstract (arXiv)

Real-time Automatic Speech Recognition (ASR) is a fundamental building block for many commercial applications of ML, including live captioning, dictation, meeting transcriptions, and medical scribes. Accuracy and latency are the most important factors when companies select a system to deploy. We present WhisperKit, an optimized on-device inference system for real-time ASR that significantly outperforms leading cloud-based systems. We benchmark against server-side systems that deploy a diverse set of models, including a frontier model (OpenAI gpt-4o-transcribe), a proprietary model (Deepgram nova-3), and an open-source model (Fireworks large-v3-turbo).Our results show that WhisperKit matches the lowest latency at 0.46s while achieving the highest accuracy 2.2% WER. The optimizations behind the WhisperKit system are described in detail in this paper.