# Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper

arXiv: https://arxiv.org/abs/2608.30927 · HTML: https://mlsanity.com/paper/2608.30927/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio and omni-modal token compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Placed in

- Efficient LLM inference and serving / [Audio and omni-modal token compression](https://mlsanity.com/inference/audio-and-omni-modal-token-compression/)

## Builds on (most cited first, 21 of 21)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2506.04779/) (2025)
- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023)
- [LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis](https://mlsanity.com/paper/2505.02625/) (2025)
- [Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts](https://mlsanity.com/paper/2311.01070/) (2023)
- [Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importance](https://mlsanity.com/paper/2504.01690/) (2025)
- [Training and Inference Efficiency of Encoder-Decoder Speech Models](https://mlsanity.com/paper/2503.05931/) (2025)
- [On the Locality of Attention in Direct Speech Translation](https://mlsanity.com/paper/2204.09028/) (2022)
- [FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Model](https://mlsanity.com/paper/2410.03007/) (2024)
- [To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation](https://mlsanity.com/paper/2406.04512/) (2024)
- [WhisperKit: On-device Real-time ASR with Billion-Scale Transformers](https://mlsanity.com/paper/2507.10860/) (2025)
- [BaldWhisper: Faster Whisper with Head Shearing and Layer Merging](https://mlsanity.com/paper/2510.08599/) (2025)
- [Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models](https://mlsanity.com/paper/2604.06871/) (2026)
- [Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion](https://mlsanity.com/paper/2603.00563/) (2026)
- [Early Attentive Sparsification Accelerates Neural Speech Transcription](https://mlsanity.com/paper/2506.15912/) (2025)
- [Pruning as Regularization: Sensitivity-Aware One-Shot Pruning in ASR](https://mlsanity.com/paper/2511.08092/) (2025)
- [Speaking Clearly: A Simplified Whisper-Based Codec for Low-Bitrate Speech Coding](https://mlsanity.com/paper/2510.20504/) (2025)

## Abstract (arXiv)

Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-token interface carries more capacity than downstream tasks require.