# FLToP CTC: Frame-Level Token Pruning via Relative Threshold for Efficient and Memory-Saving Decoding on Diverse Platforms

arXiv: https://arxiv.org/abs/2510.09085 · HTML: https://mlsanity.com/paper/2510.09085/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 13 of 13)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Pruned RNN-T for fast, memory-efficient ASR training](https://mlsanity.com/paper/2206.13236/) (2022)
- [Flashlight: Enabling Innovation in Tools for Machine Learning](https://mlsanity.com/paper/2201.12465/) (2022)
- [C-LSTM: Enabling Efficient LSTM using Structured Compression Techniques on FPGAs](https://mlsanity.com/paper/1803.06305/) (2018)
- [Accelerating RNN-T Training and Inference Using CTC guidance](https://mlsanity.com/paper/2210.16481/) (2022)
- [Speed of Light Exact Greedy Decoding for RNN-T Speech Recognition Models on GPU](https://mlsanity.com/paper/2406.03791/) (2024)
- [FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization](https://mlsanity.com/paper/2104.02882/) (2021)
- [GPU-Accelerated WFST Beam Search Decoder for CTC-based Speech Recognition](https://mlsanity.com/paper/2311.04996/) (2023)
- [A Hardware-Oriented and Memory-Efficient Method for CTC Decoding](https://mlsanity.com/paper/1905.03175/) (2019)

## Abstract (arXiv)

CTC-based ASR systems face computational and memory bottlenecks in resource-limited environments. Traditional CTC decoders, requiring up to 90% of processing time in systems (e.g., wav2vec2-large on L4 GPUs), face inefficiencies due to exhaustive token-level operations. This paper introduces Frame Level Token Pruning for Connectionist Temporal Classification (FLToP CTC), a novel decoding algorithm that employs frame-level token pruning guided by a relative threshold probability. By dynamically eliminating low-probability tokens per frame, FLToP CTC reduces compute and memory demands while maintaining negligible WER degradation. On LibriSpeech, FLToP CTC achieves a 10.5x runtime speedup and 2.78x memory reduction versus standard CTC decoders. Its simplicity enables seamless integration into CTC decoders across platforms (CPUs, GPUs, etc.). FLToP CTC addresses CTC bottlenecks, offering scalability for resource-limited environments and realtime applications, enhancing speech recognition accessibility and efficiency.