# FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling arXiv: https://arxiv.org/abs/2502.14856 · HTML: https://mlsanity.com/paper/2502.14856/ ## Key facts - Cited by 20 papers in the map (data through 14 Oct 2026). - Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. - Builds on 36 of its 50 references that are in the map. - First posted on arXiv 20 Feb 2025. ## Placed in - Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/) ## Builds on (most cited first, 30 of 36) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023) - [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022) - [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023) - [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023) - [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016) - [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024) - [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016) - [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020) - [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024) - [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023) - [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021) - [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024) - [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024) - [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017) - [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022) - [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023) - [Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies](https://mlsanity.com/paper/2407.13623/) (2024) - [TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding](https://mlsanity.com/paper/2404.11912/) (2024) - [Learning Harmonized Representations for Speculative Sampling](https://mlsanity.com/paper/2408.15766/) (2024) - [Efficient softmax approximation for GPUs](https://mlsanity.com/paper/1609.04309/) (2016) - [OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure](https://mlsanity.com/paper/2406.17276/) (2024) ## Built on by (most cited first, 20 of 20) - [MiniCPM4: Ultra-Efficient LLMs on End Devices](https://mlsanity.com/paper/2506.07900/) (2025) - [AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery](https://mlsanity.com/paper/2604.05550/) (2026) - [FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction](https://mlsanity.com/paper/2509.18362/) (2025) - [VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs](https://mlsanity.com/paper/2506.22694/) (2025) - [BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity](https://mlsanity.com/paper/2507.08771/) (2025) - [Speculative Decoding with a Speculative Vocabulary](https://mlsanity.com/paper/2602.13836/) (2026) - [DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Models](https://mlsanity.com/paper/2510.13847/) (2025) - [Trees from Marginals: Autoregressive drafting with factorized priors](https://mlsanity.com/paper/2607.06763/) (2026) - [Out-of-Vocabulary Sampling Boosts Speculative Decoding](https://mlsanity.com/paper/2506.03206/) (2025) - [SpeLLM: Character-Level Multi-Head Decoding](https://mlsanity.com/paper/2507.16323/) (2025) - [FlashHead: Efficient Drop-In Replacement for the Classification Head in Language Model Inference](https://mlsanity.com/paper/2603.14591/) (2026) - [SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving](https://mlsanity.com/paper/2512.00719/) (2025) - [NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies](https://mlsanity.com/paper/2605.26444/) (2026) - [H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache](https://mlsanity.com/paper/2609.24197/) (2026) - [COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens](https://mlsanity.com/paper/2509.06836/) (2025) - [SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding](https://mlsanity.com/paper/2605.10453/) (2026) - [DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices](https://mlsanity.com/paper/2605.10933/) (2026) - [EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation](https://mlsanity.com/paper/2605.27390/) (2026) - [Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detection](https://mlsanity.com/paper/2607.05148/) (2026) - [Acceptance-Aware Draft Model Training for Speculative Decoding](https://mlsanity.com/paper/2609.24150/) (2026) ## Abstract (arXiv) Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify mechanism to produce multiple tokens per forward pass. While state-of-the-art speculative sampling methods use only a single layer and a language modeling (LM) head as the draft model to achieve impressive layer compression, their efficiency gains are substantially reduced for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens. To address this, we present FR-Spec, a frequency-ranked speculative sampling framework that optimizes draft candidate selection through vocabulary space compression. By constraining the draft search to a frequency-prioritized token subset, our method reduces LM Head computation overhead by 75% while ensuring the equivalence of the final output distribution. Experiments across multiple datasets demonstrate an average of 1.12× speedup over the state-of-the-art speculative sampling method EAGLE-2. Code available at https://github.com/thunlp/FR-Spec.