mlsanityData to 7 Oct 2026

FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling

arXiv 2502.14856 ↗First posted 20 Feb 2025cs.CL cs.AI cs.LGcited by 20

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify mechanism to produce multiple tokens per forward pass. While state-of-the-art speculative sampling methods use only a single layer and a language modeling (LM) head as the draft model to achieve impressive layer compression, their efficiency gains are substantially reduced for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens. To address this, we present FR-Spec, a frequency-ranked speculative sampling framework that optimizes draft candidate selection through vocabulary space compression. By constraining the draft search to a frequency-prioritized token subset, our method reduces LM Head computation overhead by 75% while ensuring the equivalence of the final output distribution. Experiments across multiple datasets demonstrate an average of 1.12× speedup over the state-of-the-art speculative sampling method EAGLE-2. Code available at https://github.com/thunlp/FR-Spec.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 50 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

36 of 50 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures25 Dec
  2. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  3. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  5. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  6. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding21 Feb
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  9. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  10. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  11. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  12. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
2 more
  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  2. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 825
10 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  2. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 234
  5. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  6. 2016Efficient softmax approximation for GPUscited by 67
  7. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  8. 2024Cut Your Losses in Large-Vocabulary Language Modelscited by 35
  9. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inferencecited by 19
  10. 2024Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Trainingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 11.

20 papers.

Sort:

By area:

  1. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  2. 2026AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discoverycited by 17
  3. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  4. 2025VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMscited by 11
  5. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  6. 2026Speculative Decoding with a Speculative Vocabularycited by 6
  7. 2025DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Modelscited by 4
  8. 2025Out-of-Vocabulary Sampling Boosts Speculative Decodingcited by 2
  9. 2026FlashHead: Efficient Drop-In Replacement for the Classification Head in Language Model Inferencecited by 2
  10. 2026Trees from Marginals: Autoregressive drafting with factorized priorscited by 2
  11. 2025SpeLLM: Character-Level Multi-Head Decodingcited by 1
  12. 2025SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Servingcited by 1
8 more
  1. 2026NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabulariescited by 1
  2. 2025COMPACT: Common-token Optimized Model Pruning Across Channels and Tokenscited by 0
  3. 2026SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decodingcited by 0
  4. 2026DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devicescited by 0
  5. 2026EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptationcited by 0
  6. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0
  7. 2026Acceptance-Aware Draft Model Training for Speculative Decodingcited by 0
  8. 2026H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cachecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
  2. 2023PaSS: Parallel Speculative Sampling
  3. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  4. 2025S4C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models
  5. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
  6. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  7. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
  8. 2024Beyond the Speculative Game: A Survey of Speculative Execution in Large Language Models