mlsanityData to 7 Oct 2026

POSS: Position Specialist Generates Better Draft for Speculative Decoding

arXiv 2506.03566 ↗First posted 4 Jun 2025cs.CL cs.AI cs.LGcited by 2

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding accelerates Large Language Model (LLM) inference by using a small draft model to predict multiple tokens, and a large target model to verify these tokens in parallel. Recent studies leverage the hidden state of the target model to enhance draft model prediction accuracy. However, existing methods suffer from the degrading quality of draft token predictions at later positions, due to error accumulation in draft model generated features. In this paper, we propose Position Specialists (PosS), which consist of multiple position-specialized draft layers to generate tokens at assigned position(s). Position specialists greatly improve token acceptance rate at later positions per drafting round, as each specialist only needs to focus on handling a certain level of draft model feature deviation. Experiment results on Llama-3-8B-Instruct and Llama-2-13B-chat across six datasets demonstrate that PosS effectively improves over baselines on average acceptance length and speed-up ratio. Our codebase is available at https://github.com/shrango/PosS.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 39 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

34 of 39 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning10 Apr
  2. 2025Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding28 Mar
  3. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decoding16 Feb
  4. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment31 Jan
  5. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  6. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  7. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  8. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  9. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  10. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  11. 2024EMS-SD: Efficient Multi-sample Speculative Decoding for Accelerating Large Language Models13 May
  12. 2024Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration18 Apr
10 more
  1. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  2. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding10 Apr
  3. 2024ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding21 Feb
  4. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  10. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 12

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  5. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  6. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  7. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
  8. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  9. 2025Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Timecited by 48
  10. 2024Non-myopic Generation of Language Models for Reasoning and Planningcited by 25
  11. 2025Efficient Reasoning for LLMs through Speculative Chain-of-Thoughtcited by 23
  12. 2024Divide, Reweight, and Conquer: A Logit Arithmetic Approach for In-Context Learningcited by 9

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 36
  2. 2025Accelerated Test-Time Scaling with Model-Free Speculative Samplingcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoding Speculative Decoding
  2. 2025Speculative Decoding: Performance or Illusion?
  3. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
  4. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs
  5. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  6. 2024S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models
  7. 2024Accelerating Production LLMs with Combined Token/Embedding Speculators
  8. 2025Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs