mlsanityData to 7 Oct 2026

Learning Harmonized Representations for Speculative Sampling

arXiv 2408.15766 ↗First posted 28 Aug 2024cs.LG cs.CLcited by 69

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative sampling is a promising approach to accelerate the decoding stage for Large Language Models (LLMs). Recent advancements that leverage target LLM's contextual information, such as hidden states and KV cache, have shown significant practical improvements. However, these approaches suffer from inconsistent context between training and decoding. We also observe another discrepancy between the training and decoding objectives in existing speculative sampling methods. In this work, we propose a solution named HArmonized Speculative Sampling (HASS) that learns harmonized representations to address these issues. HASS accelerates the decoding stage without adding inference overhead through harmonized objective distillation and harmonized context alignment. Experiments on four LLaMA models demonstrate that HASS achieves 2.81x-4.05x wall-clock time speedup ratio averaging across three datasets, surpassing EAGLE-2 by 8%-20%. The code is available at https://github.com/HArmonizedSS/HASS.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 29 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

24 of 29 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding1 Aug
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  3. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters24 Jun
  4. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  5. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  6. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  8. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  9. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  11. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  12. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 12

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  9. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  10. 2018Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender Systemcited by 60
  11. 2021Recall@k Surrogate Loss with Large Batches and Similarity Mixupcited by 49
  12. 2024BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 32; 2026: 36.

69 papers.

Sort:

By area:

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  2. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptancecited by 31
  3. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  4. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 28
  5. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  6. 2026DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inferencecited by 23
  7. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  8. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  9. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 16
  10. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 15
  11. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 15
  12. 2025Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoffcited by 14
38 more
  1. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculationcited by 13
  2. 2026TALON: Confidence-Aware Speculative Decoding with Adaptive Token Treescited by 12
  3. 2025ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMscited by 11
  4. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 11
  5. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 11
  6. 2025KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimizationcited by 10
  7. 2025CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Draftercited by 9
  8. 2025Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decodingcited by 9
  9. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 9
  10. 2025Enabling Approximate Joint Sampling in Diffusion LMscited by 9
  11. 2025SpecExit: Accelerating Large Reasoning Model via Speculative Exitcited by 9
  12. 2026KERV: Kinematic-Rectified Speculative Decoding for Embodied VLA Modelscited by 9
  13. 2026DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decodingcited by 9
  14. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decodingcited by 7
  15. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 7
  16. 2025Verifying LLM Inference to Detect Model Weight Exfiltrationcited by 6
  17. 2026Speculative Decoding with a Speculative Vocabularycited by 6
  18. 2026P-EAGLE: Parallel-Drafting EAGLE with Scalable Trainingcited by 5
  19. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5
  20. 2026Verification-Aware Training for Speculative Decodingcited by 5
  21. 2025Scaling Laws for Speculative Decodingcited by 4
  22. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Modelscited by 4
  23. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  24. 2026Make Every Draft Count: Hidden State based Speculative Decodingcited by 4
  25. 2026OxyGen: Unified KV Cache Management for VLA Inference under Multi-Task Parallelismcited by 4
  26. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decodingcited by 4
  27. 2026When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?cited by 4
  28. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decodingcited by 3
  29. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarioscited by 3
  30. 2026SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Draftingcited by 3
  31. 2026Teaching Diffusion to Speculate Left-to-Rightcited by 3
  32. 2025POSS: Position Specialist Generates Better Draft for Speculative Decodingcited by 2
  33. 2025DiFR: Inference Verification Despite Nondeterminismcited by 2
  34. 2026Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptancecited by 2
  35. 2025Fast Inference of Visual Autoregressive Model with Adjacency-Adaptive Dynamical Draft Treescited by 1
  36. 2026TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verificationcited by 1
  37. 2026SMART: When is it Actually Worth Expanding a Speculative Tree?cited by 1
  38. 2026UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decodingcited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification
  2. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  3. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
  4. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  5. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  6. 2025Speculative Decoding: Performance or Illusion?
  7. 2024Decoding Speculative Decoding
  8. 2026A Practical Investigation of Training-free Relaxed Speculative Decoding