mlsanityData to 7 Oct 2026

Mixture of Attentions For Speculative Decoding

arXiv 2410.03804 ↗First posted 4 Oct 2024cs.CL cs.AI cs.LGcited by 17

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The growth in the number of parameters of Large Language Models (LLMs) has led to a significant surge in computational requirements, making them challenging and costly to deploy. Speculative decoding (SD) leverages smaller models to efficiently propose future tokens, which are then verified by the LLM in parallel. Small models that utilise activations from the LLM currently achieve the fastest decoding speeds. However, we identify several limitations of SD models including the lack of on-policyness during training and partial observability. To address these shortcomings, we propose a more grounded architecture for small models by introducing a Mixture of Attentions for SD. Our novel architecture can be applied in two scenarios: a conventional single device deployment and a novel client-server deployment where the small model is hosted on a consumer device and the LLM on a server. In a single-device scenario, we demonstrate state-of-the-art speedups improving EAGLE-2 by 9.5% and its acceptance length by 25%. In a client-server setting, our experiments demonstrate: 1) state-of-the-art latencies with minimal calls to the server for different network conditions, and 2) in the event of a complete disconnection, our approach can maintain higher accuracy compared to other SD methods and demonstrates advantages over API calls to LLMs, which would otherwise be unable to continue the generation process.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 42 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

32 of 42 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  2. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters24 Jun
  3. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  4. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attribution29 May
  5. 2024Accelerating Production LLMs with Combined Token/Embedding Speculators29 Apr
  6. 2024CLLMs: Consistency Large Language Models28 Feb
  7. 2024Tandem Transformers for Inference Efficient LLMs13 Feb
  8. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  9. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  10. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
6 more
  1. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  2. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  3. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  4. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  5. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  6. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  10. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  11. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  12. 2018Model compression via distillation and quantizationcited by 287
2 more
  1. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  2. 2021Sparse Attention with Linear Unitscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  2. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 27
  3. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  4. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  5. 2025VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMscited by 11
  6. 2025Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decodingcited by 9
  7. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  8. 2026P-EAGLE: Parallel-Drafting EAGLE with Scalable Trainingcited by 5
  9. 2025Towards Pervasive Distributed Agentic Generative AI -- A State of The Artcited by 4
  10. 2025Steering Pretrained Drafters during Speculative Decodingcited by 4
  11. 2026ConfSpec: Efficient Step-Level Speculative Reasoning via Confidence-Gated Verificationcited by 4
  12. 2026The Y-Combinator for LLMs: Solving Long-Context Rot with λ-Calculuscited by 3
5 more
  1. 2025HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decodingcited by 2
  2. 2026TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verificationcited by 1
  3. 2026FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Servingcited by 1
  4. 2025Cross-Attention Speculative Decodingcited by 0
  5. 2026The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding: Performance or Illusion?
  2. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
  3. 2024Decoding Speculative Decoding
  4. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  5. 2024S2D: Sorted Speculative Decoding For More Efficient Deployment of Nested Large Language Models
  6. 2026An Interpretable Latency Model for Speculative Decoding in LLM Serving
  7. 2025SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
  8. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts