mlsanityData to 14 Oct 2026

The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation

arXiv 2411.03786 ↗First posted 6 Nov 2024cs.LGcited by 14

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding aims to speed up autoregressive generation of a language model by verifying in parallel the tokens generated by a smaller draft model.In this work, we explore the effectiveness of learning-free, negligible-cost draft strategies, namely N-grams obtained from the model weights and the context. While the predicted next token of the base model is rarely the top prediction of these simple strategies, we observe that it is often within their top-k predictions for small k. Based on this, we show that combinations of simple strategies can achieve significant inference speedups over different tasks. The overall performance is comparable to more complex methods, yet does not require expensive preprocessing or modification of the base model, and allows for seamless `plug-and-play' integration into pipelines.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 32 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

25 of 32 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  6. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  7. 2023Online Speculative Decoding11 Oct
  8. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  9. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  10. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  11. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
2 more
  1. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  2. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  2. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decodingcited by 15
  3. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 14
  4. 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMscited by 10
  5. 2025DReSD: Dense Retrieval for Speculative Decodingcited by 6
  6. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 6
  7. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Modelscited by 5
  8. 2025Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problemcited by 3
  9. 2025Polybasic Speculative Decoding Through a Theoretical Perspectivecited by 1
  10. 2026SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decodingcited by 1
  11. 2026Task-Centric Acceleration of Small-Language Modelscited by 0
  12. 2026N-gram Injection into Transformers for Dynamic Language Model Adaptation in Handwritten Text Recognitioncited by 0
2 more
  1. 2026DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Executioncited by 0
  2. 2026AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fillcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Draft, Verify, and Improve: Toward Training-Aware Speculative Decoding
  2. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  4. 2026SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
  5. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
  6. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  7. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  8. 2026Speculative Speculative Decoding