mlsanityData to 7 Oct 2026

Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding

arXiv 2404.08698 ↗First posted 10 Apr 2024cs.CL cs.LGcited by 21

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

While Large Language Models (LLMs) have shown remarkable abilities, they are hindered by significant resource consumption and considerable latency due to autoregressive processing. In this study, we introduce Adaptive N-gram Parallel Decoding (ANPD), an innovative and lossless approach that accelerates inference by allowing the simultaneous generation of multiple tokens. ANPD incorporates a two-stage approach: it begins with a rapid drafting phase that employs an N-gram module, which adapts based on the current interactive context, followed by a verification phase, during which the original LLM assesses and confirms the proposed tokens. Consequently, ANPD preserves the integrity of the LLM's original output while enhancing processing speed. We further leverage a multi-level architecture for the N-gram module to enhance the precision of the initial draft, consequently reducing inference latency. ANPD eliminates the need for retraining or extra GPU memory, making it an efficient and plug-and-play enhancement. In our experiments, models such as LLaMA and its fine-tuned variants have shown speed improvements up to 3.67x, validating the effectiveness of our proposed ANPD.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 42 references that are in the map. First posted on arXiv 10 Apr 2024.

Builds on

26 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  2. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  3. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference5 Jul
  4. 2023Speculative Decoding with Big Little Decoder15 Feb
  5. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  6. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  12. 2015Teaching Machines to Read and Comprehendcited by 1,039
8 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  5. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  6. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  7. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  8. 2022Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformerscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  2. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 34
  3. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  4. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  5. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  6. 2025Towards Optimal Multi-draft Speculative Decodingcited by 19
  7. 2025Towards AI Search Paradigmcited by 19
  8. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  9. 2024Inevitable Trade-off between Watermark Strength and Speculative Sampling Efficiency for Language Modelscited by 7
  10. 2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decodingcited by 6
  11. 2024Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inferencecited by 5
  12. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 5
9 more
  1. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Modelscited by 4
  2. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decodingcited by 4
  3. 2025Accelerated Test-Time Scaling with Model-Free Speculative Samplingcited by 3
  4. 2024SSSD: Simply-Scalable Speculative Decodingcited by 2
  5. 2025POSS: Position Specialist Generates Better Draft for Speculative Decodingcited by 2
  6. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inferencecited by 0
  7. 2026DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Executioncited by 0
  8. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Servingcited by 0
  9. 2026ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMs
  2. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  3. 2024Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration
  4. 2025Accelerating Large Language Model Inference via Early-Exiting Algorithms
  5. 2024Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement
  6. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding
  7. 2024Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens
  8. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference