mlsanityData to 14 Oct 2026

Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement

arXiv 2410.13344 ↗First posted 17 Oct 2024cs.CL cs.AIcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models (LLMs) often face a bottleneck in inference speed due to their reliance on auto-regressive decoding. Recently, parallel decoding has shown significant promise in enhancing inference efficiency. However, we have identified two key issues with existing parallel decoding frameworks: (1) decoding heads fail to balance prediction accuracy and the parallelism of execution, and (2) parallel decoding is not a universal solution, as it can bring unnecessary overheads at some challenging decoding steps. To address these issues, we propose Cerberus, an adaptive parallel decoding framework introduces the gating mechanism to enable the LLMs to adaptively choose appropriate decoding approaches at each decoding step, along with introducing a new paradigm of decoding heads that introduce the sequential knowledge while maintaining execution parallelism. The experiment results demonstrate that the Cerberus can achieve up to 2.12x speed up compared to auto-regressive decoding, and outperforms one of the leading parallel decoding frameworks, Medusa, with a 10% - 30% increase in acceleration and superior generation quality.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 31 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

23 of 31 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  6. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  10. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  12. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
1 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decodingcited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2025ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMs
  2. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  3. 2024Plato: Plan to Efficiently Decode for Large Language Model Inference
  4. 2025Accelerating Large Language Model Inference via Early-Exiting Algorithms
  5. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference
  6. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  7. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding
  8. 2023Fast Chain-of-Thought: A Glance of Future from Parallel Decoding Leads to Answers Faster