mlsanityData to 14 Oct 2026

Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration

arXiv 2404.12022 ↗First posted 18 Apr 2024cs.CLcited by 3

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models (LLMs) have recently shown remarkable performance across a wide range of tasks. However, the substantial number of parameters in LLMs contributes to significant latency during model inference. This is particularly evident when utilizing autoregressive decoding methods, which generate one token in a single forward process, thereby not fully capitalizing on the parallel computing capabilities of GPUs. In this paper, we propose a novel parallel decoding approach, namely textithidden transfer, which decodes multiple successive tokens simultaneously in a single forward pass. The idea is to transfer the intermediate hidden states of the previous context to the textitpseudo hidden states of the future tokens to be generated, and then the pseudo hidden states will pass the following transformer layers thereby assimilating more semantic information and achieving superior predictive accuracy of the future tokens. Besides, we use the novel tree attention mechanism to simultaneously generate and verify multiple candidates of output sequences, which ensure the lossless generation and further improves the generation efficiency of our method. Experiments demonstrate the effectiveness of our method. We conduct a lot of analytic experiments to prove our motivation. In terms of acceleration metrics, we outperform all the single-model acceleration techniques, including Medusa and Self-Speculative decoding.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 34 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

27 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  2. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
11 more
  1. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  2. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  3. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  4. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  5. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  6. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  7. 2019On the Efficacy of Knowledge Distillationcited by 381
  8. 2019Depth-Adaptive Transformercited by 236
  9. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  10. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 151
  11. 2023LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recoverycited by 38

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 22
  2. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  3. 2025POSS: Position Specialist Generates Better Draft for Speculative Decodingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference
  2. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  3. 2026Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
  4. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding
  5. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding
  6. 2025Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
  7. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding
  8. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding