mlsanityData to 7 Oct 2026

Hierarchical Skip Decoding for Efficient Autoregressive Text Generation

arXiv 2403.14919 ↗First posted 22 Mar 2024cs.CL cs.AIcited by 7

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Autoregressive decoding strategy is a commonly used method for text generation tasks with pre-trained language models, while early-exiting is an effective approach to speedup the inference stage. In this work, we propose a novel decoding strategy named Hierarchical Skip Decoding (HSD) for efficient autoregressive text generation. Different from existing methods that require additional trainable components, HSD is a plug-and-play method applicable to autoregressive text generation models, it adaptively skips decoding layers in a hierarchical manner based on the current sequence length, thereby reducing computational workload and allocating computation resources. Comprehensive experiments on five text generation datasets with pre-trained language models demonstrate HSD's advantages in balancing efficiency and text quality. With almost half of the layers skipped, HSD can sustain 90% of the text quality compared to vanilla autoregressive decoding, outperforming the competitive approaches.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 49 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

40 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference5 Jul

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
27 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  2. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  4. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  5. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  6. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  7. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  8. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  9. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  10. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  11. 2022Confident Adaptive Language Modelingcited by 297
  12. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  13. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  14. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  15. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  16. 2019Depth-Adaptive Transformercited by 227
  17. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  18. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  19. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  20. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  21. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  22. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  23. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  24. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  25. 2020Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domainscited by 33
  26. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  27. 2023Copy Is All You Needcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  2. 2025AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inferencecited by 12
  3. 2025IG-Pruning: Input-Guided Block Pruning for Large Language Modelscited by 2
  4. 2026SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planningcited by 1
  5. 2025Making Language Model a Hierarchical Classifiercited by 0
  6. 2026Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generationcited by 0
  7. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models
  2. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniques
  3. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping
  4. 2022A Self-Paced Mixed Distillation Method for Non-Autoregressive Generation
  5. 2025Fast Inference via Hierarchical Speculative Decoding
  6. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models
  7. 2024Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
  8. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control