Data to 7 Oct 2026

Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding

arXiv 2310.05424 ↗First posted 9 Oct 2023cs.CLcited by 80

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

To tackle the high inference latency exhibited by autoregressive language models, previous studies have proposed an early-exiting framework that allocates adaptive computation paths for each token based on the complexity of generating the subsequent token. However, we observed several shortcomings, including performance degradation caused by a state copying mechanism or numerous exit paths, and sensitivity to exit confidence thresholds. Consequently, we propose a Fast and Robust Early-Exiting (FREE) framework, which incorporates a shallow-deep module and a synchronized parallel decoding. Our framework enables faster inference by synchronizing the decoding process of the current token with previously stacked early-exited tokens. Furthermore, as parallel decoding allows us to observe predictions from both shallow and deep models, we present a novel adaptive threshold estimator that exploits a Beta mixture model to determine suitable confidence thresholds. We empirically demonstrated the superiority of our proposed framework on extensive generation tasks.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 54 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

40 of 54 references.

Closest ancestors · 3

Same direction, newest first

Other references · 37

2017Attention Is All You Needcited by 59,932
2020Language Models are Few-Shot Learnerscited by 32,199
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
2019The Curious Case of Neural Text Degenerationcited by 2,881
2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281

25 more

2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
2019Contrastive Representation Distillationcited by 734
2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
2021Datasets: A Community Library for Natural Language Processingcited by 526
2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
2017Non-Autoregressive Neural Machine Translationcited by 457
2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
2022Confident Adaptive Language Modelingcited by 297
2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
2019Depth-Adaptive Transformercited by 227
2022Structured Pruning Learns Compact and Accurate Modelscited by 167
2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
2023Human-like Summarization Evaluation with ChatGPTcited by 126
2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
2021Distilling Linguistic Context for Language Model Compressioncited by 33
2019SCAN: A Scalable Neural Networks Framework Towards Compact and Efficient Modelscited by 22
2021Self-Contrastive Learning: Single-viewed Supervised Contrastive Framework using Sub-networkcited by 5
2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspectivecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 33; 2025: 22; 2026: 17.

80 papers.

Sort:

By area:

2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
2024A Survey on Transformer Compressioncited by 44
2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37

38 more

2024Decoding Speculative Decodingcited by 33
2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 31
2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 21
2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
2023Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Servingcited by 15
2023ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inferencecited by 13
2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Modelscited by 13
2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculationcited by 13
2024ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decodingcited by 12
2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafterscited by 12
2025AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inferencecited by 12
2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 12
2023SPIN: Sparsifying and Integrating Internal Neurons in Large Language Models for Text Classificationcited by 10
2025The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exitcited by 10
2024When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Modelscited by 9
2025Map the Flow: Revealing Hidden Pathways of Information in VideoLLMscited by 9
2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 8
2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inferencecited by 8
2023DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Modelscited by 7
2024HiRE: High Recall Approximate Top-k Estimation for Efficient LLM Inferencecited by 7
2024A-VL: Adaptive Attention for Large Vision-Language Modelscited by 7
2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelismcited by 7
2026Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readoutscited by 7
2024S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUscited by 6
2025Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlapscited by 6
2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decodingcited by 6
2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Modelscited by 5
2024Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inferencecited by 5

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way