mlsanityData to 7 Oct 2026

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism

arXiv 2312.04916 ↗First posted 8 Dec 2023cs.LG cs.AI cs.DCcited by 69

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

We present EE-LLM, a framework for large-scale training and inference of early-exit large language models (LLMs). While recent works have shown preliminary evidence for the efficacy of early exiting in accelerating LLM inference, EE-LLM makes a foundational step towards scaling up early-exit LLMs by supporting their training and inference with massive 3D parallelism. Built upon Megatron-LM, EE-LLM implements a variety of algorithmic innovations and performance optimizations tailored to early exiting, including a lightweight method that facilitates backpropagation for the early-exit training objective with pipeline parallelism, techniques of leveraging idle resources in the original pipeline schedule for computation related to early-exit layers, and two approaches of early-exit inference that are compatible with KV caching for autoregressive generation. Our analytical and empirical study shows that EE-LLM achieves great training efficiency with negligible computational overhead compared to standard LLM training, as well as outstanding inference speedup without compromising output quality. To facilitate further research and adoption, we release EE-LLM at https://github.com/pan-x-c/EE-LLM.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 73 of its 101 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

73 of 101 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models15 Nov
  2. 2023SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference16 Mar
  3. 2022Confident Adaptive Language Modeling14 Jul
  4. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions9 Jun
  5. 2021Accelerating BERT Inference for Sequence Labeling via Early-Exit28 May

Other references · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2016Layer Normalizationcited by 5,579
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2014Going Deeper with Convolutionscited by 4,231
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  13. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  14. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  15. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  16. 2020Efficient Transformers: A Surveycited by 784
  17. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  18. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  19. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  20. 2022Efficiently Scaling Transformer Inferencecited by 529
  21. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  22. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  23. 2021Dynamic Neural Networks: A Surveycited by 405
  24. 2014Deeply-Supervised Netscited by 364
  25. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  26. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  27. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  28. 2016Using the Output Embedding to Improve Language Modelscited by 266
  29. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  30. 2019Depth-Adaptive Transformercited by 227
  31. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  32. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  33. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  34. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  35. 2023Speculative Decoding with Big Little Decodercited by 163
  36. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  37. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  38. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 29; 2026: 24.

69 papers.

Sort:

By area:

  1. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  3. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  4. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  5. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 96
  6. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  7. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 56
  8. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  9. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  10. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  11. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 14
  12. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Modelscited by 13
38 more
  1. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM modelscited by 12
  2. 2025AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inferencecited by 12
  3. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMscited by 12
  4. 2025From Prompts to Power: Measuring the Energy Footprint of LLM Inferencecited by 12
  5. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 12
  6. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  7. 2025The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exitcited by 10
  8. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 9
  9. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Modelscited by 9
  10. 2026AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depthcited by 9
  11. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  12. 2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exitscited by 8
  13. 2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inferencecited by 8
  14. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspectivecited by 8
  15. 2024Skipping Computations in Multimodal LLMscited by 7
  16. 2026ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skippingcited by 6
  17. 2024A Comprehensive Survey of Accelerated Generation Techniques in Large Language Modelscited by 5
  18. 2025Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inferencecited by 5
  19. 2025Zero Token-Driven Deep Thinking in LLMs: Unlocking the Full Potential of Existing Parameters via Cyclic Refinementcited by 5
  20. 2025FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharingcited by 4
  21. 2025SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decodingcited by 4
  22. 2026Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascadecited by 4
  23. 2026Depth-adaptive Inference of Looped Language Models via Continuous Depth Batchingcited by 4
  24. 2024PrisonBreak: Jailbreaking Large Language Models with at Most Twenty-Five Targeted Bit-flipscited by 3
  25. 2025DYNAMAX: Dynamic computing for Transformers and Mamba based architecturescited by 3
  26. 2025Energy-Aware Deep Learning on Resource-Constrained Hardwarecited by 3
  27. 2025EvoP: Robust LLM Inference via Evolutionary Pruningcited by 2
  28. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  29. 2025Dynamic Rebatching for Efficient Early-Exit Inference with DREXcited by 2
  30. 2026Ruyi2 Technical Reportcited by 2
  31. 2026The Detection-Extraction Gap: Models Know the Answer Before They Can Say Itcited by 2
  32. 2026Gated Recurrent Transformers: Expressive Depth through Recurrent Modulationcited by 2
  33. 2024MoDification: Mixture of Depths Made Easycited by 1
  34. 2025M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inferencecited by 1
  35. 2025Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decodingcited by 1
  36. 2026MAXS: Meta-Adaptive Exploration with LLM Agentscited by 1
  37. 2026SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planningcited by 1
  38. 2026The Diminishing Returns of Early-Exit Decoding in Modern LLMscited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024An Efficient Inference Framework for Early-exit Large Language Models
  2. 2025Leveraging the true depth of LLMs
  3. 2023Efficient LLM Inference on CPUs
  4. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  5. 2024A Survey on Efficient Inference for Large Language Models
  6. 2025Seesaw: High-throughput LLM Inference via Model Re-sharding
  7. 2024Accelerating Large Language Model Inference with Self-Supervised Early Exits
  8. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective