mlsanityData to 7 Oct 2026

DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs

arXiv 2407.11030 ↗First posted 3 Jul 2024cs.LG cs.AI cs.CLcited by 9

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

In this paper, we introduce Dynamic Layer Operations (DLO), a novel approach for vertically scaling transformer-based Large Language Models (LLMs) by dynamically expanding, activating, or skipping layers using a sophisticated routing policy based on layerwise feature similarity. Unlike traditional Mixture-of-Experts (MoE) methods that focus on extending the model width, our approach targets model depth, addressing the redundancy observed across layer representations for various input samples. Our framework is integrated with the Supervised Fine-Tuning (SFT) stage, eliminating the need for resource-intensive Continual Pre-Training (CPT). Experimental results demonstrate that DLO not only outperforms the original unscaled models but also achieves comparable results to densely expanded models with significantly improved efficiency. Our work offers a promising direction for building efficient yet powerful LLMs. We will release our implementation and model weights upon acceptance.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 41 of its 64 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

41 of 64 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training24 Jun
  2. 2024Mixtral of Experts8 Jan
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 36

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
24 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  4. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  5. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  6. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  7. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  8. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  9. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  10. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  11. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  12. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  13. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  14. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  15. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  16. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
  17. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
  18. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49
  19. 2022Staged Training for Transformer Language Modelscited by 49
  20. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 42
  21. 2018Recurrent Stacking of Layers for Compact Neural Machine Translation Modelscited by 38
  22. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 32
  23. 2024Facial Affective Behavior Analysis with Instruction Tuningcited by 26
  24. 2023LEMON: Lossless model expansioncited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 2.

9 papers.

  1. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  2. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 13
  3. 2024Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformerscited by 8
  4. 2025Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Trainingcited by 2
  5. 2025IG-Pruning: Input-Guided Block Pruning for Large Language Modelscited by 2
  6. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  7. 2025MIDUS: Memory-Infused Depth Up-Scalingcited by 0
  8. 2026End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inferencecited by 0
  9. 2026Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DND: Boosting Large Language Models with Dynamic Nested Depth
  2. 2024Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models
  3. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  4. 2025LLM Inference Acceleration via Efficient Operation Fusion
  5. 2025Leveraging the true depth of LLMs
  6. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
  7. 2024Streamlining Redundant Layers to Compress Large Language Models
  8. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA