mlsanityData to 7 Oct 2026

Learning to Skip for Language Modeling

arXiv 2311.15436 ↗First posted 26 Nov 2023cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Overparameterized large-scale language models have impressive generalization performance of in-context few-shot learning. However, most language models allocate the same amount of parameters or computation to each token, disregarding the complexity or importance of the input data. We argue that in language model pretraining, a variable amount of computation should be assigned to different tokens, and this can be efficiently achieved via a simple routing mechanism. Different from conventional early stopping techniques where tokens can early exit at only early layers, we propose a more general method that dynamically skips the execution of a layer (or module) for any input token with a binary router. In our extensive evaluation across 24 NLP tasks, we demonstrate that the proposed method can significantly improve the 1-shot performance compared to other competitive baselines only at mild extra cost for inference.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 35 of its 41 references that are in the map. First posted on arXiv 26 Nov 2023.

Builds on

35 of 41 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  12. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
23 more
  1. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  3. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  4. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  5. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  6. 2022Confident Adaptive Language Modelingcited by 297
  7. 2018DropBlock: A regularization method for convolutional networkscited by 261
  8. 2015Highway Networkscited by 257
  9. 2021Hash Layers For Large Sparse Modelscited by 236
  10. 2019Depth-Adaptive Transformercited by 227
  11. 2014Efficient Object Localization Using Convolutional Networkscited by 226
  12. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  13. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  14. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  15. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  16. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  17. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  18. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  19. 2017IDK Cascades: Fast Deep Learning by Learning not to Overthinkcited by 47
  20. 2018Batch DropBlock Network for Person Re-identification and Beyondcited by 45
  21. 2020Controlling Computation versus Quality for Neural Sequence Modelscited by 27
  22. 2022The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimescited by 6
  23. 2020Balancing Cost and Benefit with Tied-Multi Transformerscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 8; 2026: 4.

19 papers.

Sort:

By area:

  1. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  2. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  3. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
  4. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  5. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  7. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  8. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  9. 2026BAGEN: Are LLM Agents Budget-Aware?cited by 12
  10. 2024Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modulescited by 11
  11. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  12. 2025Learning to Skip the Middle Layers of Transformerscited by 8
7 more
  1. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  2. 2024Dynamic layer selection in decoder-only transformerscited by 3
  3. 2025Position-Aware Depth Decay Decoding (D3): Boosting Large Language Model Inference Efficiencycited by 2
  4. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  5. 2026SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibrationcited by 1
  6. 2026End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inferencecited by 0
  7. 2026From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Skipping Computations in Multimodal LLMs
  2. 2023Think before you speak: Training Language Models With Pause Tokens
  3. 2026VALSE: Vertical Adaptive Layer Skipping for Efficient Inference in Large Language Models
  4. 2026Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
  5. 2026Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference
  6. 2024Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review
  7. 2024Many-Shot In-Context Learning
  8. 2025MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining