mlsanityData to 7 Oct 2026

A Survey on Dynamic Neural Networks for Natural Language Processing

arXiv 2202.07101 ↗First posted 15 Feb 2022cs.CL cs.AI cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Effectively scaling large Transformer models is a main driver of recent advances in natural language processing. Dynamic neural networks, as an emerging research direction, are capable of scaling up neural networks with sub-linear increases in computation and time by dynamically adjusting their computational path based on the input. Dynamic neural networks could be a promising solution to the growing parameter numbers of pretrained language models, allowing both model pretraining with trillions of parameters and faster inference on mobile devices. In this survey, we summarize progress of three types of dynamic neural networks in NLP: skimming, mixture of experts, and early exit. We also highlight current challenges in dynamic neural networks and directions for future research.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 54 of its 80 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

54 of 80 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
38 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  2. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  3. 2014Distributed Representations of Sentences and Documentscited by 677
  4. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  6. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  7. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  8. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  9. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  10. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  11. 2021Hash Layers For Large Sparse Modelscited by 236
  12. 2019Depth-Adaptive Transformercited by 227
  13. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  14. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  15. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  16. 2021Learned Token Pruning for Transformerscited by 138
  17. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  18. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  19. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  20. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
  21. 2021Scaling Laws for Neural Machine Translationcited by 110
  22. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  23. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  24. 2021M6: A Chinese Multimodal Pretrainercited by 91
  25. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  26. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  27. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  28. 2017Skip RNN: Learning to Skip State Updates in Recurrent Neural Networkscited by 49
  29. 2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
  30. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  31. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  32. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  33. 2021Accelerating BERT Inference for Sequence Labeling via Early-Exitcited by 29
  34. 2016ReasoNet: Learning to Stop Reading in Machine Comprehensioncited by 21
  35. 2021RomeBERT: Robust Training of Multi-Exit BERTcited by 19
  36. 2021Early Exiting with Ensemble Internal Classifierscited by 19
  37. 2017Learning to Skim Textcited by 18
  38. 2019Neural Speed Reading with Structural-Jump-LSTMcited by 17

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 8; 2025: 3; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  3. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  4. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  5. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  6. 2022Generative Knowledge Graph Construction: A Reviewcited by 49
  7. 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
  8. 2023Sparse Modular Activation for Efficient Sequence Modelingcited by 20
  9. 2023F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Taskscited by 18
  10. 2023Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicitycited by 16
  11. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Modelscited by 13
  12. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Modelscited by 11
12 more
  1. 2024NACHOS: Neural Architecture Search for Hardware Constrained Early Exit Neural Networkscited by 9
  2. 2025On-device edge learning for IoT data streams: a surveycited by 8
  3. 2023Early-Exit Neural Networks with Nested Prediction Setscited by 6
  4. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  5. 2023A comprehensive review of automatic text summarization techniques: method, data, evaluation and codingcited by 4
  6. 2024Tiny Models are the Computational Saver for Large Modelscited by 4
  7. 2024Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exitingcited by 2
  8. 2023Simple is Better and Large is Not Enough: Towards Ensembling of Foundational Language Modelscited by 1
  9. 2024Rule Based Learning with Dynamic (Graph) Neural Networkscited by 0
  10. 2025Neuroplasticity-inspired dynamic ANNs for multi-task demand forecastingcited by 0
  11. 2026MME: Mixture of Mesh Experts with Random Walk Transformer Gatingcited by 0
  12. 2026Rethinking Neural Width for Alternating Current Optimal Power Flow Proxiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dynamic Short Convolutions Improve Transformers
  2. 2024Dynamic layer selection in decoder-only transformers
  3. 2024A Survey of Transformer-based Language Models with Focus on Efficiency
  4. 2026Natural Language Processing: A Comprehensive Practical Guide from Tokenisation to RLHF
  5. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
  6. 2024A critical review of methods and challenges in large language models
  7. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  8. 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective