mlsanityData to 7 Oct 2026

Dynamic Neural Networks: A Survey

arXiv 2102.04906 ↗First posted 9 Feb 2021cs.CVcited by 405

Not on the map. Why?

Cited by 44 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Dynamic neural network is an emerging research topic in deep learning. Compared to static models which have fixed computational graphs and parameters at the inference stage, dynamic networks can adapt their structures or parameters to different inputs, leading to notable advantages in terms of accuracy, computational efficiency, adaptiveness, etc. In this survey, we comprehensively review this rapidly developing area by dividing dynamic networks into three main categories: 1) instance-wise dynamic models that process each instance with data-dependent architectures or parameters; 2) spatial-wise dynamic networks that conduct adaptive computation with respect to different spatial locations of image data and 3) temporal-wise dynamic models that perform adaptive inference along the temporal dimension for sequential data such as videos and texts. The important research problems of dynamic networks, e.g., architecture design, decision making scheme, optimization technique and applications, are reviewed systematically. Finally, we discuss the open problems in this field together with interesting future research directions.

Key facts. Cited by 405 papers in the map (data through 7 Oct 2026). Builds on 187 of its 287 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

187 of 287 references.

References · 187

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2016Densely Connected Convolutional Networkscited by 6,691
  11. 2017On Calibration of Modern Neural Networkscited by 4,860
  12. 2017Squeeze-and-Excitation Networkscited by 4,763
38 more
  1. 2014Going Deeper with Convolutionscited by 4,231
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  8. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  13. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  14. 2017Deformable Convolutional Networkscited by 1,481
  15. 2015Spatial Transformer Networkscited by 1,423
  16. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  17. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  18. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  19. 2021Transformers in Vision: A Surveycited by 1,009
  20. 2016HyperNetworkscited by 1,003
  21. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  22. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  23. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  24. 2017Dynamic Routing Between Capsulescited by 689
  25. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  26. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 666
  27. 2018Universal Transformerscited by 613
  28. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  29. 2019ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networkscited by 572
  30. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  31. 2019PointRend: Image Segmentation as Renderingcited by 439
  32. 2019Selective Kernel Networkscited by 401
  33. 2017Residual Attention Network for Image Classificationcited by 385
  34. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  35. 2014Recurrent Models of Visual Attentioncited by 344
  36. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  37. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  38. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322

50 most cited of 187 All 187 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 34; 2022: 65; 2023: 104; 2024: 78; 2025: 68; 2026: 56.

405 papers.

Sort:

By area:

  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  3. 2021Domain Generalization: A Surveycited by 760
  4. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  5. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  6. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  7. 2023FLatten Transformer: Vision Transformer using Focused Linear Attentioncited by 170
  8. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  9. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 143
  10. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  11. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  12. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 95
38 more
  1. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  2. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  3. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  4. 2021Dynamic Slimmable Networkcited by 83
  5. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
  6. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  7. 2021Adaptive Focus for Efficient Video Recognitioncited by 76
  8. 2021Adaptive Inference through Early-Exit Networks: Design, Challenges and Directionscited by 74
  9. 2021Nonlinear Transform Source-Channel Coding for Semantic Communicationscited by 73
  10. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  11. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  12. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  13. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  14. 2024Dynamic Diffusion Transformercited by 60
  15. 2023Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantizationcited by 59
  16. 2023When Does Confidence-Based Cascade Deferral Suffice?cited by 58
  17. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  18. 2022Dynamic Multimodal Fusioncited by 54
  19. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53
  20. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  21. 2021A Survey on Green Deep Learningcited by 52
  22. 2023Adaptive Rotated Convolution for Rotated Object Detectioncited by 51
  23. 2023Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipelinecited by 50
  24. 2023Rank-DETR for High Quality Object Detectioncited by 49
  25. 2022Learning to Weight Samples for Dynamic Early-exiting Networkscited by 44
  26. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  27. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  28. 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 42
  29. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  30. 2022Glance and Focus Networks for Dynamic Visual Recognitioncited by 41
  31. 2023Mask Grounding for Referring Image Segmentationcited by 40
  32. 2023SEENN: Towards Temporal Spiking Early-Exit Neural Networkscited by 39
  33. 2023ADAPT: Efficient Multi-Agent Trajectory Prediction with Adaptationcited by 39
  34. 2021AdaFocus V2: End-to-End Training of Spatial Dynamic Networks for Video Recognitioncited by 37
  35. 2023HyperE2VID: Improving Event-Based Video Reconstruction via Hypernetworkscited by 37
  36. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  37. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  38. 2021CondenseNet V2: Sparse Feature Reactivation for Deep Networkscited by 34

50 most cited of 405 All 405 (JSON)