mlsanityData to 7 Oct 2026

Conditional computation in neural networks: principles and research trends

arXiv 2403.07965 ↗First posted 12 Mar 2024cs.LG cs.AIcited by 10

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

This article summarizes principles and ideas from the emerging area of applying textitconditional computation methods to the design of neural networks. In particular, we focus on neural networks that can dynamically activate or de-activate parts of their computational graph conditionally on their input. Examples include the dynamic selection of, e.g., input tokens, layers (or sets of layers), and sub-modules inside each layer (e.g., channels in a convolutional filter). We first provide a general formalism to describe these techniques in an uniform way. Then, we introduce three notable implementations of these principles: mixture-of-experts (MoEs) networks, token selection mechanisms, and early-exit neural networks. The paper aims to provide a tutorial-like introduction to this growing field. To this end, we analyze the benefits of these modular designs in terms of efficiency, explainability, and transfer learning, with a focus on emerging applicative areas ranging from automated scientific discovery to semantic communication.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 99 of its 119 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

99 of 119 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning11 Sep
  2. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs9 Jun
  3. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning7 Jun

Other references · 96

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2024Mixtral of Expertscited by 1,785
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  9. 2022Token Merging: Your ViT But Fastercited by 995
  10. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  11. 2023Vision Transformers Need Registerscited by 844
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
38 more
  1. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  2. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  4. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  5. 2021Dynamic Neural Networks: A Surveycited by 405
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  7. 2023Augmented Language Models: a Surveycited by 398
  8. 2015Neural Module Networkscited by 375
  9. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  10. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  11. 2022Confident Adaptive Language Modelingcited by 297
  12. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  13. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  14. 2023From Sparse to Soft Mixtures of Expertscited by 245
  15. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  16. 2023Token Merging for Fast Stable Diffusioncited by 220
  17. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  18. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  19. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  20. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  21. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  22. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  23. 2019Adaptively Sparse Transformerscited by 154
  24. 2019Modeling Point Clouds with Self-Attention and Gumbel Subset Samplingcited by 151
  25. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  26. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  27. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  28. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  29. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  30. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  31. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Mergingcited by 123
  32. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 114
  33. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  34. 2019Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inferencecited by 91
  35. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  36. 2021Sparse is Enough in Scaling Transformerscited by 86
  37. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  38. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84

50 most cited of 96 All 96 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 3.

10 papers.

  1. 2025Adaptive Semantic Token Communication for Transformer-based Edge Inferencecited by 13
  2. 2024Adaptive Semantic Token Selection for AI-native Goal-oriented Communicationscited by 10
  3. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  4. 2025Attention when you needcited by 2
  5. 2024AppSign: Multi-level Approximate Computing for Real-Time Traffic Sign Recognition in Autonomous Vehiclescited by 0
  6. 2025Ray-Tracing for Conditionally Activated Neural Networkscited by 0
  7. 2025PERTINENCE: Input-based Opportunistic Neural Network Dynamic Executioncited by 0
  8. 2026Motivation is Something You Needcited by 0
  9. 2026TIDE: Token-Informed Depth Execution for Per-Token Early Exit in LLM Inferencecited by 0
  10. 2026A Comparative Analysis on the Performance of Upper Confidence Bound Algorithms in Adaptive Deep Neural Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Expert Specialization in Mixture of Experts
  2. 2026DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
  3. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
  4. 2026Optimal Mixture-of-Experts Model Averaging for Conditional Generative Models
  5. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
  6. 2026∞-MoE: Generalizing Mixture of Experts to Infinite Experts
  7. 2023Mixture of Tokens: Continuous MoE through Cross-Example Aggregation
  8. 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design