mlsanityData to 14 Oct 2026

Conditional Computation in Neural Networks for faster models

arXiv 1511.06297 ↗First posted 19 Nov 2015cs.LGcited by 167

Not on the map. Why?

Cited by 58 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep learning has become the state-of-art tool in many applications, but the evaluation and training of deep models can be time-consuming and computationally expensive. The conditional computation approach has been proposed to tackle this problem (Bengio et al., 2013; Davis & Arel, 2013). It operates by selectively activating only parts of the network at a time. In this paper, we use reinforcement learning as a tool to optimize conditional computation policies. More specifically, we cast the problem of learning activation-dependent policies for dropping out blocks of units as a reinforcement learning problem. We propose a learning scheme motivated by computation speed, capturing the idea of wanting to have parsimonious activations while maintaining prediction accuracy. We apply a policy gradient algorithm for learning policies that optimize this loss function and propose a regularization mechanism that encourages diversification of the dropout policy. We present encouraging empirical results showing that this approach improves the speed of computation without impacting the quality of the approximation.

Key facts. Cited by 167 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2017: 1; 2018: 1; 2019: 1; 2020: 2; 2021: 24; 2022: 27; 2023: 28; 2024: 28; 2025: 34; 2026: 21.

167 papers.

Sort:

By area:

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  4. 2021Dynamic Neural Networks: A Surveycited by 408
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  6. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  8. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  9. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  10. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  11. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  12. 2023From Sparse to Soft Mixtures of Expertscited by 248
38 more
  1. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  2. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  3. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 157
  4. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  5. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 133
  6. 2021A Survey on Deep Learning Technique for Video Segmentationcited by 132
  7. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  8. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  9. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  10. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  11. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  12. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  13. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  14. 2023Modular Deep Learningcited by 83
  15. 2024Mixture of A Million Expertscited by 72
  16. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  17. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 66
  18. 2026LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulationcited by 61
  19. 2023Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantizationcited by 60
  20. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 57
  21. 2022Dynamic Multimodal Fusioncited by 55
  22. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  23. 2021Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitivescited by 50
  24. 2021AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognitioncited by 48
  25. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  26. 2021Shapley Explanation Networkscited by 41
  27. 2021Zero Time Waste: Recycling Predictions in Early Exit Neural Networkscited by 39
  28. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 38
  29. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  30. 2021Dynamic Network Quantization for Efficient Video Inferencecited by 36
  31. 2021Skip-Convolutions for Efficient Video Processingcited by 32
  32. 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
  33. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 30
  34. 2025Visual Instance-aware Prompt Tuningcited by 29
  35. 2024MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detectioncited by 27
  36. 2021Contextual Dropout: An Efficient Sample-Dependent Dropout Modulecited by 26
  37. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  38. 2024Unraveling the Mystery of Scaling Laws: Part Icited by 24

50 most cited of 167 All 167 (JSON)