mlsanityData to 7 Oct 2026

A Theoretical View on Sparsely Activated Networks

arXiv 2208.04461 ↗First posted 8 Aug 2022cs.LG stat.MLcited by 6

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Deep and wide neural networks successfully fit very complex functions today, but dense models are starting to be prohibitively expensive for inference. To mitigate this, one promising direction is networks that activate a sparse subgraph of the network. The subgraph is chosen by a data-dependent routing function, enforcing a fixed mapping of inputs to subnetworks (e.g., the Mixture of Experts (MoE) paradigm in Switch Transformers). However, prior work is largely empirical, and while existing routing functions work well in practice, they do not lead to theoretical guarantees on approximation ability. We aim to provide a theoretical explanation for the power of sparse networks. As our first contribution, we present a formal model of data-dependent sparse networks that captures salient aspects of popular architectures. We then introduce a routing function based on locality sensitive hashing (LSH) that enables us to reason about how well sparse networks approximate target functions. After representing LSH-based sparse networks with our model, we prove that sparse networks can match the approximation power of dense networks on Lipschitz functions. Applying LSH on the input vectors means that the experts interpolate the target function in different subregions of the input space. To support our theory, we define various datasets based on Lipschitz target functions, and we show that sparse networks give a favorable trade-off between number of active units and approximation quality.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 25 of its 46 references that are in the map. First posted on arXiv 8 Aug 2022.

Builds on

25 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2021Transformers in Vision: A Surveycited by 1,009
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  8. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  9. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  10. 2022Transformers in Time Series: A Surveycited by 489
  11. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  12. 2021Hash Layers For Large Sparse Modelscited by 236
12 more
  1. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  2. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  3. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  4. 2015Practical and Optimal LSH for Angular Distancecited by 105
  5. 2021Sparse is Enough in Scaling Transformerscited by 86
  6. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  7. 2018Approximate Nearest Neighbor Search in High Dimensionscited by 45
  8. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  9. 2016Scaling Memory-Augmented Neural Networks with Sparse Reads and Writescited by 28
  10. 2020FastFormers: Highly Efficient Transformer Models for Natural Language Understandingcited by 28
  11. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  12. 2019Recursive Sketches for Modular Deep Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 0; 2025: 1; 2026: 1.

6 papers.

  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  2. 2023Alternating Updates for Efficient Transformerscited by 13
  3. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Taskscited by 4
  4. 2026Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networkscited by 1
  5. 2023The Power of External Memory in Increasing Predictive Model Capacitycited by 0
  6. 2023Manifold-Preserving Transformers are Effective for Short-Long Range Encodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute
  2. 2026When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
  3. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routing
  4. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  5. 2026Generalization and Scaling Laws for Mixture-of-Experts Transformers
  6. 2026Expert Routing for Communication-Efficient MoE via Finite Expert Banks
  7. 2026How Sparse Probability Maps Shape Mixture-of-Experts Routing
  8. 2026Towards a Statistical Understanding of Mixture-of-Experts