# Dynamic Capacity Networks

arXiv: https://arxiv.org/abs/1511.07838 · HTML: https://mlsanity.com/paper/1511.07838/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2015.

## Built on by (most cited first, 16 of 16)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction](https://mlsanity.com/paper/2410.06245/) (2024)
- [LSEnet: Lorentz Structural Entropy Neural Network for Deep Graph Clustering](https://mlsanity.com/paper/2405.11801/) (2024)
- [Unleashing the Potential of Spiking Neural Networks by Dynamic Confidence](https://mlsanity.com/paper/2303.10276/) (2023)
- [Spatial Mixture-of-Experts](https://mlsanity.com/paper/2211.13491/) (2022)
- [Improving Video Instance Segmentation via Temporal Pyramid Routing](https://mlsanity.com/paper/2107.13155/) (2021)
- [Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning](https://mlsanity.com/paper/2209.14624/) (2022)
- [Fast Feedforward Networks](https://mlsanity.com/paper/2308.14711/) (2023)
- [Low-Rank Interconnected Adaptation across Layers](https://mlsanity.com/paper/2407.09946/) (2024)
- [Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation](https://mlsanity.com/paper/2212.07571/) (2022)
- [Dynamics-aware Adversarial Attack of Adaptive Neural Networks](https://mlsanity.com/paper/2210.08159/) (2022)
- [Measuring Sentiment Bias in Machine Translation](https://mlsanity.com/paper/2306.07152/) (2023)
- [ASIL: Augmented Structural Information Learning for Deep Graph Clustering in Hyperbolic Space](https://mlsanity.com/paper/2504.09970/) (2025)
- [Resource Efficient Neural Networks Using Hessian Based Pruning](https://mlsanity.com/paper/2306.07030/) (2023)
- [Learning Semantic-Robust Change Detection via Semantic-Invariant Self-Distillation](https://mlsanity.com/paper/2607.19000/) (2026)

## Abstract (arXiv)

We introduce the Dynamic Capacity Network (DCN), a neural network that can adaptively assign its capacity across different portions of the input data. This is achieved by combining modules of two types: low-capacity sub-networks and high-capacity sub-networks. The low-capacity sub-networks are applied across most of the input, but also provide a guide to select a few portions of the input on which to apply the high-capacity sub-networks. The selection is made using a novel gradient-based attention mechanism, that efficiently identifies input regions for which the DCN's output is most sensitive and to which we should devote more capacity. We focus our empirical evaluation on the Cluttered MNIST and SVHN image datasets. Our findings indicate that DCNs are able to drastically reduce the number of computations, compared to traditional convolutional neural networks, while maintaining similar or even better performance.