# Why Capsule Neural Networks Do Not Scale: Challenging the Dynamic Parse-Tree Assumption

arXiv: https://arxiv.org/abs/2301.01583 · HTML: https://mlsanity.com/paper/2301.01583/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 4 Jan 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Stacked Capsule Autoencoders](https://mlsanity.com/paper/1906.06818/) (2019)
- [DeepCaps: Going Deeper with Capsule Networks](https://mlsanity.com/paper/1904.09546/) (2019)
- [Efficient-CapsNet: Capsule Network with Self-Attention Routing](https://mlsanity.com/paper/2101.12491/) (2021)
- [Capsule Routing via Variational Bayes](https://mlsanity.com/paper/1905.11455/) (2019)
- [Capsules with Inverted Dot-Product Attention Routing](https://mlsanity.com/paper/2002.04764/) (2020)
- [Unsupervised part representation by Flow Capsules](https://mlsanity.com/paper/2011.13920/) (2020)
- [Capsule Network Performance on Complex Data](https://mlsanity.com/paper/1712.03480/) (2017)
- [Improving the Robustness of Capsule Networks to Image Affine Transformations](https://mlsanity.com/paper/1911.07968/) (2019)
- [Effective and Efficient Vote Attack on Capsule Networks](https://mlsanity.com/paper/2102.10055/) (2021)
- [Interpretable Graph Capsule Networks for Object Recognition](https://mlsanity.com/paper/2012.01674/) (2020)
- [Capsule Network is Not More Robust than Convolutional Network](https://mlsanity.com/paper/2103.15459/) (2021)
- [On the Vulnerability of Capsule Networks to Adversarial Attacks](https://mlsanity.com/paper/1906.03612/) (2019)
- [Capsule Networks Need an Improved Routing Algorithm](https://mlsanity.com/paper/1907.13327/) (2019)
- [Sparse Unsupervised Capsules Generalize Better](https://mlsanity.com/paper/1804.06094/) (2018)

## Built on by (most cited first, 6 of 6)

- [ProtoCaps: A Fast and Non-Iterative Capsule Network Routing Method](https://mlsanity.com/paper/2307.09944/) (2023)
- [Vanishing Activations: A Symptom of Deep Capsule Networks](https://mlsanity.com/paper/2305.11178/) (2023)
- [Capsule Network Projectors are Equivariant and Invariant Learners](https://mlsanity.com/paper/2405.14386/) (2024)
- [Towards the Characterization of Representations Learned via Capsule-based Network Architectures](https://mlsanity.com/paper/2305.05349/) (2023)
- [Mitigating Vanishing Activations in Deep CapsNets Using Channel Pruning](https://mlsanity.com/paper/2410.16908/) (2024)
- [Equivariant Imaging Biomarkers for Robust Unsupervised Segmentation of Histopathology](https://mlsanity.com/paper/2505.05689/) (2025)

## Abstract (arXiv)

Capsule neural networks replace simple, scalar-valued neurons with vector-valued capsules. They are motivated by the pattern recognition system in the human brain, where complex objects are decomposed into a hierarchy of simpler object parts. Such a hierarchy is referred to as a parse-tree. Conceptually, capsule neural networks have been defined to realize such parse-trees. The capsule neural network (CapsNet), by Sabour, Frosst, and Hinton, is the first actual implementation of the conceptual idea of capsule neural networks. CapsNets achieved state-of-the-art performance on simple image recognition tasks with fewer parameters and greater robustness to affine transformations than comparable approaches. This sparked extensive follow-up research. However, despite major efforts, no work was able to scale the CapsNet architecture to more reasonable-sized datasets. Here, we provide a reason for this failure and argue that it is most likely not possible to scale CapsNets beyond toy examples. In particular, we show that the concept of a parse-tree, the main idea behind capsule neuronal networks, is not present in CapsNets. We also show theoretically and experimentally that CapsNets suffer from a vanishing gradient problem that results in the starvation of many capsules during training.