# Mitigating Vanishing Activations in Deep CapsNets Using Channel Pruning

arXiv: https://arxiv.org/abs/2410.16908 · HTML: https://mlsanity.com/paper/2410.16908/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 20 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Builds on (most cited first, 10 of 10)

- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [DeepCaps: Going Deeper with Capsule Networks](https://mlsanity.com/paper/1904.09546/) (2019)
- [Capsule Routing via Variational Bayes](https://mlsanity.com/paper/1905.11455/) (2019)
- [Capsules for Object Segmentation](https://mlsanity.com/paper/1804.04241/) (2018)
- [Why Capsule Neural Networks Do Not Scale: Challenging the Dynamic Parse-Tree Assumption](https://mlsanity.com/paper/2301.01583/) (2023)
- [Vanishing Activations: A Symptom of Deep Capsule Networks](https://mlsanity.com/paper/2305.11178/) (2023)
- [Filter Pruning For CNN With Enhanced Linear Representation Redundancy](https://mlsanity.com/paper/2310.06344/) (2023)

## Abstract (arXiv)

Capsule Networks outperform Convolutional Neural Networks in learning the part-whole relationships with viewpoint invariance, and the credit goes to their multidimensional capsules. It was assumed that increasing the number of capsule layers in the capsule networks would enhance the model performance. However, recent studies found that Capsule Networks lack scalability due to vanishing activations in the capsules of deeper layers. This paper thoroughly investigates the vanishing activation problem in deep Capsule Networks. To analyze this issue and understand how increasing capsule dimensions can facilitate deeper networks, various Capsule Network models are constructed and evaluated with different numbers of capsules, capsule dimensions, and intermediate layers for this paper. Unlike traditional model pruning, which reduces the number of model parameters and expedites model training, this study uses pruning to mitigate the vanishing activations in the deeper capsule layers. In addition, the backbone network and capsule layers are pruned with different pruning ratios to reduce the number of inactive capsules and achieve better model accuracy than the unpruned models.