# Capsules with Inverted Dot-Product Attention Routing

arXiv: https://arxiv.org/abs/2002.04764 · HTML: https://mlsanity.com/paper/2002.04764/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 25 of 25)

- [iDisc: Internal Discretization for Monocular Depth Estimation](https://mlsanity.com/paper/2304.06334/) (2023)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Efficient-CapsNet: Capsule Network with Self-Attention Routing](https://mlsanity.com/paper/2101.12491/) (2021)
- [Visual Parser: Representing Part-whole Hierarchies with Transformers](https://mlsanity.com/paper/2107.05790/) (2021)
- [Hybrid Routing Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2203.15310/) (2022)
- [Slot State Space Models](https://mlsanity.com/paper/2406.12272/) (2024)
- [Effective and Efficient Vote Attack on Capsule Networks](https://mlsanity.com/paper/2102.10055/) (2021)
- [Capsule Network is Not More Robust than Convolutional Network](https://mlsanity.com/paper/2103.15459/) (2021)
- [Learning with Capsules: A Survey](https://mlsanity.com/paper/2206.02664/) (2022)
- [Neural Language of Thought Models](https://mlsanity.com/paper/2402.01203/) (2024)
- [An Algorithm for Routing Vectors in Sequences](https://mlsanity.com/paper/2211.11754/) (2022)
- [Spiking CapsNet: A Spiking Neural Network With A Biologically Plausible Routing Rule Between Capsules](https://mlsanity.com/paper/2111.07785/) (2021)
- [Why Capsule Neural Networks Do Not Scale: Challenging the Dynamic Parse-Tree Assumption](https://mlsanity.com/paper/2301.01583/) (2023)
- [OrthCaps: An Orthogonal CapsNet with Sparse Attention Routing and Pruning](https://mlsanity.com/paper/2403.13351/) (2024)
- [Training Deep Capsule Networks with Residual Connections](https://mlsanity.com/paper/2104.07393/) (2021)
- [Slot-BERT: Self-supervised Object Discovery in Surgical Video](https://mlsanity.com/paper/2501.12477/) (2025)
- [Deformable Capsules for Object Detection](https://mlsanity.com/paper/2104.05031/) (2021)
- [Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition](https://mlsanity.com/paper/2110.04954/) (2021)
- [Hybrid Gromov-Wasserstein Embedding for Capsule Learning](https://mlsanity.com/paper/2209.00232/) (2022)
- [WideCaps: A Wide Attention based Capsule Network for Image Classification](https://mlsanity.com/paper/2108.03627/) (2021)
- [Capsule networks with non-iterative cluster routing](https://mlsanity.com/paper/2109.09213/) (2021)
- [Reconstruction-guided attention improves the robustness and shape processing of neural networks](https://mlsanity.com/paper/2209.13620/) (2022)
- [Affordance detection with Dynamic-Tree Capsule Networks](https://mlsanity.com/paper/2211.05200/) (2022)
- [Explainability and Robustness of Deep Visual Classification Models](https://mlsanity.com/paper/2301.01343/) (2023)
- [MSPCaps: A Multi-Scale Patchify Capsule Network with Cross-Agreement Routing for Visual Recognition](https://mlsanity.com/paper/2508.16922/) (2025)

## Abstract (arXiv)

We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote. The new mechanism 1) designs routing via inverted dot-product attention; 2) imposes Layer Normalization as normalization; and 3) replaces sequential iterative routing with concurrent iterative routing. When compared to previously proposed routing algorithms, our method improves performance on benchmark datasets such as CIFAR-10 and CIFAR-100, and it performs at-par with a powerful CNN (ResNet-18) with 4x fewer parameters. On a different task of recognizing digits from overlayed digit images, the proposed capsule model performs favorably against CNNs given the same number of layers and neurons per layer. We believe that our work raises the possibility of applying capsule networks to complex real-world tasks. Our code is publicly available at: https://github.com/apple/ml-capsules-inverted-attention-routing An alternative implementation is available at: https://github.com/yaohungt/Capsules-Inverted-Attention-Routing/blob/master/README.md