# VideoCapsuleNet: A Simplified Network for Action Detection

arXiv: https://arxiv.org/abs/1805.08162 · HTML: https://mlsanity.com/paper/1805.08162/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2018.

## Built on by (most cited first, 30 of 30)

- [Efficient-CapsNet: Capsule Network with Self-Attention Routing](https://mlsanity.com/paper/2101.12491/) (2021)
- [Large-scale Robustness Analysis of Video Action Recognition Models](https://mlsanity.com/paper/2207.01398/) (2022)
- [Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules](https://mlsanity.com/paper/2105.04836/) (2021)
- [End-to-End Semi-Supervised Learning for Video Action Detection](https://mlsanity.com/paper/2203.04251/) (2022)
- [Unsupervised Motion Representation Learning with Capsule Autoencoders](https://mlsanity.com/paper/2110.00529/) (2021)
- [Video Action Detection: Analysing Limitations and Challenges](https://mlsanity.com/paper/2204.07892/) (2022)
- [Learning with Capsules: A Survey](https://mlsanity.com/paper/2206.02664/) (2022)
- [Semi-supervised Active Learning for Video Action Detection](https://mlsanity.com/paper/2312.07169/) (2023)
- [Weakly Supervised Grounding for VQA in Vision-Language Transformers](https://mlsanity.com/paper/2207.02334/) (2022)
- [YOWOv2: A Stronger yet Efficient Multi-level Detection Framework for Real-time Spatio-temporal Action Detection](https://mlsanity.com/paper/2302.06848/) (2023)
- [On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes](https://mlsanity.com/paper/2410.19553/) (2024)
- [Towards Active Vision for Action Localization with Reactive Control and Predictive Learning](https://mlsanity.com/paper/2111.05448/) (2021)
- [Routing with Self-Attention for Multimodal Capsule Networks](https://mlsanity.com/paper/2112.00775/) (2021)
- [A Survey on Deep Learning-based Spatio-temporal Action Detection](https://mlsanity.com/paper/2308.01618/) (2023)
- [Deformable Capsules for Object Detection](https://mlsanity.com/paper/2104.05031/) (2021)
- [OmViD: Omni-supervised active learning for video action detection](https://mlsanity.com/paper/2508.13983/) (2025)
- [Exploiting Instance-based Mixed Sampling via Auxiliary Source Domain Supervision for Domain-adaptive Action Detection](https://mlsanity.com/paper/2209.15439/) (2022)
- [Point3D: tracking actions as moving points with 3D CNNs](https://mlsanity.com/paper/2203.10584/) (2022)
- [Spatial-Temporal Alignment Network for Action Recognition](https://mlsanity.com/paper/2308.09897/) (2023)
- [Self-Supervised Contrastive Learning for Videos using Differentiable Local Alignment](https://mlsanity.com/paper/2409.04607/) (2024)
- [Punching Bag vs. Punching Person: Motion Transferability in Videos](https://mlsanity.com/paper/2508.00085/) (2025)
- [Single Run Action Detector over Video Stream -- A Privacy Preserving Approach](https://mlsanity.com/paper/2102.03391/) (2021)
- [Capsule networks with non-iterative cluster routing](https://mlsanity.com/paper/2109.09213/) (2021)
- ["Knights": First Place Submission for VIPriors21 Action Recognition Challenge at ICCV 2021](https://mlsanity.com/paper/2110.07758/) (2021)
- [Effectiveness of the Recent Advances in Capsule Networks](https://mlsanity.com/paper/2210.05834/) (2022)
- [Affordance detection with Dynamic-Tree Capsule Networks](https://mlsanity.com/paper/2211.05200/) (2022)
- [CapsFlow: Optical Flow Estimation with Capsule Networks](https://mlsanity.com/paper/2304.00306/) (2023)
- [Capsule network with shortcut routing](https://mlsanity.com/paper/2307.10212/) (2023)
- [EquiCaps: Predictor-Free Pose-Aware Pre-Trained Capsule Networks](https://mlsanity.com/paper/2506.09895/) (2025)
- [Multiple Additive Neural Networks for Structured and Unstructured Data](https://mlsanity.com/paper/2604.26888/) (2026)

## Abstract (arXiv)

The recent advances in Deep Convolutional Neural Networks (DCNNs) have shown extremely good results for video human action classification, however, action detection is still a challenging problem. The current action detection approaches follow a complex pipeline which involves multiple tasks such as tube proposals, optical flow, and tube classification. In this work, we present a more elegant solution for action detection based on the recently developed capsule network. We propose a 3D capsule network for videos, called VideoCapsuleNet: a unified network for action detection which can jointly perform pixel-wise action segmentation along with action classification. The proposed network is a generalization of capsule network from 2D to 3D, which takes a sequence of video frames as input. The 3D generalization drastically increases the number of capsules in the network, making capsule routing computationally expensive. We introduce capsule-pooling in the convolutional capsule layer to address this issue which makes the voting algorithm tractable. The routing-by-agreement in the network inherently models the action representations and various action characteristics are captured by the predicted capsules. This inspired us to utilize the capsules for action localization and the class-specific capsules predicted by the network are used to determine a pixel-wise localization of actions. The localization is further improved by parameterized skip connections with the convolutional capsule layers and the network is trained end-to-end with a classification as well as localization loss. The proposed network achieves sate-of-the-art performance on multiple action detection datasets including UCF-Sports, J-HMDB, and UCF-101 (24 classes) with an impressive ~20% improvement on UCF-101 and ~15% improvement on J-HMDB in terms of v-mAP scores.