# Capsules as viewpoint learners for human pose estimation

arXiv: https://arxiv.org/abs/2302.06194 · HTML: https://mlsanity.com/paper/2302.06194/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 50 references that are in the map.
- First posted on arXiv 13 Feb 2023.

## Builds on (most cited first, 30 of 32)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Semantic Graph Convolutional Networks for 3D Human Pose Regression](https://mlsanity.com/paper/1904.03345/) (2019)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [Human Pose Estimation with Iterative Error Feedback](https://mlsanity.com/paper/1507.06550/) (2015)
- [Exploiting temporal context for 3D human pose estimation in the wild](https://mlsanity.com/paper/1905.04266/) (2019)
- [V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map](https://mlsanity.com/paper/1711.07399/) (2017)
- [LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images](https://mlsanity.com/paper/1803.00455/) (2018)
- [RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimation](https://mlsanity.com/paper/1902.09868/) (2019)
- [Stacked Capsule Autoencoders](https://mlsanity.com/paper/1906.06818/) (2019)
- [Monocular 3D Human Pose Estimation by Generation and Ordinal Ranking](https://mlsanity.com/paper/1904.01324/) (2019)
- [Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image](https://mlsanity.com/paper/1701.00295/) (2017)
- [XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera](https://mlsanity.com/paper/1907.00837/) (2019)
- [A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Image](https://mlsanity.com/paper/1908.09999/) (2019)
- [Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video](https://mlsanity.com/paper/1511.09439/) (2015)
- [3D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Training](https://mlsanity.com/paper/2004.11822/) (2020)
- [Capsule Routing via Variational Bayes](https://mlsanity.com/paper/1905.11455/) (2019)
- [Trajectory Space Factorization for Deep Video-Based 3D Human Pose Estimation](https://mlsanity.com/paper/1908.08289/) (2019)
- [Feature Boosting Network For 3D Pose Estimation](https://mlsanity.com/paper/1901.04877/) (2019)
- [DECA: Deep viewpoint-Equivariant human pose estimation using Capsule Autoencoders](https://mlsanity.com/paper/2108.08557/) (2021)
- [3D Human Pose Machines with Self-supervised Learning](https://mlsanity.com/paper/1901.03798/) (2019)
- [Towards Good Practices for Deep 3D Hand Pose Estimation](https://mlsanity.com/paper/1707.07248/) (2017)
- [Anatomy-aware 3D Human Pose Estimation with Bone-based Pose Decomposition](https://mlsanity.com/paper/2002.10322/) (2020)
- [PoseNet3D: Learning Temporally Consistent 3D Human Pose via Knowledge Distillation](https://mlsanity.com/paper/2003.03473/) (2020)

## Abstract (arXiv)

The task of human pose estimation (HPE) deals with the ill-posed problem of estimating the 3D position of human joints directly from images and videos. In recent literature, most of the works tackle the problem mostly by using convolutional neural networks (CNNs), which are capable of achieving state-of-the-art results in most datasets. We show how most neural networks are not able to generalize well when the camera is subject to significant viewpoint changes. This behaviour emerges because CNNs lack the capability of modelling viewpoint equivariance, while they rather rely on viewpoint invariance, resulting in high data dependency. Recently, capsule networks (CapsNets) have been proposed in the multi-class classification field as a solution to the viewpoint equivariance issue, reducing both the size and complexity of both the training datasets and the network itself. In this work, we show how capsule networks can be adopted to achieve viewpoint equivariance in human pose estimation. We propose a novel end-to-end viewpoint-equivariant capsule autoencoder that employs a fast Variational Bayes routing and matrix capsules. We achieve state-of-the-art results for multiple tasks and datasets while retaining other desirable properties, such as greater generalization capabilities when changing viewpoints, lower data dependency and fast inference. Additionally, by modelling each joint as a capsule, the hierarchical and geometrical structure of the overall pose is retained in the feature space, independently from the viewpoint. We further test our network on multiple datasets, both in the RGB and depth domain, from seen and unseen viewpoints and in the viewpoint transfer task.