# Semi-supervised Dense Keypoints Using Unlabeled Multiview Images

arXiv: https://arxiv.org/abs/2109.09299 · HTML: https://mlsanity.com/paper/2109.09299/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 60 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Convolutional Mesh Regression for Single-Image Human Shape Reconstruction](https://mlsanity.com/paper/1905.03244/) (2019)
- [Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies](https://mlsanity.com/paper/1801.01615/) (2018)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Panoptic Studio: A Massively Multiview System for Social Interaction Capture](https://mlsanity.com/paper/1612.03153/) (2016)
- [Tex2Shape: Detailed Full Human Body Geometry From a Single Image](https://mlsanity.com/paper/1904.08645/) (2019)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [Epipolar Transformers](https://mlsanity.com/paper/2005.04551/) (2020)
- [DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare](https://mlsanity.com/paper/1910.00116/) (2019)
- [Self-Supervised Learning of 3D Human Pose using Multi-view Geometry](https://mlsanity.com/paper/1903.02330/) (2019)
- [Learning Dense Correspondence via 3D-guided Cycle Consistency](https://mlsanity.com/paper/1604.05383/) (2016)
- [Cross View Fusion for 3D Human Pose Estimation](https://mlsanity.com/paper/1909.01203/) (2019)
- [Dense Pose Transfer](https://mlsanity.com/paper/1809.01995/) (2018)
- [Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation](https://mlsanity.com/paper/1804.01110/) (2018)
- [Learning Monocular 3D Human Pose Estimation from Multi-view Images](https://mlsanity.com/paper/1803.04775/) (2018)
- [Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild](https://mlsanity.com/paper/2003.07581/) (2020)
- [3D Human Mesh Regression with Dense Correspondence](https://mlsanity.com/paper/2006.05734/) (2020)
- [Textured Neural Avatars](https://mlsanity.com/paper/1905.08776/) (2019)
- [Unsupervised learning of object landmarks by factorized spatial embeddings](https://mlsanity.com/paper/1705.02193/) (2017)
- [Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation](https://mlsanity.com/paper/2004.02186/) (2020)
- [TexturePose: Supervising Human Mesh Estimation with Texture Consistency](https://mlsanity.com/paper/1910.11322/) (2019)
- [Learning 3D Human Shape and Pose from Dense Body Parts](https://mlsanity.com/paper/1912.13344/) (2019)
- [View-Invariant Probabilistic Embedding for Human Pose](https://mlsanity.com/paper/1912.01001/) (2019)
- [AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild](https://mlsanity.com/paper/2010.13302/) (2020)
- [Unsupervised Learning of Landmarks by Descriptor Vector Exchange](https://mlsanity.com/paper/1908.06427/) (2019)
- [Delving Deep Into Hybrid Annotations for 3D Human Recovery in the Wild](https://mlsanity.com/paper/1908.06442/) (2019)

## Built on by (most cited first, 1 of 1)

- [Normal-guided Garment UV Prediction for Human Re-texturing](https://mlsanity.com/paper/2303.06504/) (2023)

## Abstract (arXiv)

This paper presents a new end-to-end semi-supervised framework to learn a dense keypoint detector using unlabeled multiview images. A key challenge lies in finding the exact correspondences between the dense keypoints in multiple views since the inverse of the keypoint mapping can be neither analytically derived nor differentiated. This limits applying existing multiview supervision approaches used to learn sparse keypoints that rely on the exact correspondences. To address this challenge, we derive a new probabilistic epipolar constraint that encodes the two desired properties. (1) Soft correspondence: we define a matchability, which measures a likelihood of a point matching to the other image's corresponding point, thus relaxing the requirement of the exact correspondences. (2) Geometric consistency: every point in the continuous correspondence fields must satisfy the multiview consistency collectively. We formulate a probabilistic epipolar constraint using a weighted average of epipolar errors through the matchability thereby generalizing the point-to-point geometric error to the field-to-field geometric error. This generalization facilitates learning a geometrically coherent dense keypoint detection model by utilizing a large number of unlabeled multiview images. Additionally, to prevent degenerative cases, we employ a distillation-based regularization by using a pretrained model. Finally, we design a new neural network architecture, made of twin networks, that effectively minimizes the probabilistic epipolar errors of all possible correspondences between two view images by building affinity matrices. Our method shows superior performance compared to existing methods, including non-differentiable bootstrapping in terms of keypoint accuracy, multiview consistency, and 3D reconstruction accuracy.