# Self-supervised Secondary Landmark Detection via 3D Representation Learning

arXiv: https://arxiv.org/abs/2110.00543 · HTML: https://mlsanity.com/paper/2110.00543/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 55 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Convolutional Pose Machines](https://mlsanity.com/paper/1602.00134/) (2016)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation](https://mlsanity.com/paper/1406.2984/) (2014)
- [Epipolar Transformers](https://mlsanity.com/paper/2005.04551/) (2020)
- [Self-Supervised Learning of 3D Human Pose using Multi-view Geometry](https://mlsanity.com/paper/1903.02330/) (2019)
- [PoseTrack: Joint Multi-Person Pose Estimation and Tracking](https://mlsanity.com/paper/1611.07727/) (2016)
- [Can 3D Pose be Learned from 2D Projections Alone?](https://mlsanity.com/paper/1808.07182/) (2018)
- [Deep learning tools for the measurement of animal behavior in neuroscience](https://mlsanity.com/paper/1909.13868/) (2019)
- [Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis](https://mlsanity.com/paper/2004.04400/) (2020)
- [Learning visual groups from co-occurrences in space and time](https://mlsanity.com/paper/1511.06811/) (2015)
- [Thin-Slicing Network: A Deep Structured Model for Pose Estimation in Videos](https://mlsanity.com/paper/1703.10898/) (2017)
- [RGBD-Dog: Predicting Canine Pose from RGBD Sensors](https://mlsanity.com/paper/2004.07788/) (2020)
- [Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animals](https://mlsanity.com/paper/2001.08601/) (2020)
- [Self-Supervised 3D Human Pose Estimation with Multiple-View Geometry](https://mlsanity.com/paper/2108.07777/) (2021)
- [Semi- and Weakly-supervised Human Pose Estimation](https://mlsanity.com/paper/1906.01399/) (2019)
- [PoseNet3D: Learning Temporally Consistent 3D Human Pose via Knowledge Distillation](https://mlsanity.com/paper/2003.03473/) (2020)

## Built on by (most cited first, 3 of 3)

- [SuperAnimal pretrained pose estimation models for behavioral analysis](https://mlsanity.com/paper/2203.07436/) (2022)
- [OpenApePose: a database of annotated ape photographs for pose estimation](https://mlsanity.com/paper/2212.00741/) (2022)
- [Multi-person 3D pose estimation from unlabelled data](https://mlsanity.com/paper/2212.08731/) (2022)

## Abstract (arXiv)

Recent technological developments have spurred great advances in the computerized tracking of joints and other landmarks in moving animals, including humans. Such tracking promises important advances in biology and biomedicine. Modern tracking models depend critically on labor-intensive annotated datasets of primary landmarks by non-expert humans. However, such annotation approaches can be costly and impractical for secondary landmarks, that is, ones that reflect fine-grained geometry of animals, and that are often specific to customized behavioral tasks. Due to visual and geometric ambiguity, nonexperts are often not qualified for secondary landmark annotation, which can require anatomical and zoological knowledge. These barriers significantly impede downstream behavioral studies because the learned tracking models exhibit limited generalizability. We hypothesize that there exists a shared representation between the primary and secondary landmarks because the range of motion of the secondary landmarks can be approximately spanned by that of the primary landmarks. We present a method to learn this spatial relationship of the primary and secondary landmarks in three dimensional space, which can, in turn, self-supervise the secondary landmark detector. This 3D representation learning is generic, and can therefore be applied to various multiview settings across diverse organisms, including macaques, flies, and humans.