# BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos

arXiv: https://arxiv.org/abs/2212.07401 · HTML: https://mlsanity.com/paper/2212.07401/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 50 references that are in the map.
- First posted on arXiv 14 Dec 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [3D Human Pose Estimation = 2D Pose Estimation + Matching](https://mlsanity.com/paper/1612.06524/) (2016)
- [V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map](https://mlsanity.com/paper/1711.07399/) (2017)
- [Self-Supervised Learning of 3D Human Pose using Multi-view Geometry](https://mlsanity.com/paper/1903.02330/) (2019)
- [Unsupervised 3D Pose Estimation with Geometric Self-Supervision](https://mlsanity.com/paper/1904.04812/) (2019)
- [Exploiting temporal information for 3D pose estimation](https://mlsanity.com/paper/1711.08585/) (2017)
- [Unsupervised Discovery of Object Landmarks as Structural Representations](https://mlsanity.com/paper/1804.04412/) (2018)
- [Learning Monocular 3D Human Pose Estimation from Multi-view Images](https://mlsanity.com/paper/1803.04775/) (2018)
- [Unsupervised Part-Based Disentangling of Object Shape and Appearance](https://mlsanity.com/paper/1903.06946/) (2019)
- [Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild](https://mlsanity.com/paper/2003.07581/) (2020)
- [CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wild](https://mlsanity.com/paper/2011.14679/) (2020)
- [Cross-View Tracking for Multi-Human 3D Pose Estimation at over 100 FPS](https://mlsanity.com/paper/2003.03972/) (2020)
- [The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions](https://mlsanity.com/paper/2104.02710/) (2021)
- [Unsupervised Learning of Visual 3D Keypoints for Control](https://mlsanity.com/paper/2106.07643/) (2021)
- [Self-Supervised Keypoint Discovery in Behavioral Videos](https://mlsanity.com/paper/2112.05121/) (2021)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [Learning recurrent representations for hierarchical behavior modeling](https://mlsanity.com/paper/1611.00094/) (2016)
- [MetaPose: Fast 3D Pose from Multiple Views without 3D Supervision](https://mlsanity.com/paper/2108.04869/) (2021)
- [Leaving Flatland: Advances in 3D behavioral measurement](https://mlsanity.com/paper/2112.01987/) (2021)
- [Weakly Supervised Keypoint Discovery](https://mlsanity.com/paper/2109.13423/) (2021)

## Built on by (most cited first, 10 of 10)

- [Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos](https://mlsanity.com/paper/2312.13604/) (2023)
- [3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking](https://mlsanity.com/paper/2308.15316/) (2023)
- [UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues](https://mlsanity.com/paper/2404.14634/) (2024)
- [DIMO: Diverse 3D Motion Generation for Arbitrary Objects](https://mlsanity.com/paper/2511.07409/) (2025)
- [Pose Prior Learner: Unsupervised Categorical Prior Learning for Pose Estimation](https://mlsanity.com/paper/2410.03858/) (2024)
- [Mask as Supervision: Leveraging Unified Mask Information for Unsupervised 3D Pose Estimation](https://mlsanity.com/paper/2312.07051/) (2023)
- [Learning Keypoints for Multi-Agent Behavior Analysis using Self-Supervision](https://mlsanity.com/paper/2409.09455/) (2024)
- [Tracking Object Positions in Reinforcement Learning: A Metric for Keypoint Detection (extended version)](https://mlsanity.com/paper/2312.00592/) (2023)
- [X as Supervision: Contending with Depth Ambiguity in Unsupervised Monocular 3D Pose Estimation](https://mlsanity.com/paper/2411.13026/) (2024)
- [Unsupervised Monocular 3D Keypoint Discovery from Multi-View Diffusion Priors](https://mlsanity.com/paper/2507.12336/) (2025)

## Abstract (arXiv)

Quantifying motion in 3D is important for studying the behavior of humans and other animals, but manual pose annotations are expensive and time-consuming to obtain. Self-supervised keypoint discovery is a promising strategy for estimating 3D poses without annotations. However, current keypoint discovery approaches commonly process single 2D views and do not operate in the 3D space. We propose a new method to perform self-supervised keypoint discovery in 3D from multi-view videos of behaving agents, without any keypoint or bounding box supervision in 2D or 3D. Our method, BKinD-3D, uses an encoder-decoder architecture with a 3D volumetric heatmap, trained to reconstruct spatiotemporal differences across multiple views, in addition to joint length constraints on a learned 3D skeleton of the subject. In this way, we discover keypoints without requiring manual supervision in videos of humans and rats, demonstrating the potential of 3D keypoint discovery for studying behavior.