mlsanityData to 7 Oct 2026

BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos

arXiv 2212.07401 ↗First posted 14 Dec 2022cs.CV cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Quantifying motion in 3D is important for studying the behavior of humans and other animals, but manual pose annotations are expensive and time-consuming to obtain. Self-supervised keypoint discovery is a promising strategy for estimating 3D poses without annotations. However, current keypoint discovery approaches commonly process single 2D views and do not operate in the 3D space. We propose a new method to perform self-supervised keypoint discovery in 3D from multi-view videos of behaving agents, without any keypoint or bounding box supervision in 2D or 3D. Our method, BKinD-3D, uses an encoder-decoder architecture with a 3D volumetric heatmap, trained to reconstruct spatiotemporal differences across multiple views, in addition to joint length constraints on a learned 3D skeleton of the subject. In this way, we discover keypoints without requiring manual supervision in videos of humans and rats, demonstrating the potential of 3D keypoint discovery for studying behavior.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

30 of 50 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  4. 2015Spatial Transformer Networkscited by 1,423
  5. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  6. 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
  7. 2017Cascaded Pyramid Network for Multi-Person Pose Estimationcited by 341
  8. 2017Integral Human Pose Regressioncited by 255
  9. 2018Markerless tracking of user-defined features with deep learningcited by 203
  10. 2019Learnable Triangulation of Human Posecited by 160
  11. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 113
  12. 20163D Human Pose Estimation = 2D Pose Estimation + Matchingcited by 112
18 more
  1. 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
  2. 2019Self-Supervised Learning of 3D Human Pose using Multi-view Geometrycited by 98
  3. 2019Unsupervised 3D Pose Estimation with Geometric Self-Supervisioncited by 95
  4. 2017Exploiting temporal information for 3D pose estimationcited by 89
  5. 2018Unsupervised Discovery of Object Landmarks as Structural Representationscited by 73
  6. 2018Learning Monocular 3D Human Pose Estimation from Multi-view Imagescited by 70
  7. 2019Unsupervised Part-Based Disentangling of Object Shape and Appearancecited by 67
  8. 2020Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wildcited by 66
  9. 2020CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wildcited by 65
  10. 2020Cross-View Tracking for Multi-Human 3D Pose Estimation at over 100 FPScited by 36
  11. 2021The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactionscited by 31
  12. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  13. 2021Self-Supervised Keypoint Discovery in Behavioral Videoscited by 23
  14. 2022AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypointscited by 17
  15. 2016Learning recurrent representations for hierarchical behavior modelingcited by 16
  16. 2021MetaPose: Fast 3D Pose from Multiple Views without 3D Supervisioncited by 12
  17. 2021Leaving Flatland: Advances in 3D behavioral measurementcited by 10
  18. 2021Weakly Supervised Keypoint Discoverycited by 5

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 4; 2025: 2.

10 papers.

  1. 2023Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videoscited by 21
  2. 20233D-MuPPET: 3D Multi-Pigeon Pose Estimation and Trackingcited by 12
  3. 2024UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cuescited by 6
  4. 2025DIMO: Diverse 3D Motion Generation for Arbitrary Objectscited by 6
  5. 2024Pose Prior Learner: Unsupervised Categorical Prior Learning for Pose Estimationcited by 3
  6. 2023Mask as Supervision: Leveraging Unified Mask Information for Unsupervised 3D Pose Estimationcited by 2
  7. 2024Learning Keypoints for Multi-Agent Behavior Analysis using Self-Supervisioncited by 2
  8. 2023Tracking Object Positions in Reinforcement Learning: A Metric for Keypoint Detection (extended version)cited by 0
  9. 2024X as Supervision: Contending with Depth Ambiguity in Unsupervised Monocular 3D Pose Estimationcited by 0
  10. 2025Unsupervised Monocular 3D Keypoint Discovery from Multi-View Diffusion Priorscited by 0