# Depth-Guided Self-Supervised Human Keypoint Detection via Cross-Modal Distillation

arXiv: https://arxiv.org/abs/2410.14700 · HTML: https://mlsanity.com/paper/2410.14700/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [Unsupervised Discovery of Object Landmarks as Structural Representations](https://mlsanity.com/paper/1804.04412/) (2018)
- [Unsupervised Part-Based Disentangling of Object Shape and Appearance](https://mlsanity.com/paper/1903.06946/) (2019)
- [Unsupervised learning of object landmarks by factorized spatial embeddings](https://mlsanity.com/paper/1705.02193/) (2017)
- [Deep Feature Factorization For Concept Discovery](https://mlsanity.com/paper/1806.10206/) (2018)
- [XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning](https://mlsanity.com/paper/2211.13929/) (2022)
- [Motion-supervised Co-Part Segmentation](https://mlsanity.com/paper/2004.03234/) (2020)
- [Unsupervised Human Pose Estimation through Transforming Shape Templates](https://mlsanity.com/paper/2105.04154/) (2021)
- [Cross-modal Knowledge Distillation for Vision-to-Sensor Action Recognition](https://mlsanity.com/paper/2112.01849/) (2021)
- [Differentiable Drawing and Sketching](https://mlsanity.com/paper/2103.16194/) (2021)
- [GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation](https://mlsanity.com/paper/2112.01036/) (2021)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [LatentKeypointGAN: Controlling Images via Latent Keypoints](https://mlsanity.com/paper/2103.15812/) (2021)
- [Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations](https://mlsanity.com/paper/2309.04849/) (2023)
- [A Comprehensive Review of Automated Data Annotation Techniques in Human Activity Recognition](https://mlsanity.com/paper/2307.05988/) (2023)

## Abstract (arXiv)

Existing unsupervised keypoint detection methods apply artificial deformations to images such as masking a significant portion of images and using reconstruction of original image as a learning objective to detect keypoints. However, this approach lacks depth information in the image and often detects keypoints on the background. To address this, we propose Distill-DKP, a novel cross-modal knowledge distillation framework that leverages depth maps and RGB images for keypoint detection in a self-supervised setting. During training, Distill-DKP extracts embedding-level knowledge from a depth-based teacher model to guide an image-based student model with inference restricted to the student. Experiments show that Distill-DKP significantly outperforms previous unsupervised methods by reducing mean L2 error by 47.15% on Human3.6M, mean average error by 5.67% on Taichi, and improving keypoints accuracy by 1.3% on DeepFashion dataset. Detailed ablation studies demonstrate the sensitivity of knowledge distillation across different layers of the network. Project Page: https://23wm13.github.io/distill-dkp/