# Sketch2PoseNet: Efficient and Generalized Sketch to 3D Human Pose Prediction

arXiv: https://arxiv.org/abs/2510.26196 · HTML: https://mlsanity.com/paper/2510.26196/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 33 of its 66 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop](https://mlsanity.com/paper/1909.12828/) (2019)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Humans in 4D: Reconstructing and Tracking Humans with Transformers](https://mlsanity.com/paper/2305.20091/) (2023)
- [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020)
- [DeepPose: Human Pose Estimation via Deep Neural Networks](https://mlsanity.com/paper/1312.4659/) (2013)
- [Unleashing Text-to-Image Diffusion Models for Visual Perception](https://mlsanity.com/paper/2303.02153/) (2023)
- [PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop](https://mlsanity.com/paper/2103.16507/) (2021)
- [CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation](https://mlsanity.com/paper/2208.00571/) (2022)
- [Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose](https://mlsanity.com/paper/1611.07828/) (2016)
- [Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach](https://mlsanity.com/paper/1704.02447/) (2017)
- [Detailed Human Shape Estimation from a Single Image by Hierarchical Mesh Deformation](https://mlsanity.com/paper/1904.10506/) (2019)
- [Human-Art: A Versatile Human-Centric Dataset Bridging Natural and Artificial Scenes](https://mlsanity.com/paper/2303.02760/) (2023)
- [Learning to Estimate Robust 3D Human Mesh from In-the-Wild Crowded Scenes](https://mlsanity.com/paper/2104.07300/) (2021)
- [AvatarBooth: High-Quality and Customizable 3D Human Avatar Generation](https://mlsanity.com/paper/2306.09864/) (2023)
- [Direct Prediction of 3D Body Poses from Motion Compensated Sequences](https://mlsanity.com/paper/1511.06692/) (2015)
- [A Method for Animating Children's Drawings of the Human Figure](https://mlsanity.com/paper/2303.12741/) (2023)
- [3D Human Pose Machines with Self-supervised Learning](https://mlsanity.com/paper/1901.03798/) (2019)
- [Self-Supervised Human Depth Estimation from Monocular Videos](https://mlsanity.com/paper/2005.03358/) (2020)
- [DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery](https://mlsanity.com/paper/2404.01424/) (2024)
- [Detailed Avatar Recovery from Single Image](https://mlsanity.com/paper/2108.02931/) (2021)
- [Enhancing Human Pose Estimation in Ancient Vase Paintings via Perceptually-grounded Style Transfer Learning](https://mlsanity.com/paper/2012.05616/) (2020)
- [JOTR: 3D Joint Contrastive Learning with Transformers for Occluded Human Mesh Recovery](https://mlsanity.com/paper/2307.16377/) (2023)

## Built on by (most cited first, 2 of 2)

- [UIKA: Fast Universal Head Avatar from Pose-Free Images](https://mlsanity.com/paper/2601.07603/) (2026)
- [Bringing Your Portrait to 3D Presence](https://mlsanity.com/paper/2511.22553/) (2025)

## Abstract (arXiv)

3D human pose estimation from sketches has broad applications in computer animation and film production. Unlike traditional human pose estimation, this task presents unique challenges due to the abstract and disproportionate nature of sketches. Previous sketch-to-pose methods, constrained by the lack of large-scale sketch-3D pose annotations, primarily relied on optimization with heuristic rules-an approach that is both time-consuming and limited in generalizability. To address these challenges, we propose a novel approach leveraging a "learn from synthesis" strategy. First, a diffusion model is trained to synthesize sketch images from 2D poses projected from 3D human poses, mimicking disproportionate human structures in sketches. This process enables the creation of a synthetic dataset, SKEP-120K, consisting of 120k accurate sketch-3D pose annotation pairs across various sketch styles. Building on this synthetic dataset, we introduce an end-to-end data-driven framework for estimating human poses and shapes from diverse sketch styles. Our framework combines existing 2D pose detectors and generative diffusion priors for sketch feature extraction with a feed-forward neural network for efficient 2D pose estimation. Multiple heuristic loss functions are incorporated to guarantee geometric coherence between the derived 3D poses and the detected 2D poses while preserving accurate self-contacts. Qualitative, quantitative, and subjective evaluations collectively show that our model substantially surpasses previous ones in both estimation accuracy and speed for sketch-to-pose tasks.