mlsanityData to 14 Oct 2026

Unsupervised Monocular 3D Keypoint Discovery from Multi-View Diffusion Priors

arXiv 2507.12336 ↗First posted 16 Jul 2025cs.CVcited by 0

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Most existing 3D keypoint estimation methods rely on manual annotations or calibrated multi-view images, both of which are expensive to collect. This paper introduces KeyDiff3D, a framework that can accurately predict 3D keypoints from a single image, thus eliminating the need for such expensive data acquisitions. To achieve this, we leverage powerful geometric priors embedded in a pretrained multi-view diffusion model. In our framework, the diffusion model generates multi-view images from a single image, serving as supervision signals to provide 3D geometric cues to our model. We also introduce a 3D feature extractor that transforms implicit 3D priors embedded in the diffusion features into explicit 3D feature volumes. Beyond accurate keypoint estimation, we further introduce a pipeline that enables manipulation of 3D objects generated by the diffusion model. Experimental results on diverse datasets, including Human3.6M, CUB-200-2011, Stanford Dogs, and several in-the-wild and out-of-domain inputs, highlight the effectiveness of our method in terms of accuracy, generalization, and its ability to enable manipulation of 3D objects generated by the diffusion model from a single image.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 59 of its 74 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

59 of 74 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion18 Mar
  2. 2024EscherNet: A Generative Model for Scalable View Synthesis6 Feb
  3. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation2 Dec
  4. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep

Other references · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2020First Order Motion Model for Image Animationcited by 704
  6. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 612
  7. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  8. 2023Emergent Correspondence from Image Diffusioncited by 446
  9. 2017A simple yet effective baseline for 3d human pose estimationcited by 401
  10. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  11. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 301
  12. 2017Integral Human Pose Regressioncited by 256
38 more
  1. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 214
  2. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 208
  3. 2019Learnable Triangulation of Human Posecited by 161
  4. 2021AP-10K: A Benchmark for Animal Pose Estimation in the Wildcited by 123
  5. 2023Unsupervised Semantic Correspondence Using Stable Diffusioncited by 120
  6. 2017Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approachcited by 115
  7. 2018Unsupervised Learning of Object Landmarks through Conditional Image Generationcited by 114
  8. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  9. 2017V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Mapcited by 102
  10. 2019SCOPS: Self-Supervised Co-Part Segmentationcited by 100
  11. 2019Self-Supervised Learning of 3D Human Pose using Multi-view Geometrycited by 98
  12. 2019Unsupervised 3D Pose Estimation with Geometric Self-Supervisioncited by 95
  13. 2023Diffusion Model as Representation Learnercited by 95
  14. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  15. 2019RepNet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimationcited by 85
  16. 2018Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoningcited by 79
  17. 2018Unsupervised Discovery of Object Landmarks as Structural Representationscited by 73
  18. 2019Unsupervised Part-Based Disentangling of Object Shape and Appearancecited by 67
  19. 2020Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wildcited by 66
  20. 2020CanonPose: Self-Supervised Monocular 3D Human Pose Estimation in the Wildcited by 65
  21. 2017Unsupervised learning of object landmarks by factorized spatial embeddingscited by 64
  22. 2021Unsupervised Part Discovery from Contrastive Reconstructioncited by 57
  23. 2017Harvesting Multiple Views for Marker-less 3D Human Pose Annotationscited by 48
  24. 2018Can 3D Pose be Learned from 2D Projections Alone?cited by 44
  25. 2020Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesiscited by 44
  26. 2024EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Modelscited by 43
  27. 2019Unsupervised Keypoint Learning for Guiding Class-Conditional Video Predictioncited by 36
  28. 2024Exploring Diffusion Time-steps for Unsupervised Representation Learningcited by 36
  29. 2021ElePose: Unsupervised 3D Human Pose Estimation by Predicting Camera Elevation and Learning Normalizing Flows on 2D Posescited by 35
  30. 20163D Human Pose Estimation Using Convolutional Neural Networks with 2D Pose Informationcited by 28
  31. 2020Motion-supervised Co-Part Segmentationcited by 28
  32. 2020Kinematic-Structure-Preserved Representation for Unsupervised 3D Human Pose Estimationcited by 26
  33. 2021Self-Supervised Keypoint Discovery in Behavioral Videoscited by 23
  34. 2021GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generationcited by 19
  35. 2023Multiple View Geometry Transformers for 3D Human Pose Estimationcited by 19
  36. 2024Gaussian Frosting: Editable Complex Radiance Fields with Real-Time Renderingcited by 19
  37. 2022AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypointscited by 17
  38. 2023Unsupervised Keypoints from Pretrained Diffusion Modelscited by 15

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025KeyPointDiffuser: Unsupervised 3D Keypoint Learning via Latent Diffusion Models
  2. 2024Diffusion Models in 3D Vision: A Survey
  3. 2024MVD2: Efficient Multiview 3D Reconstruction for Multiview Diffusion
  4. 2023GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and Consistent 3D Generation
  5. 2024Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior
  6. 2024MVDiff: Scalable and Flexible Multi-View Diffusion for 3D Object Reconstruction from Single-View
  7. 2025Consistency Diffusion Models for Single-Image 3D Reconstruction with Priors
  8. 2025Can Video Diffusion Model Reconstruct 4D Geometry?