# KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control

arXiv: https://arxiv.org/abs/2104.11224 · HTML: https://mlsanity.com/paper/2104.11224/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 22 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [Neural Cages for Detail-Preserving 3D Deformations](https://mlsanity.com/paper/1912.06395/) (2019)
- [DualSDF: Semantic Shape Manipulation using a Two-Level Representation](https://mlsanity.com/paper/2004.02869/) (2020)
- [Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning](https://mlsanity.com/paper/1807.03146/) (2018)
- [Unsupervised Discovery of Object Landmarks as Structural Representations](https://mlsanity.com/paper/1804.04412/) (2018)
- [Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2009.05085/) (2020)
- [Unsupervised learning of object landmarks by factorized spatial embeddings](https://mlsanity.com/paper/1705.02193/) (2017)
- [3DN: 3D Deformation Network](https://mlsanity.com/paper/1903.03322/) (2019)
- [KeypointNet: A Large-scale 3D Keypoint Dataset Aggregated from Numerous Human Annotations](https://mlsanity.com/paper/2002.12687/) (2020)
- [Self-supervised learning of a facial attribute embedding from video](https://mlsanity.com/paper/1808.06882/) (2018)
- [Unsupervised Learning of Landmarks by Descriptor Vector Exchange](https://mlsanity.com/paper/1908.06427/) (2019)
- [Unsupervised Learning of Intrinsic Structural Representation Points](https://mlsanity.com/paper/2003.01661/) (2020)
- [kPAM-SC: Generalizable Manipulation Planning using KeyPoint Affordance and Shape Completion](https://mlsanity.com/paper/1909.06980/) (2019)
- [Learning Generative Models of Shape Handles](https://mlsanity.com/paper/2004.03028/) (2020)
- [Unsupervised Learning of Category-Specific Symmetric 3D Keypoints from Point Sets](https://mlsanity.com/paper/2003.07619/) (2020)

## Built on by (most cited first, 30 of 51)

- [Generative Neural Articulated Radiance Fields](https://mlsanity.com/paper/2206.14314/) (2022)
- [TextDeformer: Geometry Manipulation using Text Guidance](https://mlsanity.com/paper/2304.13348/) (2023)
- [Deforming Radiance Fields with Cages](https://mlsanity.com/paper/2207.12298/) (2022)
- [Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats](https://mlsanity.com/paper/2212.14474/) (2022)
- [Controllable Mesh Generation Through Sparse Latent Point Diffusion Models](https://mlsanity.com/paper/2303.07938/) (2023)
- [NeuralEditor: Editing Neural Radiance Fields via Manipulating Point Clouds](https://mlsanity.com/paper/2305.03049/) (2023)
- [Skeleton-free Pose Transfer for Stylized 3D Characters](https://mlsanity.com/paper/2208.00790/) (2022)
- [Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes](https://mlsanity.com/paper/2205.02904/) (2022)
- [LAKe-Net: Topology-Aware Point Cloud Completion by Localizing Aligned Keypoints](https://mlsanity.com/paper/2203.16771/) (2022)
- [Neural Shape Deformation Priors](https://mlsanity.com/paper/2210.05616/) (2022)
- [Anymate: A Dataset and Baselines for Learning 3D Object Rigging](https://mlsanity.com/paper/2505.06227/) (2025)
- [CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation](https://mlsanity.com/paper/2311.01447/) (2023)
- [As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors](https://mlsanity.com/paper/2311.16739/) (2023)
- [NCP: Neural Correspondence Prior for Effective Unsupervised Shape Matching](https://mlsanity.com/paper/2301.05839/) (2023)
- [Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Features](https://mlsanity.com/paper/2311.18113/) (2023)
- [Point2SSM: Learning Morphological Variations of Anatomies from Point Cloud](https://mlsanity.com/paper/2305.14486/) (2023)
- [Weakly-supervised 3D Pose Transfer with Keypoints](https://mlsanity.com/paper/2307.13459/) (2023)
- [Creative Birds: Self-Supervised Single-View 3D Style Transfer](https://mlsanity.com/paper/2307.14127/) (2023)
- [SC3K: Self-supervised and Coherent 3D Keypoints Estimation from Rotated, Noisy, and Decimated Point Cloud Data](https://mlsanity.com/paper/2308.05410/) (2023)
- [Category-Agnostic Neural Object Rigging](https://mlsanity.com/paper/2505.20283/) (2025)
- [ShapeMatcher: Self-Supervised Joint Shape Canonicalization, Segmentation, Retrieval and Deformation](https://mlsanity.com/paper/2311.11106/) (2023)
- [Structure-Aware 3D VR Sketch to 3D Shape Retrieval](https://mlsanity.com/paper/2209.09043/) (2022)
- [Point-Based Radiance Fields for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2310.03375/) (2023)
- [Spice-E : Structural Priors in 3D Diffusion using Cross-Entity Attention](https://mlsanity.com/paper/2311.17834/) (2023)
- [Fancy123: One Image to High-Quality 3D Mesh Generation via Plug-and-Play Deformation](https://mlsanity.com/paper/2411.16185/) (2024)
- [KP-RED: Exploiting Semantic Keypoints for Joint 3D Shape Retrieval and Deformation](https://mlsanity.com/paper/2403.10099/) (2024)
- [NeuralMLS: Geometry-Aware Control Point Deformation](https://mlsanity.com/paper/2201.01873/) (2022)
- [Unsupervised Learning of 3D Semantic Keypoints with Mutual Reconstruction](https://mlsanity.com/paper/2203.10212/) (2022)
- [Point2SSM++: Self-Supervised Learning of Anatomical Shape Models from Point Clouds](https://mlsanity.com/paper/2405.09707/) (2024)
- [Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features](https://mlsanity.com/paper/2410.02237/) (2024)

## Abstract (arXiv)

We introduce KeypointDeformer, a novel unsupervised method for shape control through automatically discovered 3D keypoints. We cast this as the problem of aligning a source 3D object to a target 3D object from the same object category. Our method analyzes the difference between the shapes of the two objects by comparing their latent representations. This latent representation is in the form of 3D keypoints that are learned in an unsupervised way. The difference between the 3D keypoints of the source and the target objects then informs the shape deformation algorithm that deforms the source object into the target object. The whole model is learned end-to-end and simultaneously discovers 3D keypoints while learning to use them for deforming object shapes. Our approach produces intuitive and semantically consistent control of shape deformations. Moreover, our discovered 3D keypoints are consistent across object category instances despite large shape variations. As our method is unsupervised, it can be readily deployed to new object categories without requiring annotations for 3D keypoints and deformations.