# Skeleton Merger: an Unsupervised Aligned Keypoint Detector

arXiv: https://arxiv.org/abs/2103.10814 · HTML: https://mlsanity.com/paper/2103.10814/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 34 references that are in the map.
- First posted on arXiv 19 Mar 2021.

## Builds on (most cited first, 15 of 15)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [Relation-Shape Convolutional Neural Network for Point Cloud Analysis](https://mlsanity.com/paper/1904.07601/) (2019)
- [SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters](https://mlsanity.com/paper/1803.11527/) (2018)
- [Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era](https://mlsanity.com/paper/1906.06543/) (2019)
- [Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge](https://mlsanity.com/paper/1609.09475/) (2016)
- [USIP: Unsupervised Stable Interest Point Detection from 3D Point Clouds](https://mlsanity.com/paper/1904.00229/) (2019)
- [KeypointNet: A Large-scale 3D Keypoint Dataset Aggregated from Numerous Human Annotations](https://mlsanity.com/paper/2002.12687/) (2020)
- [Unsupervised Learning of Category-Specific Symmetric 3D Keypoints from Point Sets](https://mlsanity.com/paper/2003.07619/) (2020)
- [Object Skeleton Extraction in Natural Images by Fusing Scale-associated Deep Side Outputs](https://mlsanity.com/paper/1603.09446/) (2016)
- [Performance Evaluation of 3D Correspondence Grouping Algorithms](https://mlsanity.com/paper/1804.02085/) (2018)
- [3D Interest Point Detection via Discriminative Learning](https://mlsanity.com/paper/1312.6826/) (2013)

## Built on by (most cited first, 27 of 27)

- [Skeleton-free Pose Transfer for Stylized 3D Characters](https://mlsanity.com/paper/2208.00790/) (2022)
- [LAKe-Net: Topology-Aware Point Cloud Completion by Localizing Aligned Keypoints](https://mlsanity.com/paper/2203.16771/) (2022)
- [USEEK: Unsupervised SE(3)-Equivariant 3D Keypoints for Generalizable Manipulation](https://mlsanity.com/paper/2209.13864/) (2022)
- [ClothesNet: An Information-Rich 3D Garment Model Repository with Simulated Clothes Environment](https://mlsanity.com/paper/2308.09987/) (2023)
- [UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence](https://mlsanity.com/paper/2405.06903/) (2024)
- [NCP: Neural Correspondence Prior for Effective Unsupervised Shape Matching](https://mlsanity.com/paper/2301.05839/) (2023)
- [SOCS: Semantically-aware Object Coordinate Space for Category-Level 6D Object Pose Estimation under Large Shape Variations](https://mlsanity.com/paper/2303.10346/) (2023)
- [Localization with Sampling-Argmax](https://mlsanity.com/paper/2110.08825/) (2021)
- [ArrayBot: Reinforcement Learning for Generalizable Distributed Manipulation through Touch](https://mlsanity.com/paper/2306.16857/) (2023)
- [ManiFoundation Model for General-Purpose Robotic Manipulation of Contact Synthesis with Arbitrary Objects and Robots](https://mlsanity.com/paper/2405.06964/) (2024)
- [SC3K: Self-supervised and Coherent 3D Keypoints Estimation from Rotated, Noisy, and Decimated Point Cloud Data](https://mlsanity.com/paper/2308.05410/) (2023)
- [SPEAL: Skeletal Prior Embedded Attention Learning for Cross-Source Point Cloud Registration](https://mlsanity.com/paper/2312.08664/) (2023)
- [Object Wake-up: 3D Object Rigging from a Single Image](https://mlsanity.com/paper/2108.02708/) (2021)
- [Unsupervised Learning of 3D Semantic Keypoints with Mutual Reconstruction](https://mlsanity.com/paper/2203.10212/) (2022)
- [General-purpose Clothes Manipulation with Semantic Keypoints](https://mlsanity.com/paper/2408.08160/) (2024)
- [Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features](https://mlsanity.com/paper/2410.02237/) (2024)
- [ESCAPE: Equivariant Shape Completion via Anchor Point Encoding](https://mlsanity.com/paper/2412.00952/) (2024)
- [Instant-Fold: In-Context Imitation Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2606.04269/) (2026)
- [Semantic keypoint extraction for scanned animals using multi-depth-camera systems](https://mlsanity.com/paper/2211.08634/) (2022)
- [3D Keypoint Estimation Using Implicit Representation Learning](https://mlsanity.com/paper/2306.11529/) (2023)
- [SelfGeo: Self-supervised and Geodesic-consistent Estimation of Keypoints on Deformable Shapes](https://mlsanity.com/paper/2408.02291/) (2024)
- [CLASP: General-Purpose Clothes Manipulation with Semantic Keypoints](https://mlsanity.com/paper/2507.19983/) (2025)
- [FCBV-Net: Category-Level Robotic Garment Smoothing via Feature-Conditioned Bimanual Value Prediction](https://mlsanity.com/paper/2508.05153/) (2025)
- [Skeleton-based Robust Registration Framework for Corrupted 3D Point Clouds](https://mlsanity.com/paper/2509.24273/) (2025)
- [KeyPointDiffuser: Unsupervised 3D Keypoint Learning via Latent Diffusion Models](https://mlsanity.com/paper/2512.03450/) (2025)
- [Learned Suppression for 3D Keypoint Detection with a Graph-Transformer Backbone](https://mlsanity.com/paper/2605.15088/) (2026)
- [KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalization](https://mlsanity.com/paper/2609.28818/) (2026)

## Abstract (arXiv)

Detecting aligned 3D keypoints is essential under many scenarios such as object tracking, shape retrieval and robotics. However, it is generally hard to prepare a high-quality dataset for all types of objects due to the ambiguity of keypoint itself. Meanwhile, current unsupervised detectors are unable to generate aligned keypoints with good coverage. In this paper, we propose an unsupervised aligned keypoint detector, Skeleton Merger, which utilizes skeletons to reconstruct objects. It is based on an Autoencoder architecture. The encoder proposes keypoints and predicts activation strengths of edges between keypoints. The decoder performs uniform sampling on the skeleton and refines it into small point clouds with pointwise offsets. Then the activation strengths are applied and the sub-clouds are merged. Composite Chamfer Distance (CCD) is proposed as a distance between the input point cloud and the reconstruction composed of sub-clouds masked by activation strengths. We demonstrate that Skeleton Merger is capable of detecting semantically-rich salient keypoints with good alignment, and shows comparable performance to supervised methods on the KeypointNet dataset. It is also shown that the detector is robust to noise and subsampling. Our code is available at https://github.com/eliphatfs/SkeletonMerger.