# StarMap for Category-Agnostic Keypoint and Viewpoint Estimation

arXiv: https://arxiv.org/abs/1803.09331 · HTML: https://mlsanity.com/paper/1803.09331/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2018.

## Built on by (most cited first, 30 of 42)

- [Sequential Modeling Enables Scalable Learning for Large Vision Models](https://mlsanity.com/paper/2312.00785/) (2023)
- [Zero-Shot Category-Level Object Pose Estimation](https://mlsanity.com/paper/2204.03635/) (2022)
- [Templates for 3D Object Pose Estimation Revisited: Generalization to New Objects and Robustness to Occlusions](https://mlsanity.com/paper/2203.17234/) (2022)
- [OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images](https://mlsanity.com/paper/2111.14341/) (2021)
- [NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation](https://mlsanity.com/paper/2101.12378/) (2021)
- [Pose for Everything: Towards Category-Agnostic Pose Estimation](https://mlsanity.com/paper/2207.10387/) (2022)
- [ImageNet3D: Towards General-Purpose Object-Level 3D Understanding](https://mlsanity.com/paper/2406.09613/) (2024)
- [Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Features](https://mlsanity.com/paper/2209.05624/) (2022)
- [VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis](https://mlsanity.com/paper/2205.15401/) (2022)
- [PoseContrast: Class-Agnostic Object Viewpoint Estimation in the Wild with Pose-Aware Contrastive Learning](https://mlsanity.com/paper/2105.05643/) (2021)
- [Generating Images with 3D Annotations Using Diffusion Models](https://mlsanity.com/paper/2306.08103/) (2023)
- [Self-Supervised Learning for Medical Image Data with Anatomy-Oriented Imaging Planes](https://mlsanity.com/paper/2403.16499/) (2024)
- [FisherMatch: Semi-Supervised Rotation Regression via Entropy-based Filtering](https://mlsanity.com/paper/2203.15765/) (2022)
- [3D-Aware Visual Question Answering about Parts, Poses and Occlusions](https://mlsanity.com/paper/2310.17914/) (2023)
- [Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose](https://mlsanity.com/paper/2110.14213/) (2021)
- [PFRL: Pose-Free Reinforcement Learning for 6D Pose Estimation](https://mlsanity.com/paper/2102.12096/) (2021)
- [DISP6D: Disentangled Implicit Shape and Pose Learning for Scalable 6D Pose Estimation](https://mlsanity.com/paper/2107.12549/) (2021)
- [Certifiable 3D Object Pose Estimation: Foundations, Learning Models, and Self-Training](https://mlsanity.com/paper/2206.11215/) (2022)
- [OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images](https://mlsanity.com/paper/2304.10266/) (2023)
- [NOVUM: Neural Object Volumes for Robust Object Classification](https://mlsanity.com/paper/2305.14668/) (2023)
- [iNeMo: Incremental Neural Mesh Models for Robust Class-Incremental Learning](https://mlsanity.com/paper/2407.09271/) (2024)
- [Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space](https://mlsanity.com/paper/2504.21749/) (2025)
- [Semantic keypoint-based pose estimation from single RGB frames](https://mlsanity.com/paper/2204.05864/) (2022)
- [Source-Free and Image-Only Unsupervised Domain Adaptation for Category Level Object Pose Estimation](https://mlsanity.com/paper/2401.10848/) (2024)
- [Fully Convolutional Geometric Features for Category-level Object Alignment](https://mlsanity.com/paper/2103.04494/) (2021)
- [Semi-supervised Viewpoint Estimation with Geometry-aware Conditional Generation](https://mlsanity.com/paper/2104.01103/) (2021)
- [Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos](https://mlsanity.com/paper/2407.04384/) (2024)
- [Localization and Mapping using Instance-specific Mesh Models](https://mlsanity.com/paper/2103.04493/) (2021)
- [Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis](https://mlsanity.com/paper/2306.00118/) (2023)
- [Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models](https://mlsanity.com/paper/2411.05005/) (2024)

## Abstract (arXiv)

Semantic keypoints provide concise abstractions for a variety of visual understanding tasks. Existing methods define semantic keypoints separately for each category with a fixed number of semantic labels in fixed indices. As a result, this keypoint representation is in-feasible when objects have a varying number of parts, e.g. chairs with varying number of legs. We propose a category-agnostic keypoint representation, which combines a multi-peak heatmap (StarMap) for all the keypoints and their corresponding features as 3D locations in the canonical viewpoint (CanViewFeature) defined for each instance. Our intuition is that the 3D locations of the keypoints in canonical object views contain rich semantic and compositional information. Using our flexible representation, we demonstrate competitive performance in keypoint detection and localization compared to category-specific state-of-the-art methods. Moreover, we show that when augmented with an additional depth channel (DepthMap) to lift the 2D keypoints to 3D, our representation can achieve state-of-the-art results in viewpoint estimation. Finally, we show that our category-agnostic keypoint representation can be generalized to novel categories.