mlsanityData to 7 Oct 2026

Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Features

arXiv 2311.18113 ↗First posted 29 Nov 2023cs.CV cs.GRcited by 17

Not on the map. Why?

Abstract (arXiv)

With the immense growth of dataset sizes and computing resources in recent years, so-called foundation models have become popular in NLP and vision tasks. In this work, we propose to explore foundation models for the task of keypoint detection on 3D shapes. A unique characteristic of keypoint detection is that it requires semantic and geometric awareness while demanding high localization accuracy. To address this problem, we propose, first, to back-project features from large pre-trained 2D vision models onto 3D shapes and employ them for this task. We show that we obtain robust 3D features that contain rich semantic information and analyze multiple candidate features stemming from different 2D foundation models. Second, we employ a keypoint candidate optimization module which aims to match the average observed distribution of keypoints on the shape and is guided by the back-projected features. The resulting approach achieves a new state of the art for few-shot keypoint detection on the KeyPointNet dataset, almost doubling the performance of the previous best methods.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

37 of 51 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Segment Anythingcited by 8,158
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  10. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  11. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  12. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
25 more
  1. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  2. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  3. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  4. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  5. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  6. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  7. 2020D3Feat: Joint Learning of Dense Detection and Description of 3D Local Featurescited by 175
  8. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 117
  9. 2023TextDeformer: Geometry Manipulation using Text Guidancecited by 94
  10. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  11. 2019USIP: Unsupervised Stable Interest Point Detection from 3D Point Cloudscited by 70
  12. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  13. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  14. 2020KeypointNet: A Large-scale 3D Keypoint Dataset Aggregated from Numerous Human Annotationscited by 50
  15. 2021KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Controlcited by 50
  16. 2020ManifoldPlus: A Robust and Scalable Watertight Manifold Surface Generation Method for Triangle Soupscited by 38
  17. 2023Zero-Shot 3D Shape Correspondencecited by 33
  18. 20223D Highlighter: Localizing Regions on 3D Shapes via Text Descriptionscited by 32
  19. 2023PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learningcited by 31
  20. 2021Unsupervised Learning of Visual 3D Keypoints for Controlcited by 30
  21. 2020Learning Implicit Functions for Topology-Varying Dense 3D Shape Correspondencecited by 27
  22. 2021Learning 3D Dense Correspondence via Canonical Point Autoencodercited by 21
  23. 2023NCP: Neural Correspondence Prior for Effective Unsupervised Shape Matchingcited by 18
  24. 2020UKPGAN: A General Self-Supervised Keypoint Detectorcited by 15
  25. 2023Neural Semantic Surface Mapscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 10; 2026: 4.

17 papers.

Sort:

By area:

  1. 2025Do It Yourself: Learning Semantic Correspondence from Pseudo-Labelscited by 13
  2. 2024SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimationcited by 12
  3. 2024Beyond Complete Shapes: A Benchmark for Quantitative Evaluation of 3D Shape Surface Matching Algorithmscited by 11
  4. 2024DV-Matcher: Deformation-based Non-Rigid Point Cloud Matching Guided by Pre-trained Visual Featurescited by 10
  5. 2025Surface-Aware Distilled 3D Semantic Featurescited by 5
  6. 2025Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondencecited by 5
  7. 2025RARE: Refine Any Registration of Pairwise Point Clouds via Zero-Shot Learningcited by 5
  8. 2026PatchAlign3D: Local Feature Alignment for Dense 3D Shape Understandingcited by 5
  9. 2025A dataset-free approach for self-supervised learning of 3D reflectional symmetriescited by 3
  10. 2025WIR3D: Visually-Informed and Geometry-Aware 3D Shape Abstractioncited by 3
  11. 2025Towards Geometry-Grounded Dense Semantic Matching with VGGT Priorscited by 3
  12. 2025Channel-adaptive Cross-modal Generative Semantic Communication for Point Cloud Transmissioncited by 1
5 more
  1. 2026Deep Feature Deformation Weightscited by 1
  2. 2026GLASS: Graph and Vision-Language Assisted Semantic Shape Correspondencecited by 1
  3. 2025Training-free zero-shot 3D symmetry detection with visual features back-projected to geometrycited by 0
  4. 2025Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detectioncited by 0
  5. 2026PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Viewscited by 0