mlsanityData to 13 Oct 2026

Understanding Pixel-level 2D Image Semantics with 3D Keypoint Knowledge Engine

arXiv 2111.10817 ↗First posted 21 Nov 2021cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Pixel-level 2D object semantic understanding is an important topic in computer vision and could help machine deeply understand objects (e.g. functionality and affordance) in our daily life. However, most previous methods directly train on correspondences in 2D images, which is end-to-end but loses plenty of information in 3D spaces. In this paper, we propose a new method on predicting image corresponding semantics in 3D domain and then projecting them back onto 2D images to achieve pixel-level understanding. In order to obtain reliable 3D semantic labels that are absent in current image datasets, we build a large scale keypoint knowledge engine called KeypointNet, which contains 103,450 keypoints and 8,234 3D models from 16 object categories. Our method leverages the advantages in 3D vision and can explicitly reason about objects self-occlusion and visibility. We show that our method gives comparative and even superior results on standard semantic benchmarks.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 69 of its 100 references that are in the map. First posted on arXiv 21 Nov 2021.

Builds on

69 of 100 references.

References · 69

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  3. 2017Mask R-CNNcited by 7,266
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,027
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,840
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,240
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,216
  8. 2015Fast R-CNNcited by 2,933
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,889
  10. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,686
  11. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  12. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,039
38 more
  1. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  2. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 976
  3. 2018Panoptic Segmentationcited by 798
  4. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  5. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  6. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  7. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 524
  8. 2018Spherical CNNscited by 452
  9. 2017Neural 3D Mesh Renderercited by 437
  10. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 403
  11. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 342
  12. 2019NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detectioncited by 290
  13. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  14. 2017Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputscited by 189
  15. 2017PoseTrack: A Benchmark for Human Pose Estimation and Trackingcited by 173
  16. 2018Neighbourhood Consensus Networkscited by 173
  17. 20183D-CODED : 3D Correspondences by Deep Deformationcited by 160
  18. 2019SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimizationcited by 159
  19. 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 142
  20. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  21. 2017Convolutional neural network architecture for geometric matchingcited by 138
  22. 2017Learning Efficient Point Cloud Generation for Dense 3D Object Reconstructioncited by 133
  23. 2014Hypercolumns for Object Segmentation and Fine-grained Localizationcited by 128
  24. 2019SDM-NET: Deep Generative Network for Structured Deformable Meshcited by 113
  25. 2019Seamless Scene Segmentationcited by 112
  26. 2017MarrNet: 3D Shape Reconstruction via 2.5D Sketchescited by 108
  27. 2019Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networkscited by 106
  28. 2019Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformationcited by 103
  29. 2019Learning to Infer Implicit Surfaces without 3D Supervisioncited by 103
  30. 2016Learning Dense Correspondence via 3D-guided Cycle Consistencycited by 91
  31. 20176-DoF Object Pose from Semantic Keypointscited by 89
  32. 2016Universal Correspondence Networkcited by 82
  33. 2018Unsupervised Deep Learning for Structured Shape Matchingcited by 81
  34. 2016SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentationcited by 70
  35. 2018Learning Shape Priors for Single-View 3D Completion and Reconstructioncited by 68
  36. 2019Learning to Infer and Execute 3D Shape Programscited by 68
  37. 2019Canonical Surface Mapping via Geometric Cycle Consistencycited by 62
  38. 2017End-to-end weakly-supervised semantic alignmentcited by 60

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 0; 2026: 1.

2 papers.

  1. 2022Highlighting Object Category Immunity for the Generalization of Human-Object Interaction Detectioncited by 9
  2. 2026Category-Level 3D Correspondence in Camera Space via Morphable Object Priorscited by 0