mlsanityData to 14 Oct 2026

End-to-End Learning of Multi-category 3D Pose and Shape Estimation

arXiv 2112.10196 ↗First posted 19 Dec 2021cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the representation of the shape and pose of objects using their keypoints. Therefore, we propose an end-to-end method that simultaneously detects 2D keypoints from an image and lifts them to 3D. The proposed method learns both 2D detection and 3D lifting only from 2D keypoints annotations. In addition to being end-to-end from images to 3D keypoints, our method also handles objects from multiple categories using a single neural network. We use a Transformer-based architecture to detect the keypoints, as well as to summarize the visual context of the image. This visual context information is then used while lifting the keypoints to 3D, to allow context-based reasoning for better performance. Our method can handle occlusions as well as a wide variety of object classes. Our experiments on three benchmarks demonstrate that our method performs better than the state-of-the-art. Our source code will be made publicly available.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 59 references that are in the map. First posted on arXiv 19 Dec 2021.

Builds on

28 of 59 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020End-to-End Object Detection with Transformerscited by 7,414
  3. 2017Mask R-CNNcited by 7,277
  4. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  5. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 695
  6. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  7. 2013DeepPose: Human Pose Estimation via Deep Neural Networkscited by 305
  8. 2019Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wildcited by 180
  9. 20183DFeat-Net: Weakly Supervised Local 3D Features for Point Cloud Registrationcited by 103
  10. 2019Unsupervised 3D Pose Estimation with Geometric Self-Supervisioncited by 95
  11. 2018Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoningcited by 79
  12. 20163D Human Pose Estimation from a Single Image via Distance Matrix Regressioncited by 73
16 more
  1. 2019C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motioncited by 61
  2. 2014Viewpoints and Keypointscited by 59
  3. 2015Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Videocited by 54
  4. 2019Deep Non-Rigid Structure from Motioncited by 38
  5. 2019Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generationcited by 34
  6. 2019Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model using Deep Non-Rigid Structure from Motioncited by 23
  7. 2021To The Point: Correspondence-driven monocular 3D category reconstructioncited by 21
  8. 2018Unsupervised Adversarial Learning of 3D Human Pose from 2D Joint Locationscited by 18
  9. 2019In Perfect Shape: Certifiably Optimal 3D Shape Reconstruction from 2D Landmarkscited by 17
  10. 2021Optimal Pose and Shape Estimation for Category-level 3D Object Perceptioncited by 15
  11. 2019Distill Knowledge from NRSfM for Weakly Supervised 3D Pose Learningcited by 13
  12. 2020Procrustean Regression Networks: Learning 3D Structure of Non-Rigid Objects from 2D Annotationscited by 13
  13. 2020Self-Supervised Multi-View Synchronization Learning for 3D Pose Estimationcited by 9
  14. 2021PAUL: Procrustean Autoencoder for Unsupervised Liftingcited by 9
  15. 2021PR-RRN: Pairwise-Regularized Residual-Recursive Networks for Non-rigid Structure-from-Motioncited by 9
  16. 20203D Registration for Self-Occluded Objects in Contextcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Unsupervised 2D-3D lifting of non-rigid objects using local constraintscited by 0