mlsanityData to 7 Oct 2026

End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization

arXiv 1909.06043 ↗First posted 13 Sep 2019cs.CVcited by 65

Not on the map. Why?

Abstract (arXiv)

Deep networks excel in learning patterns from large amounts of data. On the other hand, many geometric vision tasks are specified as optimization problems. To seamlessly combine deep learning and geometric vision, it is vital to perform learning and geometric optimization end-to-end. Towards this aim, we present BPnP, a novel network module that backpropagates gradients through a Perspective-n-Points (PnP) solver to guide parameter updates of a neural network. Based on implicit differentiation, we show that the gradients of a "self-contained" PnP solver can be derived accurately and efficiently, as if the optimizer block were a differentiable function. We validate BPnP by incorporating it in a deep model that can learn camera intrinsics, camera extrinsics (poses) and 3D structure from training datasets. Further, we develop an end-to-end trainable pipeline for object pose estimation, which achieves greater accuracy by combining feature-based heatmap losses with 2D-3D reprojection errors. Since our approach can be extended to other optimization problems, our work helps to pave the way to perform learnable geometric vision in a principled manner. Our PyTorch implementation of BPnP is available on http://github.com/BoChenYS/BPnP.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 13; 2023: 16; 2024: 15; 2025: 6; 2026: 5.

65 papers.

Sort:

By area:

  1. 2021Back to the Feature: Learning Robust Camera Localization from Pixels to Posecited by 199
  2. 2021GDRNPP: A Geometry-guided and Fully Learning-based Object Pose Estimatorcited by 148
  3. 2021SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimationcited by 88
  4. 2021MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagationcited by 82
  5. 2023EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimationcited by 79
  6. 2021ROCA: Robust CAD Model Retrieval and Alignment from a Single Imagecited by 66
  7. 2021RePOSE: Fast 6D Object Pose Refinement via Deep Texture Renderingcited by 63
  8. 2024GeoCalib: Learning Single-image Calibration with Geometric Optimizationcited by 50
  9. 2022Coupled Iterative Refinement for 6D Multi-Object Pose Estimationcited by 48
  10. 2022RNNPose: Recurrent 6-DoF Object Pose Refinement with Robust Correspondence Field Estimation and Pose Optimizationcited by 38
  11. 2023Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matchingcited by 35
  12. 2022Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimationcited by 34
38 more
  1. 2023Object Pose Estimation with Statistical Guarantees: Conformal Keypoint Detection and Geometric Uncertainty Propagationcited by 33
  2. 2021Deep Learning on Monocular Object Pose Detection and Tracking: A Comprehensive Overviewcited by 31
  3. 20236D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimationcited by 24
  4. 2023Markerless Camera-to-Robot Pose Estimation via Self-supervised Sim-to-Real Transfercited by 23
  5. 2021To The Point: Correspondence-driven monocular 3D category reconstructioncited by 21
  6. 2023Unsupervised Volumetric Animationcited by 18
  7. 2022YOLOPose: Transformer-based Multi-Object 6D Pose Estimation using Keypoint Regressioncited by 17
  8. 2023Shape-Constraint Recurrent Flow for 6D Object Pose Estimationcited by 17
  9. 2022CRT-6D: Fast 6D Object Pose Estimation with Cascaded Refinement Transformerscited by 15
  10. 2023CheckerPose: Progressive Dense Keypoint Localization for Object Pose Estimation with Graph Neural Networkcited by 15
  11. 2024CMRNext: Camera to LiDAR Matching in the Wild for Localization and Extrinsic Calibrationcited by 15
  12. 2021DSC-PoseNet: Learning 6DoF Object Pose Estimation via Dual-scale Consistencycited by 14
  13. 2023SPADES: A Realistic Spacecraft Pose Estimation Dataset using Event Sensingcited by 13
  14. 2022DGECN: A Depth-Guided Edge Convolutional Network for End-to-End 6D Pose Estimationcited by 12
  15. 2022Spacecraft Pose Estimation Based on Unsupervised Domain Adaptation and on a 3D-Guided Loss Combinationcited by 12
  16. 2022COPE: End-to-end trainable Constant Runtime Object Pose Estimationcited by 10
  17. 2024CtRNet-X: Camera-to-Robot Pose Estimation in Real-world Conditions Using a Single Cameracited by 10
  18. 2024RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Trainingcited by 10
  19. 2023Pseudo Flow Consistency for Self-Supervised 6D Object Pose Estimationcited by 9
  20. 2024SDFit: 3D Object Pose and Shape by Fitting a Morphable SDF to a Single Imagecited by 9
  21. 2025Diff2I2P: Differentiable Image-to-Point Cloud Registration with Diffusion Priorcited by 8
  22. 2023Probabilistic Triangulation for Uncalibrated Multi-View 3D Human Pose Estimationcited by 7
  23. 2024HandDGP: Camera-Space Hand Mesh Prediction with Differentiable Global Positioningcited by 7
  24. 2021Occlusion-Robust Object Pose Estimation with Holistic Representationcited by 6
  25. 2022Exploiting Problem Structure in Deep Declarative Networks: Two Case Studiescited by 6
  26. 2023Linear-Covariance Loss for End-to-End Learning of 6D Pose Estimationcited by 6
  27. 2023Deep Declarative Dynamic Time Warping for End-to-End Learning of Alignment Pathscited by 5
  28. 2024QCQP-Net: Reliably Learning Feasible Alternating Current Optimal Power Flow Solutions Under Constraintscited by 5
  29. 2022Be Careful with Rotation: A Uniform Backdoor Pattern for 3D Shapecited by 4
  30. 2022Rigidity Preserving Image Transformations and Equivariance in Perspectivecited by 3
  31. 2023YOLOPose V2: Understanding and Improving Transformer-based 6D Pose Estimationcited by 3
  32. 2024WALT3D: Generating Realistic Training Data from Time-Lapse Imagery for Reconstructing Dynamic Objects under Occlusioncited by 3
  33. 2024Test-Time Certifiable Self-Supervision to Bridge the Sim2Real Gap in Event-Based Satellite Pose Estimationcited by 3
  34. 2024Improving 2D-3D Dense Correspondences with Diffusion Models for 6D Object Pose Estimationcited by 2
  35. 2024UAV (Unmanned Aerial Vehicles): Diverse Applications of UAV Datasets in Segmentation, Classification, Detection, and Trackingcited by 2
  36. 2022EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimationcited by 1
  37. 2024MinBackProp -- Backpropagating through Minimal Solverscited by 1
  38. 2024From Variance to Veracity: Unbundling and Mitigating Gradient Variance in Differentiable Bundle Adjustment Layerscited by 1

50 most cited of 65 All 65 (JSON)