mlsanityData to 7 Oct 2026

3DP3: 3D Scene Perception via Probabilistic Programming

arXiv 2111.00312 ↗First posted 30 Oct 2021cs.CV cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

We present 3DP3, a framework for inverse graphics that uses inference in a structured generative model of objects, scenes, and images. 3DP3 uses (i) voxel models to represent the 3D shape of objects, (ii) hierarchical scene graphs to decompose scenes into objects and the contacts between them, and (iii) depth image likelihoods based on real-time graphics. Given an observed RGB-D image, 3DP3's inference algorithm infers the underlying latent 3D scene, including the object poses and a parsimonious joint parametrization of these poses, using fast bottom-up pose proposals, novel involutive MCMC updates of the scene graph structure, and, optionally, neural object detectors and pose estimators. We show that 3DP3 enables scene understanding that is aware of 3D shape, occlusion, and contact structure. Our results demonstrate that 3DP3 is more accurate at 6DoF object pose estimation from real images than deep learning baselines and shows better generalization to challenging scenes with novel viewpoints, contact, and partial observability.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 15 of its 48 references that are in the map. First posted on arXiv 30 Oct 2021.

Builds on

15 of 48 references.

References · 15

  1. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 768
  2. 2019DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusioncited by 352
  3. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  4. 2018BOP: Benchmark for 6D Object Pose Estimationcited by 254
  5. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 237
  6. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  7. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  8. 2019PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Trackingcited by 91
  9. 2019Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsensecited by 71
  10. 2018Holistic 3D Scene Parsing and Reconstruction from a Single RGB Imagecited by 69
  11. 20193D-RelNet: Joint Object and Relational Network for 3D Predictioncited by 28
  12. 2020Robust 6D Object Pose Estimation by Learning RGB-D Featurescited by 25
3 more
  1. 2013Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programscited by 10
  2. 2019GRIP: Generative Robust Inference and Perception for Semantic Robot Manipulation in Adversarial Environmentscited by 6
  3. 2019Generative Hierarchical Models for Parts, Objects, and Scenescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 8; 2024: 7; 2025: 3; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  2. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  3. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 46
  4. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  5. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 32
  6. 2022VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programmingcited by 31
  7. 2023Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologiescited by 25
  8. 2024Partially Observable Task and Motion Planning with Uncertainty and Risk Awarenesscited by 17
  9. 2023Rapid Motor Adaptation for Robotic Manipulator Armscited by 13
  10. 2022Designing Perceptual Puzzles by Differentiating Probabilistic Programscited by 12
  11. 2022ProbNeRF: Uncertainty-Aware Inference of 3D Shapes from 2D Imagescited by 12
  12. 2024Active Inference as a Model of Agencycited by 12
14 more
  1. 2022Solving the Baby Intuitions Benchmark with a Hierarchically Bayesian Theory of Mindcited by 9
  2. 2025World Modeling with Probabilistic Structure Integrationcited by 7
  3. 2022Vitruvio: 3D Building Meshes via Single Perspective Sketchescited by 5
  4. 20233D Neural Embedding Likelihood: Probabilistic Inverse Graphics for Robust 6D Pose Estimationcited by 5
  5. 2025PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Taskscited by 5
  6. 2023Learning-based Relational Object Matching Across Viewscited by 4
  7. 2026Picasso: Holistic Scene Reconstruction with Physics-Constrained Samplingcited by 4
  8. 2023Generating by Understanding: Neural Visual Generation with Logical Symbol Groundingscited by 2
  9. 2024Bayesian Inverse Graphics for Few-Shot Concept Learningcited by 2
  10. 2024Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Imagecited by 1
  11. 2024Robust Bayesian Scene Reconstruction with Retrieval-Augmented Priors for Precise Grasping and Planningcited by 0
  12. 2025CoL3D: Collaborative Learning of Single-view Depth and Camera Intrinsics for Metric 3D Shape Recoverycited by 0
  13. 2026GenMatter: Perceiving Physical Objects with Generative Matter Modelscited by 0
  14. 2026AIA: A 16nm Multicore SoC for Approximate Inference Acceleration Exploiting Non-normalized Knuth-Yao Sampling and Inter-Core Register Sharingcited by 0