mlsanityData to 14 Oct 2026

Learning 3D Object Spatial Relationships from Pre-trained 2D Diffusion Models

arXiv 2503.19914 ↗First posted 25 Mar 2025cs.CVcited by 5

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present a method for learning 3D spatial relationships between object pairs, referred to as object-object spatial relationships (OOR), by leveraging synthetically generated 3D samples from pre-trained 2D diffusion models. We hypothesize that images synthesized by 2D diffusion models inherently capture realistic OOR cues, enabling efficient collection of a 3D dataset to learn OOR for various unbounded object categories. Our approach synthesizes diverse images that capture plausible OOR cues, which we then uplift into 3D samples. Leveraging our diverse collection of 3D samples for the object pairs, we train a score-based OOR diffusion model to learn the distribution of their relative spatial relationships. Additionally, we extend our pairwise OOR to multi-object OOR by enforcing consistency across pairwise relations and preventing object collisions. Extensive experiments demonstrate the robustness of our method across various object-object spatial relationships, along with its applicability to 3D scene arrangement tasks and human motion synthesis using our OOR diffusion model.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 59 of its 83 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

59 of 83 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion18 Mar
  2. 2023DPoser: Diffusion Model as Robust 3D Human Pose Prior9 Dec
  3. 2023GenPose: Generative Category-level Object Pose Estimation via Diffusion Models18 Jun

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2023Segment Anythingcited by 8,234
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
38 more
  1. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
  3. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
  4. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  5. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  6. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  7. 2023Vision Transformers Need Registerscited by 871
  8. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  9. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  10. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  11. 2023Segment Anything in High Qualitycited by 376
  12. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 301
  13. 2023Visual Geometry Grounded Deep Structure From Motioncited by 227
  14. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 200
  15. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 153
  16. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 130
  17. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  18. 2024Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detectioncited by 122
  19. 2023GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphscited by 97
  20. 2018LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Imagecited by 95
  21. 2023Telling Left from Right: Identifying Geometry-Aware Semantic Correspondencecited by 86
  22. 2023Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregationcited by 82
  23. 2016IM2CADcited by 78
  24. 2021ROCA: Robust CAD Model Retrieval and Alignment from a Single Imagecited by 67
  25. 2022GFPose: Learning 3D Human Pose Prior with Gradient Fieldscited by 57
  26. 2024ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactionscited by 57
  27. 2022Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Modelscited by 54
  28. 2023DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Imagecited by 48
  29. 2024SceneTeller: Language-to-3D Scene Generationcited by 43
  30. 2017RoomNet: End-to-End Room Layout Estimationcited by 38
  31. 2023Improving Semantic Correspondence with Viewpoint-Guided Spherical Mapscited by 34
  32. 2020Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videoscited by 33
  33. 2024Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Modelscited by 32
  34. 2021Scene Synthesis via Uncertainty-Driven Attribute Synchronizationcited by 29
  35. 2022SPARC: Sparse Render-and-Compare for CAD model alignment in a single RGB imagecited by 19
  36. 2023CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Imagescited by 19
  37. 2023Confronting Ambiguity in 6D Object Pose Estimation via Score-Based Diffusion on SE(3)cited by 17
  38. 2024SceneMotifCoder: Example-driven Visual Program Learning for Generating 3D Object Arrangementscited by 17

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generationcited by 3
  2. 2025Learning to Generate Human-Human-Object Interactions from Textual Descriptionscited by 2
  3. 2025Human Centric General Physical Intelligence for Agile Manufacturing Automationcited by 1
  4. 2026Copy-Trasform-Paste: Zero-Shot Object-Object Alignment Guided by Vision-Language and Geometric Constraintscited by 1
  5. 2026GIF: Agentic Generation of Interactive and Functional Object Compositions for Robot Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Pair2Scene: Learning Local Object Relations for Procedural Scene Generation
  2. 2026Relational Abstractions for Spatial Reasoning with Diffusion Models
  3. 2025Video Spatial Reasoning with Object-Centric 3D Rollout
  4. 2026Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence
  5. 2026Beyond Single Object: Learning 3D Relations with Large Language Models
  6. 2025DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion
  7. 2025Learning 3D Scene Analogies with Neural Contextual Scene Maps
  8. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships