mlsanityData to 7 Oct 2026

Enhancing Diffusion Models with 3D Perspective Geometry Constraints

arXiv 2312.00944 ↗First posted 1 Dec 2023cs.CV cs.GRcited by 16

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

While perspective is a well-studied topic in art, it is generally taken for granted in images. However, for the recent wave of high-quality image synthesis methods such as latent diffusion models, perspective accuracy is not an explicit requirement. Since these methods are capable of outputting a wide gamut of possible images, it is difficult for these synthesized images to adhere to the principles of linear perspective. We introduce a novel geometric constraint in the training process of generative models to enforce perspective accuracy. We show that outputs of models trained with this constraint both appear more realistic and improve performance of downstream models trained on generated images. Subjective human trials show that images generated with latent diffusion models trained with our constraint are preferred over images from the Stable Diffusion V2 model 70% of the time. SOTA monocular depth estimation models such as DPT and PixelFormer, fine-tuned on our images, outperform the original models trained on real images by up to 7.03% in RMSE and 19.3% in SqRel on the KITTI test set for zero-shot transfer.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 54 of its 103 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

54 of 103 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2022Generative Adversarial Networkscited by 5,603
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  3. 2021Vision Transformers for Dense Predictioncited by 1,789
  4. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  6. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  9. 2018Which Training Methods for GANs do actually Converge?cited by 672
  10. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  11. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 402
  12. 2019From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimationcited by 379
  13. 2020Virtual KITTI 2cited by 345
  14. 2016Virtual Worlds as Proxy for Multi-Object Tracking Analysiscited by 340
  15. 2022On the detection of synthetic images generated by diffusion modelscited by 282
  16. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 256
  17. 2019DIODE: A Dense Indoor and Outdoor DEpth Datasetcited by 236
  18. 2018DeepJDOT: Deep Joint Distribution Optimal Transport for Unsupervised Domain Adaptationcited by 212
  19. 2021Conditional Image Generation with Score-Based Diffusion Modelscited by 195
  20. 2017Generate To Adapt: Aligning Domains using Generative Adversarial Networkscited by 183
  21. 2015Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fieldscited by 169
  22. 2021Paint by Wordcited by 107
  23. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 107
  24. 2022Soft Diffusion: Score Matching for General Corruptionscited by 101
  25. 2022Attention Attention Everywhere: Monocular Depth Prediction with Skip Attentioncited by 92
  26. 2016Beyond Sharing Weights for Deep Domain Adaptationcited by 80
  27. 2018Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimationcited by 78
  28. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Modelscited by 63
  29. 2021Unsupervised Depth Completion with Calibrated Backprojection Layerscited by 61
  30. 2019Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Predictioncited by 44
  31. 2021Learning Topology from Synthetic Data for Unsupervised Depth Completioncited by 38
  32. 2022Lighting (In)consistency of Paint by Textcited by 35
  33. 2022Perspective (In)consistency of Paint by Textcited by 34
  34. 2018Geo-Supervised Visual Depth Predictioncited by 33
  35. 2020HoliCity: A City-Scale Data Platform for Learning Holistic 3D Structurescited by 28
  36. 2019Temporal Coherence for Active Learning in Videoscited by 24
  37. 2021UV-SLAM: Unconstrained Line-based SLAM Using Vanishing Points for Structural Mappingcited by 23
  38. 2022SemAttNet: Towards Attention-based Semantic Aware Guided Depth Completioncited by 23

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024WorDepth: Variational Language Prior for Monocular Depth Estimationcited by 34
  2. 2024Test-Time Adaptation for Depth Completioncited by 26
  3. 2024RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptionscited by 20
  4. 2025ProtoDepth: Unsupervised Continual Depth Completion with Prototypescited by 11
  5. 2025Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokenscited by 9
  6. 2025MirrorVerse: Pushing Diffusion Models to Realistically Reflect the Worldcited by 8
  7. 2024Neural Approximate Mirror Maps for Constrained Diffusion Modelscited by 7
  8. 2025Radar-Guided Polynomial Fitting for Metric Depth Estimationcited by 5
  9. 2025ETA: Energy-based Test-time Adaptation for Depth Completioncited by 4
  10. 2025ControlVP: Interactive Geometric Refinement of AI-Generated Images with Consistent Vanishing Pointscited by 1
  11. 2026VisTa3D: A Dataset and Benchmark for Thin Object Reconstruction from Vision, Tactile, and 3D Point Cloudscited by 1
  12. 2024Back2Color: Domain-Adaptive Synthetic-to-Real Monocular Depth Estimation for Dynamic Traffic Scenescited by 0
4 more
  1. 2025VGLD: Visually-Guided Linguistic Disambiguation for Monocular Depth Scale Recoverycited by 0
  2. 2026From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentationcited by 0
  3. 2026PhysReflect: Geometry and Perception Guided Diffusion for Physically-Plausible Mirror Reflectionscited by 0
  4. 2026The Camera Inside the Editor: Reading the Implicit Camera of Image Editors with Painted Calibration Patternscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Geometric Consistency Refinement for Single Image Novel View Synthesis via Test-Time Adaptation of Diffusion Models
  2. 2024Novel View Synthesis with Pixel-Space Diffusion Models
  3. 2026Video Generative Models as Geometry Learner
  4. 2026Repurposing Geometric Foundation Models for Multi-view Diffusion
  5. 2024Diffusion Models in Low-Level Vision: A Survey
  6. 2024Fixing the Perspective: A Critical Examination of Zero-1-to-3
  7. 2024Diffusion Models are Geometry Critics: Single Image 3D Editing Using Pre-Trained Diffusion Priors
  8. 2024Diffusion Models in 3D Vision: A Survey