mlsanityData to 14 Oct 2026

Zero-Shot Novel Depth Synthesis Using 3D Foundation Models Scene Representations

arXiv 2609.04174 ↗First posted 3 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-foward transformers. The scene representations learned by these models enable strong performance on multiple 3D vision tasks. In this paper, we investigate using their internal representations to infer 3D in the scene from new views. Our hypothesis is that in order to solve the task of 3D reconstruction, these models need to learn a representation that includes a large amount of general knowledge about 3D scenes. After showing that it is possible to decode hidden surfaces from internal 3DFM representations, we propose a method, Z3D, that estimates pointmaps in unseen views by doing latent diffusion on 3DFM representation. We show that Z3D can predict realistic depth maps for new views across multiple datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

47 of 59 references.

References · 47

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2017Neural Discrete Representation Learningcited by 4,712
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  9. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  10. 2021Vision Transformers for Dense Predictioncited by 1,803
  11. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  12. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
35 more
  1. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  2. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  3. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  5. 2024Grounding Image Matching in 3D with MASt3Rcited by 983
  6. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  7. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  8. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  9. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 615
  10. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  11. 2025Continuous 3D Perception Model with Persistent Statecited by 548
  12. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
  13. 2025π3: Permutation-Equivariant Visual Geometry Learningcited by 319
  14. 2021Neural RGB-D Surface Reconstructioncited by 313
  15. 2025Diffusion Transformers with Representation Autoencoderscited by 295
  16. 20243D Reconstruction with Spatial Memorycited by 277
  17. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  18. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 227
  19. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 216
  20. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 189
  21. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  22. 2023PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustmentcited by 147
  23. 2024Cameras as Rays: Pose Estimation via Ray Diffusioncited by 131
  24. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimationcited by 116
  25. 2023Towards Zero-Shot Scale-Aware Monocular Depth Estimationcited by 101
  26. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 77
  27. 2025Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priorscited by 72
  28. 2025WorldMirror: Universal 3D World Reconstruction with Any-Prior Promptingcited by 67
  29. 2025Pixel-Perfect Depth with Semantics-Prompted Diffusion Transformerscited by 51
  30. 2022RUST: Latent Neural Scene Representations from Unposed Imagerycited by 43
  31. 2023Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Modelcited by 29
  32. 2022Depth Field Networks for Generalizable Multi-view Scene Representationcited by 17
  33. 2025Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusioncited by 12
  34. 2024GRIN: Zero-Shot Metric Depth with Pixel-Level Diffusioncited by 6
  35. 2025Emergent Extreme-View Geometry in 3D Foundation Modelscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning A Zero-shot Occupancy Network from Vision Foundation Models via Self-supervised Adaptation
  2. 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective
  3. 2025Dream-to-Recon: Monocular 3D Reconstruction with Diffusion-Depth Distillation from Single Images
  4. 2024VI3DRM:Towards meticulous 3D Reconstruction from Sparse Views via Photo-Realistic Novel View Synthesis
  5. 2023MVDD: Multi-View Depth Diffusion Models
  6. 2023Zero-1-to-3: Zero-shot One Image to 3D Object
  7. 2023Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day
  8. 2023iNVS: Repurposing Diffusion Inpainters for Novel View Synthesis