mlsanityData to 14 Oct 2026

NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Views

arXiv 2211.16431 ↗First posted 29 Nov 2022cs.CVcited by 148

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Virtual reality and augmented reality (XR) bring increasing demand for 3D content. However, creating high-quality 3D content requires tedious work that a human expert must do. In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360deg views that correspond well with the given reference image. By conditioning on the reference image, our model can fulfill the everlasting curiosity for synthesizing novel views of objects from images. Our technique sheds light on a promising direction of easing the workflows for 3D artists and XR designers. We propose a novel framework, dubbed NeuralLift-360, that utilizes a depth-aware neural radiance representation (NeRF) and learns to craft the scene guided by denoising diffusion models. By introducing a ranking loss, our NeuralLift-360 can be guided with rough depth estimation in the wild. We also adopt a CLIP-guided sampling strategy for the diffusion prior to provide coherent guidance. Extensive experiments demonstrate that our NeuralLift-360 significantly outperforms existing state-of-the-art baselines. Project page: https://vita-group.github.io/NeuralLift-360/

Key facts. Cited by 148 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 68 of its 82 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

68 of 82 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
38 more
  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  4. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  5. 2021Vision Transformers for Dense Predictioncited by 1,803
  6. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  7. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  8. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  9. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  10. 2021Variational Diffusion Modelscited by 1,153
  11. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
  12. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  13. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  14. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  15. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 788
  16. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 740
  17. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 737
  18. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 735
  19. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 703
  20. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  21. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 673
  22. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  23. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 651
  24. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  25. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 575
  26. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  27. 2020NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesiscited by 496
  28. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  29. 2020NeRD: Neural Reflectance Decomposition from Image Collectionscited by 449
  30. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  31. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  32. 2022Understanding Diffusion Models: A Unified Perspectivecited by 340
  33. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 216
  34. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 212
  35. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  36. 2021Neural Rays for Occlusion-aware Image-based Renderingcited by 192
  37. 2016Single-Image Depth Perception in the Wildcited by 180
  38. 2022SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Imagecited by 165

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 70; 2024: 52; 2025: 14; 2026: 10.

148 papers.

Sort:

By area:

  1. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  2. 2024Depth Anything V2cited by 1,422
  3. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  4. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  5. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  6. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 645
  7. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 560
  8. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 436
  9. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  10. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 397
  11. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 343
  12. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
38 more
  1. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 257
  2. 2024DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalizationcited by 256
  3. 2023SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesiscited by 247
  4. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  5. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  6. 2023A Variational Perspective on Solving Inverse Problems with Diffusion Modelscited by 222
  7. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  8. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  9. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  10. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 166
  11. 2023SparseGS: Sparse View Synthesis using 3D Gaussian Splattingcited by 134
  12. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  13. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
  14. 2023Animate124: Animating One Image to 4D Dynamic Scenecited by 115
  15. 2023In-Context Learning Unlocked for Diffusion Modelscited by 104
  16. 2023TeCH: Text-guided Reconstruction of Lifelike Clothed Humanscited by 104
  17. 2024EscherNet: A Generative Model for Scalable View Synthesiscited by 95
  18. 2023Anything-3D: Towards Single-view Anything Reconstruction in the Wildcited by 92
  19. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 87
  20. 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 82
  21. 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Modelscited by 79
  22. 2024Advances in 3D Generation: A Surveycited by 63
  23. 2024SPAD : Spatially Aware Multiview Diffuserscited by 62
  24. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  25. 2024ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidancecited by 59
  26. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Modelscited by 58
  27. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texturecited by 53
  28. 2024AGG: Amortized Generative 3D Gaussians for Single Image to 3Dcited by 53
  29. 2024Comp4D: LLM-Guided Compositional 4D Scene Generationcited by 50
  30. 2023A Comprehensive Survey on Knowledge Distillation of Diffusion Modelscited by 45
  31. 2023PERF: Panoramic Neural Radiance Field from a Single Panoramacited by 45
  32. 2024Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Videocited by 44
  33. 2024DressCode: Autoregressively Sewing and Generating Garments from Text Guidancecited by 44
  34. 2023DITTO-NeRF: Diffusion-based Iterative Text To Omni-directional 3D Modelcited by 43
  35. 2023Text2Immersion: Generative Immersive Scene with 3D Gaussianscited by 43
  36. 2023Single-Image 3D Human Digitization with Shape-Guided Diffusioncited by 42
  37. 2024SceneCraft: Layout-Guided 3D Scene Generationcited by 41
  38. 2023Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstructioncited by 39

50 most cited of 148 All 148 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LiftRefine: Progressively Refined View Synthesis from 3D Lifting with Volume-Triplane Representations
  2. 2026360Anything: Geometry-Free Lifting of Images and Videos to 360deg
  3. 2023ReconFusion: 3D Reconstruction with Diffusion Priors
  4. 2023Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D
  5. 2024Generative Lifting of Multiview to 3D from Unknown Pose: Wrapping NeRF inside Diffusion
  6. 2026Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction In-the-Wild
  7. 2024MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views
  8. 2024Denoising Diffusion via Image-Based Rendering