mlsanityData to 7 Oct 2026

ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models

arXiv 2403.01807 ↗First posted 4 Mar 2024cs.CVcited by 57

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

3D asset generation is getting massive amounts of attention, inspired by the recent success of text-guided 2D content creation. Existing text-to-3D methods use pretrained text-to-image diffusion models in an optimization problem or fine-tune them on synthetic data, which often results in non-photorealistic 3D objects without backgrounds. In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data. Concretely, we propose to integrate 3D volume-rendering and cross-frame-attention layers into each block of the existing U-Net network of the text-to-image model. Moreover, we design an autoregressive generation that renders more 3D-consistent images at any viewpoint. We train our model on real-world datasets of objects and showcase its capabilities to generate instances with a variety of high-quality shapes and textures in authentic surroundings. Compared to the existing methods, the results generated by our method are consistent, and have favorable visual quality (-30% FID, -37% KID).

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 59 of its 64 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

59 of 64 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  8. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  9. 2018Demystifying MMD GANscited by 1,268
  10. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  11. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  13. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  14. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  15. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  16. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  17. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  18. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  19. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  20. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  21. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  22. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  23. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  24. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
  25. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  26. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  27. 2022Novel View Synthesis with Diffusion Modelscited by 299
  28. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  29. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  30. 2018Learning Blind Video Temporal Consistencycited by 247
  31. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  32. 2021NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Videocited by 241
  33. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  34. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 215
  35. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  36. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
  37. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generationcited by 162
  38. 2023TextMesh: Generation of Realistic 3D Meshes From Text Promptscited by 135

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 25; 2026: 7.

57 papers.

Sort:

By area:

  1. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  2. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  3. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  4. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  5. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  6. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  7. 2024World-consistent Video Diffusion with Explicit 3D Modelingcited by 62
  8. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  9. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  10. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
  11. 2024MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Modelcited by 31
  12. 2024MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editingcited by 30
38 more
  1. 2025MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusioncited by 28
  2. 2024Novel View Extrapolation with Video Diffusion Priorscited by 23
  3. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
  4. 2025IntrinsiX: High-Quality PBR Generation using Image Priorscited by 18
  5. 2024Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representationcited by 17
  6. 2025UnCommon Objects in 3Dcited by 17
  7. 2025Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Modelscited by 17
  8. 2024SplatFormer: Point Transformer for Robust 3D Gaussian Splattingcited by 15
  9. 2024Customizing Text-to-Image Diffusion with Object Viewpoint Controlcited by 13
  10. 20243DEnhancer: Consistent Multi-View Diffusion for 3D Enhancementcited by 12
  11. 2024Diffusion Models in 3D Vision: A Surveycited by 11
  12. 2025LaRender: Training-Free Occlusion Control in Image Generation via Latent Renderingcited by 11
  13. 2024Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Modelscited by 10
  14. 2024Edify 3D: Scalable High-Quality 3D Asset Generationcited by 10
  15. 2024CPA: Camera-pose-awareness Diffusion Transformer for Video Generationcited by 10
  16. 2025Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Modelscited by 10
  17. 2025Gen3DEval: Using vLLMs for Automatic Evaluation of Generated 3D Objectscited by 10
  18. 2026One2Scene: Geometric Consistent Explorable 3D Scene Generation from a Single Imagecited by 10
  19. 2024A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervisioncited by 9
  20. 2025VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Controlcited by 9
  21. 2025CUPID: Generative 3D Reconstruction via Joint Object and Pose Modelingcited by 8
  22. 2026World Reconstruction From Inconsistent Viewscited by 8
  23. 2024MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creationcited by 7
  24. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
  25. 2026ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videoscited by 6
  26. 2024Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generationcited by 5
  27. 2025GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Designcited by 5
  28. 2024Jointly Generating Multi-view Consistent PBR Textures using Collaborative Controlcited by 4
  29. 2024Meta 3D Gencited by 3
  30. 2025F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Aggregative Gaussian Splattingcited by 3
  31. 2025Generative Refocusing: Flexible Defocus Control from a Single Imagecited by 3
  32. 2024Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospectscited by 2
  33. 2025Refining Few-Step Text-to-Multiview Diffusion via Reinforcement Learningcited by 2
  34. 2025Complete Gaussian Splats from a Single Image with Denoising Diffusion Modelscited by 2
  35. 2025MVCustom: Multi-View Customized Diffusion via Geometric Latent Rendering and Completioncited by 2
  36. 2025Dream-to-Recon: Monocular 3D Reconstruction with Diffusion-Depth Distillation from Single Imagescited by 1
  37. 2025Consistent text-to-image generation via scene de-contextualizationcited by 1
  38. 2025Grab-3D: Detecting AI-Generated Videos from 3D Geometric Temporal Consistencycited by 1

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode Multi-view Latent Diffusion
  2. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesis
  3. 2024GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation
  4. 2024VividDreamer: Towards High-Fidelity and Efficient Text-to-3D Generation
  5. 2024VideoMV: Consistent Multi-View Generation Based on Large Video Generative Model
  6. 2024BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion
  7. 2024Hunyuan3D 1.0: A Unified Framework for Text-to-3D and Image-to-3D Generation
  8. 20223DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models