mlsanityData to 7 Oct 2026

Retrieval-Augmented Score Distillation for Text-to-3D Generation

arXiv 2402.02972 ↗First posted 5 Feb 2024cs.CV cs.LGcited by 14

On the map: Retrieval-augmented generation → Multimodal RAG

also in 1 more
Abstract (arXiv)

Text-to-3D generation has achieved significant success by incorporating powerful 2D diffusion models, but insufficient 3D prior knowledge also leads to the inconsistency of 3D geometry. Recently, since large-scale multi-view datasets have been released, fine-tuning the diffusion model on the multi-view datasets becomes a mainstream to solve the 3D inconsistency problem. However, it has confronted with fundamental difficulties regarding the limited quality and diversity of 3D data, compared with 2D data. To sidestep these trade-offs, we explore a retrieval-augmented approach tailored for score distillation, dubbed ReDream. We postulate that both expressiveness of 2D diffusion models and geometric consistency of 3D assets can be fully leveraged by employing the semantically relevant assets directly within the optimization process. To this end, we introduce novel framework for retrieval-based quality enhancement in text-to-3D generation. We leverage the retrieved asset to incorporate its geometric prior in the variational objective and adapt the diffusion model's 2D prior toward view consistency, achieving drastic improvements in both geometry and fidelity of generated scenes. We conduct extensive experiments to demonstrate that ReDream exhibits superior quality with increased geometric consistency. Project page is available at https://ku-cvlab.github.io/ReDream/.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 52 of its 61 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

52 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model3 Apr
  2. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generator29 Sep
  3. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models26 Jul

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  12. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
37 more
  1. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  2. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  5. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  6. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  7. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  8. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  9. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  10. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  11. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  12. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 516
  13. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  14. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  15. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  16. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  17. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  18. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  19. 2022Novel View Synthesis with Diffusion Modelscited by 299
  20. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 240
  21. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  22. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  23. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  24. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  25. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  26. 2023TextMesh: Generation of Realistic 3D Meshes From Text Promptscited by 135
  27. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3Dcited by 134
  28. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  29. 2023Text-to-3D with Classifier Score Distillationcited by 127
  30. 2023Delta Denoising Scorecited by 120
  31. 2021Instance-Conditioned GANcited by 119
  32. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 99
  33. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  34. 2018A Unified Particle-Optimization Framework for Scalable Bayesian Samplingcited by 44
  35. 2019Function Space Particle Optimization for Bayesian Neural Networkscited by 41
  36. 2020RetrieveGAN: Image Synthesis via Differentiable Patch Retrievalcited by 35
  37. 2022Particle-based Variational Inference with Preconditioned Functional Gradient Flowcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 1.

14 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2025VDocRAG: Retrieval-Augmented Generation over Visually-Rich Documentscited by 57
  3. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attentioncited by 55
  4. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  5. 2025RANGE: Retrieval Augmented Neural Fields for Multi-Resolution Geo-Embeddingscited by 18
  6. 2024RAGDiffusion: Faithful Cloth Generation via External Knowledge Assimilationcited by 9
  7. 2024Benchmarking and Learning Multi-Dimensional Quality Evaluator for Text-to-3D Generationcited by 7
  8. 2025Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generationcited by 6
  9. 2025Retrievals Can Be Detrimental: Unveiling the Backdoor Vulnerability of Retrieval-Augmented Diffusion Modelscited by 5
  10. 2024Perturb-and-Revise: Flexible 3D Editing with Generative Trajectoriescited by 2
  11. 2025Towards Fine-Grained Text-to-3D Quality Assessment: A Benchmark and A Two-Stage Rank-Learning Metriccited by 2
  12. 2025Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generationcited by 1
2 more
  1. 2025MV-RAG: Retrieval Augmented Multiview Diffusioncited by 0
  2. 2026RAGMesh with FaME-G2E: Long-Form Text-Driven 3D Face Generation and Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Text-to-3D Generation with Linearized Lookahead Variational Score Distillation
  2. 2025Bridging Geometry-Coherent Text-to-3D Generation with Multi-View Diffusion Priors and Gaussian Splatting
  3. 2024PlacidDreamer: Advancing Harmony in Text-to-3D Generation
  4. 2024Geometry-Aware Score Distillation via 3D Consistent Noising and Gradient Consistency Modeling
  5. 2025Dive3D: Diverse Distillation-based Text-to-3D Generation via Score Implicit Matching
  6. 2024Semantic Score Distillation Sampling for Compositional Text-to-3D Generation
  7. 2023Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation
  8. 2024BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion