mlsanityData to 14 Oct 2026

PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion

arXiv 2312.09069 ↗First posted 14 Dec 2023cs.CVcited by 20

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Diffusion models trained on large-scale text-image datasets have demonstrated a strong capability of controllable high-quality image generation from arbitrary text prompts. However, the generation quality and generalization ability of 3D diffusion models is hindered by the scarcity of high-quality and large-scale 3D datasets. In this paper, we present PI3D, a framework that fully leverages the pre-trained text-to-image diffusion models' ability to generate high-quality 3D shapes from text prompts in minutes. The core idea is to connect the 2D and 3D domains by representing a 3D shape as a set of Pseudo RGB Images. We fine-tune an existing text-to-image diffusion model to produce such pseudo-images using a small number of text-3D pairs. Surprisingly, we find that it can already generate meaningful and consistent 3D shapes given complex text descriptions. We further take the generated shapes as the starting point for a lightweight iterative refinement using score distillation sampling to achieve high-quality generation under a low budget. PI3D generates a single 3D shape from text in only 3 minutes and the quality is validated to outperform existing 3D generative models by a large margin.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 48 of its 57 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

48 of 57 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  2. 2023Large-Vocabulary 3D Diffusion Model with Transformer14 Sep
  3. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  4. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  5. 2023HoloFusion: Towards Photo-realistic 3D Generative Modeling28 Aug
  6. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  7. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  8. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  9. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Images29 Mar
  10. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  11. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  12. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
4 more
  1. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  2. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  3. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  4. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
20 more
  1. 2022Video Diffusion Modelscited by 2,305
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  3. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  4. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  5. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  7. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  8. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  9. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  10. 2020Convolutional Occupancy Networkscited by 761
  11. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  12. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 460
  13. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  14. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
  15. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 288
  16. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  17. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 212
  18. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  19. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  20. 2022Neural Wavelet-domain Diffusion for 3D Shape Generationcited by 132

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 8; 2026: 2.

20 papers.

Sort:

By area:

  1. 2024View Selection for 3D Captioning via Diffusion Rankingcited by 32
  2. 2024ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillationcited by 27
  3. 2024Collaborative Control for Geometry-Conditioned PBR Image Generationcited by 22
  4. 20243DSceneEditor: Controllable 3D Scene Editing with Gaussian Splattingcited by 22
  5. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 15
  6. 2025Token Perturbation Guidance for Diffusion Modelscited by 15
  7. 2024A Survey On Text-to-3D Contents Generation In The Wildcited by 14
  8. 2025Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generationcited by 14
  9. 2025MeshCraft: Exploring Efficient and Controllable Mesh Generation with Flow-based DiTscited by 14
  10. 2024Envision3D: One Image to 3D with Anchor Views Interpolationcited by 12
  11. 2024DiffTF++: 3D-aware Diffusion Transformer for Large-Vocabulary 3D Generationcited by 9
  12. 2024DreamCraft3D++: Efficient Hierarchical 3D Generation with Multi-Plane Reconstruction Modelcited by 7
8 more
  1. 2025Repurposing 2D Diffusion Models with Gaussian Atlas for 3D Generationcited by 7
  2. 2025Progressive Rendering Distillation: Adapting Stable Diffusion for Instant Text-to-Mesh Generation without 3D Datacited by 7
  3. 2024Direct and Explicit 3D Generation from a Single Imagecited by 6
  4. 2025Bridging Geometry-Coherent Text-to-3D Generation with Multi-View Diffusion Priors and Gaussian Splattingcited by 1
  5. 2025Repurposing 2D Diffusion Models for 3D Shape Completioncited by 1
  6. 2026Fake3DGS: A Benchmark for 3D Manipulation Detection in Neural Renderingcited by 1
  7. 2024Fused-Planes: Why Train a Thousand Tri-Planes When You Can Share?cited by 0
  8. 2026ForgeDreamer: Industrial Text-to-3D Generation with Multi-Expert LoRA and Cross-View Hypergraphcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior
  2. 2024DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data
  3. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts
  4. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
  5. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models
  6. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesis
  7. 2026Block3D: Efficient Text-to-3D Generation via Block-Wise Diffusion
  8. 2023T3Bench: Benchmarking Current Progress in Text-to-3D Generation