mlsanityData to 7 Oct 2026

Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model

arXiv 2311.06214 ↗First posted 10 Nov 2023cs.CVcited by 376

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Text-to-3D with diffusion models has achieved remarkable progress in recent years. However, existing methods either rely on score distillation-based optimization which suffer from slow inference, low diversity and Janus problems, or are feed-forward methods that generate low-quality results due to the scarcity of 3D training data. In this paper, we propose Instant3D, a novel method that generates high-quality and diverse 3D assets from text prompts in a feed-forward manner. We adopt a two-stage paradigm, which first generates a sparse set of four structured and consistent views from text in one shot with a fine-tuned 2D text-to-image diffusion model, and then directly regresses the NeRF from the generated images with a novel transformer-based sparse-view reconstructor. Through extensive experiments, we demonstrate that our method can generate diverse 3D assets of high visual quality within 20 seconds, which is two orders of magnitude faster than previous optimization-based methods that can take 1 to 10 hours. Our project webpage: https://jiahao.ai/instant3d/.

Key facts. Cited by 376 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 77 of its 90 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

77 of 90 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023AutoDecoding Latent 3D Diffusion Models7 Jul
  2. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  3. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  4. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  5. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction13 Apr
  6. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  7. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  8. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  9. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  10. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  11. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  12. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
1 more
  1. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  13. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  14. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  15. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  16. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  17. 2017Mixed Precision Trainingcited by 1,135
  18. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  19. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  20. 2016Pixel Recurrent Neural Networkscited by 814
  21. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  22. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  23. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  24. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  25. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  26. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  27. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  28. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  29. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  30. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  31. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  32. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  33. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  34. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  35. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  36. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  37. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  38. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 180; 2025: 124; 2026: 58.

376 papers.

Sort:

By area:

  1. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  2. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  3. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  4. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  5. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  6. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 359
  7. 2024GS-LRM: Large Reconstruction Model for 3D Gaussian Splattingcited by 344
  8. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  9. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  10. 2024TripoSR: Fast 3D Object Reconstruction from a Single Imagecited by 227
  11. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
  12. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
38 more
  1. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
  2. 2025TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Modelscited by 204
  3. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  4. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  5. 2025FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Viewscited by 176
  6. 2023PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Predictioncited by 165
  7. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  8. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  9. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  10. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 143
  11. 2024MeshAnything: Artist-Created Mesh Generation with Autoregressive Transformerscited by 140
  12. 2024MeshLRM: Large Reconstruction Model for High-Quality Meshescited by 138
  13. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  14. 2024CraftsMan3D: High-fidelity Mesh Generation with 3D Native Generation and Interactive Geometry Refinercited by 120
  15. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  16. 2024Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Imagecited by 113
  17. 2024Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splatscited by 111
  18. 2024MV-Adapter: Multi-view Consistent Image Generation Made Easycited by 111
  19. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generationcited by 104
  20. 2025Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attentioncited by 103
  21. 2024MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh Tokenizationcited by 102
  22. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 100
  23. 2024Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attentioncited by 99
  24. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  25. 2024Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoderscited by 93
  26. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
  27. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  28. 2024LLaMA-Mesh: Unifying 3D Mesh Generation with Language Modelscited by 87
  29. 2024Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scalecited by 87
  30. 2024IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generationcited by 84
  31. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 84
  32. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 81
  33. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  34. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  35. 2025SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modelingcited by 76
  36. 2024Scaling Mesh Generation via Compressive Tokenizationcited by 71
  37. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  38. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 66

50 most cited of 376 All 376 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesis
  2. 2023EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior
  3. 2024BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion
  4. 2024DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data
  5. 2024Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode Multi-view Latent Diffusion
  6. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models
  7. 2024Grounded Compositional and Diverse Text-to-3D with Pretrained Multi-View Diffusion Model
  8. 2023TextMesh: Generation of Realistic 3D Meshes From Text Prompts