# Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D priors

arXiv: https://arxiv.org/abs/2312.04963 · HTML: https://mlsanity.com/paper/2312.04963/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 45 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors](https://mlsanity.com/paper/2306.17843/) (2023)

## Built on by (most cited first, 24 of 24)

- [DreamReward: Text-to-3D Generation with Human Preference](https://mlsanity.com/paper/2403.14613/) (2024)
- [View Selection for 3D Captioning via Diffusion Ranking](https://mlsanity.com/paper/2404.07984/) (2024)
- [MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusion](https://mlsanity.com/paper/2503.10289/) (2025)
- [ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillation](https://mlsanity.com/paper/2407.02040/) (2024)
- [From One to More: Contextual Part Latents for 3D Generation](https://mlsanity.com/paper/2507.08772/) (2025)
- [A Survey of Sustainability in Large Language Models: Applications, Economics, and Challenges](https://mlsanity.com/paper/2412.04782/) (2024)
- [FullPart: Generating each 3D Part at Full Resolution](https://mlsanity.com/paper/2510.26140/) (2025)
- [Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy](https://mlsanity.com/paper/2412.06698/) (2024)
- [FlexiDreamer: Single Image-to-3D Generation with FlexiCubes](https://mlsanity.com/paper/2404.00987/) (2024)
- [DH-FaceVid-1K: A Large-Scale High-Quality Dataset for Face Video Generation](https://mlsanity.com/paper/2410.07151/) (2024)
- [A General Framework to Boost 3D GS Initialization for Text-to-3D Generation by Lexical Richness](https://mlsanity.com/paper/2408.01269/) (2024)
- [GENA3D: Generative Amodal 3D Modeling by Bridging 2D Priors and 3D Coherence](https://mlsanity.com/paper/2511.21945/) (2025)
- [Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control](https://mlsanity.com/paper/2410.06985/) (2024)
- [ID-NeRF: Indirect Diffusion-guided Neural Radiance Fields for Generalizable View Synthesis](https://mlsanity.com/paper/2402.01217/) (2024)
- [Magic-Boost: Boost 3D Generation with Multi-View Conditioned Diffusion](https://mlsanity.com/paper/2404.06429/) (2024)
- [ModeDreamer: Mode Guiding Score Distillation for Text-to-3D Generation using Reference Image Prompts](https://mlsanity.com/paper/2411.18135/) (2024)
- [MOC-3D: Manifold-Order Consistency for Text-to-3D Generation](https://mlsanity.com/paper/2605.01743/) (2026)
- [Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusion](https://mlsanity.com/paper/2411.10369/) (2024)
- [DreamTexture: Shape from Virtual Texture with Analysis by Augmentation](https://mlsanity.com/paper/2503.16412/) (2025)
- [DiffIER: Optimizing Diffusion Models with Iterative Error Reduction](https://mlsanity.com/paper/2508.13628/) (2025)
- [FREE: Uncertainty-Aware Autoregression for Parallel Diffusion Transformers](https://mlsanity.com/paper/2511.20390/) (2025)
- [Debiasing Diffusion Priors via 3D Attention for Consistent Gaussian Splatting](https://mlsanity.com/paper/2512.07345/) (2025)
- [Focus on Neighbors and Know the Whole: Towards Consistent Dense Multiview Text-to-Image Generator for 3D Creation](https://mlsanity.com/paper/2408.13149/) (2024)
- [CorGi: Contribution-Guided Block-Wise Interval Caching for Training-Free Acceleration of Diffusion Transformers](https://mlsanity.com/paper/2512.24195/) (2025)

## Abstract (arXiv)

Most 3D generation research focuses on up-projecting 2D foundation models into the 3D space, either by minimizing 2D Score Distillation Sampling (SDS) loss or fine-tuning on multi-view datasets. Without explicit 3D priors, these methods often lead to geometric anomalies and multi-view inconsistency. Recently, researchers have attempted to improve the genuineness of 3D objects by directly training on 3D datasets, albeit at the cost of low-quality texture generation due to the limited texture diversity in 3D datasets. To harness the advantages of both approaches, we propose Bidirectional Diffusion(BiDiff), a unified framework that incorporates both a 3D and a 2D diffusion process, to preserve both 3D fidelity and 2D texture richness, respectively. Moreover, as a simple combination may yield inconsistent generation results, we further bridge them with novel bidirectional guidance. In addition, our method can be used as an initialization of optimization-based models to further improve the quality of 3D model and efficiency of optimization, reducing the generation process from 3.4 hours to 20 minutes. Experimental results have shown that our model achieves high-quality, diverse, and scalable 3D generation. Project website: https://bidiff.github.io/.