# Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion

arXiv: https://arxiv.org/abs/2311.15980 · HTML: https://mlsanity.com/paper/2311.15980/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [SyncDreamer: Generating Multiview-consistent Images from a Single-view Image](https://mlsanity.com/paper/2309.03453/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference](https://mlsanity.com/paper/1902.10556/) (2019)
- [Text2Tex: Text-driven Texture Synthesis via Diffusion Models](https://mlsanity.com/paper/2303.11396/) (2023)

## Built on by (most cited first, 30 of 50)

- [CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model](https://mlsanity.com/paper/2403.05034/) (2024)
- [Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer](https://mlsanity.com/paper/2405.14832/) (2024)
- [Direct3D-S2: Gigascale 3D Generation Made Easy with Spatial Sparse Attention](https://mlsanity.com/paper/2505.17412/) (2025)
- [STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians](https://mlsanity.com/paper/2403.14939/) (2024)
- [Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models](https://mlsanity.com/paper/2405.16645/) (2024)
- [Hi3DGen: High-fidelity 3D Geometry Generation from Images via Normal Bridging](https://mlsanity.com/paper/2503.22236/) (2025)
- [V3D: Video Diffusion Models are Effective 3D Generators](https://mlsanity.com/paper/2403.06738/) (2024)
- [Instant3dit: Multiview Inpainting for Fast Editing of 3D Objects](https://mlsanity.com/paper/2412.00518/) (2024)
- [Matrix3D: Large Photogrammetry Model All-in-One](https://mlsanity.com/paper/2502.07685/) (2025)
- [Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels](https://mlsanity.com/paper/2405.16822/) (2024)
- [MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusion](https://mlsanity.com/paper/2503.10289/) (2025)
- [Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention](https://mlsanity.com/paper/2410.10774/) (2024)
- [MVGamba: Unify 3D Content Generation as State Space Sequence Modeling](https://mlsanity.com/paper/2406.06367/) (2024)
- [MVD2: Efficient Multiview 3D Reconstruction for Multiview Diffusion](https://mlsanity.com/paper/2402.14253/) (2024)
- [BrightDreamer: Generic 3D Gaussian Generative Framework for Fast Text-to-3D Synthesis](https://mlsanity.com/paper/2403.11273/) (2024)
- [Simulating the Real World: A Unified Survey of Multimodal Generative Models](https://mlsanity.com/paper/2503.04641/) (2025)
- [A Survey On Text-to-3D Contents Generation In The Wild](https://mlsanity.com/paper/2405.09431/) (2024)
- [Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation](https://mlsanity.com/paper/2503.01370/) (2025)
- [Edit360: 2D Image Edits to 3D Assets from Any Angle](https://mlsanity.com/paper/2506.10507/) (2025)
- [Diffusion Models in 3D Vision: A Survey](https://mlsanity.com/paper/2410.04738/) (2024)
- [Auto-Regressively Generating Multi-View Consistent Images](https://mlsanity.com/paper/2506.18527/) (2025)
- [MVGBench: Comprehensive Benchmark for Multi-view Generation Models](https://mlsanity.com/paper/2507.00006/) (2025)
- [Splat4D: Diffusion-Enhanced 4D Gaussian Splatting for Temporally and Spatially Consistent Content Creation](https://mlsanity.com/paper/2508.07557/) (2025)
- [Wonder3D++: Cross-domain Diffusion for High-fidelity 3D Generation from a Single Image](https://mlsanity.com/paper/2511.01767/) (2025)
- [Text-guided Controllable Mesh Refinement for Interactive 3D Modeling](https://mlsanity.com/paper/2406.01592/) (2024)
- [Hyper3D: Efficient 3D Representation via Hybrid Triplane and Octree Feature for Enhanced 3D Shape Variational Auto-Encoders](https://mlsanity.com/paper/2503.10403/) (2025)
- [GECO: Generative Image-to-3D within a SECOnd](https://mlsanity.com/paper/2405.20327/) (2024)
- [CHROME: Clothed Human Reconstruction with Occlusion-Resilience and Multiview-Consistency from a Single Image](https://mlsanity.com/paper/2503.15671/) (2025)
- [DiMeR: Disentangled Mesh Reconstruction Model](https://mlsanity.com/paper/2504.17670/) (2025)
- [StdGEN: Semantic-Decomposed 3D Character Generation from Single Images](https://mlsanity.com/paper/2411.05738/) (2024)

## Abstract (arXiv)

Recent advances in generative AI have unveiled significant potential for the creation of 3D content. However, current methods either apply a pre-trained 2D diffusion model with the time-consuming score distillation sampling (SDS), or a direct 3D diffusion model trained on limited 3D data losing generation diversity. In this work, we approach the problem by employing a multi-view 2.5D diffusion fine-tuned from a pre-trained 2D diffusion model. The multi-view 2.5D diffusion directly models the structural distribution of 3D data, while still maintaining the strong generalization ability of the original 2D diffusion model, filling the gap between 2D diffusion-based and direct 3D diffusion-based methods for 3D content generation. During inference, multi-view normal maps are generated using the 2.5D diffusion, and a novel differentiable rasterization scheme is introduced to fuse the almost consistent multi-view normal maps into a consistent 3D model. We further design a normal-conditioned multi-view image generation module for fast appearance generation given the 3D geometry. Our method is a one-pass diffusion process and does not require any SDS optimization as post-processing. We demonstrate through extensive experiments that, our direct 2.5D generation with the specially-designed fusion scheme can achieve diverse, mode-seeking-free, and high-fidelity 3D content generation in only 10 seconds. Project page: https://nju-3dv.github.io/projects/direct25.