# A General Framework to Boost 3D GS Initialization for Text-to-3D Generation by Lexical Richness arXiv: https://arxiv.org/abs/2408.01269 ยท HTML: https://mlsanity.com/paper/2408.01269/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 45 references that are in the map. - First posted on arXiv 2 Aug 2024. ## Builds on (most cited first, 30 of 32) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023) - [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022) - [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023) - [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023) - [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022) - [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022) - [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021) - [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023) - [Text-to-3D using Gaussian Splatting](https://mlsanity.com/paper/2309.16585/) (2023) - [LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching](https://mlsanity.com/paper/2311.11284/) (2023) - [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024) - [RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D](https://mlsanity.com/paper/2311.16918/) (2023) - [SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D](https://mlsanity.com/paper/2310.02596/) (2023) - [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024) - [HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance](https://mlsanity.com/paper/2305.18766/) (2023) - [ATT3D: Amortized Text-to-3D Object Synthesis](https://mlsanity.com/paper/2306.07349/) (2023) - [Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior](https://mlsanity.com/paper/2401.09050/) (2024) - [CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layout](https://mlsanity.com/paper/2303.13843/) (2023) - [DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior](https://mlsanity.com/paper/2312.06439/) (2023) - [Taming Mode Collapse in Score Distillation for Text-to-3D Generation](https://mlsanity.com/paper/2401.00909/) (2023) - [VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation](https://mlsanity.com/paper/2403.17001/) (2024) - [Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D priors](https://mlsanity.com/paper/2312.04963/) (2023) - [SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity](https://mlsanity.com/paper/2401.00604/) (2023) - [Hyper-3DG: Text-to-3D Gaussian Generation via Hypergraph](https://mlsanity.com/paper/2403.09236/) (2024) - [DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling](https://mlsanity.com/paper/2311.17082/) (2023) - [Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior](https://mlsanity.com/paper/2403.09140/) (2024) ## Built on by (most cited first, 8 of 8) - [RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning](https://mlsanity.com/paper/2502.00848/) (2025) - [Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusion](https://mlsanity.com/paper/2507.04403/) (2025) - [FlowDreamer: Exploring High Fidelity Text-to-3D Generation via Rectified Flow](https://mlsanity.com/paper/2408.05008/) (2024) - [DiMeR: Disentangled Mesh Reconstruction Model](https://mlsanity.com/paper/2504.17670/) (2025) - [Flash Sculptor: Modular 3D Worlds from Objects](https://mlsanity.com/paper/2504.06178/) (2025) - [Sat2City v2: Native 3D City Asset Generation from a Single Satellite Image](https://mlsanity.com/paper/2606.24138/) (2026) - [Graph-Guided Dual-Level Augmentation for 3D Scene Segmentation](https://mlsanity.com/paper/2507.22668/) (2025) - [SAP: Segment Any 4K Panorama](https://mlsanity.com/paper/2603.12759/) (2026) ## Abstract (arXiv) Text-to-3D content creation has recently received much attention, especially with the prevalence of 3D Gaussians Splatting. In general, GS-based methods comprise two key stages: initialization and rendering optimization. To achieve initialization, existing works directly apply random sphere initialization or 3D diffusion models, e.g., Point-E, to derive the initial shapes. However, such strategies suffer from two critical yet challenging problems: 1) the final shapes are still similar to the initial ones even after training; 2) shapes can be produced only from simple texts, e.g., "a dog", not for lexically richer texts, e.g., "a dog is sitting on the top of the airplane". To address these problems, this paper proposes a novel general framework to boost the 3D GS Initialization for text-to-3D generation upon the lexical richness. Our key idea is to aggregate 3D Gaussians into spatially uniform voxels to represent complex shapes while enabling the spatial interaction among the 3D Gaussians and semantic interaction between Gaussians and texts. Specifically, we first construct a voxelized representation, where each voxel holds a 3D Gaussian with its position, scale, and rotation fixed while setting opacity as the sole factor to determine a position's occupancy. We then design an initialization network mainly consisting of two novel components: 1) Global Information Perception (GIP) block and 2) Gaussians-Text Fusion (GTF) block. Such a design enables each 3D Gaussian to assimilate the spatial information from other areas and semantic information from texts. Extensive experiments show the superiority of our framework of high-quality 3D GS initialization against the existing methods, e.g., Shap-E, by taking lexically simple, medium, and hard texts. Also, our framework can be seamlessly plugged into SoTA training frameworks, e.g., LucidDreamer, for semantically consistent text-to-3D generation.