SeMv-3D: Towards Concurrency of Semantic and Multi-view Consistency in General Text-to-3D Generation
Not on the map. Why?
Abstract (arXiv)
General Text-to-3D (GT23D) generation is crucial for creating diverse 3D content across objects and scenes, yet it faces two key challenges: 1) ensuring semantic consistency between input text and generated 3D models, and 2) maintaining multi-view consistency across different perspectives within 3D. Existing approaches typically address only one of these challenges, often leading to suboptimal results in semantic fidelity and structural coherence. To overcome these limitations, we propose SeMv-3D, a novel framework that jointly enhances semantic alignment and multi-view consistency in GT23D generation. At its core, we introduce Triplane Prior Learning (TPL), which effectively learns triplane priors by capturing spatial correspondences across three orthogonal planes using a dedicated Orthogonal Attention mechanism, thereby ensuring geometric consistency across viewpoints. Additionally, we present Prior-based Semantic Aligning in Triplanes (SAT), which enables consistent any-view synthesis by leveraging attention-based feature alignment to reinforce the correspondence between textual semantics and triplane representations. Extensive experiments demonstrate that our method sets a new state-of-the-art in multi-view consistency, while maintaining competitive performance in semantic consistency compared to methods focused solely on semantic alignment. These results emphasize the remarkable ability of our approach to effectively balance and excel in both dimensions, establishing a new benchmark in the field.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 59 of its 81 references that are in the map. First posted on arXiv 10 Oct 2024.
Builds on
59 of 81 references.
References · 59
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023GPT-4 Technical Reportcited by 18,132
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
- 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
- 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
- 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
38 more
- 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
- 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
- 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
- 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
- 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
- 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
- 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
- 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
- 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
- 2023HexPlane: A Fast Representation for Dynamic Scenescited by 582
- 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
- 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 552
- 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
- 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
- 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
- 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
- 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
- 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
- 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
- 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
- 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 256
- 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
- 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
- 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
- 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
- 20233DGen: Triplane Latent Diffusion for Textured Mesh Generationcited by 181
- 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
- 2023TextMesh: Generation of Realistic 3D Meshes From Text Promptscited by 135
- 2023Vox-E: Text-guided Voxel Editing of 3D Objectscited by 114
- 2024Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attentioncited by 99
- 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
- 2023SKED: Sketch-guided Text-based 3D Editingcited by 72
- 20243DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priorscited by 72
- 2024Advances in 3D Generation: A Surveycited by 62
- 2024SPAD : Spatially Aware Multiview Diffuserscited by 62
- 2023Control3D: Towards Controllable Text-to-3D Generationcited by 46
- 2023Instant3D: Instant Text-to-3D Generationcited by 45
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.