# 3D-TOGO: Towards Text-Guided Cross-Category 3D Object Generation arXiv: https://arxiv.org/abs/2212.01103 ยท HTML: https://mlsanity.com/paper/2212.01103/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 44 of its 58 references that are in the map. - First posted on arXiv 2 Dec 2022. ## Builds on (most cited first, 30 of 44) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021) - [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021) - [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020) - [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016) - [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013) - [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020) - [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019) - [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019) - [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018) - [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016) - [Autoregressive Image Generation using Residual Quantization](https://mlsanity.com/paper/2203.01941/) (2022) - [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021) - [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020) - [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017) - [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021) - [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://mlsanity.com/paper/2108.10904/) (2021) - [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016) - [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021) - [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021) - [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021) - [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021) - [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021) - [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019) ## Built on by (most cited first, 8 of 8) - [Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models](https://mlsanity.com/paper/2303.11989/) (2023) - [EXIM: A Hybrid Explicit-Implicit Representation for Text-Guided 3D Shape Generation](https://mlsanity.com/paper/2311.01714/) (2023) - [Text-Guided 3D Face Synthesis -- From Generation to Editing](https://mlsanity.com/paper/2312.00375/) (2023) - [Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priors](https://mlsanity.com/paper/2406.00687/) (2024) - [Dynamic Appearance Particle Neural Radiance Field](https://mlsanity.com/paper/2310.07916/) (2023) - [DreamDrone: Text-to-Image Diffusion Models are Zero-shot Perpetual View Generators](https://mlsanity.com/paper/2312.08746/) (2023) - [Regret Minimization in Population Network Games: Vanishing Heterogeneity and Convergence to Equilibria](https://mlsanity.com/paper/2507.17183/) (2025) - [RAGMesh with FaME-G2E: Long-Form Text-Driven 3D Face Generation and Editing](https://mlsanity.com/paper/2608.09186/) (2026) ## Abstract (arXiv) Text-guided 3D object generation aims to generate 3D objects described by user-defined captions, which paves a flexible way to visualize what we imagined. Although some works have been devoted to solving this challenging task, these works either utilize some explicit 3D representations (e.g., mesh), which lack texture and require post-processing for rendering photo-realistic views; or require individual time-consuming optimization for every single case. Here, we make the first attempt to achieve generic text-guided cross-category 3D object generation via a new 3D-TOGO model, which integrates a text-to-views generation module and a views-to-3D generation module. The text-to-views generation module is designed to generate different views of the target 3D object given an input caption. prior-guidance, caption-guidance and view contrastive learning are proposed for achieving better view-consistency and caption similarity. Meanwhile, a pixelNeRF model is adopted for the views-to-3D generation module to obtain the implicit 3D neural representation from the previously-generated views. Our 3D-TOGO model generates 3D objects in the form of the neural radiance field with good texture and requires no time-cost optimization for every single caption. Besides, 3D-TOGO can control the category, color and shape of generated 3D objects with the input caption. Extensive experiments on the largest 3D object dataset (i.e., ABO) are conducted to verify that 3D-TOGO can better generate high-quality 3D objects according to the input captions across 98 different categories, in terms of PSNR, SSIM, LPIPS and CLIP-score, compared with text-NeRF and Dreamfields.