# TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision

arXiv: https://arxiv.org/abs/2303.13273 · HTML: https://mlsanity.com/paper/2303.13273/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 56 references that are in the map.
- First posted on arXiv 23 Mar 2023.

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis](https://mlsanity.com/paper/2110.08985/) (2021)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation](https://mlsanity.com/paper/2112.11427/) (2021)
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)

## Built on by (most cited first, 28 of 28)

- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [Scalable 3D Captioning with Pretrained Models](https://mlsanity.com/paper/2306.07279/) (2023)
- [AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes](https://mlsanity.com/paper/2312.06644/) (2023)
- [Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era](https://mlsanity.com/paper/2305.06131/) (2023)
- [REACTO: Reconstructing Articulated Objects from a Single Video](https://mlsanity.com/paper/2404.11151/) (2024)
- [DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors](https://mlsanity.com/paper/2406.01476/) (2024)
- [DreamControl: Control-Based Text-to-3D Generation with 3D Self-Prior](https://mlsanity.com/paper/2312.06439/) (2023)
- [VolumeDiffusion: Flexible Text-to-3D Generation with Efficient Volumetric Encoder](https://mlsanity.com/paper/2312.11459/) (2023)
- [CADCrafter: Generating Computer-Aided Design Models from Unconstrained Images](https://mlsanity.com/paper/2504.04753/) (2025)
- [View Selection for 3D Captioning via Diffusion Ranking](https://mlsanity.com/paper/2404.07984/) (2024)
- [Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior](https://mlsanity.com/paper/2403.09140/) (2024)
- [EXIM: A Hybrid Explicit-Implicit Representation for Text-Guided 3D Shape Generation](https://mlsanity.com/paper/2311.01714/) (2023)
- [MoDA: Modeling Deformable 3D Objects from Casual Videos](https://mlsanity.com/paper/2304.08279/) (2023)
- [Text-to-3D Shape Generation](https://mlsanity.com/paper/2403.13289/) (2024)
- [Single Mesh Diffusion Models with Field Latents for Texture Generation](https://mlsanity.com/paper/2312.09250/) (2023)
- [TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields](https://mlsanity.com/paper/2309.17175/) (2023)
- [X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation](https://mlsanity.com/paper/2312.00085/) (2023)
- [TPA3D: Triplane Attention for Fast Text-to-3D Generation](https://mlsanity.com/paper/2312.02647/) (2023)
- [Magic3DSketch: Create Colorful 3D Models From Sketch-Based 3D Modeling Guided by Text and Language-Image Pre-Training](https://mlsanity.com/paper/2407.19225/) (2024)
- [Deep Geometric Moments Promote Shape Consistency in Text-to-3D Generation](https://mlsanity.com/paper/2408.05938/) (2024)
- [TeRA: Rethinking Text-guided Realistic 3D Avatar Generation](https://mlsanity.com/paper/2509.02466/) (2025)
- [UV-free Texture Generation with Denoising and Geodesic Heat Diffusions](https://mlsanity.com/paper/2408.16762/) (2024)
- [GenUDC: High Quality 3D Mesh Generation with Unsigned Dual Contouring Representation](https://mlsanity.com/paper/2410.17802/) (2024)
- [Magic-Boost: Boost 3D Generation with Multi-View Conditioned Diffusion](https://mlsanity.com/paper/2404.06429/) (2024)
- [Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation](https://mlsanity.com/paper/2312.11774/) (2023)
- [HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generation](https://mlsanity.com/paper/2407.14419/) (2024)
- [DecompDreamer: A Composition-Aware Curriculum for Structured 3D Asset Generation](https://mlsanity.com/paper/2503.11981/) (2025)
- [RealCAD: Towards Real-World Image-to-CAD Reconstruction under Domain Shift and Parameter Bias](https://mlsanity.com/paper/2608.30617/) (2026)

## Abstract (arXiv)

In this paper, we investigate an open research task of generating controllable 3D textured shapes from the given textual descriptions. Previous works either require ground truth caption labeling or extensive optimization time. To resolve these issues, we present a novel framework, TAPS3D, to train a text-guided 3D shape generator with pseudo captions. Specifically, based on rendered 2D images, we retrieve relevant words from the CLIP vocabulary and construct pseudo captions using templates. Our constructed captions provide high-level semantic supervision for generated 3D shapes. Further, in order to produce fine-grained textures and increase geometry diversity, we propose to adopt low-level image regularization to enable fake-rendered images to align with the real ones. During the inference phase, our proposed model can generate 3D textured shapes from the given text without any additional optimization. We conduct extensive experiments to analyze each of our proposed components and show the efficacy of our framework in generating high-fidelity 3D textured and text-relevant shapes.