ZeroForge: Feedforward Text-to-Shape Without 3D Supervision
Not on the map. Why?
Abstract (arXiv)
Current state-of-the-art methods for text-to-shape generation either require supervised training using a labeled dataset of pre-defined 3D shapes, or perform expensive inference-time optimization of implicit neural representations. In this work, we present ZeroForge, an approach for zero-shot text-to-shape generation that avoids both pitfalls. To achieve open-vocabulary shape generation, we require careful architectural adaptation of existing feed-forward approaches, as well as a combination of data-free CLIP-loss and contrastive losses to avoid mode collapse. Using these techniques, we are able to considerably expand the generative ability of existing feed-forward text-to-shape models such as CLIP-Forge. We support our method via extensive qualitative and quantitative evaluations
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 14 Jun 2023.
Builds on
41 of 50 references.
References · 41
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2017Neural Discrete Representation Learningcited by 4,650
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
- 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
29 more
- 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
- 2016Density estimation using Real NVPcited by 2,029
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
- 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
- 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
- 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
- 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
- 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
- 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
- 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
- 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
- 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 267
- 2023Multimodal Deep Learningcited by 257
- 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
- 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
- 2019Real-time Scene Text Detection with Differentiable Binarizationcited by 212
- 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
- 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
- 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
- 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
- 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
- 2020Learning Visual Representations with Caption Annotationscited by 123
- 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
- 2018Point Cloud GANcited by 79
- 2019Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Toolcited by 20
- 2021Parts2Words: Learning Joint Embedding of Point Clouds and Texts by Bidirectional Matching between Parts and Wordscited by 14
- 2019An Acceleration Framework for High Resolution Image Synthesiscited by 6
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.