# Learning Continuous 3D Words for Text-to-Image Generation

arXiv: https://arxiv.org/abs/2402.08654 · HTML: https://mlsanity.com/paper/2402.08654/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 13 Feb 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [A Neural Space-Time Representation for Text-to-Image Personalization](https://mlsanity.com/paper/2305.15391/) (2023)
- [Cones 2: Customizable Image Synthesis with Multiple Subjects](https://mlsanity.com/paper/2305.19327/) (2023)
- [Learning to generate line drawings that convey geometry and semantics](https://mlsanity.com/paper/2203.12691/) (2022)
- [DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Views](https://mlsanity.com/paper/2306.03414/) (2023)

## Built on by (most cited first, 16 of 16)

- [ZeST: Zero-Shot Material Transfer from a Single Image](https://mlsanity.com/paper/2404.06425/) (2024)
- [Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editing](https://mlsanity.com/paper/2510.08532/) (2025)
- [Customizing Text-to-Image Diffusion with Object Viewpoint Control](https://mlsanity.com/paper/2404.12333/) (2024)
- [PreciseCam: Precise Camera Control for Text-to-Image Generation](https://mlsanity.com/paper/2501.12910/) (2025)
- [Compass Control: Multi Object Orientation Control for Text-to-Image Generation](https://mlsanity.com/paper/2504.06752/) (2025)
- [Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.08434/) (2025)
- [ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generation](https://mlsanity.com/paper/2503.22194/) (2025)
- [SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation](https://mlsanity.com/paper/2511.16666/) (2025)
- [ParSEL: Parameterized Shape Editing with Language](https://mlsanity.com/paper/2405.20319/) (2024)
- [Generative Refocusing: Flexible Defocus Control from a Single Image](https://mlsanity.com/paper/2512.16923/) (2025)
- [SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation](https://mlsanity.com/paper/2602.23359/) (2026)
- [ShapeWords: Guiding Text-to-Image Synthesis with 3D Shape-Aware Prompts](https://mlsanity.com/paper/2412.02912/) (2024)
- [Pattern Analogies: Learning to Perform Programmatic Image Edits by Analogy](https://mlsanity.com/paper/2412.12463/) (2024)
- [EOPose : Exemplar-based object reposing using Generalized Pose Correspondences](https://mlsanity.com/paper/2505.03394/) (2025)
- [Thinking in Boxes: 3D Editing in Real Images Made Easy](https://mlsanity.com/paper/2606.20556/) (2026)
- [Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation](https://mlsanity.com/paper/2608.22757/) (2026)

## Abstract (arXiv)

Current controls over diffusion models (e.g., through text or ControlNet) for image generation fall short in recognizing abstract, continuous attributes like illumination direction or non-rigid shape change. In this paper, we present an approach for allowing users of text-to-image models to have fine-grained control of several attributes in an image. We do this by engineering special sets of input tokens that can be transformed in a continuous manner -- we call them Continuous 3D Words. These attributes can, for example, be represented as sliders and applied jointly with text prompts for fine-grained control over image generation. Given only a single mesh and a rendering engine, we show that our approach can be adopted to provide continuous user control over several 3D-aware attributes, including time-of-day illumination, bird wing orientation, dollyzoom effect, and object poses. Our method is capable of conditioning image creation with multiple Continuous 3D Words and text descriptions simultaneously while adding no overhead to the generative process. Project Page: https://ttchengab.github.io/continuous_3d_words