mlsanityData to 14 Oct 2026

SpaText: Spatio-Textual Representation for Controllable Image Generation

arXiv 2211.14305 ↗First posted 25 Nov 2022cs.CV cs.GR cs.LGcited by 217

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent text-to-image diffusion models are able to generate convincing results of unprecedented quality. However, it is nearly impossible to control the shapes of different regions/objects or their layout in a fine-grained fashion. Previous attempts to provide such controls were hindered by their reliance on a fixed set of labels. To this end, we present SpaText - a new method for text-to-image generation using open-vocabulary scene control. In addition to a global text prompt that describes the entire scene, the user provides a segmentation map where each region of interest is annotated by a free-form natural language description. Due to lack of large-scale datasets that have a detailed textual description for each region in the image, we choose to leverage the current large-scale text-to-image datasets and base our approach on a novel CLIP-based spatio-textual representation, and show its effectiveness on two state-of-the-art diffusion models: pixel-based and latent-based. In addition, we show how to extend the classifier-free guidance method in diffusion models to the multi-conditional case and present an alternative accelerated inference algorithm. Finally, we offer several automatic evaluation metrics and use them, in addition to FID scores and a user study, to evaluate our method and show that it achieves state-of-the-art results on image generation with free-form textual scene control.

Key facts. Cited by 217 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 77 of its 97 references that are in the map. First posted on arXiv 25 Nov 2022.

Builds on

77 of 97 references.

Closest ancestors · 11

Same direction, newest first

  1. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  2. 2022High-Resolution Image Editing via Multi-Stage Blended Diffusion24 Oct
  3. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  4. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  6. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Models26 Jul
  7. 2022Blended Latent Diffusion6 Jun
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  11. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov

Other references · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2020Denoising Diffusion Implicit Modelscited by 9,781
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  12. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  13. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  14. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  15. 2014Conditional Generative Adversarial Netscited by 2,162
  16. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  17. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  18. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  19. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  20. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  21. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  22. 2022Diffusion Models in Vision: A Surveycited by 1,015
  23. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  24. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  25. 2016Generative Adversarial Text to Image Synthesiscited by 735
  26. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  27. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  28. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  29. 2017Unsupervised Image-to-Image Translation Networkscited by 603
  30. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  31. 2022Diffusion Models already have a Semantic Latent Spacecited by 328
  32. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  33. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  34. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  35. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  36. 2019Few-Shot Unsupervised Image-to-Image Translationcited by 222
  37. 2022Re-Imagen: Retrieval-Augmented Text-to-Image Generatorcited by 210
  38. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 74; 2024: 93; 2025: 41; 2026: 9.

217 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  2. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  3. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  4. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  5. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  6. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 282
  7. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  8. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  9. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 231
  10. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  11. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  12. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 191
38 more
  1. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  2. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 164
  3. 2023Dense Text-to-Image Generation with Attention Modulationcited by 163
  4. 2023LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-Oncited by 152
  5. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  6. 2023Cross-Image Attention for Zero-Shot Appearance Transfercited by 136
  7. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 124
  8. 2024One-Step Image Translation with Text-to-Image Modelscited by 116
  9. 2024Stable Flow: Vital Layers for Training-Free Image Editingcited by 113
  10. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  11. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 109
  12. 2024Improving Diffusion Models for Authentic Virtual Try-on in the Wildcited by 109
  13. 2023Inversion-Free Image Editing with Natural Languagecited by 106
  14. 2023BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layoutcited by 102
  15. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  16. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  17. 2023FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Conditioncited by 91
  18. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  19. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  20. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 83
  21. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  22. 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 81
  23. 2023Generating images of rare concepts using pre-trained diffusion modelscited by 79
  24. 2023PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalizationcited by 77
  25. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidancecited by 69
  26. 2024Be Yourself: Bounded Attention for Multi-Subject Text-to-Image Generationcited by 69
  27. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  28. 2023Inst-Inpaint: Instructing to Remove Objects with Diffusion Modelscited by 66
  29. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 66
  30. 2023Zero-shot spatial layout conditioning for text-to-image diffusion modelscited by 64
  31. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Modelcited by 63
  32. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 63
  33. 2023Multimodal Garment Designer: Human-Centric Latent Diffusion Models for Fashion Image Editingcited by 62
  34. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  35. 2023Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3Dcited by 59
  36. 2024ControlVAR: Exploring Controllable Visual Autoregressive Modelingcited by 58
  37. 2024Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Taskscited by 48
  38. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47

50 most cited of 217 All 217 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation
  2. 2023SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis
  3. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  4. 2023SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation
  5. 2023Unlocking Spatial Comprehension in Text-to-Image Diffusion Models
  6. 2023Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model
  7. 2025Setting the Stage: Text-Driven Scene-Consistent Image Generation
  8. 2025A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models