mlsanityData to 7 Oct 2026

SEGA: Instructing Text-to-Image Models using Semantic Guidance

arXiv 2301.12247 ↗First posted 28 Jan 2023cs.CV cs.AI cs.LGcited by 4

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image diffusion models have recently received a lot of interest for their astonishing ability to produce high-fidelity images from text only. However, achieving one-shot generation that aligns with the user's intent is nearly impossible, yet small changes to the input prompt often result in very different images. This leaves the user with little semantic control. To put the user in control, we show how to interact with the diffusion process to flexibly steer it along semantic directions. This semantic guidance (SEGA) generalizes to any generative architecture using classifier-free guidance. More importantly, it allows for subtle and extensive edits, changes in composition and style, as well as optimizing the overall artistic conception. We demonstrate SEGA's effectiveness on both latent and pixel-based diffusion models such as Stable Diffusion, Paella, and DeepFloyd-IF using a variety of tasks, thus providing strong evidence for its versatility, flexibility, and improvements over existing methods.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 31 of its 35 references that are in the map. First posted on arXiv 28 Jan 2023.

Builds on

31 of 35 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models16 Dec
  2. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  3. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  4. 2022Blended Latent Diffusion6 Jun
  5. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
11 more
  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2021Variational Diffusion Modelscited by 1,145
  5. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  6. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  7. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  8. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  9. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  10. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  11. 2015Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learningcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 1.

4 papers.

  1. 2024FluxSpace: Disentangled Semantic Editing in Rectified Flow Transformerscited by 25
  2. 2024Safeguard Text-to-Image Diffusion Models with Human Feedback Inversioncited by 10
  3. 2025CE-SDWV: Effective and Efficient Concept Erasure for Text-to-Image Diffusion Models via a Semantic-Driven Word Vocabularycited by 7
  4. 2026Representation Learning in Diffusion and Flow-based Model: An Application Aspectcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022The Stable Artist: Steering Semantics in Diffusion Latent Space
  2. 2023Semantic Guidance Tuning for Text-To-Image Diffusion Models
  3. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
  4. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance
  5. 2023Text-to-image Diffusion Models in Generative AI: A Survey
  6. 2025Decoder-Only LLMs are Better Controllers for Diffusion Models
  7. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths
  8. 2022LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models