mlsanityData to 7 Oct 2026

Concept Algebra for (Score-Based) Text-Controlled Generative Models

arXiv 2302.03693 ↗First posted 7 Feb 2023cs.CL cs.LG stat.MLcited by 59

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

This paper concerns the structure of learned representations in text-guided generative models, focusing on score-based models. A key property of such models is that they can compose disparate concepts in a `disentangled' manner. This suggests these models have internal representations that encode concepts in a `disentangled' manner. Here, we focus on the idea that concepts are encoded as subspaces of some representation space. We formalize what this means, show there's a natural choice for the representation, and develop a simple method for identifying the part of the representation corresponding to a given concept. In particular, this allows us to manipulate the concepts expressed by the model through algebraic manipulation of the representation. We demonstrate the idea with examples using Stable Diffusion. Code in https://github.com/zihao12/concept-algebra-code

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 29 of its 42 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

29 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models1 Dec
  2. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  3. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  9. 2022Toy Models of Superpositioncited by 928
  10. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  11. 2022Understanding Diffusion Models: A Unified Perspectivecited by 337
  12. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
12 more
  1. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  2. 2018Towards a Definition of Disentangled Representationscited by 294
  3. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  4. 2016Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICAcited by 270
  5. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  6. 2021Contrastive Learning Inverts the Data Generating Processcited by 220
  7. 2014word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodcited by 109
  8. 2021Learning to Compose Visual Relationscited by 72
  9. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  10. 2019Analogies Explained: Towards Understanding Word Embeddingscited by 63
  11. 2020Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAEcited by 48
  12. 2022DCI-ES: An Extended Disentanglement Framework with Connections to Identifiabilitycited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 22; 2025: 25; 2026: 10.

59 papers.

Sort:

By area:

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  2. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  3. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  4. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  5. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  6. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 39
  7. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 33
  8. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  9. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 31
  10. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 22
  11. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  12. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
38 more
  1. 2023Uncovering Meanings of Embeddings via Partial Orthogonalitycited by 15
  2. 2024Global Counterfactual Directionscited by 15
  3. 2024Compositional Risk Minimizationcited by 13
  4. 2025Mechanisms of Projective Composition of Diffusion Modelscited by 13
  5. 2025Does Data Scaling Lead to Visual Compositional Generalization?cited by 13
  6. 2025On the Emergence of Linear Analogies in Word Embeddingscited by 9
  7. 2025Fair Generation without Unfair Distortions: Debiasing Text-to-Image Generation with Entanglement-Free Attentioncited by 9
  8. 2024Automated Discovery of Pairwise Interactions from Unstructured Datacited by 8
  9. 2025Responsible Diffusion Models via Constraining Text Embeddings within Safe Regionscited by 8
  10. 2025Diffusion Counterfactual Generation with Semantic Abductioncited by 8
  11. 2024Towards Compositionality in Concept Learningcited by 7
  12. 2025Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Modelscited by 7
  13. 2024Concept-based Analysis of Neural Networks via Vision-Language Modelscited by 6
  14. 2024pOps: Photo-Inspired Diffusion Operatorscited by 6
  15. 2025On Fairness of Unified Multimodal Large Language Model for Image Generationcited by 6
  16. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  17. 2025Semantic Surgery: Zero-Shot Concept Erasure in Diffusion Modelscited by 6
  18. 2023Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensemblescited by 4
  19. 2024Few-Shot Task Learning through Inverse Generative Modelingcited by 4
  20. 2025Concept Lancet: Image Editing with Compositional Representation Transplantcited by 4
  21. 2025Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocolscited by 4
  22. 2025Debugging and Runtime Analysis of Neural Networks with VLMs (A Case Study)cited by 3
  23. 2025Multi-Group Proportional Representation for Text-to-Image Modelscited by 3
  24. 2025Concept-TRAK: Understanding how diffusion models learn concepts through concept-level attributioncited by 3
  25. 2025Fully Unsupervised Self-debiasing of Text-to-Image Diffusion Modelscited by 3
  26. 2026Conditioned Activation Transport for T2I Safety Steeringcited by 3
  27. 2025The Linear Geometry of Interpretable Tokens: Jailbreaking Attacks and Defenses for Unlearned Diffusion Modelscited by 2
  28. 2025Unveiling the Latent Directions of Reflection in Large Language Modelscited by 2
  29. 2025SCALEX: Scalable Concept and Latent Exploration for Diffusion Modelscited by 2
  30. 2024MoESD: Mixture of Experts Stable Diffusion to Mitigate Gender Biascited by 1
  31. 2024Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Promptscited by 1
  32. 2025Mapping Semantic & Syntactic Relationships with Geometric Rotationcited by 1
  33. 2025Lifting Biomolecular Data Acquisitioncited by 1
  34. 2026Beyond Forgetting: Representation Misdirection Elicits Controllable Side Behaviors and Capabilitiescited by 1
  35. 2026Factored Diffusion Policies:Compositionally Generalized Robot Control with a Single Score Networkcited by 1
  36. 2026Catastrophic Compositional Generation: Why Vanilla Diffusion Models Fail to Extrapolatecited by 1
  37. 2024FSViewFusion: Few-Shots View Generation of Novel Objectscited by 0
  38. 2024Understanding the Limitations of Diffusion Concept Algebra Through Foodcited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Divide and Compose with Score Based Generative Models
  2. 2025Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimality
  3. 2023The Hidden Language of Diffusion Models
  4. 2024Learning Discrete Concepts in Latent Hierarchical Models
  5. 2025Concept Steerers: Leveraging K-Sparse Autoencoders for Test-Time Controllable Generations
  6. 2024How to Blend Concepts in Diffusion Models
  7. 2024OmniPrism: Learning Disentangled Visual Concept for Image Generation
  8. 2023Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else