mlsanityData to 7 Oct 2026

ReFACT: Updating Text-to-Image Models by Editing the Text Encoder

arXiv 2306.00738 ↗First posted 1 Jun 2023cs.CL cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Our world is marked by unprecedented technological, global, and socio-political transformations, posing a significant challenge to text-to-image generative models. These models encode factual associations within their parameters that can quickly become outdated, diminishing their utility for end-users. To that end, we introduce ReFACT, a novel approach for editing factual associations in text-to-image models without relaying on explicit input from end-users or costly re-training. ReFACT updates the weights of a specific layer in the text encoder, modifying only a tiny portion of the model's parameters and leaving the rest of the model unaffected. We empirically evaluate ReFACT on an existing benchmark, alongside a newly curated dataset. Compared to other methods, ReFACT achieves superior performance in both generalization to related concepts and preservation of unrelated concepts. Furthermore, ReFACT maintains image generation quality, making it a practical tool for updating and correcting factual information in text-to-image models.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

44 of 67 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  12. 2022Locating and Editing Factual Associations in GPTcited by 2,609
32 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  7. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  9. 2022Mass-Editing Memory in a Transformercited by 976
  10. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  11. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  12. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  13. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  14. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  15. 2021Editing Factual Knowledge in Language Modelscited by 548
  16. 2021Fast Model Editing at Scalecited by 491
  17. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  18. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 256
  19. 2020Modifying Memories in Transformer Modelscited by 222
  20. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  21. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  22. 2023Editing Implicit Assumptions in Text-to-Image Diffusion Modelscited by 128
  23. 2021Paint by Wordcited by 107
  24. 2020Rewriting a Deep Generative Modelcited by 105
  25. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  26. 2022Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidancecited by 67
  27. 2021Editing a classifier by rewriting its prediction rulescited by 64
  28. 2022Multiresolution Textual Inversioncited by 33
  29. 2022Rewriting Geometric Rules of a GANcited by 25
  30. 2023A Friendly Face: Do Text-to-Image Systems Rely on Stereotypes when the Input is Under-Specified?cited by 25
  31. 2021CreativeGAN: Editing Generative Adversarial Networks for Creative Design Synthesiscited by 24
  32. 2022Rank-One Editing of Encoder-Decoder Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 12; 2026: 8.

32 papers.

Sort:

By area:

  1. 2023Unified Concept Editing in Diffusion Modelscited by 377
  2. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 298
  3. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  4. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  5. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  6. 2023Fair Text-to-Image Diffusion via Fair Mappingcited by 31
  7. 2024T2IShield: Defending Against Backdoors on Text-to-Image Diffusion Modelscited by 29
  8. 2024Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelinescited by 28
  9. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  10. 2024Editing Massive Concepts in Text-to-Image Diffusion Modelscited by 15
  11. 2025TRCE: Towards Reliable Malicious Concept Erasure in Text-to-Image Diffusion Modelscited by 14
  12. 2025Side Effects of Erasing Concepts from Diffusion Modelscited by 12
20 more
  1. 2024Unveiling Concept Attribution in Diffusion Modelscited by 11
  2. 2025Continual Unlearning for Foundational Text-to-Image Models without Generalization Erosioncited by 11
  3. 2025A Comprehensive Survey on Concept Erasure in Text-to-Image Diffusion Modelscited by 9
  4. 2025EraseFlow: Learning Concept Erasure Policies via GFlowNet-Driven Alignmentcited by 8
  5. 2025TarPro: Targeted Protection against Malicious Image Editingcited by 7
  6. 2026Interpretability Can Be Actionablecited by 5
  7. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 4
  8. 2024Combinational Backdoor Attack against Customized Text-to-Image Modelscited by 2
  9. 2025Multimodal Prompt Decoupling Attack on the Safety Filters in Text-to-Image Modelscited by 2
  10. 2024Implicit Priors Editing in Stable Diffusion via Targeted Token Adjustmentcited by 1
  11. 2025Safe Vision-Language Models via Unsafe Weights Manipulationcited by 1
  12. 2025Replace in Translation: Boost Concept Alignment in Counterfactual Text-to-Imagecited by 1
  13. 2025How Robust is Model Editing after Fine-Tuning? An Empirical Study on Text-to-Image Diffusion Modelscited by 1
  14. 2025Practical, Generalizable and Robust Backdoor Attacks on Text-to-Image Diffusion Modelscited by 1
  15. 2026Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMscited by 1
  16. 2026Backdoor Sentinel: Detecting and Detoxifying Backdoors in Diffusion Models via Temporal Noise Consistencycited by 0
  17. 2026Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understandingcited by 0
  18. 2026When Backdoors Go Beyond Triggers: Semantic Drift in Diffusion Models Under Encoder Attackscited by 0
  19. 2026Compensation-free Machine Unlearning in Text-to-Image Diffusion Models by Eliminating the Mutual Informationcited by 0
  20. 2026Multimodal QUD: Inquisitive Questions from Scientific Figurescited by 0