# FABRIC: Personalizing Diffusion Models with Iterative Feedback

arXiv: https://arxiv.org/abs/2307.10159 · HTML: https://mlsanity.com/paper/2307.10159/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 19 Jul 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16381/) (2023)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [StyleDrop: Text-to-Image Generation in Any Style](https://mlsanity.com/paper/2306.00983/) (2023)
- [GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution](https://mlsanity.com/paper/2012.00739/) (2020)
- [Self-attention Does Not Need O(n2) Memory](https://mlsanity.com/paper/2112.05682/) (2021)
- [Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16223/) (2023)
- [Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles](https://mlsanity.com/paper/2303.03751/) (2023)

## Built on by (most cited first, 19 of 19)

- [InstructVideo: Instructing Video Diffusion Models with Human Feedback](https://mlsanity.com/paper/2312.12490/) (2023)
- [Sora as a World Model? A Complete Survey on Text-to-Video Generation](https://mlsanity.com/paper/2403.05131/) (2024)
- [Margin-aware Preference Optimization for Aligning Diffusion Models without Reference](https://mlsanity.com/paper/2406.06424/) (2024)
- [ViPer: Visual Personalization of Generative Models via Individual Preference Learning](https://mlsanity.com/paper/2407.17365/) (2024)
- [TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generation](https://mlsanity.com/paper/2401.11094/) (2024)
- [Manipulating Embeddings of Stable Diffusion Prompts](https://mlsanity.com/paper/2308.12059/) (2023)
- [SpaceControl: Introducing Test-Time Spatial Control to 3D Generative Modeling](https://mlsanity.com/paper/2512.05343/) (2025)
- [Preference Adaptive and Sequential Text-to-Image Generation](https://mlsanity.com/paper/2412.10419/) (2024)
- [CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representations](https://mlsanity.com/paper/2404.16482/) (2024)
- [Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.03481/) (2025)
- [Guidance Watermarking for Diffusion Models](https://mlsanity.com/paper/2509.22126/) (2025)
- [Personalized Reward Modeling for Text-to-Image Generation](https://mlsanity.com/paper/2511.19458/) (2025)
- [Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation](https://mlsanity.com/paper/2406.16807/) (2024)
- [CoCoG-2: Controllable generation of visual stimuli for understanding human concept representation](https://mlsanity.com/paper/2407.14949/) (2024)
- [Human Aesthetic Preference-Based Large Text-to-Image Model Personalization: Kandinsky Generation as an Example](https://mlsanity.com/paper/2402.06389/) (2024)
- [PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation](https://mlsanity.com/paper/2607.06440/) (2026)
- [Few-shot multi-token DreamBooth with LoRa for style-consistent character generation](https://mlsanity.com/paper/2510.09475/) (2025)
- [Dynamic Population Distribution Aware Human Trajectory Generation with Diffusion Model](https://mlsanity.com/paper/2511.01929/) (2025)
- [ZIPP:Zero-shot Image Personalization from Personas](https://mlsanity.com/paper/2606.08841/) (2026)

## Abstract (arXiv)

In an era where visual content generation is increasingly driven by machine learning, the integration of human feedback into generative models presents significant opportunities for enhancing user experience and output quality. This study explores strategies for incorporating iterative human feedback into the generative process of diffusion-based text-to-image models. We propose FABRIC, a training-free approach applicable to a wide range of popular diffusion models, which exploits the self-attention layer present in the most widely used architectures to condition the diffusion process on a set of feedback images. To ensure a rigorous assessment of our approach, we introduce a comprehensive evaluation methodology, offering a robust mechanism to quantify the performance of generative visual models that integrate human feedback. We show that generation results improve over multiple rounds of iterative feedback through exhaustive analysis, implicitly optimizing arbitrary user preferences. The potential applications of these findings extend to fields such as personalized content creation and customization.