# Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models

arXiv: https://arxiv.org/abs/2508.03481 · HTML: https://mlsanity.com/paper/2508.03481/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 28 of 28)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models](https://mlsanity.com/paper/2308.07863/) (2023)
- [Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization](https://mlsanity.com/paper/2309.04669/) (2023)
- [Diffusion Models for Generative Outfit Recommendation](https://mlsanity.com/paper/2402.17279/) (2024)
- [A New Creative Generation Pipeline for Click-Through Rate with Stable Diffusion Model](https://mlsanity.com/paper/2401.10934/) (2024)
- [Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewriting](https://mlsanity.com/paper/2310.08129/) (2023)
- [FABRIC: Personalizing Diffusion Models with Iterative Feedback](https://mlsanity.com/paper/2307.10159/) (2023)
- [PMG : Personalized Multimodal Generation with Large Language Models](https://mlsanity.com/paper/2404.08677/) (2024)
- [ViPer: Visual Personalization of Generative Models via Individual Preference Learning](https://mlsanity.com/paper/2407.17365/) (2024)
- [Personalized Image Generation with Large Multimodal Models](https://mlsanity.com/paper/2410.14170/) (2024)
- [Investigating Personalization Methods in Text to Music Generation](https://mlsanity.com/paper/2309.11140/) (2023)
- [Chaining text-to-image and large language model: A novel approach for generating personalized e-commerce banners](https://mlsanity.com/paper/2403.05578/) (2024)
- [Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalization](https://mlsanity.com/paper/2405.14132/) (2024)
- [Personalized Popular Music Generation Using Imitation and Structure](https://mlsanity.com/paper/2105.04709/) (2021)

## Built on by (most cited first, 5 of 5)

- [Personalized Reward Modeling for Text-to-Image Generation](https://mlsanity.com/paper/2511.19458/) (2025)
- [Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generation](https://mlsanity.com/paper/2603.20725/) (2026)
- [PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation](https://mlsanity.com/paper/2607.06440/) (2026)
- [Visual Personalization Turing Test](https://mlsanity.com/paper/2601.22680/) (2026)
- [ZIPP:Zero-shot Image Personalization from Personas](https://mlsanity.com/paper/2606.08841/) (2026)

## Abstract (arXiv)

Personalized generation in T2I diffusion models aims to naturally incorporate individual user preferences into the generation process with minimal user intervention. However, existing studies primarily rely on prompt-level modeling with large-scale models, often leading to inaccurate personalization due to the limited input token capacity of T2I diffusion models. To address these limitations, we propose DrUM, a novel method that integrates user profiling with a transformer-based adapter to enable personalized generation through condition-level modeling in the latent space. DrUM demonstrates strong performance on large-scale datasets and seamlessly integrates with open-source text encoders, making it compatible with widely used foundation T2I models without requiring additional fine-tuning.