# ViPer: Visual Personalization of Generative Models via Individual Preference Learning

arXiv: https://arxiv.org/abs/2407.17365 · HTML: https://mlsanity.com/paper/2407.17365/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Directly Fine-Tuning Diffusion Models on Differentiable Rewards](https://mlsanity.com/paper/2309.17400/) (2023)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023)
- [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://mlsanity.com/paper/2303.14420/) (2023)
- [Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer](https://mlsanity.com/paper/2312.09008/) (2023)
- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles](https://mlsanity.com/paper/2303.03751/) (2023)
- [UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance](https://mlsanity.com/paper/2210.16031/) (2022)
- [scDiffusion: conditional generation of high-quality single-cell data using diffusion model](https://mlsanity.com/paper/2401.03968/) (2024)
- [FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models](https://mlsanity.com/paper/2401.15636/) (2024)
- [FABRIC: Personalizing Diffusion Models with Iterative Feedback](https://mlsanity.com/paper/2307.10159/) (2023)
- [Controlled Training Data Generation with Diffusion Models](https://mlsanity.com/paper/2403.15309/) (2024)

## Built on by (most cited first, 17 of 17)

- [OmniStyle: Filtering High Quality Style Transfer Data at Scale](https://mlsanity.com/paper/2505.14028/) (2025)
- [Group Diffusion Transformers are Unsupervised Multitask Learners](https://mlsanity.com/paper/2410.15027/) (2024)
- [Preference Adaptive and Sequential Text-to-Image Generation](https://mlsanity.com/paper/2412.10419/) (2024)
- [Personalized Image Editing in Text-to-Image Diffusion Models via Collaborative Direct Preference Optimization](https://mlsanity.com/paper/2511.05616/) (2025)
- [Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.03481/) (2025)
- [Learning User Preferences for Image Generation Model](https://mlsanity.com/paper/2508.08220/) (2025)
- [GMTRouter: Personalized LLM Router over Multi-turn User Interactions](https://mlsanity.com/paper/2511.08590/) (2025)
- [Instant Preference Alignment for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.17718/) (2025)
- [Personalized Reward Modeling for Text-to-Image Generation](https://mlsanity.com/paper/2511.19458/) (2025)
- [PrefGen: Multimodal Preference Learning for Preference-Conditioned Image Generation](https://mlsanity.com/paper/2512.06020/) (2025)
- [Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generation](https://mlsanity.com/paper/2603.20725/) (2026)
- [ImageGem: In-the-wild Generative Image Interaction Dataset for Generative Model Personalization](https://mlsanity.com/paper/2510.18433/) (2025)
- [PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation](https://mlsanity.com/paper/2607.06440/) (2026)
- [PreferThinker: Reasoning-based Personalized Image Preference Assessment](https://mlsanity.com/paper/2511.00609/) (2025)
- [Visual Personalization Turing Test](https://mlsanity.com/paper/2601.22680/) (2026)
- [Personalizing Text-to-Image Generation to Individual Taste](https://mlsanity.com/paper/2604.07427/) (2026)
- [ZIPP:Zero-shot Image Personalization from Personas](https://mlsanity.com/paper/2606.08841/) (2026)

## Abstract (arXiv)

Different users find different images generated for the same prompt desirable. This gives rise to personalized image generation which involves creating images aligned with an individual's visual preference. Current generative models are, however, unpersonalized, as they are tuned to produce outputs that appeal to a broad audience. Using them to generate images aligned with individual users relies on iterative manual prompt engineering by the user which is inefficient and undesirable. We propose to personalize the image generation process by first capturing the generic preferences of the user in a one-time process by inviting them to comment on a small selection of images, explaining why they like or dislike each. Based on these comments, we infer a user's structured liked and disliked visual attributes, i.e., their visual preference, using a large language model. These attributes are used to guide a text-to-image model toward producing images that are tuned towards the individual user's visual preference. Through a series of user studies and large language model guided evaluations, we demonstrate that the proposed method results in generations that are well aligned with individual users' visual preferences.