# U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation

arXiv: https://arxiv.org/abs/2403.20231 · HTML: https://mlsanity.com/paper/2403.20231/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 50 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?](https://mlsanity.com/paper/1904.03189/) (2019)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Closed-Form Factorization of Latent Semantics in GANs](https://mlsanity.com/paper/2007.06600/) (2020)
- [InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs](https://mlsanity.com/paper/2005.09635/) (2020)
- [Image2StyleGAN++: How to Edit the Embedded Images?](https://mlsanity.com/paper/1911.11544/) (2019)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [Inversion-Based Style Transfer with Diffusion Models](https://mlsanity.com/paper/2211.13203/) (2022)
- [P+: Extended Textual Conditioning in Text-to-Image Generation](https://mlsanity.com/paper/2303.09522/) (2023)
- [Break-A-Scene: Extracting Multiple Concepts from a Single Image](https://mlsanity.com/paper/2305.16311/) (2023)
- [Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning](https://mlsanity.com/paper/2004.11660/) (2020)
- [Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2302.12228/) (2023)

## Built on by (most cited first, 7 of 7)

- [UnZipLoRA: Separating Content and Style from a Single Image](https://mlsanity.com/paper/2412.04465/) (2024)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance](https://mlsanity.com/paper/2405.17532/) (2024)
- [Diffusion-Based Visual Art Creation: A Survey and New Perspectives](https://mlsanity.com/paper/2408.12128/) (2024)
- [CSD-VAR: Content-Style Decomposition in Visual Autoregressive Models](https://mlsanity.com/paper/2507.13984/) (2025)
- [Consistent text-to-image generation via scene de-contextualization](https://mlsanity.com/paper/2510.14553/) (2025)
- [DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer](https://mlsanity.com/paper/2605.15681/) (2026)

## Abstract (arXiv)

Concept personalization methods enable large text-to-image models to learn specific subjects (e.g., objects/poses/3D models) and synthesize renditions in new contexts. Given that the image references are highly biased towards visual attributes, state-of-the-art personalization models tend to overfit the whole subject and cannot disentangle visual characteristics in pixel space. In this study, we proposed a more challenging setting, namely fine-grained visual appearance personalization. Different from existing methods, we allow users to provide a sentence describing the desired attributes. A novel decoupled self-augmentation strategy is proposed to generate target-related and non-target samples to learn user-specified visual attributes. These augmented data allow for refining the model's understanding of the target attribute while mitigating the impact of unrelated attributes. At the inference stage, adjustments are conducted on semantic space through the learned target and non-target embeddings to further enhance the disentanglement of target attributes. Extensive experiments on various kinds of visual attributes with SOTA personalization methods show the ability of the proposed method to mimic target visual appearance in novel contexts, thus improving the controllability and flexibility of personalization.