# Customizing Text-to-Image Models with a Single Image Pair

arXiv: https://arxiv.org/abs/2405.01536 · HTML: https://mlsanity.com/paper/2405.01536/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 89 of its 110 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 30 of 89)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)

## Built on by (most cited first, 30 of 36)

- [Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach](https://mlsanity.com/paper/2412.03017/) (2024)
- [Multi-subject Open-set Personalization in Video Generation](https://mlsanity.com/paper/2501.06187/) (2025)
- [SMooDi: Stylized Motion Diffusion Model](https://mlsanity.com/paper/2407.12783/) (2024)
- [Attention Distillation: A Unified Approach to Visual Characteristics Transfer](https://mlsanity.com/paper/2502.20235/) (2025)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [Group Diffusion Transformers are Unsupervised Multitask Learners](https://mlsanity.com/paper/2410.15027/) (2024)
- [Nested Attention: Semantic-aware Attention Values for Concept Personalization](https://mlsanity.com/paper/2501.01407/) (2025)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [Dynamic Concepts Personalization from Single Videos](https://mlsanity.com/paper/2502.14844/) (2025)
- [Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation](https://mlsanity.com/paper/2412.01027/) (2024)
- [Learning an Image Editing Model without Image Editing Pairs](https://mlsanity.com/paper/2510.14978/) (2025)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation](https://mlsanity.com/paper/2406.05000/) (2024)
- [ConsisLoRA: Enhancing Content and Style Consistency for LoRA-based Style Transfer](https://mlsanity.com/paper/2503.10614/) (2025)
- [PairEdit: Learning Semantic Variations for Exemplar-based Image Editing](https://mlsanity.com/paper/2506.07992/) (2025)
- [Steering Guidance for Personalized Text-to-Image Diffusion Models](https://mlsanity.com/paper/2508.00319/) (2025)
- [DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models](https://mlsanity.com/paper/2410.11208/) (2024)
- [Preventing Shortcuts in Adapter Training via Providing the Shortcuts](https://mlsanity.com/paper/2510.20887/) (2025)
- [OmniPrism: Learning Disentangled Visual Concept for Image Generation](https://mlsanity.com/paper/2412.12242/) (2024)
- [AlcheMinT: Fine-grained Temporal Control for Multi-Reference Consistent Video Generation](https://mlsanity.com/paper/2512.10943/) (2025)
- [Tuning-free Visual Effect Transfer across Videos](https://mlsanity.com/paper/2601.07833/) (2026)
- [MulSMo: Multimodal Stylized Motion Generation by Bidirectional Control Flow](https://mlsanity.com/paper/2412.09901/) (2024)
- [Stencil: Subject-Driven Generation with Context Guidance](https://mlsanity.com/paper/2509.17120/) (2025)
- [Replace in Translation: Boost Concept Alignment in Counterfactual Text-to-Image](https://mlsanity.com/paper/2505.14341/) (2025)
- [Delta-Adapter: Scalable Exemplar-Based Image Editing with Single-Pair Supervision](https://mlsanity.com/paper/2605.07940/) (2026)
- [One-shot Embroidery Customization via Contrastive LoRA Modulation](https://mlsanity.com/paper/2509.18948/) (2025)
- [SCEESR: Semantic-Control Edge Enhancement for Diffusion-Based Super-Resolution](https://mlsanity.com/paper/2510.19272/) (2025)
- [Expanding the Content-Style Frontier: a Balanced Subspace Blending Approach for Content-Style LoRA Fusion](https://mlsanity.com/paper/2511.01355/) (2025)
- [Training for Identity, Inference for Controllability: A Unified Approach to Tuning-Free Face Personalization](https://mlsanity.com/paper/2512.03964/) (2025)
- [Few-Shot-Based Modular Image-to-Video Adapter for Diffusion Models](https://mlsanity.com/paper/2512.20000/) (2025)

## Abstract (arXiv)

Art reinterpretation is the practice of creating a variation of a reference work, making a paired artwork that exhibits a distinct artistic style. We ask if such an image pair can be used to customize a generative model to capture the demonstrated stylistic difference. We propose Pair Customization, a new customization method that learns stylistic difference from a single image pair and then applies the acquired style to the generation process. Unlike existing methods that learn to mimic a single concept from a collection of images, our method captures the stylistic difference between paired images. This allows us to apply a stylistic change without overfitting to the specific image content in the examples. To address this new task, we employ a joint optimization method that explicitly separates the style and content into distinct LoRA weight spaces. We optimize these style and content weights to reproduce the style and content images while encouraging their orthogonality. During inference, we modify the diffusion process via a new style guidance based on our learned weights. Both qualitative and quantitative experiments show that our method can effectively learn style while avoiding overfitting to image content, highlighting the potential of modeling such stylistic differences from a single image pair.