# CustomContrast: A Multilevel Contrastive Perspective For Subject-Driven Text-to-Image Customization

arXiv: https://arxiv.org/abs/2409.05606 · HTML: https://mlsanity.com/paper/2409.05606/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [Generative Multimodal Models are In-Context Learners](https://mlsanity.com/paper/2312.13286/) (2023)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding](https://mlsanity.com/paper/2312.04461/) (2023)
- [FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention](https://mlsanity.com/paper/2305.10431/) (2023)
- [P+: Extended Textual Conditioning in Text-to-Image Generation](https://mlsanity.com/paper/2303.09522/) (2023)
- [MVImgNet: A Large-scale Dataset of Multi-view Images](https://mlsanity.com/paper/2303.06042/) (2023)
- [MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance](https://mlsanity.com/paper/2406.07209/) (2024)
- [SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation](https://mlsanity.com/paper/2312.16272/) (2023)
- [A Neural Space-Time Representation for Text-to-Image Personalization](https://mlsanity.com/paper/2305.15391/) (2023)
- [Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion](https://mlsanity.com/paper/2310.03502/) (2023)
- [Kosmos-G: Generating Images in Context with Multimodal Large Language Models](https://mlsanity.com/paper/2310.02992/) (2023)
- [ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models](https://mlsanity.com/paper/2305.16225/) (2023)
- [Consistency-guided Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2306.01195/) (2023)
- [Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization](https://mlsanity.com/paper/2305.11718/) (2023)

## Built on by (most cited first, 13 of 13)

- [CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design](https://mlsanity.com/paper/2505.19114/) (2025)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [RealGeneral: Unifying Visual Generation via Temporal In-Context Learning with Video Models](https://mlsanity.com/paper/2503.10406/) (2025)
- [Pro3D-Editor : A Progressive-Views Perspective for Consistent and Precise 3D Editing](https://mlsanity.com/paper/2506.00512/) (2025)
- [DualReal: Adaptive Joint Training for Lossless Identity-Motion Fusion in Video Customization](https://mlsanity.com/paper/2505.02192/) (2025)
- [D2iT: Dynamic Diffusion Transformer for Accurate Image Generation](https://mlsanity.com/paper/2504.09454/) (2025)
- [TF-TI2I: Training-Free Text-and-Image-to-Image Generation via Multi-Modal Implicit-Context Learning in Text-to-Image Models](https://mlsanity.com/paper/2503.15283/) (2025)
- [LongAnimation: Long Animation Generation with Dynamic Global-Local Memory](https://mlsanity.com/paper/2507.01945/) (2025)
- [FlexIP: Dynamic Control of Preservation and Personality for Customized Image Generation](https://mlsanity.com/paper/2504.07405/) (2025)
- [NearID: Identity Representation Learning via Near-identity Distractors](https://mlsanity.com/paper/2604.01973/) (2026)
- [Stage-Aware Adaptation and Distribution Calibration for Subject-Driven Personalized Text-to-Image Generation](https://mlsanity.com/paper/2607.07173/) (2026)
- [UniScale: Arbitrary-Scale Industrial Anomaly Generation](https://mlsanity.com/paper/2608.07864/) (2026)
- [V-Engram: Trigger-Indexed External Memory for Modular Text-to-Image Personalization](https://mlsanity.com/paper/2609.37198/) (2026)

## Abstract (arXiv)

Subject-driven text-to-image (T2I) customization has drawn significant interest in academia and industry. This task enables pre-trained models to generate novel images based on unique subjects. Existing studies adopt a self-reconstructive perspective, focusing on capturing all details of a single image, which will misconstrue the specific image's irrelevant attributes (e.g., view, pose, and background) as the subject intrinsic attributes. This misconstruction leads to both overfitting or underfitting of irrelevant and intrinsic attributes of the subject, i.e., these attributes are over-represented or under-represented simultaneously, causing a trade-off between similarity and controllability. In this study, we argue an ideal subject representation can be achieved by a cross-differential perspective, i.e., decoupling subject intrinsic attributes from irrelevant attributes via contrastive learning, which allows the model to focus more on intrinsic attributes through intra-consistency (features of the same subject are spatially closer) and inter-distinctiveness (features of different subjects have distinguished differences). Specifically, we propose CustomContrast, a novel framework, which includes a Multilevel Contrastive Learning (MCL) paradigm and a Multimodal Feature Injection (MFI) Encoder. The MCL paradigm is used to extract intrinsic features of subjects from high-level semantics to low-level appearance through crossmodal semantic contrastive learning and multiscale appearance contrastive learning. To facilitate contrastive learning, we introduce the MFI encoder to capture cross-modal representations. Extensive experiments show the effectiveness of CustomContrast in subject similarity and text controllability.