# λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space

arXiv: https://arxiv.org/abs/2402.05195 · HTML: https://mlsanity.com/paper/2402.05195/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 50 of its 56 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Generative Multimodal Models are In-Context Learners](https://mlsanity.com/paper/2312.13286/) (2023)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning](https://mlsanity.com/paper/2304.03411/) (2023)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attention](https://mlsanity.com/paper/2305.10431/) (2023)
- [Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models](https://mlsanity.com/paper/2305.18292/) (2023)
- [P+: Extended Textual Conditioning in Text-to-Image Generation](https://mlsanity.com/paper/2303.09522/) (2023)
- [Break-A-Scene: Extracting Multiple Concepts from a Single Image](https://mlsanity.com/paper/2305.16311/) (2023)
- [HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2307.06949/) (2023)
- [Subject-driven Text-to-Image Generation via Apprenticeship Learning](https://mlsanity.com/paper/2304.00186/) (2023)
- [Re-Imagen: Retrieval-Augmented Text-to-Image Generator](https://mlsanity.com/paper/2209.14491/) (2022)

## Built on by (most cited first, 30 of 51)

- [MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance](https://mlsanity.com/paper/2406.07209/) (2024)
- [GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing](https://mlsanity.com/paper/2407.05600/) (2024)
- [R.A.C.E.: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Model](https://mlsanity.com/paper/2405.16341/) (2024)
- [Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator](https://mlsanity.com/paper/2411.15466/) (2024)
- [Steering Rectified Flow Models in the Vector Field for Controlled Image Generation](https://mlsanity.com/paper/2412.00100/) (2024)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [Personalize Anything for Free with Diffusion Transformer](https://mlsanity.com/paper/2503.12590/) (2025)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [Personalized Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2412.02142/) (2024)
- [LoRA.rar: Learning to Merge LoRAs via Hypernetworks for Subject-Style Conditioned Image Generation](https://mlsanity.com/paper/2412.05148/) (2024)
- [PatchDPO: Patch-level DPO for Finetuning-free Personalized Image Generation](https://mlsanity.com/paper/2412.03177/) (2024)
- [RealCustom++: Representing Images as Real Textual Word for Real-Time Customization](https://mlsanity.com/paper/2408.09744/) (2024)
- [Object-level Visual Prompts for Compositional Image Generation](https://mlsanity.com/paper/2501.01424/) (2025)
- [CustomContrast: A Multilevel Contrastive Perspective For Subject-Driven Text-to-Image Customization](https://mlsanity.com/paper/2409.05606/) (2024)
- [GroundingBooth: Grounding Text-to-Image Customization](https://mlsanity.com/paper/2409.08520/) (2024)
- [Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adapter](https://mlsanity.com/paper/2505.18612/) (2025)
- [CustomVideoX: 3D Reference Attention Driven Dynamic Adaptation for Zero-Shot Customized Video Diffusion Transformers](https://mlsanity.com/paper/2502.06527/) (2025)
- [Flux Already Knows -- Activating Subject-Driven Image Generation without Training](https://mlsanity.com/paper/2504.11478/) (2025)
- [RefEdit: A Benchmark and Method for Improving Instruction-based Image Editing Model on Referring Expressions](https://mlsanity.com/paper/2506.03448/) (2025)
- [MICo-150K: A Comprehensive Dataset Advancing Multi-Image Composition](https://mlsanity.com/paper/2512.07348/) (2025)
- [DreamRelation: Bridging Customization and Relation Generation](https://mlsanity.com/paper/2410.23280/) (2024)
- [MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans](https://mlsanity.com/paper/2506.20879/) (2025)
- [MoEdit: On Learning Quantity Perception for Multi-object Image Editing](https://mlsanity.com/paper/2503.10112/) (2025)
- [ComposeMe: Attribute-Specific Image Prompts for Controllable Human Image Generation](https://mlsanity.com/paper/2509.18092/) (2025)
- [DreamCache: Finetuning-Free Lightweight Personalized Image Generation via Feature Caching](https://mlsanity.com/paper/2411.17786/) (2024)
- [Piece it Together: Part-Based Concepting with IP-Priors](https://mlsanity.com/paper/2503.10365/) (2025)
- [TF-TI2I: Training-Free Text-and-Image-to-Image Generation via Multi-Modal Implicit-Context Learning in Text-to-Image Models](https://mlsanity.com/paper/2503.15283/) (2025)
- [Personalized Text-to-Image Generation with Auto-Regressive Models](https://mlsanity.com/paper/2504.13162/) (2025)
- [MUSE: Multi-Subject Unified Synthesis via Explicit Layout Semantic Expansion](https://mlsanity.com/paper/2508.14440/) (2025)
- [FlexIP: Dynamic Control of Preservation and Personality for Customized Image Generation](https://mlsanity.com/paper/2504.07405/) (2025)

## Abstract (arXiv)

Despite the recent advances in personalized text-to-image (P-T2I) generative models, it remains challenging to perform finetuning-free multi-subject-driven T2I in a resource-efficient manner. Predominantly, contemporary approaches, involving the training of Hypernetworks and Multimodal Large Language Models (MLLMs), require heavy computing resources that range from 600 to 12300 GPU hours of training. These subject-driven T2I methods hinge on Latent Diffusion Models (LDMs), which facilitate T2I mapping through cross-attention layers. While LDMs offer distinct advantages, P-T2I methods' reliance on the latent space of these diffusion models significantly escalates resource demands, leading to inconsistent results and necessitating numerous iterations for a single desired image. In this paper, we present λ-ECLIPSE, an alternative prior-training strategy that works in the latent space of a pre-trained CLIP model without relying on the diffusion UNet models. λ-ECLIPSE leverages the image-text interleaved pre-training for fast and effective multi-subject-driven P-T2I. Through extensive experiments, we establish that λ-ECLIPSE surpasses existing baselines in composition alignment while preserving concept alignment performance, even with significantly lower resource utilization. λ-ECLIPSE performs multi-subject driven P-T2I with just 34M parameters and is trained on a mere 74 GPU hours. Additionally, λ-ECLIPSE demonstrates the unique ability to perform multi-concept interpolations.