# ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglement

arXiv: https://arxiv.org/abs/2407.07197 · HTML: https://mlsanity.com/paper/2407.07197/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 48 of its 58 references that are in the map.
- First posted on arXiv 9 Jul 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 48)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Muse: Text-To-Image Generation via Masked Generative Transformers](https://mlsanity.com/paper/2301.00704/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning](https://mlsanity.com/paper/2304.03411/) (2023)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [P+: Extended Textual Conditioning in Text-to-Image Generation](https://mlsanity.com/paper/2303.09522/) (2023)
- [Break-A-Scene: Extracting Multiple Concepts from a Single Image](https://mlsanity.com/paper/2305.16311/) (2023)
- [Subject-driven Text-to-Image Generation via Apprenticeship Learning](https://mlsanity.com/paper/2304.00186/) (2023)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models](https://mlsanity.com/paper/2305.16807/) (2023)
- [Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment](https://mlsanity.com/paper/2306.08877/) (2023)

## Built on by (most cited first, 28 of 28)

- [AnyText2: Visual Text Generation and Editing With Customizable Attributes](https://mlsanity.com/paper/2411.15245/) (2024)
- [FonTS: Text Rendering with Typography and Style Controls](https://mlsanity.com/paper/2412.00136/) (2024)
- [WordCon: Word-level Typography Control in Scene Text Rendering](https://mlsanity.com/paper/2506.21276/) (2025)
- [Multi-Class Textual-Inversion Secretly Yields a Semantic-Agnostic Classifier](https://mlsanity.com/paper/2410.22317/) (2024)
- [Anchor Token Matching: Implicit Structure Locking for Training-free AR Image Editing](https://mlsanity.com/paper/2504.10434/) (2025)
- [Free-Lunch Color-Texture Disentanglement for Stylized Image Generation](https://mlsanity.com/paper/2503.14275/) (2025)
- [Leveraging Semantic Attribute Binding for Free-Lunch Color Control in Diffusion Models](https://mlsanity.com/paper/2503.09864/) (2025)
- [CDST: Color Disentangled Style Transfer for Universal Style Reference Customization](https://mlsanity.com/paper/2506.13770/) (2025)
- [One-Way Ticket:Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models](https://mlsanity.com/paper/2505.21960/) (2025)
- [Exploring Palette based Color Guidance in Diffusion Models](https://mlsanity.com/paper/2508.08754/) (2025)
- [GenColorBench: A Color Evaluation Benchmark for Text-to-Image Generation Models](https://mlsanity.com/paper/2510.20586/) (2025)
- [A Controllable Appearance Representation for Flexible Transfer and Editing](https://mlsanity.com/paper/2504.15028/) (2025)
- [InstructAttribute: Fine-grained Object Attributes editing with Instruction](https://mlsanity.com/paper/2505.00751/) (2025)
- [Color Bind: Exploring Color Perception in Text-to-Image Models](https://mlsanity.com/paper/2508.19791/) (2025)
- [Color Me Correctly: Bridging Perceptual Color Spaces and Text Embeddings for Improved Diffusion Generation](https://mlsanity.com/paper/2509.10058/) (2025)
- [BBQ-to-Image: Numeric Bounding Box and Qolor Control in Large-Scale Text-to-Image Models](https://mlsanity.com/paper/2602.20672/) (2026)
- [Controllable diffusion-based generation for multi-channel biological data](https://mlsanity.com/paper/2507.02902/) (2025)
- [Editable Noise Map Inversion: Encoding Target-image into Noise For High-Fidelity Image Manipulation](https://mlsanity.com/paper/2509.25776/) (2025)
- [Adversarial Concept Distillation for One-Step Diffusion Personalization](https://mlsanity.com/paper/2510.20512/) (2025)
- [LumiCtrl : Learning Illuminant Prompts for Lighting Control in Personalized Text-to-Image Models](https://mlsanity.com/paper/2512.17489/) (2025)
- [NumColor: Precise Numeric Color Control in Text-to-Image Generation](https://mlsanity.com/paper/2603.13547/) (2026)
- [Recolour What Matters: Region-Aware Colour Editing via Token-Level Diffusion](https://mlsanity.com/paper/2603.18466/) (2026)
- [MULTI: Disentangling Camera Lens, Sensor, View, and Domain for Novel Image Generation](https://mlsanity.com/paper/2605.12134/) (2026)
- [Exploring the AI Obedience: Why is Generating a Pure Color Image Harder than CyberPunk?](https://mlsanity.com/paper/2603.00166/) (2026)
- [The Latent Color Subspace: Emergent Order in High-Dimensional Chaos](https://mlsanity.com/paper/2603.12261/) (2026)
- [RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement](https://mlsanity.com/paper/2607.05088/) (2026)
- [X-MULTI: VLM-based Imaging Factor Disentanglement for Factor-Aware Image Synthesis](https://mlsanity.com/paper/2608.24563/) (2026)
- [Paint-Anything: Unified Any-Color Control for Image Generation and Editing](https://mlsanity.com/paper/2609.20816/) (2026)

## Abstract (arXiv)

Text-to-Image (T2I) generation has made significant advancements with the advent of diffusion models. These models exhibit remarkable abilities to produce images based on textual prompts. Current T2I models allow users to specify object colors using linguistic color names. However, these labels encompass broad color ranges, making it difficult to achieve precise color matching. To tackle this challenging task, named color prompt learning, we propose to learn specific color prompts tailored to user-selected colors. Existing T2I personalization methods tend to result in color-shape entanglement. To overcome this, we generate several basic geometric objects in the target color, allowing for color and shape disentanglement during the color prompt learning. Our method, denoted as ColorPeel, successfully assists the T2I models to peel off the novel color prompts from these colored shapes. In the experiments, we demonstrate the efficacy of ColorPeel in achieving precise color generation with T2I models. Furthermore, we generalize ColorPeel to effectively learn abstract attribute concepts, including textures, materials, etc. Our findings represent a significant step towards improving precision and versatility of T2I models, offering new opportunities for creative applications and design tasks. Our project is available at https://moatifbutt.github.io/colorpeel/.