# Exploring Palette based Color Guidance in Diffusion Models

arXiv: https://arxiv.org/abs/2508.08754 · HTML: https://mlsanity.com/paper/2508.08754/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 14 of 14)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Inversion-Based Style Transfer with Diffusion Models](https://mlsanity.com/paper/2211.13203/) (2022)
- [Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer](https://mlsanity.com/paper/2312.09008/) (2023)
- [Cross-Image Attention for Zero-Shot Appearance Transfer](https://mlsanity.com/paper/2311.03335/) (2023)
- [ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglement](https://mlsanity.com/paper/2407.07197/) (2024)
- [RecolorNeRF: Layer Decomposed Radiance Fields for Efficient Color Editing of 3D Scenes](https://mlsanity.com/paper/2301.07958/) (2023)
- [Diffusing Colors: Image Colorization with Text Guided Diffusion](https://mlsanity.com/paper/2312.04145/) (2023)
- [Color Recommendation for Vector Graphic Documents based on Multi-Palette Representation](https://mlsanity.com/paper/2209.10820/) (2022)

## Built on by (most cited first, 4 of 4)

- [NumColor: Precise Numeric Color Control in Text-to-Image Generation](https://mlsanity.com/paper/2603.13547/) (2026)
- [Recolour What Matters: Region-Aware Colour Editing via Token-Level Diffusion](https://mlsanity.com/paper/2603.18466/) (2026)
- [Histogram-constrained Image Generation](https://mlsanity.com/paper/2606.31683/) (2026)
- [Paint-Anything: Unified Any-Color Control for Image Generation and Editing](https://mlsanity.com/paper/2609.20816/) (2026)

## Abstract (arXiv)

With the advent of diffusion models, Text-to-Image (T2I) generation has seen substantial advancements. Current T2I models allow users to specify object colors using linguistic color names, and some methods aim to personalize color-object association through prompt learning. However, existing models struggle to provide comprehensive control over the color schemes of an entire image, especially for background elements and less prominent objects not explicitly mentioned in prompts. This paper proposes a novel approach to enhance color scheme control by integrating color palettes as a separate guidance mechanism alongside prompt instructions. We investigate the effectiveness of palette guidance by exploring various palette representation methods within a diffusion-based image colorization framework. To facilitate this exploration, we construct specialized palette-text-image datasets and conduct extensive quantitative and qualitative analyses. Our results demonstrate that incorporating palette guidance significantly improves the model's ability to generate images with desired color schemes, enabling a more controlled and refined colorization process.