# Palette Aligned Image Diffusion

arXiv: https://arxiv.org/abs/2509.02000 · HTML: https://mlsanity.com/paper/2509.02000/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 7 of its 30 references that are in the map.
- First posted on arXiv 2 Sep 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 7 of 7)

- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [Analyzing Bias in Diffusion-based Face Generation Models](https://mlsanity.com/paper/2305.06402/) (2023)
- [Color Conditional Generation with Sliced Wasserstein Guidance](https://mlsanity.com/paper/2503.19034/) (2025)
- [Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generation](https://mlsanity.com/paper/2412.19853/) (2024)

## Built on by (most cited first, 1 of 1)

- [Paint-Anything: Unified Any-Color Control for Image Generation and Editing](https://mlsanity.com/paper/2609.20816/) (2026)

## Abstract (arXiv)

We introduce the Palette-Adapter, a novel method for conditioning text-to-image diffusion models on a user-specified color palette. While palettes are a compact and intuitive tool widely used in creative workflows, they introduce significant ambiguity and instability when used for conditioning image generation. Our approach addresses this challenge by interpreting palettes as sparse histograms and introducing two scalar control parameters: histogram entropy and palette-to-histogram distance, which allow flexible control over the degree of palette adherence and color variation. We further introduce a negative histogram mechanism that allows users to suppress specific undesired hues, improving adherence to the intended palette under the standard classifier-free guidance mechanism. To ensure broad generalization across the color space, we train on a carefully curated dataset with balanced coverage of rare and common colors. Our method enables stable, semantically coherent generation across a wide range of palettes and prompts. We evaluate our method qualitatively, quantitatively, and through a user study, and show that it consistently outperforms existing approaches in achieving both strong palette adherence and high image quality.