# ColorGPT: Leveraging Large Language Models for Multimodal Color Recommendation

arXiv: https://arxiv.org/abs/2508.08987 · HTML: https://mlsanity.com/paper/2508.08987/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [CanvasVAE: Learning to Generate Vector Graphic Documents](https://mlsanity.com/paper/2108.01249/) (2021)
- [Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding](https://mlsanity.com/paper/1703.10186/) (2017)
- [Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation](https://mlsanity.com/paper/1804.04128/) (2018)
- [Gromov-Wasserstein unsupervised alignment reveals structural correspondences between the color similarity structures of humans and large language models](https://mlsanity.com/paper/2308.04381/) (2023)
- [InfoColorizer: Interactive Recommendation of Color Palettes for Infographics](https://mlsanity.com/paper/2102.02041/) (2021)
- [Color Recommendation for Vector Graphic Documents based on Multi-Palette Representation](https://mlsanity.com/paper/2209.10820/) (2022)
- [Generative Colorization of Structured Mobile Web Pages](https://mlsanity.com/paper/2212.11541/) (2022)
- [C2Ideas: Supporting Creative Interior Color Design Ideation with Large Language Model](https://mlsanity.com/paper/2401.12586/) (2024)
- [Serif or Sans: Visual Font Analytics on Book Covers and Online Advertisements](https://mlsanity.com/paper/1906.10269/) (2019)
- [Character Sequence Models for ColorfulWords](https://mlsanity.com/paper/1609.08777/) (2016)
- [Multimodal Color Recommendation in Vector Graphic Documents](https://mlsanity.com/paper/2308.04118/) (2023)
- [Perceptual Structure in the Absence of Grounding for LLMs: The Impact of Abstractedness and Subjectivity in Color Language](https://mlsanity.com/paper/2311.13105/) (2023)
- [Generating Compositional Color Representations from Text](https://mlsanity.com/paper/2109.10477/) (2021)

## Abstract (arXiv)

Colors play a crucial role in the design of vector graphic documents by enhancing visual appeal, facilitating communication, improving usability, and ensuring accessibility. In this context, color recommendation involves suggesting appropriate colors to complete or refine a design when one or more colors are missing or require alteration. Traditional methods often struggled with these challenges due to the complex nature of color design and the limited data availability. In this study, we explored the use of pretrained Large Language Models (LLMs) and their commonsense reasoning capabilities for color recommendation, raising the question: Can pretrained LLMs serve as superior designers for color recommendation tasks? To investigate this, we developed a robust, rigorously validated pipeline, ColorGPT, that was built by systematically testing multiple color representations and applying effective prompt engineering techniques. Our approach primarily targeted color palette completion by recommending colors based on a set of given colors and accompanying context. Moreover, our method can be extended to full palette generation, producing an entire color palette corresponding to a provided textual description. Experimental results demonstrated that our LLM-based pipeline outperformed existing methods in terms of color suggestion accuracy and the distribution of colors in the color palette completion task. For the full palette generation task, our approach also yielded improvements in color diversity and similarity compared to current techniques.