# COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Design

arXiv: https://arxiv.org/abs/2311.16974 · HTML: https://mlsanity.com/paper/2311.16974/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://mlsanity.com/paper/2305.08891/) (2023)
- [CM3: A Causal Masked Multimodal Model of the Internet](https://mlsanity.com/paper/2201.07520/) (2022)
- [VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models](https://mlsanity.com/paper/2211.11319/) (2022)
- [CanvasVAE: Learning to Generate Vector Graphic Documents](https://mlsanity.com/paper/2108.01249/) (2021)
- [IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformers](https://mlsanity.com/paper/2304.14400/) (2023)
- [Word-As-Image for Semantic Typography](https://mlsanity.com/paper/2303.01818/) (2023)
- [DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models](https://mlsanity.com/paper/2306.14685/) (2023)
- [LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models](https://mlsanity.com/paper/2303.11589/) (2023)
- [AutoPoster: A Highly Automatic and Content-aware Design System for Advertising Poster Generation](https://mlsanity.com/paper/2308.01095/) (2023)
- [Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation](https://mlsanity.com/paper/2310.08541/) (2023)
- [Towards Flexible Multi-modal Document Models](https://mlsanity.com/paper/2303.18248/) (2023)
- [DS-Fusion: Artistic Typography via Discriminated and Stylized Diffusion](https://mlsanity.com/paper/2303.09604/) (2023)
- [Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models](https://mlsanity.com/paper/2310.07653/) (2023)
- [DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design](https://mlsanity.com/paper/2310.15144/) (2023)
- [WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Models](https://mlsanity.com/paper/2310.18332/) (2023)
- [Towards Diverse and Consistent Typography Generation](https://mlsanity.com/paper/2309.02099/) (2023)
- [Boosting GUI Prototyping with Diffusion Models](https://mlsanity.com/paper/2306.06233/) (2023)
- [Text-Guided Vector Graphics Customization](https://mlsanity.com/paper/2309.12302/) (2023)

## Built on by (most cited first, 30 of 43)

- [OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion](https://mlsanity.com/paper/2507.06165/) (2025)
- [PosterMaker: Towards High-Quality Product Poster Generation with Accurate Text Rendering](https://mlsanity.com/paper/2504.06632/) (2025)
- [POSTA: A Go-to Framework for Customized Artistic Poster Generation](https://mlsanity.com/paper/2503.14908/) (2025)
- [CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design](https://mlsanity.com/paper/2505.19114/) (2025)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [Multimodal Markup Document Models for Graphic Design Completion](https://mlsanity.com/paper/2409.19051/) (2024)
- [DreamPoster: A Unified Framework for Image-Conditioned Generative Poster Design](https://mlsanity.com/paper/2507.04218/) (2025)
- [PrismLayers: Open Data for High-Quality Multi-Layer Transparent Image Generative Models](https://mlsanity.com/paper/2505.22523/) (2025)
- [BannerAgency: Advertising Banner Design with Multimodal LLM Agents](https://mlsanity.com/paper/2503.11060/) (2025)
- [PosterVerse: A Full-Workflow Framework for Commercial-Grade Poster Generation with HTML-Based Scalable Typography](https://mlsanity.com/paper/2601.03993/) (2026)
- [AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models](https://mlsanity.com/paper/2503.00591/) (2025)
- [IGD: Instructional Graphic Design with Multimodal Layer Generation](https://mlsanity.com/paper/2507.09910/) (2025)
- [AutoPP: Towards Automated Product Poster Generation and Optimization](https://mlsanity.com/paper/2512.21921/) (2025)
- [Graphic-Design-Bench: A Comprehensive Benchmark for Evaluating AI on Graphic Design Tasks](https://mlsanity.com/paper/2604.04192/) (2026)
- [PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Design](https://mlsanity.com/paper/2512.04082/) (2025)
- [UTDesign: A Unified Framework for Stylized Text Editing and Generation in Graphic Design Images](https://mlsanity.com/paper/2512.20479/) (2025)
- [InnoAds-Composer: Efficient Condition Composition for E-Commerce Poster Generation](https://mlsanity.com/paper/2603.05898/) (2026)
- [Design-o-meter: Towards Evaluating and Refining Graphic Designs](https://mlsanity.com/paper/2411.14959/) (2024)
- [DesignAsCode: Bridging Structural Editability and Visual Fidelity in Graphic Design Generation](https://mlsanity.com/paper/2602.17690/) (2026)
- [LaDe: Unified Multi-Layered Graphic Media Generation and Decomposition](https://mlsanity.com/paper/2603.17965/) (2026)
- [Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling](https://mlsanity.com/paper/2604.28185/) (2026)
- [Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement](https://mlsanity.com/paper/2603.22187/) (2026)
- [Relation-Rich Visual Document Generator for Visual Information Extraction](https://mlsanity.com/paper/2504.10659/) (2025)
- [Agentic Design Review System](https://mlsanity.com/paper/2508.10745/) (2025)
- [PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow](https://mlsanity.com/paper/2603.25738/) (2026)
- [Mirror in the Model: Ad Banner Image Generation via Reflective Multi-LLM and Multi-modal Agents](https://mlsanity.com/paper/2507.03326/) (2025)
- [DepthArb: Training-Free Depth-Arbitrated Generation for Occlusion-Robust Image Synthesis](https://mlsanity.com/paper/2603.23924/) (2026)
- [PosterIQ: A Design Perspective Benchmark for Poster Understanding and Generation](https://mlsanity.com/paper/2603.24078/) (2026)
- [AnyDoc: Enhancing Document Generation via Large-Scale HTML/CSS Data Synthesis and Height-Aware Reinforcement Optimization](https://mlsanity.com/paper/2603.25118/) (2026)
- [Does Synthetic Layered Design Data Benefit Layered Design Decomposition?](https://mlsanity.com/paper/2605.15167/) (2026)

## Abstract (arXiv)

Graphic design, which has been evolving since the 15th century, plays a crucial role in advertising. The creation of high-quality designs demands design-oriented planning, reasoning, and layer-wise generation. Unlike the recent CanvaGPT, which integrates GPT-4 with existing design templates to build a custom GPT, this paper introduces the COLE system - a hierarchical generation framework designed to comprehensively address these challenges. This COLE system can transform a vague intention prompt into a high-quality multi-layered graphic design, while also supporting flexible editing based on user input. Examples of such input might include directives like ``design a poster for Hisaishi's concert.'' The key insight is to dissect the complex task of text-to-design generation into a hierarchy of simpler sub-tasks, each addressed by specialized models working collaboratively. The results from these models are then consolidated to produce a cohesive final output. Our hierarchical task decomposition can streamline the complex process and significantly enhance generation reliability. Our COLE system comprises multiple fine-tuned Large Language Models (LLMs), Large Multimodal Models (LMMs), and Diffusion Models (DMs), each specifically tailored for design-aware layer-wise captioning, layout planning, reasoning, and the task of generating images and text. Furthermore, we construct the DESIGNINTENTION benchmark to demonstrate the superiority of our COLE system over existing methods in generating high-quality graphic designs from user intent. Last, we present a Canva-like multi-layered image editing tool to support flexible editing of the generated multi-layered graphic design images. We perceive our COLE system as an important step towards addressing more complex and multi-layered graphic design generation tasks in the future.