# HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation

arXiv: https://arxiv.org/abs/2410.14324 · HTML: https://mlsanity.com/paper/2410.14324/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 46 of its 53 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [SDXL-Lightning: Progressive Adversarial Diffusion Distillation](https://mlsanity.com/paper/2402.13929/) (2024)

## Built on by (most cited first, 30 of 41)

- [WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generation](https://mlsanity.com/paper/2503.08153/) (2025)
- [CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation](https://mlsanity.com/paper/2412.03859/) (2024)
- [CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design](https://mlsanity.com/paper/2505.19114/) (2025)
- [PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models](https://mlsanity.com/paper/2503.10127/) (2025)
- [U-StyDiT: Ultra-high Quality Artistic Style Transfer Using Diffusion Transformers](https://mlsanity.com/paper/2503.08157/) (2025)
- [DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Models](https://mlsanity.com/paper/2502.14779/) (2025)
- [Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control](https://mlsanity.com/paper/2506.00596/) (2025)
- [InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention](https://mlsanity.com/paper/2509.16691/) (2025)
- [OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps](https://mlsanity.com/paper/2509.19282/) (2025)
- [coDrawAgents: A Multi-Agent Dialogue Framework for Compositional Image Generation](https://mlsanity.com/paper/2603.12829/) (2026)
- [MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidance](https://mlsanity.com/paper/2505.20525/) (2025)
- [Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generation](https://mlsanity.com/paper/2508.08949/) (2025)
- [ConsistCompose: Unified Multimodal Layout Control for Image Composition](https://mlsanity.com/paper/2511.18333/) (2025)
- [Training-Free Layout-to-Image Generation with Marginal Attention Constraints](https://mlsanity.com/paper/2411.10495/) (2024)
- [NAMI: Efficient Image Generation via Bridged Progressive Rectified Flow Transformers](https://mlsanity.com/paper/2503.09242/) (2025)
- [Mixture of Global and Local Experts with Diffusion Transformer for Controllable Face Generation](https://mlsanity.com/paper/2509.00428/) (2025)
- [HiCoGen: Hierarchical Compositional Text-to-Image Generation in Diffusion Models via Reinforcement Learning](https://mlsanity.com/paper/2511.19965/) (2025)
- [Control and Realism: Best of Both Worlds in Layout-to-Image without Training](https://mlsanity.com/paper/2506.15563/) (2025)
- [Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Masking](https://mlsanity.com/paper/2509.12046/) (2025)
- [CanvasComposer: Personalized Group Photo Generation via a Multi-Reference Canvas](https://mlsanity.com/paper/2510.20820/) (2025)
- [CyCLeGen: Cycle-Consistent Layout Prediction and Image Generation in Vision Foundation Models](https://mlsanity.com/paper/2603.14957/) (2026)
- [DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Data](https://mlsanity.com/paper/2501.02048/) (2025)
- [CoProSketch: Controllable and Progressive Sketch Generation with Diffusion Model](https://mlsanity.com/paper/2504.08259/) (2025)
- [Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers](https://mlsanity.com/paper/2504.10148/) (2025)
- [ChangeBridge: Spatiotemporal Image Generation with Multimodal Controls for Remote Sensing](https://mlsanity.com/paper/2507.04678/) (2025)
- [MMViR: A Multi-Modal and Multi-Granularity Representation for Long-range Video Understanding](https://mlsanity.com/paper/2601.05495/) (2026)
- [RefTon: Reference person shot assist virtual Try-on](https://mlsanity.com/paper/2511.00956/) (2025)
- [Visual Prototype Conditioned Focal Region Generation for UAV-Based Object Detection](https://mlsanity.com/paper/2604.02966/) (2026)
- [RevealLayer: Disentangling Hidden and Visible Layers via Occlusion-Aware Image Decomposition](https://mlsanity.com/paper/2605.11818/) (2026)
- [OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generation](https://mlsanity.com/paper/2605.21343/) (2026)

## Abstract (arXiv)

The task of layout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object missing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, we propose a Hierarchical Controllable (HiCo) diffusion model for layout-to-image generation, featuring object seperable conditioning branch structure. Our key insight is to achieve spatial disentanglement through hierarchical modeling of layouts. We use a multi branch structure to represent hierarchy and aggregate them in fusion module. To evaluate the performance of multi-objective controllable layout generation in natural scenes, we introduce the HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned. https://github.com/360CVGroup/HiCoT2I.