mlsanityData to 14 Oct 2026

HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation

arXiv 2410.14324 ↗First posted 18 Oct 2024cs.CVcited by 41

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

The task of layout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object missing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, we propose a Hierarchical Controllable (HiCo) diffusion model for layout-to-image generation, featuring object seperable conditioning branch structure. Our key insight is to achieve spatial disentanglement through hierarchical modeling of layouts. We use a multi branch structure to represent hierarchy and aggregate them in fusion module. To evaluate the performance of multi-objective controllable layout generation in natural scenes, we introduce the HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned. https://github.com/360CVGroup/HiCoT2I.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 46 of its 53 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

46 of 53 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024InstanceDiffusion: Instance-level Control for Image Generation5 Feb
  2. 2023Self-correcting LLM-controlled Diffusion Models27 Nov
  3. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis21 Nov
  4. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models13 Aug
  5. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  6. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  7. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation30 Mar
  8. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation16 Feb
  9. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation16 Feb
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  12. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
5 more
  1. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  3. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2016Improved Techniques for Training GANscited by 2,970
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
17 more
  1. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  3. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  4. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  5. 2017Generative Adversarial Networks: An Overviewcited by 590
  6. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  7. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  8. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 266
  9. 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
  10. 2023ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAscited by 194
  11. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  12. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 124
  13. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latentscited by 104
  14. 2024Multi-LoRA Composition for Image Generationcited by 80
  15. 2020Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesiscited by 49
  16. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  17. 2022Interactive Image Synthesis with Panoptic Layout Generationcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 26; 2026: 13.

41 papers.

Sort:

By area:

  1. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 90
  2. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 60
  3. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  4. 2025PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Modelscited by 18
  5. 2025U-StyDiT: Ultra-high Quality Artistic Style Transfer Using Diffusion Transformerscited by 12
  6. 2025DC-ControlNet: Decoupling Inter- and Intra-Element Conditions in Image Generation with Diffusion Modelscited by 10
  7. 2025Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Controlcited by 10
  8. 2025InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attentioncited by 10
  9. 2025OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlapscited by 10
  10. 2026coDrawAgents: A Multi-Agent Dialogue Framework for Compositional Image Generationcited by 7
  11. 2025MultLFG: Training-free Multi-LoRA composition using Frequency-domain Guidancecited by 6
  12. 2025Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generationcited by 6
29 more
  1. 2025ConsistCompose: Unified Multimodal Layout Control for Image Compositioncited by 5
  2. 2024Training-Free Layout-to-Image Generation with Marginal Attention Constraintscited by 4
  3. 2025NAMI: Efficient Image Generation via Bridged Progressive Rectified Flow Transformerscited by 4
  4. 2025Mixture of Global and Local Experts with Diffusion Transformer for Controllable Face Generationcited by 4
  5. 2025HiCoGen: Hierarchical Compositional Text-to-Image Generation in Diffusion Models via Reinforcement Learningcited by 4
  6. 2025Control and Realism: Best of Both Worlds in Layout-to-Image without Trainingcited by 3
  7. 2025Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Maskingcited by 3
  8. 2025CanvasComposer: Personalized Group Photo Generation via a Multi-Reference Canvascited by 3
  9. 2026CyCLeGen: Cycle-Consistent Layout Prediction and Image Generation in Vision Foundation Modelscited by 3
  10. 2025DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Datacited by 2
  11. 2025CoProSketch: Controllable and Progressive Sketch Generation with Diffusion Modelcited by 2
  12. 2025Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformerscited by 2
  13. 2025ChangeBridge: Spatiotemporal Image Generation with Multimodal Controls for Remote Sensingcited by 2
  14. 2026MMViR: A Multi-Modal and Multi-Granularity Representation for Long-range Video Understandingcited by 2
  15. 2025RefTon: Reference person shot assist virtual Try-oncited by 1
  16. 2026Visual Prototype Conditioned Focal Region Generation for UAV-Based Object Detectioncited by 1
  17. 2026RevealLayer: Disentangling Hidden and Visible Layers via Occlusion-Aware Image Decompositioncited by 1
  18. 2026OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generationcited by 1
  19. 2025Efficient Multi-Instance Generation with Janus-Pro-Dirven Prompt Parsingcited by 0
  20. 2025ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generationcited by 0
  21. 2025Enhancing Fake-News Detection with Node-Level Topological Featurescited by 0
  22. 2025Text-Conditioned Background Generation for Editable Multi-Layer Documentscited by 0
  23. 2026MoGen: A Unified Collaborative Framework for Controllable Multi-Object Image Generationcited by 0
  24. 2026DEIG: Detail-Enhanced Instance Generation with Fine-Grained Semantic Controlcited by 0
  25. 2026Benchmarking Layout-Guided Diffusion Models through Unified Semantic-Spatial Evaluation in Closed and Open Settingscited by 0
  26. 2026COLLAR: Cascaded Object-Level Latent Refinement for High-Fidelity Conditional Generationcited by 0
  27. 2026InstanceControl: Controllable Complex Image Generation without Instance Labelingcited by 0
  28. 2026Bridging Language and Spherical Space: Object-Centric Control for Text-to-Panorama Generationcited by 0
  29. 2026OverLay++: Dense-Overlap Layout-to-Image Generation Datasetcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation
  2. 2025A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models
  3. 2025STAY Diffusion: Styled Layout Diffusion Model for Diverse Layout-to-Image Generation
  4. 2024Layout Control and Semantic Guidance with Attention Loss Backward for T2I Diffusion Model
  5. 2023LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts
  6. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation
  7. 2025ToLo: A Two-Stage, Training-Free Layout-To-Image Generation Framework For High-Overlap Layouts
  8. 2023Unifying Layout Generation with a Decoupled Diffusion Model