mlsanityData to 14 Oct 2026

LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

arXiv 2305.15393 ↗First posted 24 May 2023cs.CV cs.AIcited by 347

Not on the map. Why?

Cited by 67 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Attaining a high degree of user controllability in visual generation often requires intricate, fine-grained inputs like layouts. However, such inputs impose a substantial burden on users when compared to simple text inputs. To address the issue, we study how Large Language Models (LLMs) can serve as visual planners by generating layouts from text conditions, and thus collaborate with visual generative models. We propose LayoutGPT, a method to compose in-context visual demonstrations in style sheet language to enhance the visual planning skills of LLMs. LayoutGPT can generate plausible layouts in multiple domains, ranging from 2D images to 3D indoor scenes. LayoutGPT also shows superior performance in converting challenging language concepts like numerical and spatial relations to layout arrangements for faithful text-to-image generation. When combined with a downstream image generation model, LayoutGPT outperforms text-to-image models/systems by 20-40% and achieves comparable performance as human users in designing visual layouts for numerical and spatial correctness. Lastly, LayoutGPT achieves comparable performance to supervised methods in 3D indoor scene synthesis, demonstrating its effectiveness and potential in multiple visual domains.

Key facts. Cited by 347 papers in the map (data through 14 Oct 2026). Builds on 54 of its 65 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

54 of 65 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  2. 2021Grounded Language-Image Pre-trainingcited by 1,156
  3. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  4. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  5. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  6. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  7. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  8. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  9. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  10. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  11. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  12. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  13. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  14. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  15. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  16. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  17. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 311
  18. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  19. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 301
  20. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  21. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 200
  22. 2023Teaching CLIP to Count to Tencited by 198
  23. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  24. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 179
  25. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 158
  26. 2019LayoutGAN: Generating Graphic Layouts with Wireframe Discriminatorscited by 131
  27. 2020LayoutTransformer: Layout Generation and Completion with Self-attentioncited by 123
  28. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  29. 2018Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Modelscited by 106
  30. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 95
  31. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 87
  32. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  33. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 72
  34. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 70
  35. 2021BLT: Bidirectional Layout Transformer for Controllable Layout Generationcited by 69
  36. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidancecited by 69
  37. 2018Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instructioncited by 61
  38. 2020End-to-End Optimization of Scene Layoutcited by 52

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 31; 2024: 87; 2025: 119; 2026: 110.

347 papers.

Sort:

By area:

  1. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  3. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 282
  4. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
  5. 2023Challenges and Applications of Large Language Modelscited by 254
  6. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  7. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  8. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  9. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 164
  10. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  11. 2023TextDiffuser-2: Unleashing the Power of Language Models for Text Renderingcited by 140
  12. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
38 more
  1. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 132
  2. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 124
  3. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
  4. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 117
  5. 2023Self-correcting LLM-controlled Diffusion Modelscited by 113
  6. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 109
  7. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 104
  8. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  9. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  10. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 95
  11. 2025HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixelscited by 94
  12. 2024I-Design: Personalized LLM Interior Designercited by 87
  13. 2023LLM-grounded Video Diffusion Modelscited by 83
  14. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 82
  15. 2023Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Followingcited by 77
  16. 2024SINGAPO: Single Image Controlled Generation of Articulated Parts in Objectscited by 69
  17. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 67
  18. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  19. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  20. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 60
  21. 2025SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agentcited by 56
  22. 2023LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Promptscited by 54
  23. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 54
  24. 2026SAGE: Scalable Agentic 3D Scene Generation for Embodied AIcited by 50
  25. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  26. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Modelscited by 45
  27. 2024SceneTeller: Language-to-3D Scene Generationcited by 43
  28. 2024Graphic Design with Large Multimodal Modelcited by 40
  29. 2023Paragraph-to-Image Generation with Information-Enriched Diffusion Modelcited by 39
  30. 2023AutoStory: Generating Diverse Storytelling Images with Minimal Human Effortcited by 38
  31. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 38
  32. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 38
  33. 2024Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinementcited by 37
  34. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  35. 2023Retrieval-Augmented Layout Transformer for Content-Aware Layout Generationcited by 35
  36. 2024PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generationcited by 35
  37. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  38. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35

50 most cited of 347 All 347 (JSON)