mlsanityData to 14 Oct 2026

PlanGen: Towards Unified Layout Planning and Image Generation in Auto-Regressive Vision Language Models

arXiv 2503.10127 ↗First posted 13 Mar 2025cs.CVcited by 18

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

In this paper, we propose a unified layout planning and image generation model, PlanGen, which can pre-plan spatial layout conditions before generating images. Unlike previous diffusion-based models that treat layout planning and layout-to-image as two separate models, PlanGen jointly models the two tasks into one autoregressive transformer using only next-token prediction. PlanGen integrates layout conditions into the model as context without requiring specialized encoding of local captions and bounding box coordinates, which provides significant advantages over the previous embed-and-pool operations on layout conditions, particularly when dealing with complex layouts. Unified prompting allows PlanGen to perform multitasking training related to layout, including layout planning, layout-to-image generation, image layout understanding, etc. In addition, PlanGen can be seamlessly expanded to layout-guided image manipulation thanks to the well-designed modeling, with teacher-forcing content manipulation policy and negative layout guidance. Extensive experiments verify the effectiveness of our PlanGen in multiple layoutrelated tasks, showing its great potential. Code is available at: https://360cvgroup.github.io/PlanGen.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 99 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

75 of 99 references.

References · 75

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2023Segment Anythingcited by 8,234
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  12. 2022Generative Adversarial Networkscited by 5,627
38 more
  1. 2025Qwen2.5-VL Technical Reportcited by 5,587
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2016Improved Techniques for Training GANscited by 2,970
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  11. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  13. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  14. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  15. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  16. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  17. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  18. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  19. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  20. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  21. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  22. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  23. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  24. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  25. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  26. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  27. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  28. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 331
  29. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  30. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  31. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 282
  32. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  33. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  34. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 195
  35. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  36. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  37. 2023InstructDiffusion: A Generalist Modeling Interface for Vision Taskscited by 164
  38. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2025InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attentioncited by 10
  2. 2026Re-Align: Structured Reasoning-guided Alignment for In-Context Image Generation and Editingcited by 8
  3. 2025Lay2Story: Extending Diffusion Transformers for Layout-Togglable Story Generationcited by 6
  4. 2026InnoAds-Composer: Efficient Condition Composition for E-Commerce Poster Generationcited by 6
  5. 2025ConsistCompose: Unified Multimodal Layout Control for Image Compositioncited by 5
  6. 2026HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Imagescited by 5
  7. 2026Layer-wise Instance Binding for Regional and Occlusion Control in Text-to-Image Diffusion Transformerscited by 5
  8. 2026TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Modelscited by 4
  9. 2025Layout-Conditioned Autoregressive Text-to-Image Generation via Structured Maskingcited by 3
  10. 2026CyCLeGen: Cycle-Consistent Layout Prediction and Image Generation in Vision Foundation Modelscited by 3
  11. 2026Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Renderingcited by 3
  12. 2025RefTon: Reference person shot assist virtual Try-oncited by 1
6 more
  1. 2026Large Language Models are Universal Reasoners for Visual Generationcited by 1
  2. 2026OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generationcited by 1
  3. 2026MusicLayout: Explicit Structural Planning for Controllable Text-to-Music Generationcited by 0
  4. 2026WithEveryone: Unified Planning and Identity Grounding for Group Image Generationcited by 0
  5. 2026Bridging Language and Spherical Space: Object-Centric Control for Text-to-Panorama Generationcited by 0
  6. 2026Planning and Rendering in Concert: DeepFusion of Autoregressive Layouts and Diffusion for Visual Text Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EchoGen: Cycle-Consistent Learning for Unified Layout-Image Generation and Understanding
  2. 2025A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models
  3. 2026Think, Plan, Paint: Layout-Aware Reasoning for Controllable Image Generation in Unified Models
  4. 2025LayoutAgent: A Vision-Language Agent Guided Compositional Diffusion for Spatial Layout Planning
  5. 2024DivCon: Divide and Conquer for Complex Numerical and Spatial Reasoning in Text-to-Image Generation
  6. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation
  7. 2023Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis
  8. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation