ConsistCompose: Unified Multimodal Layout Control for Image Composition
Not on the map. Why?
Abstract (arXiv)
Unified multimodal models that couple visual understanding with image generation have advanced rapidly, yet most systems still focus on visual grounding-aligning language with image regions-while their generative counterpart, linguistic-embedded layout-grounded generation (LELG) for layout-controllable multi-instance generation, remains underexplored and limits precise compositional control. We present ConsistCompose, a unified multimodal framework that embeds layout coordinates directly into language prompts, enabling layout-controlled multi-instance image generation from Interleaved Image-Text within a single generative interface. We further construct ConsistCompose3M, a 3.4M multi-instance generation dataset with layout and identity annotations (2.6M text-guided and 0.8M image-guided data pairs) that provides large-scale supervision for layout-conditioned generation. Within this framework, LELG is instantiated through instance-coordinate binding prompts and coordinate-aware classifier-free guidance, which translate linguistic layout cues into precise spatial control without task-specific branches. Experiments on COCO-Position and MS-Bench show that ConsistCompose substantially improves spatial accuracy over layout-controlled baselines while preserving identity fidelity and competitive general multimodal understanding, establishing a unified paradigm for layout-controllable multimodal image generation.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 59 of its 62 references that are in the map. First posted on arXiv 23 Nov 2025.
Builds on
59 of 62 references.
References · 59
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
- 2024GPT-4o System Cardcited by 4,588
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2021Zero-Shot Text-to-Image Generationcited by 4,288
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
38 more
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 2025Qwen-Image Technical Reportcited by 1,044
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
- 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
- 2024Emu3: Next-Token Prediction is All You Needcited by 781
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
- 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
- 2023Generative Multimodal Models are In-Context Learnerscited by 446
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
- 2024OmniGen: Unified Image Generationcited by 394
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
- 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 340
- 2021VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalizationcited by 287
- 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
- 2025Transfer between Modalities with MetaQueriescited by 236
- 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
- 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
- 2025Less-to-More Generalization: Unlocking More Controllability by In-Context Generationcited by 154
- 2025HunyuanImage 3.0 Technical Reportcited by 149
- 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 144
- 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 124
- 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 109
- 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
- 2025Detect Anything via Next Point Predictioncited by 73
- 2025GPT-IMAGE-EDIT-1.5M: A Million-Scale, GPT-Generated Image Datasetcited by 70
- 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 60
- 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 57
- 2025OminiControl2: Efficient Conditioning for Diffusion Transformerscited by 57
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026Vision as Unified Multimodal Generationcited by 7
- 2026RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Detailscited by 5
- 2026Composing Driving Worlds through Disentangled Control for Adversarial Scenario Generationcited by 2
- 2026Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generationcited by 0
- 2026InstanceControl: Controllable Complex Image Generation without Instance Labelingcited by 0
- 2026Vision as Unified Multimodal Generation7 Jul 2026
- 2026InstanceControl: Controllable Complex Image Generation without Instance Labeling30 Jun 2026
- 2026Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation25 May 2026
- 2026RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Details8 Apr 2026
- 2026Composing Driving Worlds through Disentangled Control for Adversarial Scenario Generation13 Mar 2026