mlsanityData to 7 Oct 2026

NoiseCollage: A Layout-Aware Text-to-Image Diffusion Model Based on Noise Cropping and Merging

arXiv 2403.03485 ↗First posted 6 Mar 2024cs.CVcited by 25

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Layout-aware text-to-image generation is a task to generate multi-object images that reflect layout conditions in addition to text conditions. The current layout-aware text-to-image diffusion models still have several issues, including mismatches between the text and layout conditions and quality degradation of generated images. This paper proposes a novel layout-aware text-to-image diffusion model called NoiseCollage to tackle these issues. During the denoising process, NoiseCollage independently estimates noises for individual objects and then crops and merges them into a single noise. This operation helps avoid condition mismatches; in other words, it can put the right objects in the right places. Qualitative and quantitative evaluations show that NoiseCollage outperforms several state-of-the-art models. These successful results indicate that the crop-and-merge operation of noises is a reasonable strategy to control image generation. We also show that NoiseCollage can be integrated with ControlNet to use edges, sketches, and pose skeletons as additional conditions. Experimental results show that this integration boosts the layout accuracy of ControlNet. The code is available at https://github.com/univ-esuty/noisecollage.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 43 of its 43 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

43 of 43 references.

Closest ancestors · 23

Same direction, newest first

  1. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation13 Oct
  2. 2023Dense Text-to-Image Generation with Attention Modulation24 Aug
  3. 2023SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation20 Aug
  4. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  5. 2023Zero-shot spatial layout conditioning for text-to-image diffusion models23 Jun
  6. 2023Continuous Layout Editing of Single Images with Diffusion Models22 Jun
  7. 2023Paste, Inpaint and Harmonize via Denoising: Subject-Driven Image Editing with Pre-Trained Diffusion Model13 Jun
  8. 2023Grounded Text-to-Image Synthesis with Attention Refocusing8 Jun
  9. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Model5 May
  10. 2023Freestyle Layout-to-Image Synthesis25 Mar
  11. 2023Collage Diffusion1 Mar
  12. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation16 Feb
11 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  2. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  3. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Models23 Nov
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  5. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  6. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  7. 2022Blended Latent Diffusion6 Jun
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  10. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  11. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
8 more
  1. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  2. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  3. 2023Zero-shot Image-to-Image Translationcited by 503
  4. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  5. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
  6. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  7. 2023Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Modelscited by 61
  8. 2023Training-Free Location-Aware Text-to-Image Synthesiscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 15; 2026: 3.

25 papers.

Sort:

By area:

  1. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 65
  2. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  3. 2025CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Designcited by 36
  4. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29
  5. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  6. 2024TKG-DM: Training-free Chroma Key Content Generation Diffusion Modelcited by 11
  7. 2025Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Controlcited by 10
  8. 2025InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attentioncited by 9
  9. 2025Improving Editability in Image Generation with Layer-wise Memorycited by 5
  10. 2025MUSE: Multi-Subject Unified Synthesis via Explicit Layout Semantic Expansioncited by 4
  11. 2024Generative Photomontagecited by 3
  12. 2025NoiseController: Towards Consistent Multi-view Video Generation via Noise Decomposition and Collaborationcited by 3
13 more
  1. 2025LORE: Latent Optimization for Precise Semantic Control in Rectified Flow-based Image Editingcited by 3
  2. 2025From Image Generation to Infrastructure Design: a Multi-agent Pipeline for Street Design Generationcited by 3
  3. 2025CanvasComposer: Personalized Group Photo Generation via a Multi-Reference Canvascited by 3
  4. 2025AGSwap: Overcoming Category Boundaries in Object Fusion via Adaptive Group Swappingcited by 2
  5. 2025TAUE: Training-free Noise Transplant and Cultivation Diffusion Modelcited by 2
  6. 2024P3S-Diffusion:A Selective Subject-driven Generation Framework via Point Supervisioncited by 1
  7. 2025AnimatePainter: A Self-Supervised Rendering Framework for Reconstructing Painting Processcited by 1
  8. 2024PRIMEdit: Probability Redistribution for Instance-aware Multi-object Video Editing with Benchmark Datasetcited by 0
  9. 2025Guidance-base Diffusion Models for Improving Photoacoustic Image Qualitycited by 0
  10. 2025ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generationcited by 0
  11. 2026COLLAR: Cascaded Object-Level Latent Refinement for High-Fidelity Conditional Generationcited by 0
  12. 2026Appearance Pointers -- Multimodal Region Control of Diffusion Transformerscited by 0
  13. 2026Spatially-Grounded Text-to-Video Generation via Inference-Time Gradient-Free Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Be Decisive: Noise-Induced Layouts for Multi-Subject Generation
  2. 2024Spatial-Aware Latent Initialization for Controllable Image Generation
  3. 2023LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models
  4. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis
  5. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization
  6. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation
  7. 2024Training-free Composite Scene Generation for Layout-to-Image Synthesis
  8. 2023Design Booster: A Text-Guided Diffusion Model for Image Translation with Spatial Layout Preservation