mlsanityData to 7 Oct 2026

The Lottery Ticket Hypothesis in Denoising: Towards Semantic-Driven Initialization

arXiv 2312.08872 ↗First posted 13 Dec 2023cs.CVcited by 22

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image diffusion models allow users control over the content of generated images. Still, text-to-image generation occasionally leads to generation failure requiring users to generate dozens of images under the same text prompt before they obtain a satisfying result. We formulate the lottery ticket hypothesis in denoising: randomly initialized Gaussian noise images contain special pixel blocks (winning tickets) that naturally tend to be denoised into specific content independently. The generation failure in standard text-to-image synthesis is caused by the gap between optimal and actual spatial distribution of winning tickets in initial noisy images. To this end, we implement semantic-driven initial image construction creating initial noise from known winning tickets for each concept mentioned in the prompt. We conduct a series of experiments that verify the properties of winning tickets and demonstrate their generalizability across images and prompts. Our results show that aggregating winning tickets into the initial noise image effectively induce the model to generate the specified object at the corresponding location. Project Page: https://ut-mao.github.io/noise.github.io

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 41 of its 47 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

41 of 47 references.

Closest ancestors · 21

Same direction, newest first

  1. 2023SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation20 Aug
  2. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion20 Jul
  3. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Model5 May
  4. 2023LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation30 Mar
  5. 2023Freestyle Layout-to-Image Synthesis25 Mar
  6. 2023LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation16 Feb
  7. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models16 Feb
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  9. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  10. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  11. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  12. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
9 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  2. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  3. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  5. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  8. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  9. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation6 Oct

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
8 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  3. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  4. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  5. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  6. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  7. 2021You Only Learn One Representation: Unified Network for Multiple Taskscited by 86
  8. 2023Training-Free Location-Aware Text-to-Image Synthesiscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 12.

22 papers.

Sort:

By area:

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  2. 2024A Noise is Worth Diffusion Guidancecited by 48
  3. 2024Zigzag Diffusion Sampling: Diffusion Models Can Self-Improve via Self-Reflectioncited by 38
  4. 2026Action-to-Action Flow Matchingcited by 15
  5. 2025FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noisecited by 11
  6. 2025Physics-Informed Distillation of Diffusion Models for PDE-Constrained Generationcited by 11
  7. 2024Training-Free Sketch-Guided Diffusion with Latent Optimizationcited by 7
  8. 2026Layer-wise Instance Binding for Regional and Occlusion Control in Text-to-Image Diffusion Transformerscited by 5
  9. 2026You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vectorcited by 5
  10. 2025NoiseController: Towards Consistent Multi-view Video Generation via Noise Decomposition and Collaborationcited by 3
  11. 2025LORE: Latent Optimization for Precise Semantic Control in Rectified Flow-based Image Editingcited by 3
  12. 2025SAGA: Learning Signal-Aligned Distributions for Improved Text-to-Image Generationcited by 2
10 more
  1. 2025Demystifying Numerosity in Diffusion Models -- Limitations and Remediescited by 2
  2. 2026COT-FM: Cluster-wise Optimal Transport Flow Matchingcited by 2
  3. 2026Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policiescited by 2
  4. 2026Where Should Action Generation Begin? A Learnable Source Prior for Generative Robot Policiescited by 2
  5. 2026LGTM: Training-Free Light-Guided Text-to-Image Diffusion Model via Initial Noise Manipulationcited by 1
  6. 2026Z2-Sampling: Zero-Cost Zigzag Trajectories for Semantic Alignment in Diffusion Modelscited by 1
  7. 2026LENS: Low-Frequency Eigen Noise Shaping for Efficient Diffusion Samplingcited by 1
  8. 2026Retrieving and Refining Winning Noise Tickets for Diffusion-Based Motion Generationcited by 1
  9. 2026MotionSpaceFlow: Representation-Aware Flow Matching in Direct Motion Spacecited by 0
  10. 2026Weeding Out Bad Seeds: Initial-Noise-Robust Unlearning for Text-to-Image Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization
  2. 2024The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noise
  3. 2025Noise Projection: Closing the Prompt-Agnostic Gap Behind Text-to-Image Misalignment in Diffusion Models
  4. 2024Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Models
  5. 2025Determinism of Randomness: Prompt-Residual Seed Shaping for Diffusion Generation
  6. 2025Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models
  7. 2024Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion
  8. 2026Toward Early Quality Assessment of Text-to-Image Diffusion Models