mlsanityData to 7 Oct 2026

Generating images of rare concepts using pre-trained diffusion models

arXiv 2304.14530 ↗First posted 27 Apr 2023cs.CV cs.LGcited by 78

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image diffusion models can synthesize high-quality images, but they have various limitations. Here we highlight a common failure mode of these models, namely, generating uncommon concepts and structured concepts like hand palms. We show that their limitation is partly due to the long-tail nature of their training data: web-crawled data sets are strongly unbalanced, causing models to under-represent concepts from the tail of the distribution. We characterize the effect of unbalanced training data on text-to-image models and offer a remedy. We show that rare concepts can be correctly generated by carefully selecting suitable generation seeds in the noise space, using a small reference set of images, a technique that we call SeedSelect. SeedSelect does not require retraining or finetuning the diffusion model. We assess the faithfulness, quality and diversity of SeedSelect in creating rare objects and generating complex formations like hand images, and find it consistently achieves superior performance. We further show the advantage of SeedSelect in semantic data augmentation. Generating semantically appropriate images can successfully improve performance in few-shot recognition benchmarks, for classes from the head and from the tail of the training data of diffusion models

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 51 of its 72 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

51 of 72 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification17 Apr
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  4. 2022Cap2Aug: Caption guided Image to Image data Augmentation11 Dec
  5. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  6. 2022SpaText: Spatio-Textual Representation for Controllable Image Generation25 Nov
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  8. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models26 Oct
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  10. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
1 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  11. 2020Supervised Contrastive Learningcited by 2,803
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
26 more
  1. 2016Matching Networks for One Shot Learningcited by 2,246
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  5. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 704
  6. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  7. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  8. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  9. 2021Deep Long-Tailed Learning: A Surveycited by 416
  10. 2018Meta-learning with differentiable closed-form solverscited by 395
  11. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  12. 2018Few-Shot Learning via Embedding Adaptation with Set-to-Set Functionscited by 293
  13. 2020Long-tailed Recognition by Routing Diverse Distribution-Aware Expertscited by 270
  14. 2021Parametric Contrastive Learningcited by 217
  15. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  16. 2021MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognitioncited by 99
  17. 2021Distributional Robustness Loss for Long-tail Learningcited by 84
  18. 2022A very preliminary analysis of DALL-E 2cited by 82
  19. 2021VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognitioncited by 65
  20. 2018Multi-level Semantic Feature Augmentation for One-shot Learningcited by 62
  21. 2018On GANs and GMMscited by 62
  22. 2022Generating Representative Samples for Few-Shot Classificationcited by 39
  23. 2019Baby steps towards few-shot learning with multiple semanticscited by 29
  24. 2023Fine-grained Visual Classification with High-temperature Refinement and Background Suppressioncited by 22
  25. 2021SEGA: Semantic Guided Attention on Visual Prototype for Few-Shot Learningcited by 15
  26. 2021Label Hallucination for Few-Shot Classificationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 27; 2025: 29; 2026: 17.

78 papers.

Sort:

By area:

  1. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  2. 2023LEDITS++: Limitless Image Editing using Text-to-Image Modelscited by 147
  3. 2024D-Flow: Differentiating through Flows for Controlled Generationcited by 115
  4. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  5. 2024Understanding Hallucinations in Diffusion Models through Mode Interpolationcited by 87
  6. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  7. 2024Diffusion Models and Representation Learning: A Surveycited by 53
  8. 2023HandRefiner: Refining Malformed Hands in Generated Images by Diffusion-based Conditional Inpaintingcited by 42
  9. 2023Lightning-Fast Image Inversion and Editing for Text-to-Image Diffusion Modelscited by 34
  10. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  11. 2024Identity Decoupling for Multi-Subject Personalization of Text-to-Image Modelscited by 33
  12. 2024A Survey on Data Augmentation in Large Model Eracited by 31
38 more
  1. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seedscited by 31
  2. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  3. 2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
  4. 2024Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelinescited by 28
  5. 2025ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generationcited by 20
  6. 2024ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Graspingcited by 18
  7. 2023Correcting Diffusion Generation through Resamplingcited by 15
  8. 2026Action-to-Action Flow Matchingcited by 15
  9. 2023Are Diffusion Models Vision-And-Language Reasoners?cited by 14
  10. 2024The Silent Assistant: NoiseQuery as Implicit Guidance for Goal-Driven Image Generationcited by 14
  11. 2025DASH: Detection and Assessment of Systematic Hallucinations of VLMscited by 14
  12. 2025DraCo: Draft as CoT for Text-to-Image Preview and Rare Concept Generationcited by 13
  13. 2024VTON-HandFit: Virtual Try-on for Arbitrary Hand Pose Guided by Hand Priors Embeddingcited by 11
  14. 2024TKG-DM: Training-free Chroma Key Content Generation Diffusion Modelcited by 11
  15. 2025Towards Understanding Text Hallucination of Diffusion Models via Local Generation Biascited by 11
  16. 2025Navigating with Annealing Guidance Scale in Diffusion Spacecited by 11
  17. 2025ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generationcited by 10
  18. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Modelscited by 9
  19. 2024Information Theoretic Text-to-Image Alignmentcited by 8
  20. 2024A Survey of Data Synthesis Approachescited by 8
  21. 2025Noise Matters: Optimizing Matching Noise for Diffusion Classifierscited by 7
  22. 2025Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Modelscited by 7
  23. 2024Repairing Catastrophic-Neglect in Text-to-Image Diffusion Models via Attention-Guided Feature Enhancementcited by 5
  24. 2024HaSPeR: An Image Repository for Hand Shadow Puppet Recognitioncited by 5
  25. 2026You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vectorcited by 5
  26. 2024Detection-Driven Object Count Optimization for Text-to-Image Diffusion Modelscited by 4
  27. 2025RewardSDS: Aligning Score Distillation via Reward-Weighted Samplingcited by 4
  28. 2025Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decompositioncited by 4
  29. 2024Optimizing Resource Consumption in Diffusion Models through Hallucination Early Detectioncited by 3
  30. 2024RAVEL: Rare Concept Generation and Editing via Graph-driven Relational Guidancecited by 3
  31. 2024E4C: Enhance Editability for Text-Based Image Editing by Harnessing Efficient CLIP Guidancecited by 2
  32. 2025Geometric Consistency Refinement for Single Image Novel View Synthesis via Test-Time Adaptation of Diffusion Modelscited by 2
  33. 2025Causal Disentanglement for Robust Long-tail Medical Image Generationcited by 2
  34. 2025Noise-Level Diffusion Guidance: Well Begun is Half Donecited by 2
  35. 2025Demystifying Numerosity in Diffusion Models -- Limitations and Remediescited by 2
  36. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  37. 2026COT-FM: Cluster-wise Optimal Transport Flow Matchingcited by 2
  38. 2026Where Should Action Generation Begin? A Learnable Source Prior for Generative Robot Policiescited by 2

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Multi-Concept Customization of Text-to-Image Diffusion
  2. 2026Rare Concept Generation via Counterfactual Inference in Diffusion Models
  3. 2023DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation
  4. 2023Diversity and Diffusion: Observations on Synthetic Image Distributions with Stable Diffusion
  5. 2023Norm-guided latent space exploration for text-to-image generation
  6. 2023Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else
  7. 2023Ablating Concepts in Text-to-Image Diffusion Models
  8. 2026A Concept is More Than a Word: Diversified Unlearning in Text-to-Image Diffusion Models