# Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance

arXiv: https://arxiv.org/abs/2410.22376 · HTML: https://mlsanity.com/paper/2410.22376/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 51 of its 62 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)

## Built on by (most cited first, 30 of 30)

- [Test-time Alignment of Diffusion Models without Reward Over-optimization](https://mlsanity.com/paper/2501.05803/) (2025)
- [Image Generation from Contextually-Contradictory Prompts](https://mlsanity.com/paper/2506.01929/) (2025)
- [CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step](https://mlsanity.com/paper/2507.04451/) (2025)
- [Steer Away From Mode Collisions: Improving Composition In Diffusion Models](https://mlsanity.com/paper/2509.25940/) (2025)
- [Mitigating Diffusion Model Hallucinations with Dynamic Guidance](https://mlsanity.com/paper/2510.05356/) (2025)
- [Guiding What Not to Generate: Automated Negative Prompting for Text-Image Alignment](https://mlsanity.com/paper/2512.07702/) (2025)
- [Fine-Grained Alignment and Noise Refinement for Compositional Text-to-Image Generation](https://mlsanity.com/paper/2503.06506/) (2025)
- [Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition](https://mlsanity.com/paper/2503.10037/) (2025)
- [Rare Text Semantics Were Always There in Your Diffusion Transformer](https://mlsanity.com/paper/2510.03886/) (2025)
- [AR-RAG: Autoregressive Retrieval Augmentation for Image Generation](https://mlsanity.com/paper/2506.06962/) (2025)
- [Personalized Image Generation via Human-in-the-loop Bayesian Optimization](https://mlsanity.com/paper/2602.02388/) (2026)
- [Improving Text-to-Image Generation with Intrinsic Self-Confidence Rewards](https://mlsanity.com/paper/2603.00918/) (2026)
- [Diversifying Long Prompt Image Generation through Structured Prompt Embedding Space Sampling](https://mlsanity.com/paper/2511.20251/) (2025)
- [InfSplign: Inference-Time Spatial Alignment of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2512.17851/) (2025)
- [Learning Sampling Parameters for Diffusion Models](https://mlsanity.com/paper/2607.23488/) (2026)
- [Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds](https://mlsanity.com/paper/2608.03135/) (2026)
- [Score Replacement with Bounded Deviation for Rare Prompt Generation](https://mlsanity.com/paper/2505.20808/) (2025)
- [ISAC: Training-Free Instance-to-Semantic Attention Control for Multi-Instance Generation](https://mlsanity.com/paper/2505.20935/) (2025)
- [Blending Concepts with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2506.23630/) (2025)
- [Imagine for Me: Creative Conceptual Blending of Real Images and Text via Blended Attention](https://mlsanity.com/paper/2506.24085/) (2025)
- [Taming Data Challenges in ML-based Security Tasks Using Generative AI](https://mlsanity.com/paper/2507.06092/) (2025)
- [Semantic Anchoring for Robust Personalization in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2511.22245/) (2025)
- [RAIGen: Rare Attribute Identification in Text-to-Image Generative Models](https://mlsanity.com/paper/2602.06806/) (2026)
- [Delta-K: Boosting Multi-Instance Generation via Cross-Attention Augmentation](https://mlsanity.com/paper/2603.10210/) (2026)
- [ADAPT: Attention Driven Adaptive Prompt Scheduling and InTerpolating Orthogonal Complements for Rare Concepts Generation](https://mlsanity.com/paper/2603.19157/) (2026)
- [Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation](https://mlsanity.com/paper/2603.19158/) (2026)
- [ELDiff: When Evidential Learning Meets Text-to-Image Diffusion](https://mlsanity.com/paper/2606.20924/) (2026)
- [RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement](https://mlsanity.com/paper/2607.05088/) (2026)
- [Rare Concept Generation via Counterfactual Inference in Diffusion Models](https://mlsanity.com/paper/2607.14765/) (2026)
- [TILT: Improving Compositional Generation in Diffusion Models with a Model-Intrinsic Reward](https://mlsanity.com/paper/2607.21606/) (2026)

## Abstract (arXiv)

State-of-the-art text-to-image (T2I) diffusion models often struggle to generate rare compositions of concepts, e.g., objects with unusual attributes. In this paper, we show that the compositional generation power of diffusion models on such rare concepts can be significantly enhanced by the Large Language Model (LLM) guidance. We start with empirical and theoretical analysis, demonstrating that exposing frequent concepts relevant to the target rare concepts during the diffusion sampling process yields more accurate concept composition. Based on this, we propose a training-free approach, R2F, that plans and executes the overall rare-to-frequent concept guidance throughout the diffusion inference by leveraging the abundant semantic knowledge in LLMs. Our framework is flexible across any pre-trained diffusion models and LLMs, and can be seamlessly integrated with the region-guided diffusion approaches. Extensive experiments on three datasets, including our newly proposed benchmark, RareBench, containing various prompts with rare compositions of concepts, R2F significantly surpasses existing models including SD3.0 and FLUX by up to 28.1%p in T2I alignment. Code is available at https://github.com/krafton-ai/Rare-to-Frequent.