# ConceptLab: Creative Concept Generation using VLM-Guided Diffusion Prior Constraints

arXiv: https://arxiv.org/abs/2308.02669 · HTML: https://mlsanity.com/paper/2308.02669/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 37 of its 52 references that are in the map.
- First posted on arXiv 3 Aug 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning](https://mlsanity.com/paper/2304.03411/) (2023)
- [CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers](https://mlsanity.com/paper/2204.14217/) (2022)
- [CLIP-Mesh: Generating textured meshes from text using pretrained image-text models](https://mlsanity.com/paper/2203.13333/) (2022)
- [Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2302.12228/) (2023)
- [Key-Locked Rank One Editing for Text-to-Image Personalization](https://mlsanity.com/paper/2305.01644/) (2023)

## Built on by (most cited first, 25 of 25)

- [MyVLM: Personalizing VLMs for User-Specific Queries](https://mlsanity.com/paper/2403.14599/) (2024)
- [Enhancing Creative Generation on Stable Diffusion-based Models](https://mlsanity.com/paper/2503.23538/) (2025)
- [Novel Object Synthesis via Adaptive Text-Image Harmony](https://mlsanity.com/paper/2410.20823/) (2024)
- [FreeBlend: Advancing Concept Blending with Staged Feedback-Driven Interpolation Diffusion](https://mlsanity.com/paper/2502.05606/) (2025)
- [Piece it Together: Part-Based Concepting with IP-Priors](https://mlsanity.com/paper/2503.10365/) (2025)
- [SYNTHIA: Novel Concept Design with Affordance Composition](https://mlsanity.com/paper/2502.17793/) (2025)
- [Distribution-Conditional Generation: From Class Distribution to Creative Generation](https://mlsanity.com/paper/2505.03667/) (2025)
- [ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipeline](https://mlsanity.com/paper/2508.06094/) (2025)
- [IP-Prompter: Training-Free Theme-Specific Image Generation via Dynamic Visual Prompting](https://mlsanity.com/paper/2501.15641/) (2025)
- [VLM-Guided Adaptive Negative Prompting for Creative Generation](https://mlsanity.com/paper/2510.10715/) (2025)
- [Redefining <Creative> in Dictionary: Towards an Enhanced Semantic Understanding of Creative Generation](https://mlsanity.com/paper/2410.24160/) (2024)
- [Detecting Malicious Concepts without Image Generation in AI-Generated Content (AIGC)](https://mlsanity.com/paper/2502.08921/) (2025)
- [PartComposer: Learning and Composing Part-Level Concepts from Single-Image Examples](https://mlsanity.com/paper/2506.03004/) (2025)
- [Category-Aware 3D Object Composition with Disentangled Texture and Shape Multi-view Diffusion](https://mlsanity.com/paper/2509.02357/) (2025)
- [AGSwap: Overcoming Category Boundaries in Object Fusion via Adaptive Group Swapping](https://mlsanity.com/paper/2509.18699/) (2025)
- [Chimera: Compositional Image Generation using Part-based Concepting](https://mlsanity.com/paper/2510.18083/) (2025)
- [VMDiff: Visual Mixing Diffusion for Limitless Cross-Object Synthesis](https://mlsanity.com/paper/2509.23605/) (2025)
- [RMLer: Synthesizing Novel Objects across Diverse Categories via Reinforcement Mixing Learning](https://mlsanity.com/paper/2512.19300/) (2025)
- [Inspiration Seeds: Learning Non-Literal Visual Combinations for Generative Exploration](https://mlsanity.com/paper/2602.08615/) (2026)
- [Imagine for Me: Creative Conceptual Blending of Real Images and Text via Blended Attention](https://mlsanity.com/paper/2506.24085/) (2025)
- [CREward: A Type-Specific Creativity Reward Model](https://mlsanity.com/paper/2511.19995/) (2025)
- [A Creative Agent is Worth a 64-Token Template](https://mlsanity.com/paper/2603.17895/) (2026)
- [Seeking the Unfamiliar but Memorable: Conceptual Creativity as Meta-Learning](https://mlsanity.com/paper/2605.16477/) (2026)
- [Self-Creative Text-to-Object Generation using Semantic-Aware Spatial Weighting](https://mlsanity.com/paper/2605.19554/) (2026)
- [Semantic Browsing: Controllable Diversity for Image Generation](https://mlsanity.com/paper/2606.23679/) (2026)

## Abstract (arXiv)

Recent text-to-image generative models have enabled us to transform our words into vibrant, captivating imagery. The surge of personalization techniques that has followed has also allowed us to imagine unique concepts in new scenes. However, an intriguing question remains: How can we generate a new, imaginary concept that has never been seen before? In this paper, we present the task of creative text-to-image generation, where we seek to generate new members of a broad category (e.g., generating a pet that differs from all existing pets). We leverage the under-studied Diffusion Prior models and show that the creative generation problem can be formulated as an optimization process over the output space of the diffusion prior, resulting in a set of "prior constraints". To keep our generated concept from converging into existing members, we incorporate a question-answering Vision-Language Model (VLM) that adaptively adds new constraints to the optimization problem, encouraging the model to discover increasingly more unique creations. Finally, we show that our prior constraints can also serve as a strong mixing mechanism allowing us to create hybrids between generated concepts, introducing even more flexibility into the creative process.