# Concept Decomposition for Visual Exploration and Inspiration

arXiv: https://arxiv.org/abs/2305.18203 · HTML: https://mlsanity.com/paper/2305.18203/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 73 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuning](https://mlsanity.com/paper/2304.03411/) (2023)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](https://mlsanity.com/paper/2112.00390/) (2021)
- [P+: Extended Textual Conditioning in Text-to-Image Generation](https://mlsanity.com/paper/2303.09522/) (2023)
- [Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts](https://mlsanity.com/paper/1406.2031/) (2014)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Key-Locked Rank One Editing for Text-to-Image Personalization](https://mlsanity.com/paper/2305.01644/) (2023)
- [StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators](https://mlsanity.com/paper/2108.00946/) (2021)
- [KNN-Diffusion: Image Generation via Large-Scale Retrieval](https://mlsanity.com/paper/2204.02849/) (2022)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Deep Fragment Embeddings for Bidirectional Image Sentence Mapping](https://mlsanity.com/paper/1406.5679/) (2014)
- [ReVersion: Diffusion-Based Relation Inversion from Images](https://mlsanity.com/paper/2303.13495/) (2023)
- [The Creativity of Text-to-Image Generation](https://mlsanity.com/paper/2206.02904/) (2022)
- [CLIPascene: Scene Sketching with Different Types and Levels of Abstraction](https://mlsanity.com/paper/2211.17256/) (2022)
- [CLIPasso: Semantically-Aware Object Sketching](https://mlsanity.com/paper/2202.05822/) (2022)
- [Semantic Object Parsing with Graph LSTM](https://mlsanity.com/paper/1603.07063/) (2016)

## Built on by (most cited first, 30 of 61)

- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models](https://mlsanity.com/paper/2403.10983/) (2024)
- [The Chosen One: Consistent Characters in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2311.10093/) (2023)
- [TokenVerse: Versatile Multi-concept Personalization in Token Modulation Space](https://mlsanity.com/paper/2501.12224/) (2025)
- [UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models](https://mlsanity.com/paper/2402.11846/) (2024)
- [PALP: Prompt Aligned Personalization of Text-to-Image Models](https://mlsanity.com/paper/2401.06105/) (2024)
- [GeoDiffuser: Geometry-Based Image Editing with Diffusion Models](https://mlsanity.com/paper/2404.14403/) (2024)
- [ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglement](https://mlsanity.com/paper/2407.07197/) (2024)
- [ConceptLab: Creative Concept Generation using VLM-Guided Diffusion Prior Constraints](https://mlsanity.com/paper/2308.02669/) (2023)
- [UnZipLoRA: Separating Content and Style from a Single Image](https://mlsanity.com/paper/2412.04465/) (2024)
- [Magic Clothing: Controllable Garment-Driven Image Synthesis](https://mlsanity.com/paper/2404.09512/) (2024)
- [Distilling Diversity and Control in Diffusion Models](https://mlsanity.com/paper/2503.10637/) (2025)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [Lego: Learning to Disentangle and Invert Personalized Concepts Beyond Object Appearance in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2311.13833/) (2023)
- [Can AI Be as Creative as Humans?](https://mlsanity.com/paper/2401.01623/) (2024)
- [ConceptExpress: Harnessing Diffusion Models for Single-image Unsupervised Concept Extraction](https://mlsanity.com/paper/2407.07077/) (2024)
- [An Image is Worth Multiple Words: Discovering Object Level Concepts using Multi-Concept Prompt Learning](https://mlsanity.com/paper/2310.12274/) (2023)
- [OneActor: Consistent Character Generation via Cluster-Conditioned Guidance](https://mlsanity.com/paper/2404.10267/) (2024)
- [IP-Composer: Semantic Composition of Visual Concepts](https://mlsanity.com/paper/2502.13951/) (2025)
- [PartCraft: Crafting Creative Objects by Parts](https://mlsanity.com/paper/2407.04604/) (2024)
- [Enhancing Creative Generation on Stable Diffusion-based Models](https://mlsanity.com/paper/2503.23538/) (2025)
- [Language-Informed Visual Concept Learning](https://mlsanity.com/paper/2312.03587/) (2023)
- [Cross Initialization for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2312.15905/) (2023)
- [InstructBrush: Learning Attention-based Instruction Optimization for Image Editing](https://mlsanity.com/paper/2403.18660/) (2024)
- [Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective](https://mlsanity.com/paper/2511.07970/) (2025)
- [CusConcept: Customized Visual Concept Decomposition with Diffusion Models](https://mlsanity.com/paper/2410.00398/) (2024)
- [AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools](https://mlsanity.com/paper/2502.18736/) (2025)
- [U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation](https://mlsanity.com/paper/2403.20231/) (2024)
- [AnyDressing: Customizable Multi-Garment Virtual Dressing via Latent Diffusion Models](https://mlsanity.com/paper/2412.04146/) (2024)
- [CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering](https://mlsanity.com/paper/2501.01371/) (2025)

## Abstract (arXiv)

A creative idea is often born from transforming, combining, and modifying ideas from existing visual examples capturing various concepts. However, one cannot simply copy the concept as a whole, and inspiration is achieved by examining certain aspects of the concept. Hence, it is often necessary to separate a concept into different aspects to provide new perspectives. In this paper, we propose a method to decompose a visual concept, represented as a set of images, into different visual aspects encoded in a hierarchical tree structure. We utilize large vision-language models and their rich latent space for concept decomposition and generation. Each node in the tree represents a sub-concept using a learned vector embedding injected into the latent space of a pretrained text-to-image model. We use a set of regularizations to guide the optimization of the embedding vectors encoded in the nodes to follow the hierarchical structure of the tree. Our method allows to explore and discover new concepts derived from the original one. The tree provides the possibility of endless visual sampling at each node, allowing the user to explore the hidden sub-concepts of the object of interest. The learned aspects in each node can be combined within and across trees to create new visual ideas, and can be used in natural language sentences to apply such aspects to new designs.