mlsanityData to 14 Oct 2026

Cones: Concept Neurons in Diffusion Models for Customized Generation

arXiv 2303.05125 ↗First posted 9 Mar 2023cs.CVcited by 150

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Human brains respond to semantic features of presented stimuli with different neurons. It is then curious whether modern deep neural networks admit a similar behavior pattern. Specifically, this paper finds a small cluster of neurons in a diffusion model corresponding to a particular subject. We call those neurons the concept neurons. They can be identified by statistics of network gradients to a stimulation connected with the given subject. The concept neurons demonstrate magnetic properties in interpreting and manipulating generation results. Shutting them can directly yield the related subject contextualized in different scenes. Concatenating multiple clusters of concept neurons can vividly generate all related concepts in a single image. A few steps of further fine-tuning can enhance the multi-concept capability, which may be the first to manage to generate up to four different subjects in a single image. For large-scale applications, the concept neurons are environmentally friendly as we only need to store a sparse cluster of int index instead of dense float32 values of the parameters, which reduces storage consumption by 90% compared with previous subject-driven generation methods. Extensive qualitative and quantitative studies on diverse scenarios show the superiority of our method in interpreting and manipulating diffusion models.

Key facts. Cited by 150 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 22 of its 33 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

22 of 33 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  2. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  3. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  4. 2022Multi-Concept Customization of Text-to-Image Diffusion8 Dec
  5. 2022Sketch-Guided Text-to-Image Diffusion Models24 Nov
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation25 Aug
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  10. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidance10 Dec

Other references · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  9. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  11. 2020Countering Language Drift with Seeded Iterated Learningcited by 57
  12. 2019Countering Language Drift via Visual Groundingcited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 37; 2024: 59; 2025: 41; 2026: 13.

150 papers.

Sort:

By area:

  1. 2025VACE: All-in-One Video Creation and Editingcited by 479
  2. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  3. 2023AnyDoor: Zero-shot Object-level Image Customizationcited by 412
  4. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 359
  5. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  6. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 328
  7. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  8. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  9. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 199
  10. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  11. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  12. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 142
38 more
  1. 2023DiffusionSat: A Generative Foundation Model for Satellite Imagerycited by 131
  2. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  3. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 109
  4. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 102
  5. 2023Cones 2: Customizable Image Synthesis with Multiple Subjectscited by 101
  6. 2023VideoLCM: Video Latent Consistency Modelcited by 92
  7. 2023Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Modelscited by 83
  8. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 83
  9. 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjectscited by 77
  10. 2023Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editingcited by 76
  11. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  12. 2024Zero-shot Image Editing with Reference Imitationcited by 65
  13. 2024IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representationcited by 64
  14. 2024CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilitiescited by 60
  15. 2023MomentDiff: Generative Video Moment Retrieval from Random to Realcited by 58
  16. 2024LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Modelscited by 55
  17. 2023FaceStudio: Put Your Face Everywhere in Secondscited by 53
  18. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  19. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  20. 2024FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Compositioncited by 49
  21. 2025K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAscited by 48
  22. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  23. 2023LivePhoto: Real Image Animation with Text-guided Motion Controlcited by 44
  24. 2024Resolving Multi-Condition Confusion for Finetuning-Free Personalized Image Generationcited by 43
  25. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 43
  26. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  27. 2023AutoStory: Generating Diverse Storytelling Images with Minimal Human Effortcited by 38
  28. 2024Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Modelscited by 37
  29. 2024Customizing Text-to-Image Models with a Single Image Paircited by 36
  30. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  31. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  32. 2024Identity Decoupling for Multi-Subject Personalization of Text-to-Image Modelscited by 33
  33. 2024How to Continually Adapt Text-to-Image Diffusion Models for Flexible Customization?cited by 33
  34. 2024FlashFace: Human Image Personalization with High-fidelity Identity Preservationcited by 31
  35. 2024InstantFamily: Masked Attention for Zero-shot Multi-ID Image Generationcited by 30
  36. 2024MagicQuill: An Intelligent Interactive Image Editing Systemcited by 30
  37. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  38. 2024Conditional Image Synthesis with Diffusion Models: A Surveycited by 29

50 most cited of 150 All 150 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations
  2. 2026Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation
  3. 2023Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models
  4. 2024LLM-assisted Concept Discovery: Automatically Identifying and Explaining Neuron Functions
  5. 2024How to Blend Concepts in Diffusion Models
  6. 2024Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representations
  7. 2024Understanding Visual Concepts Across Models
  8. 2023Do text-free diffusion models learn discriminative visual representations?