# CatVersion: Concatenating Embeddings for Diffusion-Based Text-to-Image Personalization

arXiv: https://arxiv.org/abs/2311.14631 · HTML: https://mlsanity.com/paper/2311.14631/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 45 of its 56 references that are in the map.
- First posted on arXiv 24 Nov 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [Pivotal Tuning for Latent-based Editing of Real Images](https://mlsanity.com/paper/2106.05744/) (2021)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)

## Built on by (most cited first, 15 of 15)

- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [OMG: Occlusion-friendly Personalized Multi-concept Generation in Diffusion Models](https://mlsanity.com/paper/2403.10983/) (2024)
- [λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space](https://mlsanity.com/paper/2402.05195/) (2024)
- [StableIdentity: Inserting Anybody into Anywhere at First Sight](https://mlsanity.com/paper/2401.15975/) (2024)
- [A Survey on Personalized Content Synthesis with Diffusion Models](https://mlsanity.com/paper/2405.05538/) (2024)
- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [InstructBrush: Learning Attention-based Instruction Optimization for Image Editing](https://mlsanity.com/paper/2403.18660/) (2024)
- [A Comprehensive Survey on Visual Concept Mining in Text-to-image Diffusion Models](https://mlsanity.com/paper/2503.13576/) (2025)
- [MasterWeaver: Taming Editability and Face Identity for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2405.05806/) (2024)
- [AlignIT: Enhancing Prompt Alignment in Customization of Text-to-Image Models](https://mlsanity.com/paper/2406.18893/) (2024)
- [Detecting Malicious Concepts without Image Generation in AI-Generated Content (AIGC)](https://mlsanity.com/paper/2502.08921/) (2025)
- [PersonaCraft: Personalized and Controllable Full-Body Multi-Human Scene Generation Using Occlusion-Aware 3D-Conditioned Diffusion](https://mlsanity.com/paper/2411.18068/) (2024)
- [Adversarial Concept Distillation for One-Step Diffusion Personalization](https://mlsanity.com/paper/2510.20512/) (2025)
- [Single-Reference Text-to-Image Manipulation with Dual Contrastive Denoising Score](https://mlsanity.com/paper/2508.12718/) (2025)
- [PC-Diffusion: Aligning Diffusion Models with Human Preferences via Preference Classifier](https://mlsanity.com/paper/2511.07806/) (2025)

## Abstract (arXiv)

We propose CatVersion, an inversion-based method that learns the personalized concept through a handful of examples. Subsequently, users can utilize text prompts to generate images that embody the personalized concept, thereby achieving text-to-image personalization. In contrast to existing approaches that emphasize word embedding learning or parameter fine-tuning for the diffusion model, which potentially causes concept dilution or overfitting, our method concatenates embeddings on the feature-dense space of the text encoder in the diffusion model to learn the gap between the personalized concept and its base class, aiming to maximize the preservation of prior knowledge in diffusion models while restoring the personalized concepts. To this end, we first dissect the text encoder's integration in the image generation process to identify the feature-dense space of the encoder. Afterward, we concatenate embeddings on the Keys and Values in this space to learn the gap between the personalized concept and its base class. In this way, the concatenated embeddings ultimately manifest as a residual on the original attention output. To more accurately and unbiasedly quantify the results of personalized image generation, we improve the CLIP image alignment score based on masks. Qualitatively and quantitatively, CatVersion helps to restore personalization concepts more faithfully and enables more robust editing.