# Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion

arXiv: https://arxiv.org/abs/2310.03502 · HTML: https://mlsanity.com/paper/2310.03502/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 40 of its 44 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 40)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)

## Built on by (most cited first, 30 of 114)

- [BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion](https://mlsanity.com/paper/2403.06976/) (2024)
- [MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance](https://mlsanity.com/paper/2406.07209/) (2024)
- [AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error](https://mlsanity.com/paper/2401.17879/) (2024)
- [Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding](https://mlsanity.com/paper/2510.06308/) (2025)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?](https://mlsanity.com/paper/2407.04842/) (2024)
- [λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space](https://mlsanity.com/paper/2402.05195/) (2024)
- [T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation](https://mlsanity.com/paper/2501.12612/) (2025)
- [Margin-aware Preference Optimization for Aligning Diffusion Models without Reference](https://mlsanity.com/paper/2406.06424/) (2024)
- [Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modeling](https://mlsanity.com/paper/2507.17801/) (2025)
- [Aligned Datasets Improve Detection of Latent Diffusion-Generated Images](https://mlsanity.com/paper/2410.11835/) (2024)
- [Black-box Membership Inference Attacks against Fine-tuned Diffusion Models](https://mlsanity.com/paper/2312.08207/) (2023)
- [MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation](https://mlsanity.com/paper/2404.05674/) (2024)
- [T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching](https://mlsanity.com/paper/2402.14167/) (2024)
- [BrushEdit: All-In-One Image Inpainting and Editing](https://mlsanity.com/paper/2412.10316/) (2024)
- [Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models](https://mlsanity.com/paper/2403.14291/) (2024)
- [Contrasting Deepfakes Diffusion via Contrastive Learning and Global-Local Similarities](https://mlsanity.com/paper/2407.20337/) (2024)
- [How to Trace Latent Generative Model Generated Images without Artificial Watermark?](https://mlsanity.com/paper/2405.13360/) (2024)
- [Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework](https://mlsanity.com/paper/2410.21061/) (2024)
- [ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations](https://mlsanity.com/paper/2312.04655/) (2023)
- [ZeroRF: Fast Sparse View 360deg Reconstruction with Zero Pretraining](https://mlsanity.com/paper/2312.09249/) (2023)
- [A Practitioner's Guide to Continual Multimodal Pretraining](https://mlsanity.com/paper/2408.14471/) (2024)
- [Towards Reliable Verification of Unauthorized Data Usage in Personalized Text-to-Image Diffusion Models](https://mlsanity.com/paper/2410.10437/) (2024)
- [Improving Long-Text Alignment for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2410.11817/) (2024)
- [PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference](https://mlsanity.com/paper/2410.21966/) (2024)
- [When Worse is Better: Navigating the compression-generation tradeoff in visual tokenization](https://mlsanity.com/paper/2412.16326/) (2024)
- [Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings](https://mlsanity.com/paper/2412.18645/) (2024)
- [Kiki or Bouba? Sound Symbolism in Vision-and-Language Models](https://mlsanity.com/paper/2310.16781/) (2023)
- [Evaluating and Mitigating IP Infringement in Visual Generative AI](https://mlsanity.com/paper/2406.04662/) (2024)
- [DragText: Rethinking Text Embedding in Point-based Image Editing](https://mlsanity.com/paper/2407.17843/) (2024)

## Abstract (arXiv)

Text-to-image generation is a significant domain in modern computer vision and has achieved substantial improvements through the evolution of generative architectures. Among these, there are diffusion-based models that have demonstrated essential quality enhancements. These models are generally split into two categories: pixel-level and latent-level approaches. We present Kandinsky1, a novel exploration of latent diffusion architecture, combining the principles of the image prior models with latent diffusion techniques. The image prior model is trained separately to map text embeddings to image embeddings of CLIP. Another distinct feature of the proposed model is the modified MoVQ implementation, which serves as the image autoencoder component. Overall, the designed model contains 3.3B parameters. We also deployed a user-friendly demo system that supports diverse generative modes such as text-to-image generation, image fusion, text and image fusion, image variations generation, and text-guided inpainting/outpainting. Additionally, we released the source code and checkpoints for the Kandinsky models. Experimental evaluations demonstrate a FID score of 8.03 on the COCO-30K dataset, marking our model as the top open-source performer in terms of measurable image generation quality.