# Shifted Diffusion for Text-to-image Generation

arXiv: https://arxiv.org/abs/2211.15388 · HTML: https://mlsanity.com/paper/2211.15388/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 24 Nov 2022.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers](https://mlsanity.com/paper/2204.14217/) (2022)
- [Cross-Modal Contrastive Learning for Text-to-Image Generation](https://mlsanity.com/paper/2101.04702/) (2021)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [Re-Imagen: Retrieval-Augmented Text-to-Image Generator](https://mlsanity.com/paper/2209.14491/) (2022)
- [LAFITE: Towards Language-Free Training for Text-to-Image Generation](https://mlsanity.com/paper/2111.13792/) (2021)
- [Improved Vector Quantized Diffusion Models](https://mlsanity.com/paper/2205.16007/) (2022)
- [Lafite2: Few-shot Text-to-Image Generation](https://mlsanity.com/paper/2210.14124/) (2022)

## Built on by (most cited first, 30 of 37)

- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach](https://mlsanity.com/paper/2305.13579/) (2023)
- [Scaling Diffusion Models to Real-World 3D LiDAR Scene Completion](https://mlsanity.com/paper/2403.13470/) (2024)
- [Exploiting the Signal-Leak Bias in Diffusion Models](https://mlsanity.com/paper/2309.15842/) (2023)
- [Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation](https://mlsanity.com/paper/2401.05675/) (2024)
- [Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data](https://mlsanity.com/paper/2401.08567/) (2024)
- [Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution](https://mlsanity.com/paper/2412.15213/) (2024)
- [A Survey of Diffusion Based Image Generation Models: Issues and Their Solutions](https://mlsanity.com/paper/2308.13142/) (2023)
- [Grid Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2404.00234/) (2024)
- [Highly Compressed Tokenizer Can Generate Without Training](https://mlsanity.com/paper/2506.08257/) (2025)
- [PRedItOR: Text Guided Image Editing with Diffusion Prior](https://mlsanity.com/paper/2302.07979/) (2023)
- [One Image is Worth a Thousand Words: A Usability Preservable Text-Image Collaborative Erasing Framework](https://mlsanity.com/paper/2505.11131/) (2025)
- [NFIG: Multi-Scale Autoregressive Image Generation via Frequency Ordering](https://mlsanity.com/paper/2503.07076/) (2025)
- [Novel Object Synthesis via Adaptive Text-Image Harmony](https://mlsanity.com/paper/2410.20823/) (2024)
- [LocInv: Localization-aware Inversion for Text-Guided Image Editing](https://mlsanity.com/paper/2405.01496/) (2024)
- [DiffGAD: A Diffusion-based Unsupervised Graph Anomaly Detector](https://mlsanity.com/paper/2410.06549/) (2024)
- [FreeBlend: Advancing Concept Blending with Staged Feedback-Driven Interpolation Diffusion](https://mlsanity.com/paper/2502.05606/) (2025)
- [Video Generation with Stable Transparency via Shiftable RGB-A Distribution Learner](https://mlsanity.com/paper/2509.24979/) (2025)
- [Controlled and Conditional Text to Image Generation with Diffusion Prior](https://mlsanity.com/paper/2302.11710/) (2023)
- [One-for-All: Towards Universal Domain Translation with a Single StyleGAN](https://mlsanity.com/paper/2310.14222/) (2023)
- [Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models](https://mlsanity.com/paper/2507.11554/) (2025)
- [Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving](https://mlsanity.com/paper/2503.21449/) (2025)
- [Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation](https://mlsanity.com/paper/2406.09305/) (2024)
- [Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation](https://mlsanity.com/paper/2502.17198/) (2025)
- [Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text](https://mlsanity.com/paper/2503.01261/) (2025)
- [ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models](https://mlsanity.com/paper/2406.12044/) (2024)
- [Detecting AI-generated Artwork](https://mlsanity.com/paper/2504.07078/) (2025)
- [FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching](https://mlsanity.com/paper/2604.06757/) (2026)
- [Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models](https://mlsanity.com/paper/2512.03463/) (2025)
- [AdaGen: Learning Adaptive Policy for Image Synthesis](https://mlsanity.com/paper/2603.06993/) (2026)

## Abstract (arXiv)

We present Corgi, a novel method for text-to-image generation. Corgi is based on our proposed shifted diffusion model, which achieves better image embedding generation from input text. Unlike the baseline diffusion model used in DALL-E 2, our method seamlessly encodes prior knowledge of the pre-trained CLIP model in its diffusion process by designing a new initialization distribution and a new transition step of the diffusion. Compared to the strong DALL-E 2 baseline, our method performs better in generating image embedding from the text in terms of both efficiency and effectiveness, resulting in better text-to-image generation. Extensive large-scale experiments are conducted and evaluated in terms of both quantitative measures and human evaluation, indicating a stronger generation ability of our method compared to existing ones. Furthermore, our model enables semi-supervised and language-free training for text-to-image generation, where only part or none of the images in the training dataset have an associated caption. Trained with only 1.7% of the images being captioned, our semi-supervised model obtains FID results comparable to DALL-E 2 on zero-shot text-to-image generation evaluated on MS-COCO. Corgi also achieves new state-of-the-art results across different datasets on downstream language-free text-to-image generation tasks, outperforming the previous method, Lafite, by a large margin.