mlsanityData to 7 Oct 2026

Cross-Modal Contrastive Learning for Text-to-Image Generation

arXiv 2101.04702 ↗First posted 12 Jan 2021cs.CVcited by 259

Not on the map. Why?

Cited by 87 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The output of text-to-image synthesis systems should be coherent, clear, photo-realistic scenes with high semantic fidelity to their conditioned text descriptions. Our Cross-Modal Contrastive Generative Adversarial Network (XMC-GAN) addresses this challenge by maximizing the mutual information between image and text. It does this via multiple contrastive losses which capture inter-modality and intra-modality correspondences. XMC-GAN uses an attentional self-modulation generator, which enforces strong text-image correspondence, and a contrastive discriminator, which acts as a critic as well as a feature encoder for contrastive learning. The quality of XMC-GAN's output is a major step up from previous models, as we show on three challenging datasets. On MS-COCO, not only does XMC-GAN improve state-of-the-art FID from 24.70 to 9.33, but--more importantly--people prefer XMC-GAN by 77.3 for image quality and 74.1 for image-text alignment, compared to three other recent models. XMC-GAN also generalizes to the challenging Localized Narratives dataset (which has longer, more detailed descriptions), improving state-of-the-art FID from 48.70 to 14.12. Lastly, we train and evaluate XMC-GAN on the challenging Open Images data, establishing a strong benchmark FID score of 26.91.

Key facts. Cited by 259 papers in the map (data through 7 Oct 2026). Builds on 49 of its 65 references that are in the map. First posted on arXiv 12 Jan 2021.

Builds on

49 of 65 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2013Auto-Encoding Variational Bayescited by 3,980
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  11. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
37 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  3. 2016Least Squares Generative Adversarial Networkscited by 1,000
  4. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  5. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  6. 2018Self-Attention Generative Adversarial Networkscited by 809
  7. 2016Generative Adversarial Text to Image Synthesiscited by 735
  8. 2018MINE: Mutual Information Neural Estimationcited by 680
  9. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  10. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 637
  11. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  12. 2018A Note on the Inception Scorecited by 225
  13. 2019Connecting Vision and Language with Localized Narrativescited by 225
  14. 2020Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learningcited by 219
  15. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  16. 2015DRAW: A Recurrent Neural Network For Image Generationcited by 190
  17. 2019Controllable Text-to-Image Generationcited by 187
  18. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  19. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  20. 2019Consistency Regularization for Generative Adversarial Networkscited by 134
  21. 2018Improving GANs Using Optimal Transportcited by 122
  22. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  23. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  24. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  25. 2020Image Augmentations for GAN Trainingcited by 78
  26. 2017Parallel Multiscale Autoregressive Density Estimationcited by 71
  27. 2018Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Networkcited by 64
  28. 2019Semantics Disentangling for Text-to-Image Generationcited by 61
  29. 2018Perfect match: Improved cross-modal embeddings for audio-visual synchronisationcited by 60
  30. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46
  31. 2019Generating Multiple Objects at Spatially Distinct Locationscited by 41
  32. 2020InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learningcited by 37
  33. 2020Text-to-Image Generation Grounded by Fine-Grained User Attentioncited by 36
  34. 2018On Self Modulation for Generative Adversarial Networkscited by 33
  35. 2019Dual Adversarial Inference for Text-to-Image Synthesiscited by 17
  36. 2018Geometry-Contrastive GAN for Facial Expression Transfercited by 16
  37. 2019CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesiscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 72; 2023: 83; 2024: 46; 2025: 28; 2026: 7.

259 papers.

Sort:

By area:

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  7. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  8. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  9. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  10. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  11. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  12. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
38 more
  1. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  2. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  3. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  4. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  5. 2023InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generationcited by 387
  6. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  7. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  8. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  9. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  10. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  11. 2023Generating Images with Multimodal Language Modelscited by 259
  12. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  13. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  14. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  15. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  16. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  17. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
  18. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  19. 2022DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Modelscited by 167
  20. 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 163
  21. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  22. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  23. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  24. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  25. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  26. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
  27. 2024InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimizationcited by 123
  28. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  29. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  30. 2023AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusioncited by 98
  31. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  32. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  33. 2023Improving Tuning-Free Real Image Editing with Proximal Guidancecited by 87
  34. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 84
  35. 2023A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesiscited by 78
  36. 2023Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Followingcited by 77
  37. 2022Make-A-Story: Visual Memory Conditioned Consistent Story Generationcited by 76
  38. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75

50 most cited of 259 All 259 (JSON)