# Dual Contrastive Loss and Attention for GANs

arXiv: https://arxiv.org/abs/2103.16748 · HTML: https://mlsanity.com/paper/2103.16748/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 92 of its 112 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 30 of 92)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)

## Built on by (most cited first, 30 of 30)

- [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021)
- [Data-Efficient Instance Generation from Instance Discrimination](https://mlsanity.com/paper/2106.04566/) (2021)
- [Generative Models: What Do They Know? Do They Know Things? Let's Find Out!](https://mlsanity.com/paper/2311.17137/) (2023)
- [Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis](https://mlsanity.com/paper/2105.14376/) (2021)
- [StyleGAN knows Normal, Depth, Albedo, and More](https://mlsanity.com/paper/2306.00987/) (2023)
- [A Practical Contrastive Learning Framework for Single-Image Super-Resolution](https://mlsanity.com/paper/2111.13924/) (2021)
- [Modulated Contrast for Versatile Image Synthesis](https://mlsanity.com/paper/2203.09333/) (2022)
- [Improving GANs with A Dynamic Discriminator](https://mlsanity.com/paper/2209.09897/) (2022)
- [Diverse Inpainting and Editing with GAN Inversion](https://mlsanity.com/paper/2307.15033/) (2023)
- [View Generalization for Single Image Textured 3D Models](https://mlsanity.com/paper/2106.06533/) (2021)
- [Fine Detailed Texture Learning for 3D Meshes with Generative Models](https://mlsanity.com/paper/2203.09362/) (2022)
- [LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer](https://mlsanity.com/paper/2212.09877/) (2022)
- [Image-to-Image Translation with Disentangled Latent Vectors for Face Editing](https://mlsanity.com/paper/2301.04628/) (2023)
- [Progressive Learning of 3D Reconstruction Network from 2D GAN Data](https://mlsanity.com/paper/2305.11102/) (2023)
- [Make It So: Steering StyleGAN for Any Image Inversion and Editing](https://mlsanity.com/paper/2304.14403/) (2023)
- [Unsupervised HDR Image and Video Tone Mapping via Contrastive Learning](https://mlsanity.com/paper/2303.07327/) (2023)
- [Warping the Residuals for Image Editing with StyleGAN](https://mlsanity.com/paper/2312.11422/) (2023)
- [Transferring Unconditional to Conditional GANs with Hyper-Modulation](https://mlsanity.com/paper/2112.02219/) (2021)
- [StyLitGAN: Prompting StyleGAN to Produce New Illumination Conditions](https://mlsanity.com/paper/2205.10351/) (2022)
- [PaDPaF: Partial Disentanglement with Partially-Federated GANs](https://mlsanity.com/paper/2212.03836/) (2022)
- [Structure-Guided Image Completion with Image-level and Object-level Semantic Discriminators](https://mlsanity.com/paper/2212.06310/) (2022)
- [DGFont++: Robust Deformable Generative Networks for Unsupervised Font Generation](https://mlsanity.com/paper/2212.14742/) (2022)
- [Generative Adversarial Learning via Kernel Density Discrimination](https://mlsanity.com/paper/2107.06197/) (2021)
- [SP-BatikGAN: An Efficient Generative Adversarial Network for Symmetric Pattern Generation](https://mlsanity.com/paper/2304.09384/) (2023)
- [Synthesis of Batik Motifs using a Diffusion -- Generative Adversarial Network](https://mlsanity.com/paper/2307.12122/) (2023)
- [Improving the Latent Space of Image Style Transfer](https://mlsanity.com/paper/2205.12135/) (2022)
- [StyleRes: Transforming the Residuals for Real Image Editing with StyleGAN](https://mlsanity.com/paper/2212.14359/) (2022)
- [ComGAN: Toward GANs Exploiting Multiple Samples](https://mlsanity.com/paper/2304.12098/) (2023)
- [Enhancing Robustness in Post-Processing Watermarking: An Ensemble Attack Network Using CNNs and Transformers](https://mlsanity.com/paper/2509.03006/) (2025)
- [HP-GAN: Harnessing pretrained networks for GAN improvement with FakeTwins and discriminator consistency](https://mlsanity.com/paper/2602.03039/) (2026)

## Abstract (arXiv)

Generative Adversarial Networks (GANs) produce impressive results on unconditional image generation when powered with large-scale image datasets. Yet generated images are still easy to spot especially on datasets with high variance (e.g. bedroom, church). In this paper, we propose various improvements to further push the boundaries in image generation. Specifically, we propose a novel dual contrastive loss and show that, with this loss, discriminator learns more generalized and distinguishable representations to incentivize generation. In addition, we revisit attention and extensively experiment with different attention blocks in the generator. We find attention to be still an important module for successful image generation even though it was not used in the recent state-of-the-art models. Lastly, we study different attention architectures in the discriminator, and propose a reference attention mechanism. By combining the strengths of these remedies, we improve the compelling state-of-the-art Fr'echet Inception Distance (FID) by at least 17.5% on several benchmark datasets. We obtain even more significant improvements on compositional synthetic scenes (up to 47.5% in FID). Code and models are available at https://github.com/ningyu1991/AttentionDualContrastGAN .