mlsanityData to 7 Oct 2026

ViTGAN: Training GANs with Vision Transformers

arXiv 2107.04589 ↗First posted 9 Jul 2021cs.CV cs.LG eess.IVcited by 110

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, Vision Transformers (ViTs) have shown competitive performance on image recognition while requiring less vision-specific inductive biases. In this paper, we investigate if such performance can be extended to image generation. To this end, we integrate the ViT architecture into generative adversarial networks (GANs). For ViT discriminators, we observe that existing regularization methods for GANs interact poorly with self-attention, causing serious instability during training. To resolve this issue, we introduce several novel regularization techniques for training GANs with ViTs. For ViT generators, we examine architectural choices for latent and pixel mapping layers to facilitate convergence. Empirically, our approach, named ViTGAN, achieves comparable performance to the leading CNN-based GAN models on three datasets: CIFAR-10, CelebA, and LSUN bedroom.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 57 of its 70 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

57 of 70 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
38 more
  1. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  2. 2016Improved Techniques for Training GANscited by 2,938
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  5. 2017Improved Training of Wasserstein GANscited by 2,223
  6. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  8. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  9. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  10. 2021Vision Transformers for Dense Predictioncited by 1,789
  11. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  12. 2021ViViT: A Video Vision Transformercited by 1,610
  13. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  14. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  15. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  16. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  17. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  18. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  19. 2020Pre-Trained Image Processing Transformercited by 891
  20. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  21. 2018Self-Attention Generative Adversarial Networkscited by 809
  22. 2018Which Training Methods for GANs do actually Converge?cited by 672
  23. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  24. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  25. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  26. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  27. 2021Early Convolutions Help Transformers See Bettercited by 395
  28. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  29. 2020The Lipschitz Constant of Self-Attentioncited by 171
  30. 2019Consistency Regularization for Generative Adversarial Networkscited by 134
  31. 2020A U-Net Based Discriminator for Generative Adversarial Networkscited by 128
  32. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  33. 2021Generative Adversarial Transformerscited by 120
  34. 2020Image Generators with Conditionally-Independent Pixel Synthesiscited by 105
  35. 2021Regularizing Generative Adversarial Networks under Limited Datacited by 105
  36. 2020Toward Transformer-Based Object Detectioncited by 84
  37. 2020Image Augmentations for GAN Trainingcited by 78
  38. 2020Improved Consistency Regularization for GANscited by 67

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 31; 2023: 41; 2024: 20; 2025: 12; 2026: 1.

110 papers.

Sort:

By area:

  1. 2023Consistency Modelscited by 1,778
  2. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  3. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  4. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  5. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  6. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  7. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 240
  8. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  9. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 232
  10. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  11. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  12. 2024WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wildcited by 117
38 more
  1. 2023DiffiT: Diffusion Vision Transformers for Image Generationcited by 104
  2. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  3. 2022Poisson Flow Generative Modelscited by 95
  4. 2023Reflected Diffusion Modelscited by 81
  5. 2022ActFormer: A GAN-based Transformer towards General Action-Conditioned 3D Human Motion Generationcited by 77
  6. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  7. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Modelcited by 58
  8. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 46
  9. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  10. 2022StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesiscited by 38
  11. 2022State-of-the-Art in the Architecture, Methods and Applications of StyleGANcited by 34
  12. 2024Contractive Diffusion Probabilistic Modelscited by 29
  13. 2022InstaFormer: Instance-Aware Image-to-Image Translation with Transformercited by 28
  14. 2022Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigationcited by 27
  15. 2023Conditional Denoising Diffusion for Sequential Recommendationcited by 23
  16. 2024M2D2M: Multi-Motion Generation from Text with Discrete Diffusion Modelscited by 23
  17. 2023A Recent Survey of Vision Transformers for Medical Image Segmentationcited by 22
  18. 2022NightLab: A Dual-level Architecture with Hardness Detection for Segmentation at Nightcited by 21
  19. 2022Learning Attention Propagation for Compositional Zero-Shot Learningcited by 20
  20. 2023SRFormerV2: Taking a Closer Look at Permuted Self-Attention for Image Super-Resolutioncited by 18
  21. 2023Polynomial Implicit Neural Representations For Large Diverse Datasetscited by 18
  22. 2022ITTR: Unpaired Image-to-Image Translation with Transformerscited by 15
  23. 2022Improving Diffusion Model Efficiency Through Patchingcited by 15
  24. 2022LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designercited by 15
  25. 2023Mitigating Transformer Overconfidence via Lipschitz Regularizationcited by 15
  26. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  27. 2023Human MotionFormer: Transferring Human Motions with Vision Transformerscited by 14
  28. 2024Generative AI for Unmanned Vehicle Swarms: Challenges, Applications and Opportunitiescited by 14
  29. 2021Facial-Sketch Synthesis: A New Challengecited by 13
  30. 2022Delving Globally into Texture and Structure for Image Inpaintingcited by 13
  31. 2023EGIC: Enhanced Low-Bit-Rate Generative Image Compression Guided by Semantic Segmentationcited by 13
  32. 2023Exploring Sparse MoE in GANs for Text-conditioned Image Synthesiscited by 13
  33. 2022Outpainting by Queriescited by 12
  34. 2024Decomposition-based Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentationcited by 12
  35. 2024On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Modelscited by 12
  36. 2022Multi-Tailed Vision Transformer for Efficient Inferencecited by 10
  37. 2022SCAM! Transferring humans between images with Semantic Cross Attention Modulationcited by 10
  38. 2022Infrared Image Super-Resolution: Systematic Review, and Future Trendscited by 10

50 most cited of 110 All 110 (JSON)