# StyleSwin: Transformer-based GAN for High-resolution Image Generation

arXiv: https://arxiv.org/abs/2112.10762 · HTML: https://mlsanity.com/paper/2112.10762/

## Key facts

- Cited by 152 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 75 references that are in the map.
- First posted on arXiv 20 Dec 2021.

## Builds on (most cited first, 30 of 61)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)

## Built on by (most cited first, 30 of 152)

- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion](https://mlsanity.com/paper/2212.06135/) (2022)
- [Photorealistic Video Generation with Diffusion Models](https://mlsanity.com/paper/2312.06662/) (2023)
- [LocalMamba: Visual State Space Model with Windowed Selective Scan](https://mlsanity.com/paper/2403.09338/) (2024)
- [Retinexformer: One-stage Retinex-based Transformer for Low-light Image Enhancement](https://mlsanity.com/paper/2303.06705/) (2023)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [Pretraining is All You Need for Image-to-Image Translation](https://mlsanity.com/paper/2205.12952/) (2022)
- [Improving Sample Quality of Diffusion Models Using Self-Attention Guidance](https://mlsanity.com/paper/2210.00939/) (2022)
- [Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers](https://mlsanity.com/paper/2401.11605/) (2024)
- [Wavelet Diffusion Models are fast and scalable Image Generators](https://mlsanity.com/paper/2211.16152/) (2022)
- [Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey](https://mlsanity.com/paper/2308.09388/) (2023)
- [DiffiT: Diffusion Vision Transformers for Image Generation](https://mlsanity.com/paper/2312.02139/) (2023)
- [Relay Diffusion: Unifying diffusion process across resolutions for image synthesis](https://mlsanity.com/paper/2309.03350/) (2023)
- [Structure-Aware Sparse-View X-ray 3D Reconstruction](https://mlsanity.com/paper/2311.10959/) (2023)
- [MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptation](https://mlsanity.com/paper/2212.08062/) (2022)
- [Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance](https://mlsanity.com/paper/2210.05559/) (2022)
- [Dilated Neighborhood Attention Transformer](https://mlsanity.com/paper/2209.15001/) (2022)
- [Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs](https://mlsanity.com/paper/2403.12553/) (2024)
- [Boosting Diffusion Models with Moving Average Sampling in Frequency Domain](https://mlsanity.com/paper/2403.17870/) (2024)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Is Artificial Intelligence Generated Image Detection a Solved Problem?](https://mlsanity.com/paper/2505.12335/) (2025)
- [Community Forensics: Using Thousands of Generators to Train Fake Image Detectors](https://mlsanity.com/paper/2411.04125/) (2024)
- [GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling](https://mlsanity.com/paper/2403.19655/) (2024)
- [Towards Semantic Equivalence of Tokenization in Multimodal LLM](https://mlsanity.com/paper/2406.05127/) (2024)
- [Generative Modeling through the Semi-dual Formulation of Unbalanced Optimal Transport](https://mlsanity.com/paper/2305.14777/) (2023)
- [StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis](https://mlsanity.com/paper/2206.09479/) (2022)
- [PTW: Pivotal Tuning Watermarking for Pre-Trained Image Generators](https://mlsanity.com/paper/2304.07361/) (2023)
- [ReMasker: Imputing Tabular Data with Masked Autoencoding](https://mlsanity.com/paper/2309.13793/) (2023)

## Abstract (arXiv)

Despite the tantalizing success in a broad of vision tasks, transformers have not yet demonstrated on-par ability as ConvNets in high-resolution image generative modeling. In this paper, we seek to explore using pure transformers to build a generative adversarial network for high-resolution image synthesis. To this end, we believe that local attention is crucial to strike the balance between computational efficiency and modeling capacity. Hence, the proposed generator adopts Swin transformer in a style-based architecture. To achieve a larger receptive field, we propose double attention which simultaneously leverages the context of the local and the shifted windows, leading to improved generation quality. Moreover, we show that offering the knowledge of the absolute position that has been lost in window-based transformers greatly benefits the generation quality. The proposed StyleSwin is scalable to high resolutions, with both the coarse geometry and fine structures benefit from the strong expressivity of transformers. However, blocking artifacts occur during high-resolution synthesis because performing the local attention in a block-wise manner may break the spatial coherency. To solve this, we empirically investigate various solutions, among which we find that employing a wavelet discriminator to examine the spectral discrepancy effectively suppresses the artifacts. Extensive experiments show the superiority over prior transformer-based GANs, especially on high resolutions, e.g., 1024x1024. The StyleSwin, without complex training strategies, excels over StyleGAN on CelebA-HQ 1024, and achieves on-par performance on FFHQ-1024, proving the promise of using transformers for high-resolution image generation. The code and models will be available at https://github.com/microsoft/StyleSwin.