# Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision Transformers

arXiv: https://arxiv.org/abs/2310.05400 · HTML: https://mlsanity.com/paper/2310.05400/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 49 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 30 of 44)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets](https://mlsanity.com/paper/2202.00273/) (2022)

## Built on by (most cited first, 30 of 33)

- [An Image is Worth 32 Tokens for Reconstruction and Generation](https://mlsanity.com/paper/2406.07550/) (2024)
- [Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2510.11690/) (2025)
- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior](https://mlsanity.com/paper/2410.21264/) (2024)
- [Autoregressive Models in Vision: A Survey](https://mlsanity.com/paper/2411.05902/) (2024)
- [HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes](https://mlsanity.com/paper/2401.00365/) (2023)
- [Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey](https://mlsanity.com/paper/2412.18619/) (2024)
- [StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis](https://mlsanity.com/paper/2401.17093/) (2024)
- [CAT: Content-Adaptive Image Tokenization](https://mlsanity.com/paper/2501.03120/) (2025)
- [SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.06764/) (2025)
- [DetailFlow: 1D Coarse-to-Fine Autoregressive Image Generation via Next-Detail Prediction](https://mlsanity.com/paper/2505.21473/) (2025)
- [Highly Compressed Tokenizer Can Generate Without Training](https://mlsanity.com/paper/2506.08257/) (2025)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [Quantize-then-Rectify: Efficient VQ-VAE Training](https://mlsanity.com/paper/2507.10547/) (2025)
- [Improving Flexible Image Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2601.01535/) (2026)
- [PanoLlama: Generating Endless and Coherent Panoramas with Next-Token-Prediction LLMs](https://mlsanity.com/paper/2411.15867/) (2024)
- [Scaling Image Tokenizers with Grouped Spherical Quantization](https://mlsanity.com/paper/2412.02632/) (2024)
- [Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression](https://mlsanity.com/paper/2506.09482/) (2025)
- [Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering](https://mlsanity.com/paper/2606.01911/) (2026)
- [Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation](https://mlsanity.com/paper/2411.10281/) (2024)
- [GloTok: Global Perspective Tokenizer for Image Reconstruction and Generation](https://mlsanity.com/paper/2511.14184/) (2025)
- [UniAPO: Unified Multimodal Automated Prompt Optimization](https://mlsanity.com/paper/2508.17890/) (2025)
- [SFTok: Bridging the Performance Gap in Discrete Tokenizers](https://mlsanity.com/paper/2512.16910/) (2025)
- [Beyond Patches: Global-aware Autoregressive Model for Multimodal Few-Shot Font Generation](https://mlsanity.com/paper/2601.01593/) (2026)
- [ImgCoT: Compressing Long Chain of Thought into Compact Visual Tokens for Efficient Reasoning of Large Language Model](https://mlsanity.com/paper/2601.22730/) (2026)
- [Improving Detail in Pluralistic Image Inpainting with Feature Dequantization](https://mlsanity.com/paper/2412.01046/) (2024)
- [Tokenize Image as a Set](https://mlsanity.com/paper/2503.16425/) (2025)
- [D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokens](https://mlsanity.com/paper/2503.17155/) (2025)
- [UniCode2: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2506.20214/) (2025)
- [NativeTok: Native Visual Tokenization for Improved Image Generation](https://mlsanity.com/paper/2601.22837/) (2026)

## Abstract (arXiv)

Vector-quantized image modeling has shown great potential in synthesizing high-quality images. However, generating high-resolution images remains a challenging task due to the quadratic computational overhead of the self-attention process. In this study, we seek to explore a more efficient two-stage framework for high-resolution image generation with improvements in the following three aspects. (1) Based on the observation that the first quantization stage has solid local property, we employ a local attention-based quantization model instead of the global attention mechanism used in previous methods, leading to better efficiency and reconstruction quality. (2) We emphasize the importance of multi-grained feature interaction during image generation and introduce an efficient attention mechanism that combines global attention (long-range semantic consistency within the whole image) and local attention (fined-grained details). This approach results in faster generation speed, higher generation fidelity, and improved resolution. (3) We propose a new generation pipeline incorporating autoencoding training and autoregressive generation strategy, demonstrating a better paradigm for image synthesis. Extensive experiments demonstrate the superiority of our approach in high-quality and high-resolution image reconstruction and generation.