mlsanityData to 7 Oct 2026

Vector-quantized Image Modeling with Improved VQGAN

arXiv 2110.04627 ↗First posted 9 Oct 2021cs.CV cs.LGcited by 656

Not on the map. Why?

Cited by 132 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretraining language models with next-token prediction on massive text corpora has delivered phenomenal zero-shot, few-shot, transfer learning and multi-tasking capabilities on both generative and discriminative language tasks. Motivated by this success, we explore a Vector-quantized Image Modeling (VIM) approach that involves pretraining a Transformer to predict rasterized image tokens autoregressively. The discrete image tokens are encoded from a learned Vision-Transformer-based VQGAN (ViT-VQGAN). We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. The improved ViT-VQGAN further improves vector-quantized image modeling tasks, including unconditional, class-conditioned image generation and unsupervised representation learning. When trained on ImageNet at (256×256) resolution, we achieve Inception Score (IS) of 175.1 and Fr'echet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN, which obtains 70.6 and 17.04 for IS and FID, respectively. Based on ViT-VQGAN and unsupervised pretraining, we further evaluate the pretrained Transformer by averaging intermediate features, similar to Image GPT (iGPT). This ImageNet-pretrained VIM-L significantly beats iGPT-L on linear-probe accuracy from 60.3% to 73.2% for a similar model size. VIM-L also outperforms iGPT-XL which is trained with extra web image data and larger model size.

Key facts. Cited by 656 papers in the map (data through 7 Oct 2026). Builds on 57 of its 69 references that are in the map. First posted on arXiv 9 Oct 2021.

Builds on

57 of 69 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
38 more
  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  13. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  14. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  15. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  16. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  17. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  18. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  19. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  20. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  21. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  22. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  23. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  24. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  25. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  26. 2018Self-Attention Generative Adversarial Networkscited by 809
  27. 2018Image Transformercited by 770
  28. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  29. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  30. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  31. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  32. 2016Adversarial Feature Learningcited by 350
  33. 2021Generating Images with Sparse Representationscited by 262
  34. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  35. 2019Large Scale Adversarial Representation Learningcited by 238
  36. 2015Semi-supervised Sequence Learningcited by 198
  37. 2020Adversarial Latent Autoencoderscited by 155
  38. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 41; 2023: 122; 2024: 172; 2025: 211; 2026: 105.

656 papers.

Sort:

By area:

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  6. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  7. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  8. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  9. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  10. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  11. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  12. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
38 more
  1. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  3. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  4. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  5. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  6. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  7. 2022MAGVIT: Masked Generative Video Transformercited by 341
  8. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  9. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  10. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  11. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  12. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  13. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  14. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  15. 2023The Stable Signature: Rooting Watermarks in Latent Diffusion Modelscited by 289
  16. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  17. 2022Visual Prompting via Image Inpaintingcited by 248
  18. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  19. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  20. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
  21. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  22. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  23. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  24. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  25. 2023Recommender Systems with Generative Retrievalcited by 191
  26. 2023Geometric Latent Diffusion Models for 3D Molecule Generationcited by 185
  27. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
  28. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  29. 2023Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorscited by 156
  30. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 153
  31. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  32. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  33. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  34. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  35. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  36. 2022Semantic Image Synthesis via Diffusion Modelscited by 131
  37. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  38. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129

50 most cited of 656 All 656 (JSON)