mlsanityData to 7 Oct 2026

Generating Images with Sparse Representations

arXiv 2103.03841 ↗First posted 5 Mar 2021cs.CV stat.MLcited by 262

Not on the map. Why?

Cited by 179 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The high dimensionality of images presents architecture and sampling-efficiency challenges for likelihood-based generative models. Previous approaches such as VQ-VAE use deep autoencoders to obtain compact representations, which are more practical as inputs for likelihood-based models. We present an alternative approach, inspired by common image compression methods like JPEG, and convert images to quantized discrete cosine transform (DCT) blocks, which are represented sparsely as a sequence of DCT channel, spatial location, and DCT coefficient triples. We propose a Transformer-based autoregressive architecture, which is trained to sequentially predict the conditional distribution of the next element in such sequences, and which scales effectively to high resolution images. On a range of image datasets, we demonstrate that our approach can generate high quality, diverse images, with sample metric scores competitive with state of the art methods. We additionally show that simple modifications to our method yield effective image colorization and super-resolution models.

Key facts. Cited by 262 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 5 Mar 2021.

Builds on

33 of 55 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  11. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  12. 2015Variational Inference with Normalizing Flowscited by 2,161
21 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2020Reformer: The Efficient Transformercited by 1,658
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  7. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  8. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  9. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  10. 2016Pixel Recurrent Neural Networkscited by 814
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  12. 2020Jukebox: A Generative Model for Musiccited by 535
  13. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  14. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  15. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  16. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  17. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  18. 2018Efficient Neural Audio Synthesiscited by 218
  19. 2018Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscalingcited by 74
  20. 2017Parallel Multiscale Autoregressive Density Estimationcited by 71
  21. 2020Pay Attention when Requiredcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 30; 2023: 35; 2024: 59; 2025: 75; 2026: 52.

262 papers.

Sort:

By area:

  1. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  4. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  5. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  6. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  7. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  8. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  9. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  10. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 266
  11. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  12. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
38 more
  1. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformerscited by 212
  2. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
  3. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 199
  4. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  5. 2023Fast Training of Diffusion Models with Masked Transformerscited by 172
  6. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  7. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 151
  8. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 138
  9. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  10. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functionscited by 133
  11. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  12. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  13. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 121
  14. 2022ShapeFormer: Transformer-based Shape Completion via Sparse Representationcited by 106
  15. 2022Accelerating Diffusion Models via Early Stop of the Diffusion Processcited by 100
  16. 2025DDT: Decoupled Diffusion Transformercited by 100
  17. 2022Refining Generative Process with Discriminator Guidance in Score-based Diffusion Modelscited by 98
  18. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 95
  19. 2025PixelFlow: Pixel-Space Generative Models with Flowcited by 91
  20. 2023Input Perturbation Reduces Exposure Bias in Diffusion Modelscited by 88
  21. 2025EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modelingcited by 85
  22. 2024FiT: Flexible Vision Transformer for Diffusion Modelcited by 84
  23. 2023Relay Diffusion: Unifying diffusion process across resolutions for image synthesiscited by 78
  24. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 77
  25. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 76
  26. 2023Diffusion Models Without Attentioncited by 75
  27. 2025PixNerd: Pixel Neural Field Diffusioncited by 75
  28. 2022A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Imagescited by 74
  29. 2021Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codescited by 71
  30. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  31. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  32. 2024Dynamic Diffusion Transformercited by 60
  33. 2023Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantizationcited by 59
  34. 2023LayoutDM: Transformer-based Diffusion Model for Layout Generationcited by 57
  35. 2024DiG: Scalable and Efficient Diffusion Models with Gated Linear Attentioncited by 56
  36. 2025DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generationcited by 55
  37. 2023PanoDiffusion: 360-degree Panorama Outpainting via Diffusioncited by 50
  38. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 50

50 most cited of 262 All 262 (JSON)