mlsanityData to 14 Oct 2026

Spectral Image Tokenizer

arXiv 2412.09607 ↗First posted 12 Dec 2024cs.CV cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Image tokenizers map images to sequences of discrete tokens, and are a crucial component of autoregressive transformer-based image generation. The tokens are typically associated with spatial locations in the input image, arranged in raster scan order, which is not ideal for autoregressive modeling. In this paper, we propose to tokenize the image spectrum instead, obtained from a discrete wavelet transform (DWT), such that the sequence of tokens represents the image in a coarse-to-fine fashion. Our tokenizer brings several advantages: 1) it leverages that natural images are more compressible at high frequencies, 2) it can take and reconstruct images of different resolutions without retraining, 3) it improves the conditioning for next-token prediction -- instead of conditioning on a partial line-by-line reconstruction of the image, it takes a coarse reconstruction of the full image, 4) it enables partial decoding where the first few generated tokens can reconstruct a coarse version of the image, 5) it enables autoregressive models to be used for image upsampling. We evaluate the tokenizer reconstruction metrics as well as multiscale image generation, text-guided image upsampling and editing.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 39 of its 63 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

39 of 63 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
27 more
  1. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  6. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  8. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  9. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  10. 2016Pixel Recurrent Neural Networkscited by 816
  11. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  12. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  13. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  14. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  15. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  16. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  17. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  18. 2021Generating Images with Sparse Representationscited by 266
  19. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137
  20. 2022Wavelet Diffusion Models are fast and scalable Image Generatorscited by 117
  21. 2022Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learningcited by 78
  22. 2023Matryoshka Diffusion Modelscited by 72
  23. 2022SinDDM: A Single Image Denoising Diffusion Modelcited by 60
  24. 2023Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantizationcited by 60
  25. 2017Multi-Stage Variational Auto-Encoders for Coarse-to-Fine Image Generationcited by 18
  26. 2024Wavelets Are All You Need for Autoregressive Image Generationcited by 9
  27. 2024Quantised Global Autoencoder: A Holistic Approach to Representing Visual Datacited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025Improving the Diffusability of Autoencoderscited by 80
  2. 2024DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Spacecited by 34
  3. 2025Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Modelscited by 17
  4. 2025H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Modelscited by 13
  5. 2025SpectralAR: Spectral Autoregressive Visual Generationcited by 11
  6. 2025Latent Wavelet Diffusion For Ultra-High-Resolution Image Synthesiscited by 7
  7. 2025Efficient Multi-Camera Tokenization with Triplanes for End-to-End Drivingcited by 6
  8. 2025Improving Progressive Generation with Decomposable Flow Matchingcited by 6
  9. 2026Improving Flexible Image Tokenizers for Autoregressive Image Generationcited by 6
  10. 2025Flow Along the K-Amplitude for Generative Modelingcited by 3
  11. 2026Balancing Image Compression and Generation with Bootstrapped Tokenizationcited by 1
  12. 2026Decomposing Private Image Generation via Coarse-to-Fine Wavelet Modelingcited by 0
3 more
  1. 2026Structured State-Space Regularization for Generation-Friendly Image Tokenizationcited by 0
  2. 2026Spectral Vision Transformer for Efficient Tokenization with Limited Datacited by 0
  3. 2026MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Expertscited by 0