mlsanityData to 14 Oct 2026

SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer

arXiv 2412.10958 ↗First posted 14 Dec 2024cs.CV cs.AI cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

Efficient image tokenization with high compression ratios remains a critical challenge for training generative models. We present SoftVQ-VAE, a continuous image tokenizer that leverages soft categorical posteriors to aggregate multiple codewords into each latent token, substantially increasing the representation capacity of the latent space. When applied to Transformer-based architectures, our approach compresses 256x256 and 512x512 images using as few as 32 or 64 1-dimensional tokens. Not only does SoftVQ-VAE show consistent and high-quality reconstruction, more importantly, it also achieves state-of-the-art and significantly faster image generation results across different denoising-based generative models. Remarkably, SoftVQ-VAE improves inference throughput by up to 18x for generating 256x256 images and 55x for 512x512 images while achieving competitive FID scores of 1.78 and 2.21 for SiT-XL. It also improves the training efficiency of the generative models by reducing the number of training iterations by 2.3x while maintaining comparable performance. With its fully-differentiable design and semantic-rich latent space, our experiment demonstrates that SoftVQ-VAE achieves efficient tokenization without compromising generation quality, paving the way for more efficient generative models. Code and model are released.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 99 of its 120 references that are in the map. First posted on arXiv 14 Dec 2024.

Builds on

99 of 120 references.

References · 99

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2022Scalable Diffusion Models with Transformerscited by 6,054
38 more
  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2022Flow Matching for Generative Modelingcited by 5,702
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  13. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  14. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  15. 2016Improved Techniques for Training GANscited by 2,970
  16. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  17. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  18. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  19. 2022Video Diffusion Modelscited by 2,305
  20. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  21. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  22. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  23. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  24. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  25. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  26. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  27. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  28. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  29. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  30. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  31. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  32. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  33. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  34. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  35. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  36. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  37. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  38. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 24; 2026: 31.

56 papers.

Sort:

By area:

  1. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 96
  2. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 57
  3. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Spacecited by 47
  4. 2025AToken: A Unified Tokenizer for Visioncited by 30
  5. 2024Next Patch Prediction for Autoregressive Visual Generationcited by 26
  6. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  7. 2025Latent Denoising Makes Good Tokenizerscited by 20
  8. 2025DetailFlow: 1D Coarse-to-Fine Autoregressive Image Generation via Next-Detail Predictioncited by 18
  9. 2025DC-AR: Efficient Masked Autoregressive Image Generation with Deep Compression Hybrid Tokenizercited by 13
  10. 2026Making Reconstruction FID Predictive of Diffusion Generation FIDcited by 13
  11. 2025Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesiscited by 10
  12. 2025Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspectivecited by 10
38 more
  1. 2026A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokenscited by 10
  2. 2025Diffusion Generative Recommendation with Continuous Tokenscited by 9
  3. 2025GPSToken: Gaussian Parameterized Spatially-adaptive Tokenization for Image Representation and Generationcited by 9
  4. 2025DC-Gen: Post-Training Diffusion Acceleration with Deeply Compressed Latent Spacecited by 8
  5. 2026RotVLA: Rotational Latent Action for Vision-Language-Action Modelcited by 8
  6. 2025SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenizationcited by 7
  7. 2025Rethinking Discrete Tokens: Treating Them as Conditions for Continuous Autoregressive Image Synthesiscited by 6
  8. 2026Improving Flexible Image Tokenizers for Autoregressive Image Generationcited by 6
  9. 2026DA-VAE: Plug-in Latent Compression for Diffusion via Detail Alignmentcited by 6
  10. 2025VFRTok: Variable Frame Rates Video Tokenizer with Duration-Proportional Information Assumptioncited by 5
  11. 2025Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generationcited by 5
  12. 2025VAEVQ: Enhancing Discrete Visual Tokenization through Variational Modelingcited by 5
  13. 2025HD-Prot: A Protein Language Model for Joint Sequence-Structure Modeling with Continuous Structure Tokenscited by 4
  14. 2026VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latentscited by 3
  15. 2026Adaptive 1D Video Diffusion Autoencodercited by 3
  16. 2026Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Renderingcited by 3
  17. 2025Aligning Latent Spaces with Flow Priorscited by 2
  18. 2026Language-Guided Transformer Tokenizer for Human Motion Generationcited by 2
  19. 2026MacTok: Robust Continuous Tokenization for Image Generationcited by 2
  20. 2025Decoupling Complexity from Scale in Latent Diffusion Modelcited by 1
  21. 2026Semantic-Aware Prefix Learning for Token-Efficient Image Generationcited by 1
  22. 2026VibeToken: Scaling 1D Image Tokenizers and Autoregressive Models for Dynamic Resolution Generationscited by 1
  23. 2026RankE: End-to-End Post-Training for Discrete Text-to-Image Generation with Decoder Co-Evolutioncited by 1
  24. 2026Balancing Image Compression and Generation with Bootstrapped Tokenizationcited by 1
  25. 2026TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenizationcited by 1
  26. 2026SA-VLA: State-aware tokenizer for improving Vision-Language-Action Models' performancecited by 1
  27. 2026Pixel-Space Diffusion Transformerscited by 1
  28. 2026VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizerscited by 1
  29. 2025Dual Codebook VQ: Enhanced Image Reconstruction with Reduced Codebook Sizecited by 0
  30. 2025Local-Order Auxiliary Losses Can Improve Autoencoder Reconstructioncited by 0
  31. 2025VALA: Learning Latent Anchors for Training-Free and Temporally Consistentcited by 0
  32. 2026GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Modelscited by 0
  33. 2026NativeTok: Native Visual Tokenization for Improved Image Generationcited by 0
  34. 2026Adaptive Augmentation-Aware Latent Learning for Robust LiDAR Semantic Segmentationcited by 0
  35. 2026Structure over Pixels: Learning Variable-Length Visual Programscited by 0
  36. 2026MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Mergingcited by 0
  37. 2026MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Expertscited by 0
  38. 2026DisMix: Order-Aware Mixup for Medical Imaging via Disentangling Ordinal and Non-Ordinal Featurescited by 0

50 most cited of 56 All 56 (JSON)