mlsanityData to 7 Oct 2026

Image and Video Tokenization with Binary Spherical Quantization

arXiv 2406.07548 ↗First posted 11 Jun 2024cs.CV cs.IT cs.LGcited by 110

Not on the map. Why?

Cited by 22 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a new transformer-based image and video tokenizer with Binary Spherical Quantization (BSQ). BSQ projects the high-dimensional visual embedding to a lower-dimensional hypersphere and then applies binary quantization. BSQ is (1) parameter-efficient without an explicit codebook, (2) scalable to arbitrary token dimensions, and (3) compact: compressing visual data by up to 100× with minimal distortion. Our tokenizer uses a transformer encoder and decoder with simple block-wise causal masking to support variable-length videos as input. The resulting BSQ-ViT achieves state-of-the-art visual reconstruction quality on image and video reconstruction benchmarks with 2.4× throughput compared to the best prior methods. Furthermore, by learning an autoregressive prior for adaptive arithmetic coding, BSQ-ViT achieves comparable results on video compression with state-of-the-art video compression standards. BSQ-ViT also enables masked language models to achieve competitive image synthesis quality to GAN- and diffusion-based methods.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 54 of its 79 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

54 of 79 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  10. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2021ViViT: A Video Vision Transformercited by 1,610
  13. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  14. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  15. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  16. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  17. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  18. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  19. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  20. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  21. 2017Non-Autoregressive Neural Machine Translationcited by 457
  22. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  23. 2022MAGVIT: Masked Generative Video Transformercited by 341
  24. 2018DVC: An End-to-end Deep Video Compression Frameworkcited by 324
  25. 2022ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Codingcited by 296
  26. 2020CompressAI: a PyTorch library and evaluation platform for end-to-end compression researchcited by 271
  27. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  28. 2023Language Modeling Is Compressioncited by 225
  29. 2021Deep Contextual Video Compressioncited by 216
  30. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  31. 2017Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representationscited by 149
  32. 2020Nonlinear Transform Codingcited by 132
  33. 2018Practical Full Resolution Learned Lossless Image Compressioncited by 93
  34. 2018Learned Video Compressioncited by 89
  35. 2019Practical Lossless Compression with Latent Variables using Bits Back Codingcited by 85
  36. 2022VCT: A Video Compression Transformercited by 74
  37. 2022SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantizationcited by 68
  38. 2023Online Clustered Codebookcited by 55

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 52; 2026: 50.

110 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  3. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  4. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 55
  5. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  6. 2024Restructuring Vector Quantization with the Rotation Trickcited by 49
  7. 2025InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generationcited by 43
  8. 2025Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenizationcited by 35
  9. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  10. 2025Bridging Continuous and Discrete Tokens for Autoregressive Visual Generationcited by 29
  11. 2025OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generationcited by 27
  12. 2025Kronos: A Foundation Model for the Language of Financial Marketscited by 25
38 more
  1. 2025Training-Free Text-Guided Image Editing with Visual Autoregressive Modelcited by 21
  2. 2025Visual Autoregressive Modeling for Instruction-Guided Image Editingcited by 21
  3. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 21
  4. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  5. 2024Epsilon-VAE: Denoising as Visual Decodingcited by 16
  6. 2025Hyperspherical Latents Improve Continuous-Token Autoregressive Generationcited by 16
  7. 2025TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generationcited by 15
  8. 2024When Worse is Better: Navigating the compression-generation tradeoff in visual tokenizationcited by 14
  9. 2025Fine-Tuning Visual Autoregressive Models for Subject-Driven Generationcited by 13
  10. 2024Visual Lexicon: Rich Image Features in Language Spacecited by 12
  11. 2024Multimodal Latent Diffusion Model for Complex Sewing Pattern Generationcited by 12
  12. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  13. 2025TokBench: Evaluating Your Visual Tokenizer before Visual Generationcited by 12
  14. 2025TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modelingcited by 12
  15. 2025Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropycited by 11
  16. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 11
  17. 2025Image Tokenizer Needs Post-Trainingcited by 10
  18. 2025Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesiscited by 9
  19. 2025MMAIF: Multi-task and Multi-degradation All-in-One for Image Fusion with Language Guidancecited by 9
  20. 2025WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstructioncited by 9
  21. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 8
  22. 2026Autoregressive Image Generation with Masked Bit Modelingcited by 8
  23. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  24. 2025SSDD: Single-Step Diffusion Decoder for Efficient Image Tokenizationcited by 7
  25. 2026What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusioncited by 7
  26. 2025EditInfinity: Image Editing with Binary-Quantized Generative Modelscited by 6
  27. 2025DINO-Tok: Adapting DINO for Visual Tokenizerscited by 6
  28. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  29. 2025Spherical Leech Quantization for Visual Tokenization and Generationcited by 6
  30. 2025End-to-End Vision Tokenizer Tuningcited by 5
  31. 2025Instella-T2I: Pushing the Limits of 1D Discrete Latent Space Image Generationcited by 5
  32. 2025BitMark: Watermarking Bitwise Autoregressive Image Generative Modelscited by 5
  33. 2025DiVeQ: Differentiable Vector Quantization Using the Reparameterization Trickcited by 5
  34. 2026Improving Flexible Image Tokenizers for Autoregressive Image Generationcited by 5
  35. 2024Scaling Image Tokenizers with Grouped Spherical Quantizationcited by 4
  36. 2025Virtual Full-stack Scanning of Brain MRI via Imputing Any Quantised Codecited by 4
  37. 2025Beyond Words: Advancing Long-Text Image Generation via Multimodal Autoregressive Modelscited by 4
  38. 2026BitDance: Scaling Autoregressive Generative Models with Binary Tokenscited by 4

50 most cited of 110 All 110 (JSON)