# Image and Video Tokenization with Binary Spherical Quantization

arXiv: https://arxiv.org/abs/2406.07548 · HTML: https://mlsanity.com/paper/2406.07548/

## Key facts

- Cited by 112 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 79 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)

## Built on by (most cited first, 30 of 112)

- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](https://mlsanity.com/paper/2412.04431/) (2024)
- [Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2510.11690/) (2025)
- [GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation](https://mlsanity.com/paper/2504.08736/) (2025)
- [Restructuring Vector Quantization with the Rotation Trick](https://mlsanity.com/paper/2410.06424/) (2024)
- [OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation](https://mlsanity.com/paper/2509.03498/) (2025)
- [InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation](https://mlsanity.com/paper/2511.04675/) (2025)
- [Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization](https://mlsanity.com/paper/2503.11056/) (2025)
- [UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2505.14682/) (2025)
- [OctGPT: Octree-based Multiscale Autoregressive Models for 3D Shape Generation](https://mlsanity.com/paper/2504.09975/) (2025)
- [Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation](https://mlsanity.com/paper/2503.16430/) (2025)
- [Kronos: A Foundation Model for the Language of Financial Markets](https://mlsanity.com/paper/2508.02739/) (2025)
- [Visual Autoregressive Modeling for Instruction-Guided Image Editing](https://mlsanity.com/paper/2508.15772/) (2025)
- [FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenization](https://mlsanity.com/paper/2512.04952/) (2025)
- [Training-Free Text-Guided Image Editing with Visual Autoregressive Model](https://mlsanity.com/paper/2503.23897/) (2025)
- [FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks](https://mlsanity.com/paper/2502.04465/) (2025)
- [Epsilon-VAE: Denoising as Visual Decoding](https://mlsanity.com/paper/2410.04081/) (2024)
- [Hyperspherical Latents Improve Continuous-Token Autoregressive Generation](https://mlsanity.com/paper/2509.24335/) (2025)
- [When Worse is Better: Navigating the compression-generation tradeoff in visual tokenization](https://mlsanity.com/paper/2412.16326/) (2024)
- [TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generation](https://mlsanity.com/paper/2507.18537/) (2025)
- [ELT: Elastic Looped Transformers for Visual Generation](https://mlsanity.com/paper/2604.09168/) (2026)
- [Fine-Tuning Visual Autoregressive Models for Subject-Driven Generation](https://mlsanity.com/paper/2504.02612/) (2025)
- [TokBench: Evaluating Your Visual Tokenizer before Visual Generation](https://mlsanity.com/paper/2505.18142/) (2025)
- [TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling](https://mlsanity.com/paper/2508.16790/) (2025)
- [Visual Lexicon: Rich Image Features in Language Space](https://mlsanity.com/paper/2412.06774/) (2024)
- [Multimodal Latent Diffusion Model for Complex Sewing Pattern Generation](https://mlsanity.com/paper/2412.14453/) (2024)
- [MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization](https://mlsanity.com/paper/2504.00999/) (2025)
- [Image Tokenizer Needs Post-Training](https://mlsanity.com/paper/2509.12474/) (2025)
- [Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropy](https://mlsanity.com/paper/2510.09012/) (2025)
- [Robust Latent Matters: Boosting Image Generation with Sampling Error Synthesis](https://mlsanity.com/paper/2503.08354/) (2025)

## Abstract (arXiv)

We propose a new transformer-based image and video tokenizer with Binary Spherical Quantization (BSQ). BSQ projects the high-dimensional visual embedding to a lower-dimensional hypersphere and then applies binary quantization. BSQ is (1) parameter-efficient without an explicit codebook, (2) scalable to arbitrary token dimensions, and (3) compact: compressing visual data by up to 100× with minimal distortion. Our tokenizer uses a transformer encoder and decoder with simple block-wise causal masking to support variable-length videos as input. The resulting BSQ-ViT achieves state-of-the-art visual reconstruction quality on image and video reconstruction benchmarks with 2.4× throughput compared to the best prior methods. Furthermore, by learning an autoregressive prior for adaptive arithmetic coding, BSQ-ViT achieves comparable results on video compression with state-of-the-art video compression standards. BSQ-ViT also enables masked language models to achieve competitive image synthesis quality to GAN- and diffusion-based methods.