# VidTok: A Versatile and Open-Source Video Tokenizer

arXiv: https://arxiv.org/abs/2412.13061 · HTML: https://mlsanity.com/paper/2412.13061/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 17 of 17)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2410.13848/) (2024)
- [OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation](https://mlsanity.com/paper/2406.09399/) (2024)
- [IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI](https://mlsanity.com/paper/2411.00785/) (2024)
- [CV-VAE: A Compatible Video VAE for Latent Generative Video Models](https://mlsanity.com/paper/2405.20279/) (2024)
- [UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing](https://mlsanity.com/paper/2402.13185/) (2024)
- [aMUSEd: An Open MUSE Reproduction](https://mlsanity.com/paper/2401.01808/) (2024)
- [OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model](https://mlsanity.com/paper/2409.01199/) (2024)
- [Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators](https://mlsanity.com/paper/2407.07356/) (2024)

## Built on by (most cited first, 30 of 35)

- [Video World Models with Long-term Spatial Memory](https://mlsanity.com/paper/2506.05284/) (2025)
- [MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft](https://mlsanity.com/paper/2504.08388/) (2025)
- [Improving the Diffusability of Autoencoders](https://mlsanity.com/paper/2502.14831/) (2025)
- [Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspective](https://mlsanity.com/paper/2507.08801/) (2025)
- [FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenization](https://mlsanity.com/paper/2512.04952/) (2025)
- [Playing with Transformer at 30+ FPS via Next-Frame Diffusion](https://mlsanity.com/paper/2506.01380/) (2025)
- [LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models](https://mlsanity.com/paper/2503.14325/) (2025)
- [VidTwin: Video VAE with Decoupled Structure and Dynamics](https://mlsanity.com/paper/2412.17726/) (2024)
- [H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Models](https://mlsanity.com/paper/2504.10567/) (2025)
- [Rethinking Video Tokenization: A Conditioned Diffusion-based Approach](https://mlsanity.com/paper/2503.03708/) (2025)
- [Fast Autoregressive Video Generation with Diagonal Decoding](https://mlsanity.com/paper/2503.14070/) (2025)
- [I2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting](https://mlsanity.com/paper/2507.09144/) (2025)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [Latent Wavelet Diffusion For Ultra-High-Resolution Image Synthesis](https://mlsanity.com/paper/2506.00433/) (2025)
- [EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation](https://mlsanity.com/paper/2603.12267/) (2026)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization](https://mlsanity.com/paper/2604.12887/) (2026)
- [FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space](https://mlsanity.com/paper/2602.02092/) (2026)
- [Progressive Growing of Video Tokenizers for Temporally Compact Latent Spaces](https://mlsanity.com/paper/2501.05442/) (2025)
- [iFSQ: Improving FSQ for Image Generation with 1 Line of Code](https://mlsanity.com/paper/2601.17124/) (2026)
- [REGEN: Learning Compact Video Embedding with (Re-)Generative Decoder](https://mlsanity.com/paper/2503.08665/) (2025)
- [OneVAE: Joint Discrete and Continuous Optimization Helps Discrete Video VAE Train Better](https://mlsanity.com/paper/2508.09857/) (2025)
- [VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latents](https://mlsanity.com/paper/2602.04202/) (2026)
- [Adaptive 1D Video Diffusion Autoencoder](https://mlsanity.com/paper/2602.04220/) (2026)
- [DGAE: Diffusion-Guided Autoencoder for Efficient Latent Representation Learning](https://mlsanity.com/paper/2506.09644/) (2025)
- [Autoregressive Video Autoencoder with Decoupled Temporal and Spatial Context](https://mlsanity.com/paper/2512.11293/) (2025)
- [TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization](https://mlsanity.com/paper/2606.17590/) (2026)
- [AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation](https://mlsanity.com/paper/2606.30811/) (2026)
- [FlashDecoder: Real-Time Latent-to-Pixel Streaming Decoder with Transformers](https://mlsanity.com/paper/2607.14898/) (2026)
- [VideoTok4D: A 4D-Aware Video Tokenizer for Compact World Representation](https://mlsanity.com/paper/2609.12874/) (2026)

## Abstract (arXiv)

Encoding video content into compact latent tokens has become a fundamental step in video generation and understanding, driven by the need to address the inherent redundancy in pixel-level representations. Consequently, there is a growing demand for high-performance, open-source video tokenizers as video-centric research gains prominence. We introduce VidTok, a versatile video tokenizer that delivers state-of-the-art performance in both continuous and discrete tokenizations. VidTok incorporates several key advancements over existing approaches: 1) model architecture such as convolutional layers and up/downsampling modules; 2) to address the training instability and codebook collapse commonly associated with conventional Vector Quantization (VQ), we integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3) improved training strategies, including a two-stage training process and the use of reduced frame rates. By integrating these advancements, VidTok achieves substantial improvements over existing methods, demonstrating superior performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD, under standardized evaluation settings.