mlsanityData to 14 Oct 2026

VidTok: A Versatile and Open-Source Video Tokenizer

arXiv 2412.13061 ↗First posted 17 Dec 2024cs.CV cs.AI cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Encoding video content into compact latent tokens has become a fundamental step in video generation and understanding, driven by the need to address the inherent redundancy in pixel-level representations. Consequently, there is a growing demand for high-performance, open-source video tokenizers as video-centric research gains prominence. We introduce VidTok, a versatile video tokenizer that delivers state-of-the-art performance in both continuous and discrete tokenizations. VidTok incorporates several key advancements over existing approaches: 1) model architecture such as convolutional layers and up/downsampling modules; 2) to address the training instability and codebook collapse commonly associated with conventional Vector Quantization (VQ), we integrate Finite Scalar Quantization (FSQ) into discrete video tokenization; 3) improved training strategies, including a two-stage training process and the use of reduced frame rates. By integrating these advancements, VidTok achieves substantial improvements over existing methods, demonstrating superior performance across multiple metrics, including PSNR, SSIM, LPIPS, and FVD, under standardized evaluation settings.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 17 of its 48 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

17 of 48 references.

References · 17

  1. 2016Layer Normalizationcited by 5,669
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  5. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  6. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  8. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  9. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  10. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  11. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 118
  12. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 72
5 more
  1. 2024CV-VAE: A Compatible Video VAE for Latent Generative Video Modelscited by 70
  2. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 48
  3. 2024aMUSEd: An Open MUSE Reproductioncited by 32
  4. 2024OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video Diffusion Modelcited by 30
  5. 2024Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitatorscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 19; 2026: 15.

35 papers.

Sort:

By area:

  1. 2025Video World Models with Long-term Spatial Memorycited by 136
  2. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 97
  3. 2025Improving the Diffusability of Autoencoderscited by 80
  4. 2025Lumos-1: On Autoregressive Video Generation with Discrete Diffusion from a Unified Model Perspectivecited by 22
  5. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 22
  6. 2025Playing with Transformer at 30+ FPS via Next-Frame Diffusioncited by 20
  7. 2025LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Modelscited by 17
  8. 2024VidTwin: Video VAE with Decoupled Structure and Dynamicscited by 14
  9. 2025H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Modelscited by 13
  10. 2025Rethinking Video Tokenization: A Conditioned Diffusion-based Approachcited by 9
  11. 2025Fast Autoregressive Video Generation with Diagonal Decodingcited by 8
  12. 2025I2-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecastingcited by 8
23 more
  1. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 8
  2. 2025Latent Wavelet Diffusion For Ultra-High-Resolution Image Synthesiscited by 7
  3. 2026EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generationcited by 7
  4. 2026Evolution of Video Generative Foundationscited by 7
  5. 2026VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenizationcited by 6
  6. 2026FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Spacecited by 5
  7. 2025Progressive Growing of Video Tokenizers for Temporally Compact Latent Spacescited by 4
  8. 2026iFSQ: Improving FSQ for Image Generation with 1 Line of Codecited by 4
  9. 2025REGEN: Learning Compact Video Embedding with (Re-)Generative Decodercited by 3
  10. 2025OneVAE: Joint Discrete and Continuous Optimization Helps Discrete Video VAE Train Bettercited by 3
  11. 2026VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latentscited by 3
  12. 2026Adaptive 1D Video Diffusion Autoencodercited by 3
  13. 2025DGAE: Diffusion-Guided Autoencoder for Efficient Latent Representation Learningcited by 2
  14. 2025Autoregressive Video Autoencoder with Decoupled Temporal and Spatial Contextcited by 1
  15. 2026TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenizationcited by 1
  16. 2026AVTok: 1D Unified Tokenization for Holistic Audio-Video Generationcited by 1
  17. 2026FlashDecoder: Real-Time Latent-to-Pixel Streaming Decoder with Transformerscited by 1
  18. 2026VideoTok4D: A 4D-Aware Video Tokenizer for Compact World Representationcited by 1
  19. 2025ARSS: Taming Decoder-only Autoregressive Visual Generation for View Synthesis From Single Viewcited by 0
  20. 2026Scaling Next-Brain-Token Prediction for MEGcited by 0
  21. 2026Optimality of FSQ Tokens for Continuous Diffusion for Categorical Data with Application to Text-to-Speechcited by 0
  22. 2026Loss-Resilient Wireless Video Token Communication over Block Fading Channelscited by 0
  23. 2026SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models
  2. 2026PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation
  3. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation
  4. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM
  5. 2025MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs
  6. 2025MambaVideo for Discrete Video Tokenization with Channel-Split Quantization
  7. 2026Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline
  8. 2026Visual Token Coding for Video Multimodal Large Language Models