# TVC: Tokenized Video Compression with Ultra-Low Bit Rate

arXiv: https://arxiv.org/abs/2504.16953 · HTML: https://mlsanity.com/paper/2504.16953/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 28 of 28)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [MAGVIT: Masked Generative Video Transformer](https://mlsanity.com/paper/2212.05199/) (2022)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis](https://mlsanity.com/paper/2211.09117/) (2022)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression](https://mlsanity.com/paper/2207.05894/) (2022)
- [OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation](https://mlsanity.com/paper/2406.09399/) (2024)
- [Real-World Blind Super-Resolution via Feature Matching with Implicit High-Resolution Priors](https://mlsanity.com/paper/2202.13142/) (2022)
- [MLIC: Multi-Reference Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2211.07273/) (2022)
- [Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization](https://mlsanity.com/paper/2503.11056/) (2025)
- [Extreme Image Compression using Fine-tuned VQGANs](https://mlsanity.com/paper/2307.08265/) (2023)
- [Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration](https://mlsanity.com/paper/2306.06513/) (2023)
- [HybridFlow: Infusing Continuity into Masked Codebook for Extreme Low-Bitrate Image Compression](https://mlsanity.com/paper/2404.13372/) (2024)
- [VQCNIR: Clearer Night Image Restoration with Vector-Quantized Codebook](https://mlsanity.com/paper/2312.08606/) (2023)
- [Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation](https://mlsanity.com/paper/2412.04432/) (2024)
- [Neural Image Compression Using Masked Sparse Visual Representation](https://mlsanity.com/paper/2309.11661/) (2023)
- [HDCompression: Hybrid-Diffusion Image Compression for Ultra-Low Bitrates](https://mlsanity.com/paper/2502.07160/) (2025)

## Built on by (most cited first, 7 of 7)

- [StreamingTOM: Streaming Token Compression for Efficient Video Understanding](https://mlsanity.com/paper/2510.18269/) (2025)
- [Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey](https://mlsanity.com/paper/2507.22920/) (2025)
- [SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Acceleration](https://mlsanity.com/paper/2602.04361/) (2026)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression](https://mlsanity.com/paper/2607.02562/) (2026)
- [PAT-VCM: Plug-and-Play Auxiliary Tokens for Video Coding for Machines](https://mlsanity.com/paper/2604.13294/) (2026)
- [From Pixels to Nucleotides: End-to-End Token-Based Video Compression for DNA Storage](https://mlsanity.com/paper/2604.13667/) (2026)

## Abstract (arXiv)

Tokenized visual representations have shown promise in image compression, yet their extension to video remains underexplored due to the challenges posed by complex temporal dynamics and stringent bit rate constraints. In this paper, we present tokenized video compression (TVC), a token-based dual-stream framework designed to operate effectively at ultra-low bit rates. TVC leverages the Cosmos video tokenizer to extract both discrete and continuous token streams. The discrete tokens are partially masked using a strategic masking scheme and then compressed losslessly with a discrete checkerboard context model to reduce transmission overhead. The masked tokens are reconstructed by a decoder-only Transformer with spatiotemporal token prediction. In parallel, the continuous tokens are quantized and compressed using a continuous checkerboard context model, providing complementary continuous information at ultra-low bit rates. At the decoder side, the two streams are fused with a ControlNet-based multi-scale integration module, ensuring high perceptual quality alongside stable fidelity in reconstruction. Overall, this work illustrates the practicality of tokenized video compression and points to new directions for semantics-aware, token-native approaches.