# InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression

arXiv: https://arxiv.org/abs/2512.16975 · HTML: https://mlsanity.com/paper/2512.16975/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Builds on (most cited first, 30 of 32)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Finite Scalar Quantization: VQ-VAE Made Simple](https://mlsanity.com/paper/2309.15505/) (2023)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [MAGVIT: Masked Generative Video Transformer](https://mlsanity.com/paper/2212.05199/) (2022)
- [An Image is Worth 32 Tokens for Reconstruction and Generation](https://mlsanity.com/paper/2406.07550/) (2024)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation](https://mlsanity.com/paper/2409.04410/) (2024)
- [OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation](https://mlsanity.com/paper/2406.09399/) (2024)
- [FlexTok: Resampling Images into 1D Token Sequences of Flexible Length](https://mlsanity.com/paper/2502.13967/) (2025)
- [MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh Tokenization](https://mlsanity.com/paper/2408.02555/) (2024)
- [The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/1905.00737/) (2019)
- [SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models](https://mlsanity.com/paper/2308.16692/) (2023)
- [Learning Ordered Representations with Nested Dropout](https://mlsanity.com/paper/1402.0915/) (2014)
- [One-D-Piece: Image Tokenizer Meets Quality-Controllable Compression](https://mlsanity.com/paper/2501.10064/) (2025)
- [ElasticTok: Adaptive Tokenization for Image and Video](https://mlsanity.com/paper/2410.08368/) (2024)
- [Adaptive Length Image Tokenization via Recurrent Allocation](https://mlsanity.com/paper/2411.02393/) (2024)
- [Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization](https://mlsanity.com/paper/2503.11056/) (2025)
- [Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens](https://mlsanity.com/paper/2504.14666/) (2025)

## Built on by (most cited first, 8 of 8)

- [EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation](https://mlsanity.com/paper/2603.12267/) (2026)
- [AdaCodec: A Predictive Visual Code for Video MLLMs](https://mlsanity.com/paper/2606.02569/) (2026)
- [Swift Sampling: Selecting Temporal Surprises via Taylor Series](https://mlsanity.com/paper/2605.22678/) (2026)
- [VPG: Visual Prefix Guidance for Autoregressive Image and Video Generation](https://mlsanity.com/paper/2605.30317/) (2026)
- [Structure over Pixels: Learning Variable-Length Visual Programs](https://mlsanity.com/paper/2605.27696/) (2026)
- [Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting](https://mlsanity.com/paper/2606.06158/) (2026)
- [Level-of-Token Diffusion](https://mlsanity.com/paper/2610.05816/) (2026)
- [Visual Tokens on Demand: Visual Budgeting for Scalable and Efficient Multi-Image Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2610.18858/) (2026)

## Abstract (arXiv)

Accurate and efficient discrete video tokenization is essential for long video sequences processing. Yet, the inherent complexity and variable information density of videos present a significant bottleneck for current tokenizers, which rigidly compress all content at a fixed rate, leading to redundancy or information loss. Drawing inspiration from Shannon's information theory, this paper introduces InfoTok, a principled framework for adaptive video tokenization. We rigorously prove that existing data-agnostic training methods are suboptimal in representation length, and present a novel evidence lower bound (ELBO)-based algorithm that approaches theoretical optimality. Leveraging this framework, we develop a transformer-based adaptive compressor that enables adaptive tokenization. Empirical results demonstrate state-of-the-art compression performance, saving 20% tokens without influence on performance, and achieving 2.3x compression rates while still outperforming prior heuristic adaptive approaches. By allocating tokens according to informational richness, InfoTok enables a more compressed yet accurate tokenization for video representation, offering valuable insights for future research.