mlsanityData to 14 Oct 2026

Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space

arXiv 2505.17011 ↗First posted 22 May 2025cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

We propose AdapTok, an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content. AdapTok is equipped with a block-wise masking strategy that randomly drops tail tokens of each block during training, and a block causal scorer to predict the reconstruction quality of video frames using different numbers of tokens. During inference, an adaptive token allocation strategy based on integer linear programming is further proposed to adjust token usage given predicted scores. Such design allows for sample-wise, content-aware, and temporally dynamic token allocation under a controllable overall budget. Extensive experiments for video reconstruction and generation on UCF-101 and Kinetics-600 demonstrate the effectiveness of our approach. Without additional image data, AdapTok consistently improves reconstruction quality and generation performance under different token budgets, allowing for more scalable and token-efficient generative video modeling.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

35 of 39 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2017Neural Discrete Representation Learningcited by 4,712
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  8. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  9. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  10. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  11. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  12. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
23 more
  1. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  2. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  3. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  4. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  5. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  6. 2022MAGVIT: Masked Generative Video Transformercited by 346
  7. 2018A Short Note about Kinetics-600cited by 333
  8. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  9. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  10. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  11. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 118
  12. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  13. 2025FlexTok: Resampling Images into 1D Token Sequences of Flexible Lengthcited by 110
  14. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  15. 2023Online Clustered Codebookcited by 55
  16. 2025One-D-Piece: Image Tokenizer Meets Quality-Controllable Compressioncited by 48
  17. 2023Regularized Vector Quantization for Tokenized Image Synthesiscited by 46
  18. 2024ElasticTok: Adaptive Tokenization for Image and Videocited by 46
  19. 2024LARP: Tokenizing Videos with a Learned Autoregressive Generative Priorcited by 46
  20. 2024Adaptive Length Image Tokenization via Recurrent Allocationcited by 43
  21. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  22. 2025CAT: Content-Adaptive Image Tokenizationcited by 21
  23. 2020Stochastic Bottleneck: Rateless Auto-Encoder for Flexible Dimensionality Reductioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 14.

16 papers.

Sort:

By area:

  1. 2026A Frame is Worth One Token: Efficient Generative World Modeling with Delta Tokenscited by 10
  2. 2025MotionStrata: Hierarchical Motion Latents for Compact Video Autoencodingcited by 9
  3. 2026EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generationcited by 7
  4. 2025DeRA: Decoupled Representation Alignment for Video Tokenizationcited by 6
  5. 2026One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformerscited by 4
  6. 2026Adaptive 1D Video Diffusion Autoencodercited by 3
  7. 2026TaskTok: Delving into Task Tokens for Task-driven Image Restorationcited by 3
  8. 2026Swift Sampling: Selecting Temporal Surprises via Taylor Seriescited by 2
  9. 2026Soft Tail-dropping for Adaptive Visual Tokenizationcited by 1
  10. 2026TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenizationcited by 1
  11. 2026AVTok: 1D Unified Tokenization for Holistic Audio-Video Generationcited by 1
  12. 2026ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixescited by 1
4 more
  1. 2026Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpaintingcited by 0
  2. 2026MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generationcited by 0
  3. 2026Keep-or-Drop? Adaptive Tokenizer for Compact Video Representationcited by 0
  4. 2026V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environmentscited by 0