mlsanityData to 7 Oct 2026

ElasticTok: Adaptive Tokenization for Image and Video

arXiv 2410.08368 ↗First posted 10 Oct 2024cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Efficient video tokenization remains a key bottleneck in learning general purpose vision models that are capable of processing long video sequences. Prevailing approaches are restricted to encoding videos to a fixed number of tokens, where too few tokens will result in overly lossy encodings, and too many tokens will result in prohibitively long sequence lengths. In this work, we introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. To enable this in a computationally scalable way, we propose a masking technique that drops a random number of tokens at the end of each frames's token encoding. During inference, ElasticTok can dynamically allocate tokens when needed -- more complex data can leverage more tokens, while simpler data only needs a few tokens. Our empirical evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage, paving the way for future development of more powerful multimodal models, world models, and agents.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

39 of 54 references.

References · 39

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
27 more
  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  3. 2016Deep Networks with Stochastic Depthcited by 973
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  5. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  6. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  7. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  8. 2018Joint Autoregressive and Hierarchical Priors for Learned Image Compressioncited by 642
  9. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  10. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  13. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  14. 2022MAGVIT: Masked Generative Video Transformercited by 341
  15. 2018Slimmable Neural Networkscited by 301
  16. 2017Lossy Image Compression with Compressive Autoencoderscited by 289
  17. 2022Matryoshka Representation Learningcited by 268
  18. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  19. 2021Deep Contextual Video Compressioncited by 216
  20. 2023Neural Video Compression with Diverse Contextscited by 155
  21. 2014Learning Ordered Representations with Nested Dropoutcited by 81
  22. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  23. 2022VCT: A Video Compression Transformercited by 74
  24. 2020A Technical Overview of AV1cited by 46
  25. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  26. 2021Variable-rate discrete representation learningcited by 24
  27. 2018Anytime Neural Prediction via Slicing Networks Verticallycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 20; 2026: 22.

44 papers.

Sort:

By area:

  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  2. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  3. 2024Adaptive Length Image Tokenization via Recurrent Allocationcited by 40
  4. 2025AToken: A Unified Tokenizer for Visioncited by 30
  5. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 25
  6. 2025CAT: Content-Adaptive Image Tokenizationcited by 18
  7. 2025Latent Denoising Makes Good Tokenizerscited by 18
  8. 2025Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Spacecited by 16
  9. 2025Adapting Self-Supervised Representations as a Latent Space for Efficient Generationcited by 13
  10. 2025Accelerating Vision Transformers with Adaptive Patch Sizescited by 12
  11. 2025Single-pass Adaptive Image Tokenization for Minimum Program Searchcited by 11
  12. 2025MotionStrata: Hierarchical Motion Latents for Compact Video Autoencodingcited by 9
32 more
  1. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 8
  2. 2025WorldPack: Dynamic Frame Compression for Long-context Video World Modelingcited by 7
  3. 2026EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generationcited by 7
  4. 2024Efficient Long Video Tokenization via Coordinate-based Patch Reconstructioncited by 6
  5. 2025Efficient Multi-Camera Tokenization with Triplanes for End-to-End Drivingcited by 6
  6. 2025DeRA: Decoupled Representation Alignment for Video Tokenizationcited by 6
  7. 2025InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compressioncited by 6
  8. 2026VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenizationcited by 6
  9. 2025ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generationcited by 5
  10. 2025Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentationcited by 5
  11. 2026Improving Flexible Image Tokenizers for Autoregressive Image Generationcited by 5
  12. 2025OneVAE: Joint Discrete and Continuous Optimization Helps Discrete Video VAE Train Bettercited by 3
  13. 2026VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latentscited by 3
  14. 2026Adaptive 1D Video Diffusion Autoencodercited by 3
  15. 2026One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformerscited by 3
  16. 2026DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunkingcited by 2
  17. 2025Text-Guided Semantic Image Encodercited by 1
  18. 2026Soft Tail-dropping for Adaptive Visual Tokenizationcited by 1
  19. 2026(1D) Ordered Tokens Enable Efficient Test-Time Searchcited by 1
  20. 2026Swift Sampling: Selecting Temporal Surprises via Taylor Seriescited by 1
  21. 2026PARCEL: Pool-Anchored Resampling with Conditioned Elastic Queries for Efficient Vision-Language Understandingcited by 1
  22. 2026AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokenscited by 1
  23. 2026Neural Field Tokenizations with Hierarchy and Spatial Locality Priorscited by 1
  24. 2026MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generationcited by 1
  25. 2026TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenizationcited by 1
  26. 2025Beyond Static Perception: Integrating Temporal Context into VLMs for Cloth Foldingcited by 0
  27. 2026Structure over Pixels: Learning Variable-Length Visual Programscited by 0
  28. 2026ChannelTok: Efficient Flexible-Length Vision Tokenizationcited by 0
  29. 2026Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpaintingcited by 0
  30. 2026Keep-or-Drop? Adaptive Tokenizer for Compact Video Representationcited by 0
  31. 2026ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixescited by 0
  32. 2026SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice
  2. 2026DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMs
  3. 2025DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models
  4. 2025Token Sequence Compression for Efficient Multimodal Computing
  5. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM
  6. 2025TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inference
  7. 2026Beyond Selection: Token Parameterization for Extreme Visual Token Compression
  8. 2026RUTA: Principled Visual Token Allocation via Rate-Utility Optimization