mlsanityData to 14 Oct 2026

Adaptive Length Image Tokenization via Recurrent Allocation

arXiv 2411.02393 ↗First posted 4 Nov 2024cs.CV cs.AI cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Current vision systems typically assign fixed-length representations to images, regardless of the information content. This contrasts with human intelligence - and even large language models - which allocate varying representational capacities based on entropy, context and familiarity. Inspired by this, we propose an approach to learn variable-length token representations for 2D images. Our encoder-decoder architecture recursively processes 2D image tokens, distilling them into 1D latent tokens over multiple iterations of recurrent rollouts. Each iteration refines the 2D tokens, updates the existing 1D latent tokens, and adaptively increases representational capacity by adding new tokens. This enables compression of images into a variable number of tokens, ranging from 32 to 256. We validate our tokenizer using reconstruction loss and FID metrics, demonstrating that token count aligns with image entropy, familiarity and downstream task requirements. Recurrent token processing with increasing representational capacity in each iteration shows signs of token specialization, revealing potential for object / part discovery.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

29 of 37 references.

References · 29

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  7. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  8. 2022Token Merging: Your ViT But Fastercited by 1,021
  9. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  10. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  11. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  12. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
17 more
  1. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  2. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  3. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  4. 2022Matryoshka Representation Learningcited by 282
  5. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  6. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  7. 2022Scalable Adaptive Computation for Iterative Generationcited by 195
  8. 2022FlexiViT: One Model for All Patch Sizescited by 152
  9. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 117
  10. 2024Matryoshka Multimodal Modelscited by 84
  11. 2023Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networkscited by 82
  12. 2024Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%cited by 68
  13. 2024ElasticTok: Adaptive Tokenization for Image and Videocited by 46
  14. 2016Detecting People in Artwork with CNNscited by 25
  15. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25
  16. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  17. 2018SAVOIAS: A Diverse, Multi-Category Visual Complexity Datasetcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 27.

43 papers.

Sort:

By area:

  1. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  2. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  3. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 33
  4. 2025"Principal Components" Enable A New Language of Imagescited by 27
  5. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 26
  6. 2025CAT: Content-Adaptive Image Tokenizationcited by 21
  7. 2025Latent Denoising Makes Good Tokenizerscited by 20
  8. 2025Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Spacecited by 16
  9. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 14
  10. 2025Accelerating Vision Transformers with Adaptive Patch Sizescited by 13
  11. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  12. 2025Single-pass Adaptive Image Tokenization for Minimum Program Searchcited by 12
31 more
  1. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 9
  2. 2025InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compressioncited by 8
  3. 2026EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generationcited by 7
  4. 2025ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generationcited by 6
  5. 2026Improving Flexible Image Tokenizers for Autoregressive Image Generationcited by 6
  6. 2026End-to-End Training for Unified Tokenization and Latent Denoisingcited by 6
  7. 2026VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenizationcited by 6
  8. 2026CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenizationcited by 5
  9. 2026MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generationcited by 4
  10. 2026Adaptive 1D Video Diffusion Autoencodercited by 3
  11. 2026DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunkingcited by 3
  12. 2024Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generationcited by 2
  13. 2025QLIP: A Dynamic Quadtree Vision Prior Enhances MLLM Performance Without Retrainingcited by 2
  14. 2025DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generationcited by 2
  15. 2026Learning to Theorize the World from Observationcited by 2
  16. 2026Soft Tail-dropping for Adaptive Visual Tokenizationcited by 1
  17. 2026Composable Visual Tokenizers with Generator-Free Diagnostics of Learnabilitycited by 1
  18. 2026(1D) Ordered Tokens Enable Efficient Test-Time Searchcited by 1
  19. 2026VibeToken: Scaling 1D Image Tokenizers and Autoregressive Models for Dynamic Resolution Generationscited by 1
  20. 2026AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokenscited by 1
  21. 2026Neural Field Tokenizations with Hierarchy and Spatial Locality Priorscited by 1
  22. 2026ZipTok3D: High-Fidelity 3D Tokenization with Compact Token Prefixescited by 1
  23. 2026Communication-Inspired Tokenization for Structured Image Representationscited by 0
  24. 2026Structure over Pixels: Learning Variable-Length Visual Programscited by 0
  25. 2026ChannelTok: Efficient Flexible-Length Vision Tokenizationcited by 0
  26. 2026Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformerscited by 0
  27. 2026Keep-or-Drop? Adaptive Tokenizer for Compact Video Representationcited by 0
  28. 2026NesTok: Nested Self-Aligned 1D Tokenizer for Autoregressive Image Generationcited by 0
  29. 2026Level-of-Token Diffusioncited by 0
  30. 2026MixTok: Adaptive Visual Tokenization with Global 1D Tokens and Sparse 2D Refinementcited by 0
  31. 2026Semantically Steerable Image Tokenizationcited by 0