mlsanityData to 7 Oct 2026

CAT: Content-Adaptive Image Tokenization

arXiv 2501.03120 ↗First posted 6 Jan 2025cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity. To address this, we introduce Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design a caption-based evaluation system that leverages large language models (LLMs) to predict content complexity and determine the optimal compression ratio for a given image, taking into account factors critical to human perception. Trained on images with diverse compression ratios, CAT demonstrates robust performance in image reconstruction. We also utilize its variable-length latent representations to train Diffusion Transformers (DiTs) for ImageNet generation. By optimizing token allocation, CAT improves the FID score over fixed-ratio baselines trained with the same flops and boosts the inference throughput by 18.5%.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 56 of its 78 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

56 of 78 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  12. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  13. 2022Token Merging: Your ViT But Fastercited by 995
  14. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  15. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  16. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  17. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  18. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  19. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  20. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  21. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  22. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  23. 2022Matryoshka Representation Learningcited by 268
  24. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  25. 2022Patching open-vocabulary models by interpolating weightscited by 187
  26. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  27. 2024Matryoshka Multimodal Modelscited by 82
  28. 2023Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantizationcited by 59
  29. 2024ElasticTok: Adaptive Tokenization for Image and Videocited by 44
  30. 2024Adaptive Length Image Tokenization via Recurrent Allocationcited by 40
  31. 2024ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Datacited by 39
  32. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  33. 2024Language-Guided Image Tokenization for Generationcited by 36
  34. 2023Vision Transformers with Mixed-Resolution Tokenizationcited by 35
  35. 2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
  36. 2023Efficient-VQGAN: Towards High-Resolution Image Generation with Efficient Vision Transformerscited by 33
  37. 2021NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Taskscited by 32
  38. 2024Specialized Foundation Models Struggle to Beat Supervised Baselinescited by 23

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 9.

18 papers.

Sort:

By area:

  1. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  2. 2025Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Spacecited by 16
  3. 2025Single-pass Adaptive Image Tokenization for Minimum Program Searchcited by 11
  4. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 11
  5. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  6. 2025WorldPack: Dynamic Frame Compression for Long-context Video World Modelingcited by 7
  7. 2026EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generationcited by 7
  8. 2025InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compressioncited by 6
  9. 2026ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Responsecited by 5
  10. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 3
  11. 2025RECODE: Reasoning Through Code Generation for Visual Question Answeringcited by 2
  12. 2025DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generationcited by 2
6 more
  1. 2026Adaptive Protein Tokenizationcited by 2
  2. 2026AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokenscited by 1
  3. 2026Structure over Pixels: Learning Variable-Length Visual Programscited by 0
  4. 2026ChannelTok: Efficient Flexible-Length Vision Tokenizationcited by 0
  5. 2026FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learningcited by 0
  6. 2026QuadTok: Quadtree Visual Tokenizer for Autoregressive Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice
  2. 2025AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs
  3. 2026Communication-Inspired Tokenization for Structured Image Representations
  4. 2026Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers
  5. 2026Semantic-Aware Prefix Learning for Token-Efficient Image Generation
  6. 2025Token Sequence Compression for Efficient Multimodal Computing
  7. 2026EvoTok: A Unified Image Tokenizer via Residual Latent Evolution for Visual Understanding and Generation
  8. 2026MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging