mlsanityData to 7 Oct 2026

Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice

arXiv 2605.16384 ↗First posted 11 May 2026cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Accurate and effective discrete image tokenization is crucial for long image sequence processing. However, current methods rigidly compress all content at a fixed rate, ignoring the variable information density of images and leading to either redundancy or information loss. Inspired by information entropy, we propose TaTok, a Theoretically grounded adaptive image Tokenization framework. We rigorously identify two key drawbacks in existing methods: information insufficiency when reconstructing images with patch tokens alone, and information redundancy among patch tokens. To address these, we introduce global tokens that model mutual information across patch tokens, and a Dynamic Token Filtering (DTF) algorithm based on cumulative conditional entropy to eliminate redundancy. Experiments confirm TaTok's state-of-the-art performance, delivering a 1.3x gFID improvement and 8.7x inference speedup. By allocating tokens according to information richness, TaTok enables more compressed yet accurate image tokenization, offering valuable insights for future research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 46 of its 65 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

46 of 65 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
34 more
  1. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  13. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  14. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  15. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  16. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  17. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  18. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  19. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  20. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  21. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  22. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  23. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  24. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  25. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  26. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  27. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  28. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  29. 2021Generating Images with Sparse Representationscited by 262
  30. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 153
  31. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  32. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  33. 2025Hita: Holistic Tokenizer for Autoregressive Image Generationcited by 11
  34. 2025SFTok: Bridging the Performance Gap in Discrete Tokenizerscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CAT: Content-Adaptive Image Tokenization
  2. 2026AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens
  3. 2024Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding
  4. 2026Communication-Inspired Tokenization for Structured Image Representations
  5. 2025FlexTok: Resampling Images into 1D Token Sequences of Flexible Length
  6. 2024Language-Guided Image Tokenization for Generation
  7. 2025AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs
  8. 2024ElasticTok: Adaptive Tokenization for Image and Video