mlsanityData to 7 Oct 2026

Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation

arXiv 2507.08441 ↗First posted 11 Jul 2025cs.CV cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

In this work, we present a novel direction to build an image tokenizer directly on top of a frozen vision foundation model, which is a largely underexplored area. Specifically, we employ a frozen vision foundation model as the encoder of our tokenizer. To enhance its effectiveness, we introduce two key components: (1) a region-adaptive quantization framework that reduces redundancy in the pre-trained features on regular 2D grids, and (2) a semantic reconstruction objective that aligns the tokenizer's outputs with the foundation model's representations to preserve semantic fidelity. Based on these designs, our proposed image tokenizer, VFMTok, achieves substantial improvements in image reconstruction and generation quality, while also enhancing token efficiency. It further boosts autoregressive (AR) generation -- achieving a gFID of 1.36 on ImageNet benchmarks, while accelerating model convergence by three times, and enabling high-fidelity class-conditional synthesis without the need for classifier-free guidance (CFG). The code is available at https://github.com/CVMI-Lab/VFMTok.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 57 of its 66 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

57 of 66 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2017Deformable Convolutional Networkscited by 1,481
  13. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  14. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  15. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  16. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  17. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  18. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  19. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  20. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  21. 2023Vision Transformers Need Registerscited by 844
  22. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  23. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  24. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  25. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  26. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  27. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  28. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  29. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  30. 2022MAGVIT: Masked Generative Video Transformercited by 341
  31. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  32. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  33. 2023Fast Training of Diffusion Models with Masked Transformerscited by 172
  34. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 153
  35. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125
  36. 2024Randomized Autoregressive Visual Generationcited by 125
  37. 2025FlexTok: Resampling Images into 1D Token Sequences of Flexible Lengthcited by 105
  38. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 93

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 14.

20 papers.

Sort:

By area:

  1. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  2. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  3. 2025The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencodingcited by 21
  4. 2025WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstructioncited by 9
  5. 2026Autoregressive Image Generation with Masked Bit Modelingcited by 8
  6. 2026What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusioncited by 7
  7. 2025DINO-Tok: Adapting DINO for Visual Tokenizerscited by 6
  8. 2026Improving Flexible Image Tokenizers for Autoregressive Image Generationcited by 5
  9. 2026End-to-End Autoregressive Image Generation with 1D Semantic Tokenizercited by 5
  10. 2025RecTok: Reconstruction Distillation along Rectified Flowcited by 4
  11. 2026Adaptive 1D Video Diffusion Autoencodercited by 3
  12. 2026IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncodercited by 2
8 more
  1. 2026Cubic Discrete Diffusion: Discrete Visual Generation on High-Dimensional Representation Tokenscited by 1
  2. 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
  3. 2026Vision Foundation Models as Generalist Tokenizers for Image Generationcited by 1
  4. 2026UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Modelcited by 0
  5. 2026Communication-Inspired Tokenization for Structured Image Representationscited by 0
  6. 2026DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understandingcited by 0
  7. 2026InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generationcited by 0
  8. 2026MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Mergingcited by 0