# Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation

arXiv: https://arxiv.org/abs/2507.08441 · HTML: https://mlsanity.com/paper/2507.08441/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 66 references that are in the map.
- First posted on arXiv 11 Jul 2025.

## Builds on (most cited first, 30 of 57)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)

## Built on by (most cited first, 21 of 21)

- [AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models](https://mlsanity.com/paper/2509.25162/) (2025)
- [UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation](https://mlsanity.com/paper/2510.10575/) (2025)
- [The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding](https://mlsanity.com/paper/2512.19693/) (2025)
- [WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction](https://mlsanity.com/paper/2508.05599/) (2025)
- [Autoregressive Image Generation with Masked Bit Modeling](https://mlsanity.com/paper/2602.09024/) (2026)
- [What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion](https://mlsanity.com/paper/2605.07915/) (2026)
- [DINO-Tok: Adapting DINO for Visual Tokenizers](https://mlsanity.com/paper/2511.20565/) (2025)
- [Improving Flexible Image Tokenizers for Autoregressive Image Generation](https://mlsanity.com/paper/2601.01535/) (2026)
- [End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer](https://mlsanity.com/paper/2605.00503/) (2026)
- [RecTok: Reconstruction Distillation along Rectified Flow](https://mlsanity.com/paper/2512.13421/) (2025)
- [Adaptive 1D Video Diffusion Autoencoder](https://mlsanity.com/paper/2602.04220/) (2026)
- [IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder](https://mlsanity.com/paper/2606.11096/) (2026)
- [Cubic Discrete Diffusion: Discrete Visual Generation on High-Dimensional Representation Tokens](https://mlsanity.com/paper/2603.19232/) (2026)
- [WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens](https://mlsanity.com/paper/2605.18115/) (2026)
- [Vision Foundation Models as Generalist Tokenizers for Image Generation](https://mlsanity.com/paper/2605.18390/) (2026)
- [UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Model](https://mlsanity.com/paper/2602.14178/) (2026)
- [Communication-Inspired Tokenization for Structured Image Representations](https://mlsanity.com/paper/2602.20731/) (2026)
- [DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding](https://mlsanity.com/paper/2603.19219/) (2026)
- [InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generation](https://mlsanity.com/paper/2605.14333/) (2026)
- [MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging](https://mlsanity.com/paper/2605.30904/) (2026)
- [Semantically Steerable Image Tokenization](https://mlsanity.com/paper/2610.15515/) (2026)

## Abstract (arXiv)

In this work, we present a novel direction to build an image tokenizer directly on top of a frozen vision foundation model, which is a largely underexplored area. Specifically, we employ a frozen vision foundation model as the encoder of our tokenizer. To enhance its effectiveness, we introduce two key components: (1) a region-adaptive quantization framework that reduces redundancy in the pre-trained features on regular 2D grids, and (2) a semantic reconstruction objective that aligns the tokenizer's outputs with the foundation model's representations to preserve semantic fidelity. Based on these designs, our proposed image tokenizer, VFMTok, achieves substantial improvements in image reconstruction and generation quality, while also enhancing token efficiency. It further boosts autoregressive (AR) generation -- achieving a gFID of 1.36 on ImageNet benchmarks, while accelerating model convergence by three times, and enabling high-fidelity class-conditional synthesis without the need for classifier-free guidance (CFG). The code is available at https://github.com/CVMI-Lab/VFMTok.