# VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models arXiv: https://arxiv.org/abs/2510.18457 · HTML: https://mlsanity.com/paper/2510.18457/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. - Builds on 53 of its 57 references that are in the map. - First posted on arXiv 21 Oct 2025. ## Placed in - Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/) ## Builds on (most cited first, 30 of 53) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023) - [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021) - [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022) - [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018) - [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016) - [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022) - [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023) - [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016) - [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019) - [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025) - [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016) - [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025) - [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022) - [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024) - [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019) - [Stochastic Interpolants: A Unifying Framework for Flows and Diffusions](https://mlsanity.com/paper/2303.08797/) (2023) - [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023) - [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024) - [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023) - [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024) - [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024) ## Built on by (most cited first, 17 of 17) - [One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation](https://mlsanity.com/paper/2512.07829/) (2025) - [Toward Diffusible High-Dimensional Latent Spaces: A Frequency Perspective](https://mlsanity.com/paper/2511.22249/) (2025) - [Improving Reconstruction of Representation Autoencoder](https://mlsanity.com/paper/2602.08620/) (2026) - [What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion](https://mlsanity.com/paper/2605.07915/) (2026) - [PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion](https://mlsanity.com/paper/2605.23902/) (2026) - [End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer](https://mlsanity.com/paper/2605.00503/) (2026) - [PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space](https://mlsanity.com/paper/2607.05373/) (2026) - [Geometric Autoencoder for Diffusion Models](https://mlsanity.com/paper/2603.10365/) (2026) - [OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder](https://mlsanity.com/paper/2603.16099/) (2026) - [DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders](https://mlsanity.com/paper/2605.22777/) (2026) - [ReWorld: Representation Learning for World Action Models](https://mlsanity.com/paper/2606.27504/) (2026) - [RePack then Refine: Efficient Diffusion Transformer with Vision Foundation Model](https://mlsanity.com/paper/2512.12083/) (2025) - [WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens](https://mlsanity.com/paper/2605.18115/) (2026) - [Vision Foundation Models as Generalist Tokenizers for Image Generation](https://mlsanity.com/paper/2605.18390/) (2026) - [Generate in Reconstruction Space, Match in Semantic Space: Transport Geometry for One-Step Generation](https://mlsanity.com/paper/2606.00514/) (2026) - [Pixel-Space Diffusion Transformers](https://mlsanity.com/paper/2607.17585/) (2026) - [Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models](https://mlsanity.com/paper/2608.17564/) (2026) ## Abstract (arXiv) The performance of Latent Diffusion Models (LDMs) is critically dependent on the quality of their visual tokenizers. While recent works have explored incorporating Vision Foundation Models (VFMs) into the tokenizers training via distillation, we empirically find this approach inevitably weakens the robustness of learnt representation from original VFM. In this paper, we bypass the distillation by proposing a more direct approach by leveraging the frozen VFM for the LDMs tokenizer, named VFM Variational Autoencoder (VFM-VAE).To fully exploit the potential to leverage frozen VFM for the LDMs tokenizer, we design a new decoder to reconstruct realistic images from the semantic-rich representation of VFM. With the proposed VFM-VAE, we conduct a systematic study on how the representation from different tokenizers impact the representation learning process throughout diffusion training, enabling synergistic benefits of dual-side alignment on both tokenizers and diffusion models. Our effort in tokenizer design and training strategy lead to superior performance and efficiency: our system reaches a gFID (w/o CFG) of 2.22 in merely 80 epochs (a 10× speedup over prior tokenizers). With continued training to 640 epochs, it further attains a gFID (w/o CFG) of 1.62. These results offer solid evidence for the substantial potential of VFMs to serve as visual tokenizers to accelerate the LDM training progress.