mlsanityData to 14 Oct 2026

AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models

arXiv 2509.25162 ↗First posted 29 Sep 2025cs.CVcited by 36

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

In this work, we propose aligning pretrained visual encoders to serve as tokenizers for latent diffusion models in image generation. Unlike training a variational autoencoder (VAE) from scratch, which primarily emphasizes low-level details, our approach leverages the rich semantic structure of foundation encoders. We introduce a three-stage alignment strategy called AlignTok: (1) freeze the encoder and train an adapter and a decoder to establish a semantic latent space; (2) jointly optimize all components with an additional semantic preservation loss, enabling the encoder to capture perceptual details while retaining high-level semantics; and (3) refine the decoder for improved reconstruction quality. This alignment yields semantically rich image tokenizers that benefit diffusion models. On ImageNet 256×256, our tokenizer accelerates the convergence of diffusion models, reaching a gFID of 1.90 within just 64 epochs, and improves generation both with and without classifier-free guidance. Scaling to LAION, text-to-image models trained with our tokenizer consistently outperforms FLUX VAE and VA-VAE under the same training steps. Overall, our method is simple, scalable, and establishes a semantically grounded paradigm for continuous tokenizer design.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 60 of its 66 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

60 of 66 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space1 Aug
  2. 2025Latent Diffusion Models with Masked AutoEncoders14 Jul
  3. 2025CoVAE: Consistency Training of Variational Autoencoders12 Jul
  4. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers14 Apr
  5. 2025USP: Unified Self-Supervised Pretraining for Image Generation and Understanding8 Mar
  6. 2025Improving the Diffusability of Autoencoders20 Feb
  7. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Models5 Feb
  8. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models2 Jan
  9. 2024EP-CFG: Energy-Preserving Classifier-Free Guidance13 Dec
  10. 2024FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification14 Oct
  11. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think9 Oct
  12. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
6 more
  1. 2023Analyzing and Improving the Training Dynamics of Diffusion Models5 Dec
  2. 2023Fast Training of Diffusion Models with Masked Transformers15 Jun
  3. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  4. 2022Scalable Diffusion Models with Transformers19 Dec
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  6. 2020Denoising Diffusion Implicit Models6 Oct

Other references · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  10. 2018Demystifying MMD GANscited by 1,293
  11. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  12. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
30 more
  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  2. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  3. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  4. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  5. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  6. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  7. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  8. 2024The Platonic Representation Hypothesiscited by 382
  9. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  10. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  11. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  12. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 202
  13. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 153
  14. 2025EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modelingcited by 89
  15. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  16. 2025GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generationcited by 57
  17. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 51
  18. 2025UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encodingcited by 49
  19. 2025QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generationcited by 45
  20. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
  21. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  22. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 44
  23. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  24. 2025Learnings from Scaling Visual Tokenizers for Reconstruction and Generationcited by 32
  25. 2024Causal Diffusion Transformers for Generative Modelingcited by 31
  26. 2025AToken: A Unified Tokenizer for Visioncited by 30
  27. 2025"Principal Components" Enable A New Language of Imagescited by 27
  28. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  29. 2025Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generationcited by 21
  30. 2025Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Modelcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 31.

36 papers.

Sort:

By area:

  1. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  2. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 32
  3. 2026Improved Baselines with Representation Autoencoderscited by 31
  4. 2025Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editingcited by 22
  5. 2026Making Reconstruction FID Predictive of Diffusion Generation FIDcited by 13
  6. 2025Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generatorcited by 11
  7. 2026How to Train Your Latent Diffusion Language Model Jointly With the Latent Spacecited by 11
  8. 2026Improving Reconstruction of Representation Autoencodercited by 9
  9. 2026What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusioncited by 9
  10. 2026Hyperspherical Autoencoder for High-Fidelity Image Reconstruction and Generationcited by 7
  11. 2025Distribution Matching Variational AutoEncodercited by 6
  12. 2026DA-VAE: Plug-in Latent Compression for Diffusion via Detail Alignmentcited by 6
24 more
  1. 2025REGLUE Your Latents with Global and Local Semantics for Entangled Diffusioncited by 5
  2. 2026End-to-End Autoregressive Image Generation with 1D Semantic Tokenizercited by 5
  3. 2025RecTok: Reconstruction Distillation along Rectified Flowcited by 4
  4. 2026PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Spacecited by 4
  5. 2026Adaptive 1D Video Diffusion Autoencodercited by 3
  6. 2026WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modelingcited by 3
  7. 2026Towards Predictive, Aligned, and Scalable Robot Learningcited by 3
  8. 2026Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformerscited by 3
  9. 2026Geometric Autoencoder for Diffusion Modelscited by 2
  10. 2026OneWorld: Taming Scene Generation with 3D Unified Representation Autoencodercited by 2
  11. 2026DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoderscited by 2
  12. 2026ReWorld: Representation Learning for World Action Modelscited by 2
  13. 2026RAE-AR: Taming Autoregressive Models with Representation Autoencoderscited by 1
  14. 2026WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokenscited by 1
  15. 2026Vision Foundation Models as Generalist Tokenizers for Image Generationcited by 1
  16. 2026Generate in Reconstruction Space, Match in Semantic Space: Transport Geometry for One-Step Generationcited by 1
  17. 2026FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoderscited by 1
  18. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
  19. 2026Coevolving Representations in Joint Image-Feature Diffusioncited by 0
  20. 2026SRC-Flow: Compact Semantic Representations Enable Normalizing Flows for Image Generationcited by 0
  21. 2026STREAM: Stochastic Riemannian Flow Matching with Anisotropic Decoder for Digital Histopathology Image Generationcited by 0
  22. 2026AffineTok: Semantic Affine Consistency for Diffusion-Friendly Visual Tokenizercited by 0
  23. 2026ProSR: Semantic-Prototype-Guided Discrete Modeling for Physically Consistent SAR Super-Resolutioncited by 0
  24. 2026On the Diffusibility of High-Dimensional Latentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MaskAlign: Token-Subset Representation Alignment for Efficient Diffusion Training
  2. 2026Purify Before You Align: Improving Representation Alignment for Diffusion Models
  3. 2026Boosting Latent Diffusion Models via Disentangled Representation Alignment
  4. 2026Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers
  5. 2026Training-Free Representation Guidance for Diffusion Models with a Representation Alignment Projector
  6. 2025VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models
  7. 2025Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization
  8. 2026Representation by Design in Generation: Cross-View Class-Token Alignment in Diffusion Transformers