# One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation

arXiv: https://arxiv.org/abs/2512.07829 · HTML: https://mlsanity.com/paper/2512.07829/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 50 of its 53 references that are in the map.
- First posted on arXiv 8 Dec 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)

## Built on by (most cited first, 28 of 28)

- [Improved Baselines with Representation Autoencoders](https://mlsanity.com/paper/2605.18324/) (2026)
- [Making Reconstruction FID Predictive of Diffusion Generation FID](https://mlsanity.com/paper/2603.05630/) (2026)
- [Improving Reconstruction of Representation Autoencoder](https://mlsanity.com/paper/2602.08620/) (2026)
- [What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion](https://mlsanity.com/paper/2605.07915/) (2026)
- [RPiAE: A Representation-Pivoted Autoencoder Enhancing Both Image Generation and Editing](https://mlsanity.com/paper/2603.19206/) (2026)
- [Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model](https://mlsanity.com/paper/2603.05438/) (2026)
- [WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling](https://mlsanity.com/paper/2605.06407/) (2026)
- [Geometric Autoencoder for Diffusion Models](https://mlsanity.com/paper/2603.10365/) (2026)
- [STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation](https://mlsanity.com/paper/2605.08029/) (2026)
- [Normalizing Trajectory Models](https://mlsanity.com/paper/2605.08078/) (2026)
- [RiT: Vanilla Diffusion Transformers Suffice in Representation Space](https://mlsanity.com/paper/2605.21981/) (2026)
- [DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders](https://mlsanity.com/paper/2605.22777/) (2026)
- [IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder](https://mlsanity.com/paper/2606.11096/) (2026)
- [ReWorld: Representation Learning for World Action Models](https://mlsanity.com/paper/2606.27504/) (2026)
- [RePack then Refine: Efficient Diffusion Transformer with Vision Foundation Model](https://mlsanity.com/paper/2512.12083/) (2025)
- [Vision Foundation Models as Generalist Tokenizers for Image Generation](https://mlsanity.com/paper/2605.18390/) (2026)
- [X-Foresight: A Joint Vision-Action Causal Forecasting Network via Predictive World Modeling](https://mlsanity.com/paper/2605.24892/) (2026)
- [Generate in Reconstruction Space, Match in Semantic Space: Transport Geometry for One-Step Generation](https://mlsanity.com/paper/2606.00514/) (2026)
- [Learning When to Denoise: Optimizing Asynchronous Schedules for Latent Diffusion](https://mlsanity.com/paper/2606.19662/) (2026)
- [MIMFlow: Integrating Masked Image Modeling with Normalizing Flows for End-to-End Image Generation](https://mlsanity.com/paper/2606.26016/) (2026)
- [A Tilted Seesaw: Revisiting Autoencoder Trade-off for Controllable Diffusion](https://mlsanity.com/paper/2601.21633/) (2026)
- [UniCom: Unified Multimodal Modeling via Compressed Continuous Semantic Representations](https://mlsanity.com/paper/2603.10702/) (2026)
- [SPAE: Spectrally Guided Autoencoder for Pretrained Visual Latents](https://mlsanity.com/paper/2608.01306/) (2026)
- [The Domain Is a Residue: Adapting Self-Supervised Features, Not Generators](https://mlsanity.com/paper/2609.37330/) (2026)
- [HiRAE: Hierarchical Representation Autoencoding with Residual Budgets](https://mlsanity.com/paper/2609.37775/) (2026)
- [Rethinking Generative Image Compression at Extremely Low Bitrates](https://mlsanity.com/paper/2609.39315/) (2026)
- [SAGE: Salient Factor Discovery and Generation with Visual Foundation Representations](https://mlsanity.com/paper/2609.39635/) (2026)
- [Pooling Representation Autoencoders for Efficient Diffusion](https://mlsanity.com/paper/2610.09242/) (2026)

## Abstract (arXiv)

Visual generative models (e.g., diffusion models) typically operate in compressed latent spaces to balance training efficiency and sample quality. In parallel, there has been growing interest in leveraging high-quality pre-trained visual representations, either by aligning them inside VAEs or directly within the generative model. However, adapting such representations remains challenging due to fundamental mismatches between understanding-oriented features and generation-friendly latent spaces. Representation encoders benefit from high-dimensional latents that capture diverse hypotheses for masked regions, whereas generative models favor low-dimensional latents that must faithfully preserve injected noise. This discrepancy has led prior work to rely on complex objectives and architectures. In this work, we propose FAE (Feature Auto-Encoder), a simple yet effective framework that adapts pre-trained visual representations into low-dimensional latents suitable for generation using as little as a single attention layer, while retaining sufficient information for both reconstruction and understanding. The key is to couple two separate deep decoders: one trained to reconstruct the original feature space, and a second that takes the reconstructed features as input for image generation. FAE is generic; it can be instantiated with a variety of self-supervised encoders (e.g., DINO, SigLIP) and plugged into two distinct generative families: diffusion models and normalizing flows. Across class-conditional and text-to-image benchmarks, FAE achieves strong performance. For example, on ImageNet 256x256, our diffusion model with CFG attains a near state-of-the-art FID of 1.29 (800 epochs) and 1.70 (80 epochs). Without CFG, FAE reaches the state-of-the-art FID of 1.48 (800 epochs) and 2.08 (80 epochs), demonstrating both high quality and fast learning.