# Distilling Latent Manifolds: Resolution Extrapolation by Variational Autoencoders

arXiv: https://arxiv.org/abs/2603.14536 · HTML: https://mlsanity.com/paper/2603.14536/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 19 of its 20 references that are in the map.
- First posted on arXiv 15 Mar 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 19 of 19)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.17459/) (2024)
- [Knowledge Diffusion for Distillation](https://mlsanity.com/paper/2305.15712/) (2023)
- [Variational Autoencoder with Learned Latent Structure](https://mlsanity.com/paper/2006.10597/) (2020)
- [Scale-Invariant Convolutional Neural Networks](https://mlsanity.com/paper/1411.6369/) (2014)
- [LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Models](https://mlsanity.com/paper/2405.14477/) (2024)
- [Exploring Representation-Aligned Latent Space for Better Generation](https://mlsanity.com/paper/2502.00359/) (2025)
- [Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits](https://mlsanity.com/paper/2302.08086/) (2023)
- [Data-free Knowledge Distillation with Diffusion Models](https://mlsanity.com/paper/2504.00870/) (2025)
- [Disentangled Representation Learning and Generation with Manifold Optimization](https://mlsanity.com/paper/2006.07046/) (2020)
- [Hyperbolic VAE via Latent Gaussian Distributions](https://mlsanity.com/paper/2209.15217/) (2022)
- [H3AE: High Compression, High Speed, and High Quality AutoEncoder for Video Diffusion Models](https://mlsanity.com/paper/2504.10567/) (2025)
- [FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion](https://mlsanity.com/paper/2411.18552/) (2024)
- [Towards Learning Affine-Invariant Representations via Data-Efficient CNNs](https://mlsanity.com/paper/1909.00114/) (2019)
- [Exploring the ability of CNNs to generalise to previously unseen scales over wide scale ranges](https://mlsanity.com/paper/2004.01536/) (2020)
- [Adv-KD: Adversarial Knowledge Distillation for Faster Diffusion Sampling](https://mlsanity.com/paper/2405.20675/) (2024)
- [Hierarchical Invariance for Robust and Interpretable Vision Tasks at Larger Scales](https://mlsanity.com/paper/2402.15430/) (2024)

## Abstract (arXiv)

Variational Autoencoder (VAE) encoders play a critical role in modern generative models, yet their computational cost often motivates the use of knowledge distillation or quantification to obtain compact alternatives. Existing studies typically believe that the model work better on the samples closed to their training data distribution than unseen data distribution. In this work, we report a counter-intuitive phenomenon in VAE encoder distillation: a compact encoder distilled only at low resolutions exhibits poor reconstruction performance at its native resolution, but achieves dramatically improved results when evaluated at higher, unseen input resolutions. Despite never being trained beyond 2562 resolution, the distilled encoder generalizes effectively to 5122 resolution inputs, partially inheriting the teacher model's resolution preference.We further analyze latent distributions across resolutions and find that higher-resolution inputs produce latent representations more closely aligned with the teacher's manifold. Through extensive experiments on ImageNet-256, we show that simple resolution remapping-upsampling inputs before encoding and downsampling reconstructions for evaluation-leads to substantial gains across PSNR, MSE, SSIM, LPIPS, and rFID metrics. These findings suggest that VAE encoder distillation learns resolution-consistent latent manifolds rather than resolution-specific pixel mappings. This also means that the high training cost on memory, time and high-resolution datasets are not necessary conditions for distilling a VAE with high-resolution image reconstruction capabilities. On low resolution datasets, the distillation model still could learn the detailed knowledge of the teacher model in high-resolution image reconstruction.