# Fine-structure Preserved Real-world Image Super-resolution via Transfer VAE Training

arXiv: https://arxiv.org/abs/2507.20291 · HTML: https://mlsanity.com/paper/2507.20291/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 73 of its 89 references that are in the map.
- First posted on arXiv 27 Jul 2025.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 73)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Designing a Practical Degradation Model for Deep Blind Image Super-Resolution](https://mlsanity.com/paper/2103.14006/) (2021)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)

## Built on by (most cited first, 21 of 21)

- [RobustGS: Unified Boosting of Feedforward 3D Gaussian Splatting under Low-Quality Conditions](https://mlsanity.com/paper/2508.03077/) (2025)
- [One-Step Diffusion Transformer for Controllable Real-World Image Super-Resolution](https://mlsanity.com/paper/2511.17138/) (2025)
- [VOSR: A Vision-Only Generative Model for Image Super-Resolution](https://mlsanity.com/paper/2604.03225/) (2026)
- [GDPO-SR: Group Direct Preference Optimization for One-Step Generative Image Super-Resolution](https://mlsanity.com/paper/2603.16769/) (2026)
- [Bridging Fidelity-Reality with Controllable One-Step Diffusion for Image Super-Resolution](https://mlsanity.com/paper/2512.14061/) (2025)
- [SSCM: A Spatial-Semantic Consistent Model for Multi-Contrast MRI Super-Resolution](https://mlsanity.com/paper/2509.18593/) (2025)
- [OP4KSR: One-Step Patch-Free 4K Super-Resolution with Periodic Artifact Suppression](https://mlsanity.com/paper/2605.13457/) (2026)
- [Do Protective Perturbations Really Protect Portrait Privacy under Real-world Image Transformations?](https://mlsanity.com/paper/2604.23688/) (2026)
- [Bridging Restoration and Generation in One-step Diffusion for Real-World Image Super-Resolution](https://mlsanity.com/paper/2604.24136/) (2026)
- [PGP-DiffSR: Phase-Guided Progressive Pruning for Efficient Diffusion-based Image Super-Resolution](https://mlsanity.com/paper/2512.02681/) (2025)
- [LCUDiff: Latent Capacity Upgrade Diffusion for Faithful Human Body Restoration](https://mlsanity.com/paper/2602.04406/) (2026)
- [ReconMIL: Synergizing Latent Space Reconstruction with Bi-Stream Mamba for Whole Slide Image Analysis](https://mlsanity.com/paper/2603.19925/) (2026)
- [VICR: Visual In-Context Restoration for Real-World Image Super-Resolution](https://mlsanity.com/paper/2606.00704/) (2026)
- [FreqOrtho-SR: Frequency-Guided Orthogonal Expert Learning for Real-World Image Super-Resolution](https://mlsanity.com/paper/2606.28745/) (2026)
- [Adaptive Densification for High-Fidelity and Efficient Sparse Gaussian Splatting in Arbitrary-Scale Super-Resolution](https://mlsanity.com/paper/2606.29400/) (2026)
- [Resonant Brane Splatting for Arbitrary-Scale Super-Resolution](https://mlsanity.com/paper/2606.29453/) (2026)
- [Efficient Difficulty-Aware Dynamic Routing for Diffusion-Based Real-World Image Super-Resolution](https://mlsanity.com/paper/2607.15711/) (2026)
- [Preserve More Details: Mitigating Content Drift in Real-World Image Super-Resolution](https://mlsanity.com/paper/2608.09373/) (2026)
- [SPARK: Input-Conditioned Sparse Activation Modulation for Frozen DiT-based Super-Resolution](https://mlsanity.com/paper/2609.03813/) (2026)
- [DPAMixerSR: An Efficient Degradation-Pattern-Aware Model for Image Super-Resolution](https://mlsanity.com/paper/2609.32705/) (2026)
- [LoCoVSR: Local Context Diffusion Posterior Sampling for Video Super-Resolution](https://mlsanity.com/paper/2609.32742/) (2026)

## Abstract (arXiv)

Impressive results on real-world image super-resolution (Real-ISR) have been achieved by employing pre-trained stable diffusion (SD) models. However, one critical issue of such methods lies in their poor reconstruction of image fine structures, such as small characters and textures, due to the aggressive resolution reduction of the VAE (eg., 8× downsampling) in the SD model. One solution is to employ a VAE with a lower downsampling rate for diffusion; however, adapting its latent features with the pre-trained UNet while mitigating the increased computational cost poses new challenges. To address these issues, we propose a Transfer VAE Training (TVT) strategy to transfer the 8× downsampled VAE into a 4× one while adapting to the pre-trained UNet. Specifically, we first train a 4× decoder based on the output features of the original VAE encoder, then train a 4× encoder while keeping the newly trained decoder fixed. Such a TVT strategy aligns the new encoder-decoder pair with the original VAE latent space while enhancing image fine details. Additionally, we introduce a compact VAE and compute-efficient UNet by optimizing their network architectures, reducing the computational cost while capturing high-resolution fine-scale features. Experimental results demonstrate that our TVT method significantly improves fine-structure preservation, which is often compromised by other SD-based methods, while requiring fewer FLOPs than state-of-the-art one-step diffusion models. The official code can be found at https://github.com/Joyies/TVT.