# Arbitrary-Scale Image Generation and Upsampling using Latent Diffusion Model and Implicit Neural Decoder

arXiv: https://arxiv.org/abs/2403.10255 · HTML: https://mlsanity.com/paper/2403.10255/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 32)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Learning Continuous Image Representation with Local Implicit Image Function](https://mlsanity.com/paper/2012.09161/) (2020)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models](https://mlsanity.com/paper/2003.03808/) (2020)
- [Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization](https://mlsanity.com/paper/2308.14469/) (2023)
- [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023)
- [Meta-SR: A Magnification-Arbitrary Network for Super-Resolution](https://mlsanity.com/paper/1903.00875/) (2019)
- [GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution](https://mlsanity.com/paper/2012.00739/) (2020)
- [Adversarial Generation of Continuous Images](https://mlsanity.com/paper/2011.12026/) (2020)
- [Local Texture Estimator for Implicit Representation Function](https://mlsanity.com/paper/2111.08918/) (2021)
- [Image Generators with Conditionally-Independent Pixel Synthesis](https://mlsanity.com/paper/2011.13775/) (2020)
- [Positional Encoding as Spatial Inductive Bias in GANs](https://mlsanity.com/paper/2012.05217/) (2020)
- [Pyramidal Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2208.01864/) (2022)
- [Arbitrary-Scale Image Synthesis](https://mlsanity.com/paper/2204.02273/) (2022)

## Built on by (most cited first, 25 of 25)

- [3D MedDiffusion: A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation](https://mlsanity.com/paper/2412.13059/) (2024)
- [Acquire and then Adapt: Squeezing out Text-to-Image Model for Image Restoration](https://mlsanity.com/paper/2504.15159/) (2025)
- [AutoMV: An Automatic Multi-Agent System for Music Video Generation](https://mlsanity.com/paper/2512.12196/) (2025)
- [Latent Diffusion, Implicit Amplification: Efficient Continuous-Scale Super-Resolution for Remote Sensing Images](https://mlsanity.com/paper/2410.22830/) (2024)
- [Network Bending of Diffusion Models for Audio-Visual Generation](https://mlsanity.com/paper/2406.19589/) (2024)
- [SASNet: Spatially-Adaptive Sinusoidal Networks for INRs](https://mlsanity.com/paper/2503.09750/) (2025)
- [SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit 3D Meshes](https://mlsanity.com/paper/2509.21859/) (2025)
- [Prompt Augmentation for Self-supervised Text-guided Image Manipulation](https://mlsanity.com/paper/2412.13081/) (2024)
- [Efficient Temporal Consistency in Diffusion-Based Video Editing with Adaptor Modules: A Theoretical Framework](https://mlsanity.com/paper/2504.16016/) (2025)
- [Temporal and Spatial Super Resolution with Latent Diffusion Model in Medical MRI images](https://mlsanity.com/paper/2410.23898/) (2024)
- [Self-Cascaded Diffusion Models for Arbitrary-Scale Image Super-Resolution](https://mlsanity.com/paper/2506.07813/) (2025)
- [Diffusion Transformer meets Multi-level Wavelet Spectrum for Single Image Super-Resolution](https://mlsanity.com/paper/2511.01175/) (2025)
- [OmniScaleSR: Unleashing Scale-Controlled Diffusion Prior for Faithful and Realistic Arbitrary-Scale Image Super-Resolution](https://mlsanity.com/paper/2512.04699/) (2025)
- [CASR: A Robust Cyclic Framework for Arbitrary Large-Scale Super-Resolution with Distribution Alignment and Self-Similarity Awareness](https://mlsanity.com/paper/2602.22159/) (2026)
- [Co-learning Single-Step Diffusion Upsampler and Downsampler with Two Discriminators and Distillation](https://mlsanity.com/paper/2410.07663/) (2024)
- [From Diffusion to Resolution: Leveraging 2D Diffusion Models for 3D Super-Resolution Task](https://mlsanity.com/paper/2411.16792/) (2024)
- [Stroke-based Cyclic Amplifier: Image Super-Resolution at Arbitrary Ultra-Large Scales](https://mlsanity.com/paper/2506.10774/) (2025)
- [APT: Improving Diffusion Models for High Resolution Image Generation with Adaptive Path Tracing](https://mlsanity.com/paper/2507.21690/) (2025)
- [Physically-Guided Optical Inversion Enable Non-Contact Side-Channel Attack on Isolated Screens](https://mlsanity.com/paper/2604.13419/) (2026)
- [RaPD: Resolution-Agnostic Pixel Diffusion via Semantics-Enriched Implicit Representations](https://mlsanity.com/paper/2605.15908/) (2026)
- [Fidelity- and Perception-Aware Local Implicit Attention for Arbitrary-Scale Image Super-Resolution](https://mlsanity.com/paper/2606.21910/) (2026)
- [CoIn: Comprehensive 2D-3D Inpainting with Gaussian Splatting Guidance](https://mlsanity.com/paper/2606.27584/) (2026)
- [Adaptive Densification for High-Fidelity and Efficient Sparse Gaussian Splatting in Arbitrary-Scale Super-Resolution](https://mlsanity.com/paper/2606.29400/) (2026)
- [AVSR-Diff: Scale-Agnostic Diffusion Priors for Temporally Consistent Arbitrary-Scale Video Super-Resolution](https://mlsanity.com/paper/2607.00987/) (2026)
- [Coarse-to-fine Framework for Generative MEF via Implicit Neural Representation](https://mlsanity.com/paper/2607.17611/) (2026)

## Abstract (arXiv)

Super-resolution (SR) and image generation are important tasks in computer vision and are widely adopted in real-world applications. Most existing methods, however, generate images only at fixed-scale magnification and suffer from over-smoothing and artifacts. Additionally, they do not offer enough diversity of output images nor image consistency at different scales. Most relevant work applied Implicit Neural Representation (INR) to the denoising diffusion model to obtain continuous-resolution yet diverse and high-quality SR results. Since this model operates in the image space, the larger the resolution of image is produced, the more memory and inference time is required, and it also does not maintain scale-specific consistency. We propose a novel pipeline that can super-resolve an input image or generate from a random noise a novel image at arbitrary scales. The method consists of a pretrained auto-encoder, a latent diffusion model, and an implicit neural decoder, and their learning strategies. The proposed method adopts diffusion processes in a latent space, thus efficient, yet aligned with output image space decoded by MLPs at arbitrary scales. More specifically, our arbitrary-scale decoder is designed by the symmetric decoder w/o up-scaling from the pretrained auto-encoder, and Local Implicit Image Function (LIIF) in series. The latent diffusion process is learnt by the denoising and the alignment losses jointly. Errors in output images are backpropagated via the fixed decoder, improving the quality of output images. In the extensive experiments using multiple public benchmarks on the two tasks i.e. image super-resolution and novel image generation at arbitrary scales, the proposed method outperforms relevant methods in metrics of image quality, diversity and scale consistency. It is significantly better than the relevant prior-art in the inference speed and memory usage.