# SeNM-VAE: Semi-Supervised Noise Modeling with Hierarchical Variational Autoencoder

arXiv: https://arxiv.org/abs/2403.17502 · HTML: https://mlsanity.com/paper/2403.17502/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 71 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 30 of 43)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Uformer: A General U-Shaped Transformer for Image Restoration](https://mlsanity.com/paper/2106.03106/) (2021)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Simple Baselines for Image Restoration](https://mlsanity.com/paper/2204.04676/) (2022)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Solving Inverse Problems in Medical Imaging with Score-Based Generative Models](https://mlsanity.com/paper/2111.08005/) (2021)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [Plug-and-Play Image Restoration with Deep Denoiser Prior](https://mlsanity.com/paper/2008.13751/) (2020)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Noise2Void - Learning Denoising from Single Noisy Images](https://mlsanity.com/paper/1811.10980/) (2018)
- [MAXIM: Multi-Axis MLP for Image Processing](https://mlsanity.com/paper/2201.02973/) (2022)
- [Noise2Self: Blind Denoising by Self-Supervision](https://mlsanity.com/paper/1901.11365/) (2019)
- [Toward Convolutional Blind Denoising of Real Photographs](https://mlsanity.com/paper/1807.04686/) (2018)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Benchmarking Denoising Algorithms with Real Photographs](https://mlsanity.com/paper/1707.01313/) (2017)
- [SRFlow: Learning the Super-Resolution Space with Normalizing Flow](https://mlsanity.com/paper/2006.14200/) (2020)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Variational Denoising Network: Toward Blind Noise Modeling and Removal](https://mlsanity.com/paper/1908.11314/) (2019)
- [Frequency Separation for Real-World Super-Resolution](https://mlsanity.com/paper/1911.07850/) (2019)
- [Noise Flow: Noise Modeling with Conditional Normalizing Flows](https://mlsanity.com/paper/1908.08453/) (2019)
- [Dual Adversarial Network: Toward Real-world Noise Removal and Noise Generation](https://mlsanity.com/paper/2007.05946/) (2020)
- [To learn image super-resolution, use a GAN to learn how to do image degradation first](https://mlsanity.com/paper/1807.11458/) (2018)
- [AP-BSN: Self-Supervised Denoising for Real-World Images via Asymmetric PD and Blind-Spot Network](https://mlsanity.com/paper/2203.11799/) (2022)
- [Unsupervised Real-world Image Super Resolution via Domain-distance Aware Training](https://mlsanity.com/paper/2004.01178/) (2020)
- [NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results](https://mlsanity.com/paper/2005.01996/) (2020)

## Built on by (most cited first, 4 of 4)

- [MoCLIP: Motion-Aware Fine-Tuning and Distillation of CLIP for Human Motion Generation](https://mlsanity.com/paper/2505.10810/) (2025)
- [Unsupervised Methods for Video Quality Improvement: A Survey of Restoration and Enhancement Techniques](https://mlsanity.com/paper/2507.08375/) (2025)
- [Unpaired Joint Distribution Modeling via Multi-Scale Image Representations](https://mlsanity.com/paper/2607.08198/) (2026)
- [YeTI: You Only Need Two Noisy Images for Real-World sRGB Noise Generation](https://mlsanity.com/paper/2607.09193/) (2026)

## Abstract (arXiv)

The data bottleneck has emerged as a fundamental challenge in learning based image restoration methods. Researchers have attempted to generate synthesized training data using paired or unpaired samples to address this challenge. This study proposes SeNM-VAE, a semi-supervised noise modeling method that leverages both paired and unpaired datasets to generate realistic degraded data. Our approach is based on modeling the conditional distribution of degraded and clean images with a specially designed graphical model. Under the variational inference framework, we develop an objective function for handling both paired and unpaired data. We employ our method to generate paired training samples for real-world image denoising and super-resolution tasks. Our approach excels in the quality of synthetic degraded images compared to other unpaired and paired noise modeling methods. Furthermore, our approach demonstrates remarkable performance in downstream image restoration tasks, even with limited paired data. With more paired data, our method achieves the best performance on the SIDD dataset.