Energy-Calibrated VAE with Test Time Free Lunch
Not on the map. Why?
Abstract (arXiv)
In this paper, we propose a novel generative model that utilizes a conditional Energy-Based Model (EBM) for enhancing Variational Autoencoder (VAE), termed Energy-Calibrated VAE (EC-VAE). Specifically, VAEs often suffer from blurry generated samples due to the lack of a tailored training on the samples generated in the generative direction. On the other hand, EBMs can generate high-quality samples but require expensive Markov Chain Monte Carlo (MCMC) sampling. To address these issues, we introduce a conditional EBM for calibrating the generative direction of VAE during training, without requiring it for the generation at test time. In particular, we train EC-VAE upon both the input data and the calibrated samples with adaptive weight to enhance efficacy while avoiding MCMC sampling at test time. Furthermore, we extend the calibration idea of EC-VAE to variational learning and normalizing flows, and apply EC-VAE to an additional application of zero-shot image restoration via neural transport prior and range-null theory. We evaluate the proposed method with two applications, including image generation and zero-shot image restoration, and the experimental results show that our method achieves competitive performance over single-step non-adversarial generation. Our code is available at https://github.com/DJ-LYH/EC-VAE.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 50 of its 64 references that are in the map. First posted on arXiv 7 Nov 2023.
Builds on
50 of 64 references.
References · 50
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2022Flow Matching for Generative Modelingcited by 5,426
- 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
- 2013Auto-Encoding Variational Bayescited by 3,980
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
- 2016Density estimation using Real NVPcited by 2,029
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
- 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
38 more
- 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
- 2017Adversarial Discriminative Domain Adaptationcited by 1,309
- 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
- 2014NICE: Non-linear Independent Components Estimationcited by 1,052
- 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
- 2022Denoising Diffusion Restoration Modelscited by 951
- 2016Conditional Image Generation with PixelCNN Decoderscited by 859
- 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
- 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
- 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 574
- 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
- 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
- 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
- 2017VAE with a VampPriorcited by 247
- 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
- 2016A Theory of Generative ConvNetcited by 174
- 2020Adversarial Latent Autoencoderscited by 155
- 2021Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesiscited by 140
- 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 122
- 2020VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Modelscited by 103
- 2019AutoGAN: Neural Architecture Search for Generative Adversarial Networkscited by 94
- 2018IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesiscited by 85
- 2019Flow Contrastive Estimation of Energy-Based Modelscited by 75
- 2020SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flowscited by 71
- 2019COCO-GAN: Generation by Parts via Conditional Coordinatingcited by 67
- 2020A Contrastive Learning Approach for Training Variational Autoencoder Priorscited by 65
- 2020Dual Contradistinctive Generative Autoencodercited by 58
- 2022A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Modelcited by 44
- 2020Learning Energy-Based Model with Variational Auto-Encoder as Amortized Samplercited by 43
- 2021Constrained Learning with Non-Convex Lossescited by 42
- 2020The Expressive Power of a Class of Normalizing Flow Modelscited by 35
- 2020Joint Training of Variational Auto-Encoder and Latent Energy-Based Modelcited by 30
- 2020MCMC Should Mix: Learning Energy-Based Model with Neural Transport Latent Space MCMCcited by 25
- 2018Improving MMD-GAN Training with Repulsive Loss Functioncited by 22
- 2023Guiding Energy-based Models via Contrastive Latent Variablescited by 17
- 2022Learning Probabilistic Models from Generator Latent Spaces with Hat EBMcited by 11
- 2019Quality Aware Generative Adversarial Networkscited by 10
- 2019Generative Latent Flowcited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.