# Enhancing variational generation through self-decomposition

arXiv: https://arxiv.org/abs/2202.02738 · HTML: https://mlsanity.com/paper/2202.02738/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 6 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [From Variational to Deterministic Autoencoders](https://mlsanity.com/paper/1903.12436/) (2019)
- [An Introduction to Deep Generative Modeling](https://mlsanity.com/paper/2103.05180/) (2021)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Balancing reconstruction error and Kullback-Leibler divergence in Variational Autoencoders](https://mlsanity.com/paper/2002.07514/) (2020)
- [Overpruning in Variational Bayesian Neural Networks](https://mlsanity.com/paper/1801.06230/) (2018)
- [A survey on Variational Autoencoders from a GreenAI perspective](https://mlsanity.com/paper/2103.01071/) (2021)
- [Tackling Over-pruning in Variational Autoencoders](https://mlsanity.com/paper/1706.03643/) (2017)
- [Dissecting FLOPs along input dimensions for GreenAI cost estimations](https://mlsanity.com/paper/2107.11949/) (2021)
- [Blind Deblurring using Deep Learning: A Survey](https://mlsanity.com/paper/1907.10128/) (2019)
- [Variational Autoencoders with Normalizing Flow Decoders](https://mlsanity.com/paper/2004.05617/) (2020)
- [Sparsity in Variational Autoencoders](https://mlsanity.com/paper/1812.07238/) (2018)
- [Variance Loss in Variational Autoencoders](https://mlsanity.com/paper/2002.09860/) (2020)

## Built on by (most cited first, 1 of 1)

- [Comparing the latent space of generative models](https://mlsanity.com/paper/2207.06812/) (2022)

## Abstract (arXiv)

In this article we introduce the notion of Split Variational Autoencoder (SVAE), whose output hatx is obtained as a weighted sum σ odot hatx1 + (1-σ) odot hatx2 of two generated images hatx1,hatx2, and σ is a em learned compositional map. The composing images hatx1,hatx2, as well as the σ-map are automatically synthesized by the model. The network is trained as a usual Variational Autoencoder with a negative loglikelihood loss between training and reconstructed images. No additional loss is required for hatx1,hatx2 or σ, neither any form of human tuning. The decomposition is nondeterministic, but follows two main schemes, that we may roughly categorize as either saysyntactic or saysemantic. In the first case, the map tends to exploit the strong correlation between adjacent pixels, splitting the image in two complementary high frequency sub-images. In the second case, the map typically focuses on the contours of objects, splitting the image in interesting variations of its content, with more marked and distinctive features. In this case, according to empirical observations, the Fr'echet Inception Distance (FID) of hatx1 and hatx2 is usually lower (hence better) than that of hatx, that clearly suffers from being the average of the former. In a sense, a SVAE forces the Variational Autoencoder to make choices, in contrast with its intrinsic tendency to em average between alternatives with the aim to minimize the reconstruction loss towards a specific sample. According to the FID metric, our technique, tested on typical datasets such as Mnist, Cifar10 and CelebA, allows us to outperform all previous purely variational architectures (not relying on normalization flows).