# RENs: Relevance Encoding Networks

arXiv: https://arxiv.org/abs/2205.13061 · HTML: https://mlsanity.com/paper/2205.13061/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 35)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [MoFlow: An Invertible Flow Model for Generating Molecular Graphs](https://mlsanity.com/paper/2006.10137/) (2020)
- [Variational Lossy Autoencoder](https://mlsanity.com/paper/1611.02731/) (2016)
- [Diversify and Match: A Domain Adaptive Representation Learning Paradigm for Object Detection](https://mlsanity.com/paper/1905.05396/) (2019)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Generative Models as a Data Source for Multiview Representation Learning](https://mlsanity.com/paper/2106.05258/) (2021)
- [Implicit Reparameterization Gradients](https://mlsanity.com/paper/1805.08498/) (2018)
- [Data Augmentation Using GANs](https://mlsanity.com/paper/1904.09135/) (2019)
- [On the Intrinsic Dimensionality of Image Representations](https://mlsanity.com/paper/1803.09672/) (2018)
- [Simple and Effective VAE Training with Calibrated Decoders](https://mlsanity.com/paper/2006.13202/) (2020)
- [Distribution Matching in Variational Inference](https://mlsanity.com/paper/1802.06847/) (2018)
- [Unsupervised Neural Machine Translation with Generative Language Models Only](https://mlsanity.com/paper/2110.05448/) (2021)
- [On the Latent Space of Wasserstein Auto-Encoders](https://mlsanity.com/paper/1802.03761/) (2018)
- [Variational Relevance Vector Machines](https://mlsanity.com/paper/1301.3838/) (2013)
- [Relevance Factor VAE: Learning and Identifying Disentangled Factors](https://mlsanity.com/paper/1902.01568/) (2019)
- [MARTA GANs: Unsupervised Representation Learning for Remote Sensing Image Classification](https://mlsanity.com/paper/1612.08879/) (2016)
- [Online Principal Component Analysis in High Dimension: Which Algorithm to Choose?](https://mlsanity.com/paper/1511.03688/) (2015)
- [Improving Variational Autoencoder with Deep Feature Consistent and Generative Adversarial Training](https://mlsanity.com/paper/1906.01984/) (2019)
- [Adversarial Symmetric Variational Autoencoder](https://mlsanity.com/paper/1711.04915/) (2017)
- [Hybrid Generative-Contrastive Representation Learning](https://mlsanity.com/paper/2106.06162/) (2021)
- [L0-ARM: Network Sparsification via Stochastic Binary Optimization](https://mlsanity.com/paper/1904.04432/) (2019)

## Abstract (arXiv)

The manifold assumption for high-dimensional data assumes that the data is generated by varying a set of parameters obtained from a low-dimensional latent space. Deep generative models (DGMs) are widely used to learn data representations in an unsupervised way. DGMs parameterize the underlying low-dimensional manifold in the data space using bottleneck architectures such as variational autoencoders (VAEs). The bottleneck dimension for VAEs is treated as a hyperparameter that depends on the dataset and is fixed at design time after extensive tuning. As the intrinsic dimensionality of most real-world datasets is unknown, often, there is a mismatch between the intrinsic dimensionality and the latent dimensionality chosen as a hyperparameter. This mismatch can negatively contribute to the model performance for representation learning and sample generation tasks. This paper proposes relevance encoding networks (RENs): a novel probabilistic VAE-based framework that uses the automatic relevance determination (ARD) prior in the latent space to learn the data-specific bottleneck dimensionality. The relevance of each latent dimension is directly learned from the data along with the other model parameters using stochastic gradient descent and a reparameterization trick adapted to non-Gaussian priors. We leverage the concept of DeepSets to capture permutation invariant statistical properties in both data and latent spaces for relevance determination. The proposed framework is general and flexible and can be used for the state-of-the-art VAE models that leverage regularizers to impose specific characteristics in the latent space (e.g., disentanglement). With extensive experimentation on synthetic and public image datasets, we show that the proposed model learns the relevant latent bottleneck dimensionality without compromising the representation and generation quality of the samples.