# Encoding Binary Concepts in the Latent Space of Generative Models for Enhancing Data Representation

arXiv: https://arxiv.org/abs/2303.12255 · HTML: https://mlsanity.com/paper/2303.12255/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 22 Mar 2023.

## Builds on (most cited first, 22 of 22)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1611.02648/) (2016)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [Complementary Learning for Overcoming Catastrophic Forgetting Using Experience Replay](https://mlsanity.com/paper/1903.04566/) (2019)
- [Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning](https://mlsanity.com/paper/2104.08808/) (2021)
- [Detection and Continual Learning of Novel Face Presentation Attacks](https://mlsanity.com/paper/2108.12081/) (2021)
- [Generative Continual Concept Learning](https://mlsanity.com/paper/1906.03744/) (2019)
- [GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL](https://mlsanity.com/paper/2110.15489/) (2021)

## Built on by (most cited first, 2 of 2)

- [Cognitively Inspired Cross-Modal Data Generation Using Diffusion Models](https://mlsanity.com/paper/2305.18433/) (2023)
- [Robust Internal Representations for Domain Generalization](https://mlsanity.com/paper/2309.15522/) (2023)

## Abstract (arXiv)

Binary concepts are empirically used by humans to generalize efficiently. And they are based on Bernoulli distribution which is the building block of information. These concepts span both low-level and high-level features such as "large vs small" and "a neuron is active or inactive". Binary concepts are ubiquitous features and can be used to transfer knowledge to improve model generalization. We propose a novel binarized regularization to facilitate learning of binary concepts to improve the quality of data generation in autoencoders. We introduce a binarizing hyperparameter r in data generation process to disentangle the latent space symmetrically. We demonstrate that this method can be applied easily to existing variational autoencoder (VAE) variants to encourage symmetric disentanglement, improve reconstruction quality, and prevent posterior collapse without computation overhead. We also demonstrate that this method can boost existing models to learn more transferable representations and generate more representative samples for the input distribution which can alleviate catastrophic forgetting using generative replay under continual learning settings.