# A Batch Normalized Inference Network Keeps the KL Vanishing Away

arXiv: https://arxiv.org/abs/2004.12585 · HTML: https://mlsanity.com/paper/2004.12585/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2020.

## Built on by (most cited first, 20 of 20)

- [Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment](https://mlsanity.com/paper/2312.15645/) (2023)
- [Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation](https://mlsanity.com/paper/2207.06130/) (2022)
- [Amortized Variational Inference: A Systematic Review](https://mlsanity.com/paper/2209.10888/) (2022)
- [VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding](https://mlsanity.com/paper/2211.00430/) (2022)
- [Hierarchical Distribution-Aware Testing of Deep Learning](https://mlsanity.com/paper/2205.08589/) (2022)
- [Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commenting](https://mlsanity.com/paper/2404.12782/) (2024)
- [Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness](https://mlsanity.com/paper/2110.12381/) (2021)
- [Towards Diverse, Relevant and Coherent Open-Domain Dialogue Generation via Hybrid Latent Variables](https://mlsanity.com/paper/2212.01145/) (2022)
- [On the Latent Holes of VAEs for Text Generation](https://mlsanity.com/paper/2110.03318/) (2021)
- [RVAE-LAMOL: Residual Variational Autoencoder to Enhance Lifelong Language Learning](https://mlsanity.com/paper/2205.10857/) (2022)
- [GMAIR: Unsupervised Object Detection Based on Spatial Attention and Gaussian Mixture](https://mlsanity.com/paper/2106.01722/) (2021)
- [Improving Variational Autoencoders with Density Gap-based Regularization](https://mlsanity.com/paper/2211.00321/) (2022)
- [Debunking Free Fusion Myth: Online Multi-view Anomaly Detection with Disentangled Product-of-Experts Modeling](https://mlsanity.com/paper/2310.18728/) (2023)
- [Neural Topic Modeling with Deep Mutual Information Estimation](https://mlsanity.com/paper/2203.06298/) (2022)
- [InitialGAN: A Language GAN with Completely Random Initialization](https://mlsanity.com/paper/2208.02531/) (2022)
- [Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generation](https://mlsanity.com/paper/2210.12409/) (2022)
- [Generalizable Semantic Vision Query Generation for Zero-shot Panoptic and Semantic Segmentation](https://mlsanity.com/paper/2402.13697/) (2024)
- [Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models](https://mlsanity.com/paper/2505.09062/) (2025)
- [Pureformer-VC: Non-parallel One-Shot Voice Conversion with Pure Transformer Blocks and Triplet Discriminative Training](https://mlsanity.com/paper/2409.01668/) (2024)
- [Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling](https://mlsanity.com/paper/2602.16979/) (2026)

## Abstract (arXiv)

Variational Autoencoder (VAE) is widely used as a generative model to approximate a model's posterior on latent variables by combining the amortized variational inference and deep neural networks. However, when paired with strong autoregressive decoders, VAE often converges to a degenerated local optimum known as "posterior collapse". Previous approaches consider the Kullback Leibler divergence (KL) individual for each datapoint. We propose to let the KL follow a distribution across the whole dataset, and analyze that it is sufficient to prevent posterior collapse by keeping the expectation of the KL's distribution positive. Then we propose Batch Normalized-VAE (BN-VAE), a simple but effective approach to set a lower bound of the expectation by regularizing the distribution of the approximate posterior's parameters. Without introducing any new model component or modifying the objective, our approach can avoid the posterior collapse effectively and efficiently. We further show that the proposed BN-VAE can be extended to conditional VAE (CVAE). Empirically, our approach surpasses strong autoregressive baselines on language modeling, text classification and dialogue generation, and rivals more complex approaches while keeping almost the same training time as VAE.