# On Feature Diversity in Energy-based Models

arXiv: https://arxiv.org/abs/2306.01489 · HTML: https://mlsanity.com/paper/2306.01489/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 30 of 32)

- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [A Theory of Generative ConvNet](https://mlsanity.com/paper/1602.03264/) (2016)
- [Generalization in Deep Learning](https://mlsanity.com/paper/1710.05468/) (2017)
- [Improved Contrastive Divergence Training of Energy Based Models](https://mlsanity.com/paper/2012.01316/) (2020)
- [Energy-based Generative Adversarial Network](https://mlsanity.com/paper/1609.03126/) (2016)
- [A Distributional Approach to Controlled Text Generation](https://mlsanity.com/paper/2012.11635/) (2020)
- [Deep Structured Energy Based Models for Anomaly Detection](https://mlsanity.com/paper/1605.07717/) (2016)
- [Residual Energy-Based Models for Text Generation](https://mlsanity.com/paper/2004.11714/) (2020)
- [Reducing Overfitting in Deep Networks by Decorrelating Representations](https://mlsanity.com/paper/1511.06068/) (2015)
- [Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling](https://mlsanity.com/paper/2003.06060/) (2020)
- [Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification](https://mlsanity.com/paper/2004.01301/) (2020)
- [Maximum Entropy Generators for Energy-Based Models](https://mlsanity.com/paper/1901.08508/) (2019)
- [Batch Active Learning Using Determinantal Point Processes](https://mlsanity.com/paper/1906.07975/) (2019)
- [Diverse Neural Network Learns True Target Functions](https://mlsanity.com/paper/1611.03131/) (2016)
- [Energy-Based Models for Continual Learning](https://mlsanity.com/paper/2011.12216/) (2020)
- [Training Deep Energy-Based Models with f-Divergence Minimization](https://mlsanity.com/paper/2003.03463/) (2020)
- [Leveraging Filter Correlations for Deep Model Compression](https://mlsanity.com/paper/1811.10559/) (2018)
- [How to Train Your Energy-Based Model for Regression](https://mlsanity.com/paper/2005.01698/) (2020)
- [Exact sampling of determinantal point processes with sublinear time preprocessing](https://mlsanity.com/paper/1905.13476/) (2019)
- [Universum Prescription: Regularization using Unlabeled Data](https://mlsanity.com/paper/1511.03719/) (2015)
- [Energy-Based Continuous Inverse Optimal Control](https://mlsanity.com/paper/1904.05453/) (2019)
- [Modelling heterogeneous distributions with an Uncountable Mixture of Asymmetric Laplacians](https://mlsanity.com/paper/1910.12288/) (2019)
- [Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness](https://mlsanity.com/paper/2110.12381/) (2021)
- [On the Generalization Error Bounds of Neural Networks under Diversity-Inducing Mutual Angular Regularization](https://mlsanity.com/paper/1511.07110/) (2015)

## Abstract (arXiv)

Energy-based learning is a powerful learning paradigm that encapsulates various discriminative and generative approaches. An energy-based model (EBM) is typically formed of inner-model(s) that learn a combination of the different features to generate an energy mapping for each input configuration. In this paper, we focus on the diversity of the produced feature set. We extend the probably approximately correct (PAC) theory of EBMs and analyze the effect of redundancy reduction on the performance of EBMs. We derive generalization bounds for various learning contexts, i.e., regression, classification, and implicit regression, with different energy functions and we show that indeed reducing redundancy of the feature set can consistently decrease the gap between the true and empirical expectation of the energy and boosts the performance of the model.