mlsanityData to 14 Oct 2026

On Feature Diversity in Energy-based Models

arXiv 2306.01489 ↗First posted 2 Jun 2023cs.LG cs.IT math.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

Energy-based learning is a powerful learning paradigm that encapsulates various discriminative and generative approaches. An energy-based model (EBM) is typically formed of inner-model(s) that learn a combination of the different features to generate an energy mapping for each input configuration. In this paper, we focus on the diversity of the produced feature set. We extend the probably approximately correct (PAC) theory of EBMs and analyze the effect of redundancy reduction on the performance of EBMs. We derive generalization bounds for various learning contexts, i.e., regression, classification, and implicit regression, with different energy functions and we show that indeed reducing redundancy of the feature set can consistently decrease the gap between the true and empirical expectation of the energy and boosts the performance of the model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 59 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

32 of 59 references.

References · 32

  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  2. 2016Learning without Forgettingcited by 2,496
  3. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  6. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  7. 2017Exploring Generalization in Deep Learningcited by 600
  8. 2016A Theory of Generative ConvNetcited by 174
  9. 2017Generalization in Deep Learningcited by 146
  10. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 130
  11. 2016Energy-based Generative Adversarial Networkcited by 119
  12. 2020A Distributional Approach to Controlled Text Generationcited by 119
20 more
  1. 2016Deep Structured Energy Based Models for Anomaly Detectioncited by 114
  2. 2020Residual Energy-Based Models for Text Generationcited by 95
  3. 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
  4. 2020Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Samplingcited by 71
  5. 2020Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classificationcited by 57
  6. 2019Maximum Entropy Generators for Energy-Based Modelscited by 56
  7. 2019Batch Active Learning Using Determinantal Point Processescited by 43
  8. 2016Diverse Neural Network Learns True Target Functionscited by 33
  9. 2020Energy-Based Models for Continual Learningcited by 32
  10. 2020Training Deep Energy-Based Models with f-Divergence Minimizationcited by 29
  11. 2018Leveraging Filter Correlations for Deep Model Compressioncited by 23
  12. 2020How to Train Your Energy-Based Model for Regressioncited by 23
  13. 2019Exact sampling of determinantal point processes with sublinear time preprocessingcited by 19
  14. 2015Universum Prescription: Regularization using Unlabeled Datacited by 16
  15. 2019Energy-Based Continuous Inverse Optimal Controlcited by 14
  16. 2019Modelling heterogeneous distributions with an Uncountable Mixture of Asymmetric Laplacianscited by 9
  17. 2021Regularizing Variational Autoencoder with Diversity and Uncertainty Awarenesscited by 6
  18. 2015On the Generalization Error Bounds of Neural Networks under Diversity-Inducing Mutual Angular Regularizationcited by 4
  19. 2023WLD-Reg: A Data-dependent Within-layer Diversity Regularizercited by 4
  20. 2021Learning Proposals for Practical Energy-Based Regressioncited by 3

Built on by

Not cited yet.