mlsanityData to 7 Oct 2026

Towards Bridging the Performance Gaps of Joint Energy-based Models

arXiv 2209.07959 ↗First posted 16 Sep 2022cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

Can we train a hybrid discriminative-generative model within a single network? This question has recently been answered in the affirmative, introducing the field of Joint Energy-based Model (JEM), which achieves high classification accuracy and image generation quality simultaneously. Despite recent advances, there remain two performance gaps: the accuracy gap to the standard softmax classifier, and the generation quality gap to state-of-the-art generative models. In this paper, we introduce a variety of training techniques to bridge the accuracy gap and the generation quality gap of JEM. 1) We incorporate a recently proposed sharpness-aware minimization (SAM) framework to train JEM, which promotes the energy landscape smoothness and the generalizability of JEM. 2) We exclude data augmentation from the maximum likelihood estimate pipeline of JEM, and mitigate the negative impact of data augmentation to image generation quality. Extensive experiments on multiple datasets demonstrate that our SADA-JEM achieves state-of-the-art performances and outperforms JEM in image classification, image generation, calibration, out-of-distribution detection and adversarial robustness by a notable margin. Our code is available at https://github.com/sndnyang/SADAJEM.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 16 Sep 2022.

Builds on

39 of 51 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2022Generative Adversarial Networkscited by 5,603
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2017On Calibration of Modern Neural Networkscited by 4,860
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2016Wide Residual Networkscited by 3,057
  12. 2016Improved Techniques for Training GANscited by 2,938
27 more
  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  2. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  7. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  8. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  9. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  10. 2017Countering Adversarial Images using Input Transformationscited by 517
  11. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  12. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  13. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  14. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  15. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  16. 2016A study of the effect of JPG compression on adversarial imagescited by 196
  17. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 127
  18. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 122
  19. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  20. 2020VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Modelscited by 103
  21. 2020Certified Defenses for Adversarial Patchescited by 100
  22. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  23. 2020No MCMC for me: Amortized sampling for fast and stable training of energy-based modelscited by 64
  24. 2020Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Modelscited by 58
  25. 2017Defense against Universal Adversarial Perturbationscited by 48
  26. 2021JEM++: Improved Techniques for Training JEMcited by 28
  27. 2021Towards Understanding the Generative Capability of Adversarially Robust Classifierscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Shedding More Light on Robust Classifiers under the lens of Energy-based Modelscited by 9
  2. 2024Improving Generalization with Flat Hilbert Bayesian Inferencecited by 5
  3. 2024Text-to-Image Generation Via Energy-Based CLIPcited by 3
  4. 2025Uncertainty and Energy based Loss Guided Semi-Supervised Semantic Segmentationcited by 3
  5. 2025Learning Energy-Based Generative Models via Potential Flow: A Variational Principle Approach to Probability Density Homotopy Matchingcited by 2
  6. 2025Understanding Adversarial Training with Energy-based Modelscited by 2
  7. 2024CALICO: Confident Active Learning with Integrated Calibrationcited by 1
  8. 2024Classification-Denoising Networkscited by 1
  9. 2025Your Classifier Can Do More: Towards Balancing the Gaps in Classification, Robustness, and Generationcited by 1
  10. 2025Scalable Energy-Based Models via Adversarial Training: Unifying Discrimination and Generationcited by 1
  11. 2026A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Modelscited by 1
  12. 2024Exploring Energy-Based Models for Out-of-Distribution Detection in Dialect Identificationcited by 0
3 more
  1. 2024Variational Potential Flow: A Novel Probabilistic Framework for Energy-Based Generative Modellingcited by 0
  2. 2026Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spotcited by 0
  3. 2026Comparing SGLD and a fixed-noise Predictor-Corrector adaptation in canonical Joint Energy-Based Models on CIFAR-10cited by 0