mlsanityData to 14 Oct 2026

JEM++: Improved Techniques for Training JEM

arXiv 2109.09032 ↗First posted 19 Sep 2021cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Joint Energy-based Model (JEM) is a recently proposed hybrid model that retains strong discriminative power of modern CNN classifiers, while generating samples rivaling the quality of GAN-based approaches. In this paper, we propose a variety of new training procedures and architecture features to improve JEM's accuracy, training stability, and speed altogether. 1) We propose a proximal SGLD to generate samples in the proximity of samples from the previous step, which improves the stability. 2) We further treat the approximate maximum likelihood learning of EBM as a multi-step differential game, and extend the YOPO framework to cut out redundant calculations during backpropagation, which accelerates the training substantially. 3) Rather than initializing SGLD chain from random noise, we introduce a new informative initialization that samples from a distribution estimated from training data. 4) This informative initialization allows us to enable batch normalization in JEM, which further releases the power of modern CNN architectures for hybrid modeling. Code: https://github.com/sndnyang/JEMPP

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 35 of its 61 references that are in the map. First posted on arXiv 19 Sep 2021.

Builds on

35 of 61 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2013Intriguing properties of neural networkscited by 5,155
  9. 2017On Calibration of Modern Neural Networkscited by 4,960
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2013Auto-Encoding Variational Bayescited by 4,053
  12. 2016Wide Residual Networkscited by 3,070
23 more
  1. 2016Improved Techniques for Training GANscited by 2,970
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  3. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  5. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  6. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  7. 2017Countering Adversarial Images using Input Transformationscited by 520
  8. 2018Do Deep Generative Models Know What They Don't Know?cited by 443
  9. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
  10. 2018Invertible Residual Networkscited by 365
  11. 2019Adversarial Examples Improve Image Recognitioncited by 286
  12. 2019Residual Flows for Invertible Generative Modelingcited by 257
  13. 2016A study of the effect of JPG compression on adversarial imagescited by 197
  14. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 183
  15. 2016A Theory of Generative ConvNetcited by 174
  16. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  17. 2020Certified Defenses for Adversarial Patchescited by 101
  18. 2019Flow Contrastive Estimation of Energy-Based Modelscited by 75
  19. 2020No MCMC for me: Amortized sampling for fast and stable training of energy-based modelscited by 64
  20. 2016Cooperative Training of Descriptor and Generator Networkscited by 60
  21. 2017Defense against Universal Adversarial Perturbationscited by 48
  22. 2018Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Modelcited by 34
  23. 2020Joint Training of Variational Auto-Encoder and Latent Energy-Based Modelcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 10; 2025: 5; 2026: 2.

28 papers.

Sort:

By area:

  1. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Modelcited by 58
  2. 2023TEA: Test-time Energy Adaptationcited by 25
  3. 2023Learning Data Representations with Joint Diffusion Modelscited by 22
  4. 2023Guiding Energy-based Models via Contrastive Latent Variablescited by 18
  5. 2022Towards Bridging the Performance Gaps of Joint Energy-based Modelscited by 15
  6. 2022Energy-Based Contrastive Learning of Visual Representationscited by 13
  7. 2024Shedding More Light on Robust Classifiers under the lens of Energy-based Modelscited by 9
  8. 2023M-EBM: Towards Understanding the Manifolds of Energy-Based Modelscited by 6
  9. 2024EQ-CBM: A Probabilistic Concept Bottleneck with Energy-based Models and Quantized Vectorscited by 6
  10. 2024Text-to-Image Generation Via Energy-Based CLIPcited by 3
  11. 2023Exploring the Connection between Robust and Generative Modelscited by 2
  12. 2024On Discriminative Probabilistic Modeling for Self-Supervised Representation Learningcited by 2
16 more
  1. 2025Exploring Structured Semantic Priors Underlying Diffusion Score for Test-time Adaptationcited by 2
  2. 2025Understanding Adversarial Training with Energy-based Modelscited by 2
  3. 2022Mitigating Out-of-Distribution Data Density Overestimation in Energy-Based Modelscited by 1
  4. 2024CALICO: Confident Active Learning with Integrated Calibrationcited by 1
  5. 2024Classification-Denoising Networkscited by 1
  6. 2024MITA: Bridging the Gap between Model and Data for Test-time Adaptationcited by 1
  7. 2025Your Classifier Can Do More: Towards Balancing the Gaps in Classification, Robustness, and Generationcited by 1
  8. 2025Scalable Energy-Based Models via Adversarial Training: Unifying Discrimination and Generationcited by 1
  9. 2023Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneckcited by 0
  10. 2023Non-Generative Energy Based Modelscited by 0
  11. 2024A Hybrid Generative and Discriminative PointNet on Unordered Point Setscited by 0
  12. 2024Exploring Energy-Based Models for Out-of-Distribution Detection in Dialect Identificationcited by 0
  13. 2024Critic Loss for Image Classificationcited by 0
  14. 2025Contrastive Residual Energy Test-time Adaptationcited by 0
  15. 2026Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shiftcited by 0
  16. 2026Not Too Generative, Not Too Discriminative: The Human Alignment Sweet Spotcited by 0