mlsanityData to 7 Oct 2026

Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models

arXiv 2101.06829 ↗First posted 18 Jan 2021cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

In this work, we explore joint energy-based model (EBM) training during the finetuning of pretrained text encoders (e.g., Roberta) for natural language understanding (NLU) tasks. Our experiments show that EBM training can help the model reach a better calibration that is competitive to strong baselines, with little or no loss in accuracy. We discuss three variants of energy functions (namely scalar, hidden, and sharp-hidden) that can be defined on top of a text encoder, and compare them in experiments. Due to the discreteness of text data, we adopt noise contrastive estimation (NCE) to train the energy-based model. To make NCE training more effective, we train an auto-regressive noise model with the masked language model (MLM) objective.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 14 of its 22 references that are in the map. First posted on arXiv 18 Jan 2021.

Builds on

14 of 22 references.

References · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  9. 2019Verified Uncertainty Calibrationcited by 307
  10. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  11. 2020Residual Energy-Based Models for Text Generationcited by 95
  12. 2020ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translationcited by 34
2 more
  1. 2020Posterior Calibrated Training on Sentence Classification Taskscited by 5
  2. 2016On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimationcited by 3

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 5; 2023: 3; 2024: 2; 2025: 2.

14 papers.

Sort:

By area:

  1. 2021Oops I Took A Gradient: Scalable Sampling for Discrete Distributionscited by 92
  2. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  3. 2022On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliencycited by 31
  4. 2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29
  5. 2024Building Minimal and Reusable Causal State Abstractions for Reinforcement Learningcited by 11
  6. 2022Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimatescited by 7
  7. 2023On the Calibration of Multilingual Question Answering LLMscited by 6
  8. 2022Consistent Training via Energy-Based GFlowNets for Modeling Discrete Joint Distributionscited by 4
  9. 2021Sampling from Discrete Energy-Based Models with Quality/Efficiency Trade-offscited by 2
  10. 2025LSEBMCL: A Latent Space Energy-Based Model for Continual Learningcited by 2
  11. 2023Exploring Energy-based Language Models with Different Architectures and Training Methods for Speech Recognitioncited by 1
  12. 2022EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answeringcited by 0
2 more
  1. 2024QE-EBM: Using Quality Estimators as Energy Loss for Machine Translationcited by 0
  2. 2025Entriever: Energy-based Retriever for Knowledge-Grounded Dialog Systemscited by 0