# Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models

arXiv: https://arxiv.org/abs/2101.06829 · HTML: https://mlsanity.com/paper/2101.06829/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 14 of 14)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)
- [Residual Energy-Based Models for Text Generation](https://mlsanity.com/paper/2004.11714/) (2020)
- [ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2005.00850/) (2020)
- [Posterior Calibrated Training on Sentence Classification Tasks](https://mlsanity.com/paper/2004.14500/) (2020)
- [On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation](https://mlsanity.com/paper/1602.06064/) (2016)

## Built on by (most cited first, 14 of 14)

- [Oops I Took A Gradient: Scalable Sampling for Discrete Distributions](https://mlsanity.com/paper/2102.04509/) (2021)
- [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://mlsanity.com/paper/2206.00761/) (2022)
- [On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency](https://mlsanity.com/paper/2203.07559/) (2022)
- [Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Models](https://mlsanity.com/paper/2305.19249/) (2023)
- [Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning](https://mlsanity.com/paper/2401.12497/) (2024)
- [Calibration Meets Explanation: A Simple and Effective Approach for Model Confidence Estimates](https://mlsanity.com/paper/2211.03041/) (2022)
- [On the Calibration of Multilingual Question Answering LLMs](https://mlsanity.com/paper/2311.08669/) (2023)
- [Consistent Training via Energy-Based GFlowNets for Modeling Discrete Joint Distributions](https://mlsanity.com/paper/2211.00568/) (2022)
- [Sampling from Discrete Energy-Based Models with Quality/Efficiency Trade-offs](https://mlsanity.com/paper/2112.05702/) (2021)
- [LSEBMCL: A Latent Space Energy-Based Model for Continual Learning](https://mlsanity.com/paper/2501.05495/) (2025)
- [Exploring Energy-based Language Models with Different Architectures and Training Methods for Speech Recognition](https://mlsanity.com/paper/2305.12676/) (2023)
- [EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering](https://mlsanity.com/paper/2206.14355/) (2022)
- [QE-EBM: Using Quality Estimators as Energy Loss for Machine Translation](https://mlsanity.com/paper/2410.10228/) (2024)
- [Entriever: Energy-based Retriever for Knowledge-Grounded Dialog Systems](https://mlsanity.com/paper/2506.00585/) (2025)

## Abstract (arXiv)

In this work, we explore joint energy-based model (EBM) training during the finetuning of pretrained text encoders (e.g., Roberta) for natural language understanding (NLU) tasks. Our experiments show that EBM training can help the model reach a better calibration that is competitive to strong baselines, with little or no loss in accuracy. We discuss three variants of energy functions (namely scalar, hidden, and sharp-hidden) that can be defined on top of a text encoder, and compare them in experiments. Due to the discreteness of text data, we adopt noise contrastive estimation (NCE) to train the energy-based model. To make NCE training more effective, we train an auto-regressive noise model with the masked language model (MLM) objective.