# Residual Energy-Based Models for Text

arXiv: https://arxiv.org/abs/2004.10188 · HTML: https://mlsanity.com/paper/2004.10188/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Apr 2020.

## Built on by (most cited first, 5 of 5)

- [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://mlsanity.com/paper/2206.00761/) (2022)
- [Energy-Based Models for Code Generation under Compilability Constraints](https://mlsanity.com/paper/2106.04985/) (2021)
- [Aligning language models with human preferences](https://mlsanity.com/paper/2404.12150/) (2024)
- [Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP](https://mlsanity.com/paper/2108.12522/) (2021)
- [Mast Kalandar at SemEval-2024 Task 8: On the Trail of Textual Origins: RoBERTa-BiLSTM Approach to Detect AI-Generated Text](https://mlsanity.com/paper/2407.02978/) (2024)

## Abstract (arXiv)

Current large-scale auto-regressive language models display impressive fluency and can generate convincing text. In this work we start by asking the question: Can the generations of these models be reliably distinguished from real text by statistical discriminators? We find experimentally that the answer is affirmative when we have access to the training data for the model, and guardedly affirmative even if we do not. This suggests that the auto-regressive models can be improved by incorporating the (globally normalized) discriminators into the generative process. We give a formalism for this using the Energy-Based Model framework, and show that it indeed improves the results of the generative models, measured both in terms of perplexity and in terms of human evaluation.