mlsanityData to 7 Oct 2026

Residual Energy-Based Models for Text Generation

arXiv 2004.11714 ↗First posted 22 Apr 2020cs.CL cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation. The dominant parametric approach is based on locally normalized models which predict one word at a time. While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process. In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level. In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa. Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 20; 2023: 20; 2024: 14; 2025: 14; 2026: 7.

95 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Implicit Behavioral Cloningcited by 442
  3. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401
  4. 2021How to Train Your Energy-Based Modelscited by 271
  5. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  6. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  7. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  8. 2021TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generationcited by 118
  9. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  10. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  11. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  12. 2021Oops I Took A Gradient: Scalable Sampling for Discrete Distributionscited by 92
38 more
  1. 2023Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglementcited by 89
  2. 2024Energy-Based Diffusion Language Models for Text Generationcited by 86
  3. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  4. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  5. 2021Diffusion Normalizing Flowcited by 77
  6. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  7. 2022HYPRO: A Hybridly Normalized Probabilistic Model for Long-Horizon Prediction of Event Sequencescited by 57
  8. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 52
  9. 2021ScoreGrad: Multivariate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Modelscited by 42
  10. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  11. 2021Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastingscited by 37
  12. 2024Energy-Based Concept Bottleneck Models: Unifying Prediction, Concept Intervention, and Probabilistic Interpretationscited by 33
  13. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  14. 2023Probabilistic Adaptation of Text-to-Video Modelscited by 29
  15. 2022Composable Text Controls in Latent Space with ODEscited by 28
  16. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  17. 2024Non-myopic Generation of Language Models for Reasoning and Planningcited by 25
  18. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  19. 2023No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Functioncited by 19
  20. 2023Guiding Energy-based Models via Contrastive Latent Variablescited by 17
  21. 2021Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Modelscited by 14
  22. 2021To Beam Or Not To Beam: That is a Question of Cooperation for Language GANscited by 14
  23. 2024BBox-Adapter: Lightweight Adapting for Black-Box Large Language Modelscited by 11
  24. 2021Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learningcited by 10
  25. 2021Energy-Based Models for Code Generation under Compilability Constraintscited by 10
  26. 2025On-the-fly Preference Alignment via Principle-Guided Decodingcited by 10
  27. 2021Residual Energy-Based Models for End-to-End Speech Recognitioncited by 9
  28. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
  29. 2021MRI Reconstruction Using Deep Energy-Based Modelcited by 8
  30. 2022Generative Cooperative Networks for Natural Language Generationcited by 8
  31. 2023Language Model Decoding as Direct Metrics Optimizationcited by 8
  32. 2025Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flowscited by 8
  33. 2023Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyondcited by 7
  34. 2025Inference-time Alignment in Continuous Spacecited by 7
  35. 2022Gradient-Guided Importance Sampling for Learning Binary Energy-Based Modelscited by 6
  36. 2024Preference Optimization for Molecule Synthesis with Conditional Residual Energy-based Modelscited by 6
  37. 2024Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelinescited by 6
  38. 2022Better Language Model with Hypernym Class Predictioncited by 5

50 most cited of 95 All 95 (JSON)