mlsanityData to 7 Oct 2026

RL with KL penalties is better viewed as Bayesian inference

arXiv 2205.11275 ↗First posted 23 May 2022cs.LG stat.MLcited by 130

Not on the map. Why?

Cited by 47 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning (RL) is frequently employed in fine-tuning large language models (LMs), such as GPT-3, to penalize them for undesirable features of generated sequences, such as offensiveness, social bias, harmfulness or falsehood. The RL formulation involves treating the LM as a policy and updating it to maximise the expected value of a reward function which captures human preferences, such as non-offensiveness. In this paper, we analyze challenges associated with treating a language model as an RL policy and show how avoiding those challenges requires moving beyond the RL paradigm. We start by observing that the standard RL approach is flawed as an objective for fine-tuning LMs because it leads to distribution collapse: turning the LM into a degenerate distribution. Then, we analyze KL-regularised RL, a widely used recipe for fine-tuning LMs, which additionally constrains the fine-tuned LM to stay close to its original distribution in terms of Kullback-Leibler (KL) divergence. We show that KL-regularised RL is equivalent to variational inference: approximating a Bayesian posterior which specifies how to update a prior LM to conform with evidence provided by the reward function. We argue that this Bayesian inference view of KL-regularised RL is more insightful than the typically employed RL perspective. The Bayesian inference view explains how KL-regularised RL avoids the distribution collapse problem and offers a first-principles derivation for its objective. While this objective happens to be equivalent to RL (with a particular choice of parametric reward), there exist other objectives for fine-tuning LMs which are no longer equivalent to RL. That observation leads to a more general point: RL is not an adequate formal framework for problems such as fine-tuning language models. These problems are best viewed as Bayesian inference: approximating a pre-defined target distribution.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

35 of 48 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2016Variational Inference: A Review for Statisticianscited by 1,466
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2022Red Teaming Language Models with Language Modelscited by 983
23 more
  1. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  2. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  3. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  4. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  5. 2020Artificial Intelligence, Values and Alignmentcited by 443
  6. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  7. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  10. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  11. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  12. 2020A Distributional Approach to Controlled Text Generationcited by 117
  13. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  14. 2017Deceiving Google's Perspective API Built for Detecting Toxic Commentscited by 99
  15. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  16. 2017The free energy principle for action and perception: A mathematical reviewcited by 66
  17. 2018Controllable Neural Story Plot Generation via Reward Shapingcited by 49
  18. 2016Adversarial Perturbations Against Deep Neural Networks for Malware Classificationcited by 42
  19. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  20. 2021Controlling Conditional Language Models without Catastrophic Forgettingcited by 32
  21. 2020Action and Perception as Divergence Minimizationcited by 31
  22. 2017Reinforced Video Captioning with Entailment Rewardscited by 21
  23. 2021Energy-Based Models for Code Generation under Compilability Constraintscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 17; 2024: 31; 2025: 40; 2026: 39.

130 papers.

Sort:

By area:

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  4. 2023Pretraining Language Models with Human Preferencescited by 245
  5. 2026Self-Distillation Enables Continual Learningcited by 212
  6. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  7. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  8. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  9. 2023Controlled Decoding from Language Modelscited by 133
  10. 2023Training Language Models with Language Feedback at Scalecited by 125
  11. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  12. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
38 more
  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  2. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  3. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  4. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 79
  5. 2024Decoding-time Realignment of Language Modelscited by 67
  6. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  7. 2023LoRA ensembles for large language model fine-tuningcited by 65
  8. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  9. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  10. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  11. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  12. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  13. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  14. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48
  15. 2024Asymptotics of Language Model Alignmentcited by 46
  16. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  17. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  18. 2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
  19. 2024InfAlign: Inference-aware language model alignmentcited by 34
  20. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  21. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  22. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  23. 2022Reward Gaming in Conditional Text Generationcited by 29
  24. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  25. 2024Instruction Fine-Tuning: Does Prompt Loss Matter?cited by 26
  26. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
  27. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  28. 2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFcited by 20
  29. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  30. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  31. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  32. 2025Outsourced diffusion sampling: Efficient posterior inference in latent spaces of generative modelscited by 16
  33. 2025TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generationcited by 15
  34. 2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
  35. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferencescited by 13
  36. 2023Conditioning Predictive Models: Risks and Strategiescited by 11
  37. 2025Reinforcement Fine-Tuning for Materials Designcited by 11
  38. 2023Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspectivecited by 10

50 most cited of 130 All 130 (JSON)