mlsanityData to 7 Oct 2026

On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting

arXiv 2206.00761 ↗First posted 1 Jun 2022cs.LG cs.CL stat.MLcited by 80

Not on the map. Why?

Cited by 39 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The availability of large pre-trained models is changing the landscape of Machine Learning research and practice, moving from a training-from-scratch to a fine-tuning paradigm. While in some applications the goal is to "nudge" the pre-trained distribution towards preferred outputs, in others it is to steer it towards a different distribution over the sample space. Two main paradigms have emerged to tackle this challenge: Reward Maximization (RM) and, more recently, Distribution Matching (DM). RM applies standard Reinforcement Learning (RL) techniques, such as Policy Gradients, to gradually increase the reward signal. DM prescribes to first make explicit the target distribution that the model is fine-tuned to approximate. Here we explore the theoretical connections between the two paradigms, and show that methods such as KL-control developed for RM can also be construed as belonging to DM. We further observe that while DM differs from RM, it can suffer from similar training difficulties, such as high gradient variance. We leverage connections between the two paradigms to import the concept of baseline into DM methods. We empirically validate the benefits of adding a baseline on an array of controllable language generation tasks such as constraining topic, sentiment, and gender distributions in texts sampled from a language model. We observe superior performance in terms of constraint satisfaction, stability and sample efficiency.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Builds on 59 of its 110 references that are in the map. First posted on arXiv 1 Jun 2022.

Builds on

59 of 110 references.

References · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
38 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  3. 2022Red Teaming Language Models with Language Modelscited by 983
  4. 2021Ethical and social risks of harm from Language Modelscited by 957
  5. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  6. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  10. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  11. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  13. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  14. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  15. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  16. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  17. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  18. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  19. 2019Efficient Exploration via State Marginal Matchingcited by 186
  20. 2021Challenges in Detoxifying Language Modelscited by 169
  21. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  22. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  23. 2020A Distributional Approach to Controlled Text Generationcited by 117
  24. 2014Neural Variational Inference and Learning in Belief Networkscited by 116
  25. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  26. 2018Language GANs Falling Shortcited by 97
  27. 2020Residual Energy-Based Models for Text Generationcited by 95
  28. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 84
  29. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  30. 2017The free energy principle for action and perception: A mathematical reviewcited by 66
  31. 2015Structured Prediction Energy Networkscited by 59
  32. 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
  33. 2018Controllable Neural Story Plot Generation via Reward Shapingcited by 49
  34. 2020Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Modelscited by 44
  35. 2020ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translationcited by 34
  36. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 32
  37. 2021Controlling Conditional Language Models without Catastrophic Forgettingcited by 32
  38. 2020Action and Perception as Divergence Minimizationcited by 31

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 25; 2025: 27; 2026: 16.

80 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023Pretraining Language Models with Human Preferencescited by 245
  4. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  5. 2023Class-Incremental Learning: A Surveycited by 209
  6. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  7. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  8. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  9. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  10. 2023Auditing large language models: a three-layered approachcited by 135
  11. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  12. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
38 more
  1. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  2. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
  3. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 88
  4. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  5. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  6. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  7. 2023Learning to Generate Better Than Your LLMcited by 49
  8. 2024Asymptotics of Language Model Alignmentcited by 46
  9. 2024A Survey of Mambacited by 39
  10. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  11. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 37
  12. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  13. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  14. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  15. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 29
  16. 2025LADDER: Self-Improving LLMs Through Recursive Problem Decompositioncited by 24
  17. 2023A density estimation perspective on learning from pairwise human preferencescited by 23
  18. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMscited by 21
  19. 2025R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learningcited by 21
  20. 2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFcited by 20
  21. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  22. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  23. 2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
  24. 2024Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibrationcited by 13
  25. 2024Reward Learning From Preference With Tiescited by 13
  26. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  27. 2024RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistributioncited by 10
  28. 2025Frictional Agent Alignment Framework: Slow Down and Don't Break Thingscited by 10
  29. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9
  30. 2024BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedbackcited by 7
  31. 2024Guaranteed Generation from Large Language Modelscited by 7
  32. 2026DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignmentcited by 7
  33. 2024"Yes, My LoRD." Guiding Language Model Extraction with Locality Reinforced Distillationcited by 5
  34. 2025Aligning Compound AI Systems via System-level DPOcited by 4
  35. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modelingcited by 4
  36. 2025Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Alignercited by 4
  37. 2023disco: a toolkit for Distributional Control of Generative Modelscited by 3
  38. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3

50 most cited of 80 All 80 (JSON)