mlsanityData to 7 Oct 2026

Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

arXiv 2210.01241 ↗First posted 3 Oct 2022cs.CL cs.LGcited by 244

Not on the map. Why?

Cited by 112 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We tackle the problem of aligning pre-trained large language models (LMs) with human preferences. If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework. However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, as well as a lack of open-source libraries and benchmarks customized for LM alignment. Thus, a question rises in the research community: is RL a practical paradigm for NLP? To help answer this, we first introduce an open-source modular library, RL4LMs (Reinforcement Learning for Language Models), for optimizing language generators with RL. The library consists of on-policy RL algorithms that can be used to train any encoder or encoder-decoder LM in the HuggingFace library (Wolf et al. 2020) with an arbitrary reward function. Next, we present the GRUE (General Reinforced-language Understanding Evaluation) benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions which capture automated measures of human preference. GRUE is the first leaderboard-style evaluation of RL algorithms for NLP tasks. Finally, we introduce an easy-to-use, performant RL algorithm, NLPO (Natural Language Policy Optimization) that learns to effectively reduce the combinatorial action space in language generation. We show 1) that RL techniques are generally better than supervised methods at aligning LMs to human preferences; and 2) that NLPO exhibits greater stability and performance than previous policy gradient methods (e.g., PPO (Schulman et al. 2017)), based on both automatic and human evaluations.

Key facts. Cited by 244 papers in the map (data through 7 Oct 2026). Builds on 61 of its 86 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

61 of 86 references.

References · 61

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2015Trust Region Policy Optimizationcited by 2,503
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
38 more
  1. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2015Teaching Machines to Read and Comprehendcited by 1,039
  9. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  10. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  11. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  13. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  14. 2016Self-critical Sequence Training for Image Captioningcited by 544
  15. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  16. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  17. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  18. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  19. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  20. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  21. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  22. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  23. 2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 160
  24. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  25. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  26. 2018Learning to Write with Cooperative Discriminatorscited by 119
  27. 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
  28. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
  29. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  30. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  31. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  32. 2020Graph Constrained Reinforcement Learning for Natural Language Action Spacescited by 80
  33. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 80
  34. 2018Learn What Not to Learn: Action Elimination with Deep Reinforcement Learningcited by 77
  35. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 76
  36. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  37. 2020Text Generation by Learning from Demonstrationscited by 64
  38. 2015Learning to Search Better Than Your Teachercited by 50

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 80; 2024: 96; 2025: 45; 2026: 15.

244 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  11. 2023Augmented Language Models: a Surveycited by 398
  12. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
38 more
  1. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  2. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  3. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  7. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  8. 2023Fine-tuning Language Models for Factualitycited by 219
  9. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  10. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 211
  11. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 203
  12. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  13. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  14. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  15. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  16. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  17. 2023Mitigating the Alignment Tax of RLHFcited by 181
  18. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 165
  19. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  20. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  21. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  22. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  23. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  24. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  25. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  26. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  27. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  28. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  29. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  30. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
  31. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  32. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  33. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  34. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  35. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  36. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  37. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  38. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80

50 most cited of 244 All 244 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  2. 2024Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding
  3. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  4. 2024Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting
  5. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  6. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
  7. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment
  8. 2024Natural Language Reinforcement Learning