mlsanityData to 7 Oct 2026

Learning to Generate Better Than Your LLM

arXiv 2306.11816 ↗First posted 20 Jun 2023cs.LG cs.AI cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) has emerged as a powerful paradigm for fine-tuning Large Language Models (LLMs) for text generation. In particular, recent LLMs such as ChatGPT and GPT-4 can engage in fluent conversations with users after finetuning with RL. Capitalizing on key properties of text generation, we seek to investigate RL algorithms beyond general purpose algorithms like Proximal Policy Optimization (PPO). In particular, we extend RL algorithms to allow them to interact with a dynamic black-box guide LLM and propose RL with guided feedback (RLGF), a suite of RL algorithms for LLM fine-tuning. We provide two ways for the guide LLM to interact with the LLM to be optimized for maximizing rewards. The guide LLM can generate text which serves as additional starting states for the RL optimization procedure. The guide LLM can also be used to complete the partial sentences generated by the LLM that is being optimized, treating the guide LLM as an expert to imitate and surpass eventually. We experiment on the IMDB positive sentiment, CommonGen, and TL;DR summarization tasks. We show that our RL algorithms achieve higher performance than supervised learning (SL) and the RL baseline PPO, demonstrating the benefit of interaction with the guide LLM. On both CommonGen and TL;DR, we not only outperform our SL baselines but also improve upon PPO across a variety of metrics beyond the one we optimized for. Our code can be found at https://github.com/Cornell-RL/tril.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 69 of its 107 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

69 of 107 references.

References · 69

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2015Trust Region Policy Optimizationcited by 2,503
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
38 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  5. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  7. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  10. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  12. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  13. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  14. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  15. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  16. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  17. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  18. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  19. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  20. 2022Defining and Characterizing Reward Hackingcited by 184
  21. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  22. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 170
  23. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  24. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  25. 2023Training Language Models with Language Feedback at Scalecited by 125
  26. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  27. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  28. 2020A Distributional Approach to Controlled Text Generationcited by 117
  29. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  30. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  31. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 112
  32. 2017Sentence Simplification with Deep Reinforcement Learningcited by 107
  33. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  34. 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
  35. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  36. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  37. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  38. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 27; 2025: 16; 2026: 1.

49 papers.

Sort:

By area:

  1. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  2. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  3. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  4. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  5. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  6. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  7. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  8. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  9. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  10. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  11. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  12. 2024Dataset Reset Policy Optimization for RLHFcited by 41
37 more
  1. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  2. 2023Interactive Robot Learning from Verbal Correctioncited by 28
  3. 2024Watermarking Techniques for Large Language Models: A Surveycited by 20
  4. 2025A Theory of Learning with Autoregressive Chain of Thoughtcited by 20
  5. 2025Towards AI Search Paradigmcited by 19
  6. 2023Why Can Large Language Models Generate Correct Chain-of-Thoughts?cited by 16
  7. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 16
  8. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  9. 2024Q-Probe: A Lightweight Approach to Reward Maximization for Language Modelscited by 15
  10. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  11. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  12. 2024A Survey on Human-Centric LLMscited by 13
  13. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 12
  14. 2024Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarioscited by 10
  15. 2024SWAG: Storytelling With Action Guidancecited by 8
  16. 2024Adaptive Draft-Verification for Efficient Large Language Model Decodingcited by 8
  17. 2025HalluShift: Measuring Distribution Shifts towards Hallucination Detection in LLMscited by 8
  18. 2024RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanationcited by 7
  19. 2024Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teamingcited by 6
  20. 2025Survey: Multi-Armed Bandits Meet Large Language Modelscited by 6
  21. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  22. 2025DAMR: Efficient and Adaptive Context-Aware Knowledge Graph Question Answering with LLM-Guided MCTScited by 6
  23. 2024Learning to Better Search with Language Models via Guided Reinforced Self-Trainingcited by 5
  24. 2025Enhancing PPO with Trajectory-Aware Hybrid Policiescited by 4
  25. 2025Efficient Imitation under Misspecificationcited by 4
  26. 2025EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensemblescited by 4
  27. 2025Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMscited by 4
  28. 2024The Real, the Better: Aligning Large Language Models with Online Human Behaviorscited by 3
  29. 2024Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learningcited by 3
  30. 2024Large Language Model as a Catalyst: A Paradigm Shift in Base Station Siting Optimizationcited by 3
  31. 2024An Empirical Study on Self-correcting Large Language Models for Data Science Code Generationcited by 2
  32. 2024How Privacy-Savvy Are Large Language Models? A Case Study on Compliance and Privacy Technical Reviewcited by 2
  33. 2025EMA Without the Lag: Bias-Corrected Iterate Averaging Schemescited by 2
  34. 2024GuideLight: "Industrial Solution" Guidance for More Practical Traffic Signal Control Agentscited by 1
  35. 2024EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensemblescited by 1
  36. 2024Integrating Large Language Models and Reinforcement Learning for Non-Linear Reasoningcited by 1
  37. 2025RLAP: A Reinforcement Learning Enhanced Adaptive Planning Framework for Multi-step NLP Task Solvingcited by 0