mlsanityData to 7 Oct 2026

Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond

arXiv 2310.06147 ↗First posted 9 Oct 2023cs.LG cs.AIcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have garnered wide attention and led to successful products such as ChatGPT and GPT-4. Their proficiency in adhering to instructions and delivering harmless, helpful, and honest (3H) responses can largely be attributed to the technique of Reinforcement Learning from Human Feedback (RLHF). In this paper, we aim to link the research in conventional RL to RL techniques used in LLM research. Demystify this technique by discussing why, when, and how RL excels. Furthermore, we explore potential future avenues that could either benefit from or contribute to RLHF research. Highlighted Takeaways: 1. RLHF is Online Inverse RL with Offline Demonstration Data. 2. RLHF > SFT because Imitation Learning (and Inverse RL) > Behavior Cloning (BC) by alleviating the problem of compounding error. 3. The RM step in RLHF generates a proxy of the expensive human feedback, such an insight can be generalized to other LLM tasks such as prompting evaluation and optimization where feedback is also expensive. 4. The policy learning in RLHF is more challenging than conventional problems studied in IRL due to their high action dimensionality and feedback sparsity. 5. The main superiority of PPO over off-policy value-based methods is its stability gained from (almost) on-policy data and conservative policy updates.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 37 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

26 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  4. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers10 Feb

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
10 more
  1. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  2. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  3. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  4. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  5. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  6. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  7. 2019Policy Continuation with Hindsight Inverse Dynamicscited by 22
  8. 2021Safe Exploration by Solving Early Terminated MDPcited by 13
  9. 2020Novel Policy Seeking with Constrained Optimizationcited by 11
  10. 2020Zeroth-Order Supervised Policy Improvementcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 8; 2026: 1.

20 papers.

Sort:

By area:

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  2. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  3. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  4. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  5. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  6. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  7. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  8. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  9. 2024Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHFcited by 18
  10. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  11. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  12. 2025Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexitycited by 9
8 more
  1. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  2. 2025MARCO: Multi-Agent Code Optimization with Real-Time Knowledge Integration for High-Performance Computingcited by 5
  3. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 4
  4. 2023When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspectivecited by 3
  5. 2024Using RL to Identify Divisive Perspectives Improves LLMs Abilities to Identify Communities on Social Mediacited by 1
  6. 2025Automated Meta Prompt Engineering for Alignment with the Theory of Mindcited by 0
  7. 2025Match Chat: Real Time Generative AI and Generative Computing for Tenniscited by 0
  8. 2026medR: Reward Engineering for Clinical Offline Reinforcement Learning via Tri-Drive Potential Functionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  2. 2023A Survey of Reinforcement Learning from Human Feedback
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  6. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  7. 2024Parameter Efficient Reinforcement Learning from Human Feedback
  8. 2024Reinforcement Learning for LLM Post-Training: A Survey