Data to 7 Oct 2026

Reinforcement Learning Enhanced LLMs: A Survey

arXiv 2412.10400 ↗First posted 5 Dec 2024cs.CL cs.AI cs.LGcited by 69

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning (RL) enhanced large language models (LLMs), particularly exemplified by DeepSeek-R1, have exhibited outstanding performance. Despite the effectiveness in improving LLM capabilities, its implementation remains highly complex, requiring complex algorithms, reward modeling strategies, and optimization techniques. This complexity poses challenges for researchers and practitioners in developing a systematic understanding of RL-enhanced LLMs. Moreover, the absence of a comprehensive survey summarizing existing research on RL-enhanced LLMs has limited progress in this domain, hindering further advancements. In this work, we are going to make a systematic review of the most up-to-date state of knowledge on RL-enhanced LLMs, attempting to consolidate and analyze the rapidly growing research in this field, helping researchers understand the current challenges and advancements. Specifically, we (1) detail the basics of RL; (2) introduce popular RL-enhanced LLMs; (3) review researches on two widely-used reward model-based RL techniques: Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF); and (4) explore Direct Preference Optimization (DPO), a set of methods that bypass the reward model to directly use human preference data for aligning LLM outputs with human expectations. We will also point out current challenges and deficiencies of existing methods and suggest some avenues for further improvements. Project page of this work can be found at https://github.com/ShuheWang1998/Reinforcement-Learning-Enhanced-LLMs-A-Survey.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 78 of its 110 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

78 of 110 references.

Closest ancestors · 7

Same direction, newest first

Other references · 71

2020Language Models are Few-Shot Learnerscited by 32,199
2022Training language models to follow instructions with human feedbackcited by 15,712
2024The Llama 3 Herd of Modelscited by 14,192
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2020Measuring Massive Multitask Language Understandingcited by 7,768
2020Longformer: The Long-Document Transformercited by 3,362
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
2015Trust Region Policy Optimizationcited by 2,503

38 more

2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
2016Concrete Problems in AI Safetycited by 1,582
2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
2023LIMA: Less Is More for Alignmentcited by 1,080
2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
2024Self-Rewarding Language Modelscited by 654
2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
2023Instruction Tuning for Large Language Models: A Surveycited by 617
2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
2023Large Language Model Guided Tree-of-Thoughtcited by 218
2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 153
2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 47; 2026: 22.

69 papers.

Sort:

By area:

2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
2025CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learningcited by 59
2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 30
2025ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineeringcited by 26
2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 19
2025SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimizationcited by 18
2025Adaptive Graph of Thoughts: Test-Time Adaptive Reasoning Unifying Chain, Tree, and Graph Structurescited by 16
2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16

38 more

2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Callscited by 16
2025GTM: Simulating the World of Tools for AI Agentscited by 12
2025Towards Watermarking of Open-Source LLMscited by 11
2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
2025MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Clustercited by 9
2025Can GRPO Boost Complex Multimodal Table Understanding?cited by 9
2025Three Minds, One Legend: Jailbreak Large Reasoning Model with Adaptive Stacked Cipherscited by 8
2025Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMscited by 8
2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
2025Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Modelscited by 7
2025Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuningcited by 7
2025Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Callingcited by 7
2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
2025DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Surveycited by 6
2025IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewardscited by 6
2026Paying Less Generalization Tax: A Cross-Domain Generalization Study of RL Training for LLM Agentscited by 6
2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solvingcited by 6
2026On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agentscited by 6
2025From Emergence to Control: Probing and Modulating Self-Reflection in Language Modelscited by 5
2025TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicinecited by 4
2025Reducing Belief Deviation in Reinforcement Learning for Active Reasoningcited by 4
2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
2025Retrieval-GRPO: A Multi-Objective Reinforcement Learning Framework for Dense Retrieval in Taobao Searchcited by 3
2026From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agentscited by 3
2026ReRec: Reasoning-Augmented LLM-based Recommendation Assistant via Reinforcement Fine-tuningcited by 3
2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
2026Think-to-Personalize: Unifying Reasoning and Retrieval for User-Centric Personalized Dense Retrievalcited by 2
2025Contemplative Artificial Intelligencecited by 1
2025Retrieval Augmented Learning: A Retrial-based Large Language Model Self-Supervised Learning and Autonomous Knowledge Generationcited by 1
2025Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performancecited by 1
2025Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generationcited by 1
2025Quantum Bayesian Networks Can Speed up Reinforcement Learning in Partially Observable Environmentscited by 1
2025AR2: Adversarial Reinforcement Learning for Abstract Reasoning in Large Language Modelscited by 1
2025Human-Allied Relational Reinforcement Learningcited by 1
2026GIFT: Games as Informal Training for Generalizable LLMscited by 1
2026ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluationcited by 1
2026Safety Training May Persist Through Helpfulness Optimization in LLM Agentscited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way