Data to 7 Oct 2026

Multi-turn Reinforcement Learning from Preference Human Feedback

arXiv 2405.14655 ↗First posted 23 May 2024cs.LGcited by 65

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has become the standard approach for aligning Large Language Models (LLMs) with human preferences, allowing LLMs to demonstrate remarkable abilities in various tasks. Existing methods work by emulating the preferences at the single decision (turn) level, limiting their capabilities in settings that require planning or multi-turn interactions to achieve a long-term goal. In this paper, we address this issue by developing novel methods for Reinforcement Learning (RL) from preference feedback between two full multi-turn conversations. In the tabular setting, we present a novel mirror-descent-based policy optimization algorithm for the general multi-turn preference-based RL problem, and prove its convergence to Nash equilibrium. To evaluate performance, we create a new environment, Education Dialogue, where a teacher agent guides a student in learning a random topic, and show that a deep RL variant of our algorithm outperforms RLHF baselines. Finally, we show that in an environment with explicit rewards, our algorithm recovers the same performance as a reward-based RL baseline, despite relying solely on a weaker preference signal.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 53 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

38 of 53 references.

Closest ancestors · 6

Same direction, newest first

2023Nash Learning from Human Feedback1 Dec
2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
2023Is RLHF More Difficult than Standard RL?25 Jun
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
2019Fine-Tuning Language Models from Human Preferences18 Sep
2017Deep reinforcement learning from human preferences12 Jun

Other references · 32

2020Language Models are Few-Shot Learnerscited by 32,199
2022Training language models to follow instructions with human feedbackcited by 15,712
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2017Proximal Policy Optimization Algorithmscited by 14,441
2022Scaling Instruction-Finetuned Language Modelscited by 2,997
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2020Learning to summarize from human feedbackcited by 2,562
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
2016Deep Reinforcement Learning for Dialogue Generationcited by 286

20 more

2023Preference Ranking Optimization for Human Alignmentcited by 283
2019A Theory of Regularized Markov Decision Processescited by 279
2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
2020Mirror Descent Policy Optimizationcited by 102
2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
2019Online Convex Optimization in Adversarial Markov Decision Processescited by 79
2020Optimistic Policy Optimization with Bandit Feedbackcited by 79
2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
2020Reinforcement Learning with Trajectory Feedbackcited by 40
2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
2021Online Markov Decision Processes with Aggregate Bandit Feedbackcited by 10
2024Near-Optimal Regret in Linear MDPs with Aggregate Bandit Feedbackcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 42; 2026: 15.

65 papers.

Sort:

By area:

2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
2025Self-rewarding correction for mathematical reasoningcited by 67
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
2025Consistently Simulating Human Personas with Multi-Turn Reinforcement Learningcited by 35
2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29

38 more

2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 26
2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Rewardcited by 25
2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
2025Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foragingcited by 20
2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learningcited by 16
2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
2025Don't lie to your friends: Learning what you know from collaborative self-playcited by 14
2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
2025M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Qualitycited by 11
2025POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimizationcited by 10
2024Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversitycited by 9
2025Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voicecited by 9
2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
2025Learning from Failures in Multi-Attempt Reinforcement Learningcited by 6
2025Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomescited by 6
2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 6
2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspectivecited by 5
2025Discerning minds or generic tutors? Evaluating instructional guidance capabilities in Socratic LLMscited by 5
2025DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulationcited by 5
2024Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learningcited by 4
2025Best Policy Learning from Trajectory Preference Feedbackcited by 3
2025Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guaranteescited by 3
2025ShiQ: Bringing back Bellman to LLMscited by 3
2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 3
2025ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuningcited by 3
2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
2025Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logscited by 3
2025Personalized Education with Generative AI and Digital Twins: VR, RAG, and Zero-Shot Sentiment Analysis for Industry 4.0 Workforce Developmentcited by 2
2025Opponent Shaping in LLM Agentscited by 2
2025Learning "Partner-Aware" Collaborators in Multi-Party Collaborationcited by 2
2025Iterative GRPO: Batch-Online Multi-Turn RL via Single-Turn RLHFcited by 2
2024PRISM: A Personalized, Rapid, and Immersive Skill Mastery framework for personalizing experiential learning through Generative AIcited by 1
2026LLM-MC-Affect: LLM-Based Monte Carlo Modeling of Affective Trajectories and Latent Ambiguity for Interpersonal Dynamic Insightcited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way