mlsanityData to 14 Oct 2026

Mutual-Taught for Co-adapting Policy and Reward Models

arXiv 2506.06292 ↗First posted 17 May 2025cs.LG cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

During the preference optimization of large language models (LLMs), distribution shifts may arise between newly generated model samples and the data used to train the reward model (RM). This shift reduces the efficacy of the RM, which in turn negatively impacts the performance of the policy model (PM). To address this challenge, we propose Mutual-Taught, a self-training method that iteratively improves both the PM and RM without requiring additional human annotation. Our approach mirrors the expectation-maximization (EM) algorithm. In the E-step, the PM is updated using feedback from the current RM, guiding the PM toward a better approximation of the latent optimal preference distribution. In the M-step, we update the RM by constructing training data from the outputs of the PM before and after the E-step update. This process ensures that the RM adapts to the evolving policy distribution. Experimental results demonstrate that this iterative approach leads to consistent improvements in both models. Specifically, our 8B policy model, LLaMA-3-8B-Instruct-MT, achieves a length-controlled win rate of 54.1% on AlpacaEval-2, while our 8B reward model, FsfairX-LLaMA3-RM-MT, performs on par with GPT-4o-2024-08-06 on RewardBench.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 37 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

26 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  2. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  3. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models22 Jan
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  5. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learning18 Oct
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct

Other references · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
7 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  3. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  4. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  5. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  6. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  7. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Towards Agentic Self-Learning LLMs in Search Environmentcited by 8
  2. 2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoningcited by 5
  3. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  2. 2024Fine-Tuning Language Models with Reward Learning on Policy
  3. 2025Bootstrapping LLMs via Preference-Based Policy Optimization
  4. 2026The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
  5. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  6. 2024Aligning Large Language Models by On-Policy Self-Judgment
  7. 2024Value Augmented Sampling for Language Model Alignment and Personalization
  8. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillation