mlsanityData to 7 Oct 2026

MetaRM: Shifted Distributions Alignment via Meta-Learning

arXiv 2405.00438 ↗First posted 1 May 2024cs.LG cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The success of Reinforcement Learning from Human Feedback (RLHF) in language model alignment is critically dependent on the capability of the reward model (RM). However, as the training process progresses, the output distribution of the policy model shifts, leading to the RM's reduced ability to distinguish between responses. This issue is further compounded when the RM, trained on a specific data distribution, struggles to generalize to examples outside of that distribution. These two issues can be united as a challenge posed by the shifted distribution of the environment. To surmount this challenge, we introduce MetaRM, a method leveraging meta-learning to align the RM with the shifted environment distribution. MetaRM is designed to train the RM by minimizing data loss, particularly for data that can improve the differentiation ability to examples of the shifted target distribution. Extensive experiments demonstrate that MetaRM significantly improves the RM's distinguishing ability in iterative RLHF optimization, and also provides the capacity to identify subtle differences in out-of-distribution samples.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 38 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

29 of 38 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  2. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models22 Jan
  3. 2023A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift21 Nov
  4. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learning18 Oct
  5. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  9. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  9. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  10. 2021MetaICL: Learning to Learn In Contextcited by 450
  11. 2021Meta-learning via Language Model In-context Tuningcited by 163
  12. 2018Learning to Learn from Noisy Labeled Datacited by 160
5 more
  1. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  2. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  3. 2019Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Taskscited by 48
  4. 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 42
  5. 2022Dynamics-Aware Comparison of Learned Reward Functionscited by 17

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replaycited by 19
  3. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  4. 2025The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  2. 2026Meta-Learning Preferences for Multilingual LLM Alignment
  3. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  4. 2026Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback
  5. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  6. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
  7. 2024Prior Constraints-based Reward Model Training for Aligning Large Language Models
  8. 2024Solving the Inverse Alignment Problem for Efficient RLHF