mlsanityData to 7 Oct 2026

Generalizing Reward Modeling for Out-of-Distribution Preference Learning

arXiv 2402.14760 ↗First posted 22 Feb 2024cs.LG cs.CLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference learning (PL) with large language models (LLMs) aims to align the LLMs' generations with human preferences. Previous work on reinforcement learning from human feedback (RLHF) has demonstrated promising results in in-distribution PL. However, due to the difficulty of obtaining human feedback, discretely training reward models for every encountered distribution is challenging. Thus, out-of-distribution (OOD) PL is practically useful for enhancing the generalization ability of LLMs with limited preference feedback. This work addresses OOD PL by optimizing a general reward model through a meta-learning approach. During meta-training, a bilevel optimization algorithm is utilized to learn a reward model capable of guiding policy learning to align with human preferences across various distributions. When encountering a test distribution, the meta-test procedure conducts regularized policy optimization using the learned reward model for PL. We theoretically demonstrate the convergence rate of the bilevel optimization algorithm under reasonable assumptions. Additionally, we conduct experiments on two text generation tasks across 20 held-out domains and outperform a variety of strong baselines across various evaluation metrics.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 37 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

31 of 37 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers10 Feb
  6. 2023On The Fragility of Learned Reward Functions9 Jan
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2015Trust Region Policy Optimizationcited by 2,503
  10. 2021Convex optimizationcited by 2,479
  11. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
10 more
  1. 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 762
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  3. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  4. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  6. 2018Approximation Methods for Bilevel Programmingcited by 231
  7. 2020Bilevel Optimization: Convergence Analysis and Enhanced Designcited by 219
  8. 2017A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applicationscited by 126
  9. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  10. 2019The Complexity of Finding Stationary Points with Stochastic Gradient Descentcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  2. 2026Maximum Likelihood Reinforcement Learningcited by 29
  3. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  4. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  5. 2026FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignmentcited by 1
  6. 2025UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
  2. 2024Fine-Tuning Language Models with Reward Learning on Policy
  3. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
  4. 2024ICPL: Few-shot In-context Preference Learning via LLMs
  5. 2023Nash Learning from Human Feedback
  6. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization