mlsanityData to 7 Oct 2026

Aligning LLMs with Domain Invariant Reward Models

arXiv 2501.00911 ↗First posted 1 Jan 2025cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Aligning large language models (LLMs) to human preferences is challenging in domains where preference data is unavailable. We address the problem of learning reward models for such target domains by leveraging feedback collected from simpler source domains, where human preferences are easier to obtain. Our key insight is that, while domains may differ significantly, human preferences convey domain-agnostic concepts that can be effectively captured by a reward model. We propose method, a framework that trains domain-invariant reward models by optimizing a dual loss: a domain loss that minimizes the divergence between source and target distribution, and a source loss that optimizes preferences on the source domain. We show method is a general approach that we evaluate and analyze across 4 distinct settings: (1) Cross-lingual transfer (accuracy: 0.621 rightarrow 0.661), (2) Clean-to-noisy (accuracy: 0.671 rightarrow 0.703), (3) Few-shot-to-full transfer (accuracy: 0.845 rightarrow 0.920), and (4) Simple-to-complex tasks transfer (correlation: 0.508 rightarrow 0.556). Our code, models and data are available at urlhttps://github.com/portal-cornell/dial.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 52 references that are in the map. First posted on arXiv 1 Jan 2025.

Builds on

16 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  2. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  3. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  4. 2018Scalable agent alignment via reward modeling: a research direction19 Nov

Other references · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2017Improved Training of Wasserstein GANscited by 2,223
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  7. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  8. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  9. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 80
  10. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  11. 2021UDALM: Unsupervised Domain Adaptation through Language Modelingcited by 33
  12. 2017Wasserstein Distance Guided Representation Learning for Domain Adaptationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Transforming and Combining Rewards for Aligning Large Language Models
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  4. 2024Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes
  5. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  6. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
  7. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
  8. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models