mlsanityData to 7 Oct 2026

An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift

arXiv 2601.05882 ↗First posted 9 Jan 2026cs.CL cs.AI cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference tuning aligns base language models to human judgments of quality, helpfulness, or safety by optimizing over explicit preference signals rather than likelihood alone. Prior work has shown that preference tuning degrades performance and reduces helpfulness outside the training domain. However, the extent to which adaptation strategies mitigate this domain shift remains unexplored. We address this challenge by conducting a comprehensive and systematic study of alignment generalization under domain shift. We compare five popular alignment objectives and various adaptation strategies from source to target, including target-domain supervised fine-tuning and pseudo-labeling, across summarization, question-answering helpfulness, and safety alignment tasks. Our findings reveal systematic differences in generalization across alignment objectives under domain shift. We show that adaptation strategies based on pseudo-labeling substantially reduce domain-shift degradation but induce mode collapse, revealing a generalization-diversity trade-off.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 60 of its 82 references that are in the map. First posted on arXiv 9 Jan 2026.

Builds on

60 of 82 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Rethinking DPO: The Role of Rejected Responses in Preference Misalignment15 Jun
  2. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models8 Jun
  3. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?29 May
  4. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO29 May
  5. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset4 Apr
  6. 2025Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF28 Mar
  7. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization24 Feb
  8. 2025Diverse Preference Optimization30 Jan
  9. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model24 Oct
  10. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignment25 Jun
  11. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions23 May
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
6 more
  1. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization31 Mar
  2. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  3. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  4. 2023Mitigating the Alignment Tax of RLHF12 Sep
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 42

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
30 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  4. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  6. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  7. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  8. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  9. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  10. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  11. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 114
  12. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 100
  13. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  14. 2022Semantic Diversity in Dialogue with Natural Language Inferencecited by 28
  15. 2022Rethinking and Refining the Distinct Metriccited by 27
  16. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  17. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
  18. 2025Creative Preference Optimizationcited by 24
  19. 2025Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Modelscited by 18
  20. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  21. 2025Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Datacited by 16
  22. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 13
  23. 2024AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarizationcited by 9
  24. 2024Adapting Chat Language Models Using Only Target Unlabeled Language Datacited by 6
  25. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  26. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  27. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 5
  28. 2025Distribution Shift Alignment Helps LLMs Simulate Survey Response Distributionscited by 5
  29. 2025Fine-Tuning on Noisy Instructions: Effects on Generalization and Performancecited by 2
  30. 2025Can one size fit all?: Measuring Failure in Multi-Document Summarization Domain Transfercited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Where does output diversity collapse in post-training?cited by 10
  2. 2026MIRROR: From Imitation to Internalization in LLM Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  2. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
  3. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  4. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  5. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  6. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning
  7. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  8. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs