mlsanityData to 14 Oct 2026

Everyone Deserves A Reward: Learning Customized Human Preferences

arXiv 2309.03126 ↗First posted 6 Sep 2023cs.CLcited by 43

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) are essential for aligning large language models (LLMs) with human preferences to improve interaction quality. However, the real world is pluralistic, which leads to diversified human preferences with respect to different religions, politics, cultures, etc. Moreover, each individual can have their unique preferences on various topics. Neglecting the diversity of human preferences, current human feedback aligning methods only consider a general reward model, which is below satisfaction for customized or personalized application scenarios. To explore customized preference learning, we collect a domain-specific preference (DSP) dataset, which includes preferred responses for each given query from four practical domains. Besides, from the perspective of data efficiency, we propose a three-stage customized RM learning scheme, then empirically verify its effectiveness on both general preference datasets and our DSP set. Furthermore, we test multiple training and data strategies on the three learning stages. We find several ways to better preserve the general preferring ability while training the customized RMs, especially general preference enrichment, and customized preference imitation learning. The DSP dataset and code are available at https://github.com/Linear95/DSP.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 33 references that are in the map. First posted on arXiv 6 Sep 2023.

Builds on

24 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  3. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
7 more
  1. 2023Instruction Tuning with GPT-4cited by 686
  2. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  3. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  4. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 77
  5. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 72
  6. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  7. 2023An Empirical Study on Information Extraction using Large Language Modelscited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 18; 2025: 15; 2026: 6.

43 papers.

Sort:

By area:

  1. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  2. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  3. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  4. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  5. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  6. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 74
  7. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  8. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistantscited by 48
  9. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  10. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 35
  11. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  12. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
31 more
  1. 2023Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answeringcited by 29
  2. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27
  3. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 27
  4. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  5. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
  6. 2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25
  7. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  8. 2024On the Way to LLM Personalization: Learning to Remember User Conversationscited by 21
  9. 2024SoFA: Shielded On-the-fly Alignment via Priority Rule Followingcited by 18
  10. 2025Instant Personalized Large Language Model Adaptation via Hypernetworkcited by 18
  11. 2024Contextualized Evaluations: Judging Language Model Responses to Underspecified Queriescited by 12
  12. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  13. 2025On-the-fly Preference Alignment via Principle-Guided Decodingcited by 10
  14. 2025Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Modelscited by 10
  15. 2023The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Modelscited by 7
  16. 2024Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigationcited by 7
  17. 2024Atoxia: Red-teaming Large Language Models with Target Toxic Answerscited by 7
  18. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalizationcited by 6
  19. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 6
  20. 2024Prior Constraints-based Reward Model Training for Aligning Large Language Modelscited by 4
  21. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignmentcited by 4
  22. 2025Semi-Supervised Preference Optimization with Limited Feedbackcited by 3
  23. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
  24. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  25. 2025UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimalitycited by 2
  26. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  27. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
  28. 2026CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Taskscited by 1
  29. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferencescited by 0
  30. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  31. 2026ConvoDrift: A Multi-Turn Conversational Dataset for Modeling Stylistic Tone Evolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  2. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
  4. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
  5. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
  6. 2024A Survey on Human Preference Learning for Large Language Models
  7. 2025Aligning LLMs with Domain Invariant Reward Models
  8. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment