mlsanityData to 7 Oct 2026

Improving Context-Aware Preference Modeling for Language Models

arXiv 2407.14916 ↗First posted 20 Jul 2024cs.CL cs.AI cs.LGcited by 22

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

While finetuning language models from pairwise preferences has proven remarkably effective, the underspecified nature of natural language presents critical challenges. Direct preference feedback is uninterpretable, difficult to provide where multidimensional criteria may apply, and often inconsistent, either because it is based on incomplete instructions or provided by diverse principals. To address these challenges, we consider the two-step preference modeling procedure that first resolves the under-specification by selecting a context, and then evaluates preference with respect to the chosen context. We decompose reward modeling error according to these two steps, which suggests that supervising context in addition to context-specific preference may be a viable approach to aligning models with diverse human preferences. For this to work, the ability of models to evaluate context-specific preference is critical. To this end, we contribute context-conditioned preference datasets and accompanying experiments that investigate the ability of language models to evaluate context-specific preference. We use our datasets to (1) show that existing preference models benefit from, but fail to fully consider, added context, (2) finetune a context-aware reward model with context-specific performance exceeding that of GPT-4 and Llama 3 70B on tested datasets, and (3) investigate the value of context-aware preference modeling.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 86 references that are in the map. First posted on arXiv 20 Jul 2024.

Builds on

42 of 86 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Aligning to Thousands of Preferences via System Message Generalization28 May
  2. 2024A Roadmap to Pluralistic Alignment7 Feb
  3. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications24 Oct
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Models12 Oct
  6. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  7. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
  8. 2020Learning to summarize from human feedback2 Sep

Other references · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Mistral 7Bcited by 3,065
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
22 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  4. 2018AI safety via debatecited by 384
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  6. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  7. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  8. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  9. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  10. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  11. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  12. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  13. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
  14. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  15. 2023Context-faithful Prompting for Large Language Modelscited by 76
  16. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  17. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  18. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  19. 2024Are LLM-based Evaluators Confusing NLG Quality Criteria?cited by 35
  20. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  21. 2024Mapping Social Choice Theory to RLHFcited by 26
  22. 2023A density estimation perspective on learning from pairwise human preferencescited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 14; 2026: 3.

22 papers.

Sort:

By area:

  1. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  2. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  3. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  4. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  5. 2025AI Alignment at Your Discretioncited by 20
  6. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 17
  7. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  8. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  9. 2024Contextualized Evaluations: Judging Language Model Responses to Underspecified Queriescited by 12
  10. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  11. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  12. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 9
10 more
  1. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  2. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  3. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  4. 2025Benchmarking and Improving LLM Robustness for Personalized Generationcited by 4
  5. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Modelscited by 4
  6. 2026Benchmarking LLM-as-a-Judge for Long-Form Output Evaluationcited by 3
  7. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 2
  8. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modelingcited by 2
  9. 2025Steerable Chatbots: Exploring Personalization Control Interfaces via LLM Activation Steeringcited by 1
  10. 2025User-centric Subjective Leaderboard by Customizable Reward Modelingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modeling
  3. 2024A Survey on Human Preference Learning for Large Language Models
  4. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
  5. 2024Transforming and Combining Rewards for Aligning Large Language Models
  6. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
  7. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularization
  8. 2024ICPL: Few-shot In-context Preference Learning via LLMs