mlsanityData to 7 Oct 2026

Compositional preference models for aligning LMs

arXiv 2310.13011 ↗First posted 17 Oct 2023cs.CL cs.LGcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

As language models (LMs) become more capable, it is increasingly important to align them with human preferences. However, the dominant paradigm for training Preference Models (PMs) for that purpose suffers from fundamental limitations, such as lack of transparency and scalability, along with susceptibility to overfitting the preference dataset. We propose Compositional Preference Models (CPMs), a novel PM framework that decomposes one global preference assessment into several interpretable features, obtains scalar scores for these features from a prompted LM, and aggregates these scores using a logistic regression classifier. Through these simple steps, CPMs allow to control which properties of the preference data are used to train the preference model and to build it based on features that are believed to underlie the human preference judgment. Our experiments show that CPMs not only improve generalization and are more robust to overoptimization than standard PMs, but also that best-of-n samples obtained using CPMs tend to be preferred over samples obtained using conventional PMs. Overall, our approach demonstrates the benefits of endowing PMs with priors about which features determine human preferences while relying on LM capabilities to extract those features in a scalable and robust way.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 46 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

35 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Human Feedback is not Gold Standard28 Sep
  2. 2023Pretraining Language Models with Human Preferences16 Feb
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 31

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2016Concrete Problems in AI Safetycited by 1,582
  12. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
19 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Whose Opinions Do Language Models Reflect?cited by 759
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  5. 2018AI safety via debatecited by 384
  6. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  7. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  8. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  9. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  10. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  11. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  12. 2018Supervising strong learners by amplifying weak expertscited by 139
  13. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  14. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  15. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  16. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  17. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 44
  18. 2020Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learningcited by 21
  19. 2023On The Fragility of Learned Reward Functionscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 8; 2026: 3.

25 papers.

Sort:

By area:

  1. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  2. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  3. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  4. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
  5. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
  6. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 39
  7. 2024Personality Alignment of Large Language Modelscited by 36
  8. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Modelscited by 22
  9. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  10. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  11. 2024AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligencecited by 12
  12. 2024ALaRM: Align Language Models via Hierarchical Rewards Modelingcited by 11
13 more
  1. 2024Multi-objective Reinforcement learning from AI Feedbackcited by 9
  2. 2024Interpreting Language Reward Models via Contrastive Explanationscited by 9
  3. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  4. 2025AdaBoN: Adaptive Best-of-N Alignmentcited by 8
  5. 2025Configurable Preference Tuning with Rubric-Guided Synthetic Datacited by 7
  6. 2024HAF-RM: A Hybrid Alignment Framework for Reward Model Trainingcited by 5
  7. 2025FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Datacited by 5
  8. 2025Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstructioncited by 2
  9. 2025Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decompositioncited by 2
  10. 2024Aligning language models with human preferencescited by 1
  11. 2026Discovering Implicit Large Language Model Alignment Objectivescited by 0
  12. 2026PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questionscited by 0
  13. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2024A Survey on Human Preference Learning for Large Language Models
  3. 2026Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
  4. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  5. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
  6. 2025WorldPM: Scaling Human Preference Modeling
  7. 2024Dissecting Human and LLM Preferences
  8. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignment