# PrefPalette: Personalized Preference Modeling with Latent Attributes

arXiv: https://arxiv.org/abs/2507.13541 · HTML: https://mlsanity.com/paper/2507.13541/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://mlsanity.com/paper/1912.11078/) (2019)
- [Personalized Language Modeling from Personalized Human Feedback](https://mlsanity.com/paper/2402.05133/) (2024)
- [Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization](https://mlsanity.com/paper/2310.03708/) (2023)
- [Dialogue Response Ranking Training with Large-Scale Human Feedback Data](https://mlsanity.com/paper/2009.06978/) (2020)
- [Models of human preference for learning reward functions](https://mlsanity.com/paper/2206.02231/) (2022)
- [The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains](https://mlsanity.com/paper/2507.06187/) (2025)
- [Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Sets](https://mlsanity.com/paper/2305.15407/) (2023)
- [ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning](https://mlsanity.com/paper/2502.14860/) (2025)
- [ComPO: Community Preferences for Language Model Personalization](https://mlsanity.com/paper/2410.16027/) (2024)
- [FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users](https://mlsanity.com/paper/2502.19312/) (2025)
- [ValueScope: Unveiling Implicit Norms and Values via Return Potential Model of Social Interactions](https://mlsanity.com/paper/2407.02472/) (2024)
- [When People are Floods: Analyzing Dehumanizing Metaphors in Immigration Discourse with Large Language Models](https://mlsanity.com/paper/2502.13246/) (2025)
- [Orchestrating LLMs with Different Personalizations](https://mlsanity.com/paper/2407.04181/) (2024)
- [Influencing Humans to Conform to Preference Models for RLHF](https://mlsanity.com/paper/2501.06416/) (2025)

## Built on by (most cited first, 16 of 16)

- [What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data](https://mlsanity.com/paper/2510.26202/) (2025)
- [Cognitive Foundations for Reasoning and Their Manifestation in LLMs](https://mlsanity.com/paper/2511.16660/) (2025)
- [HorizonBench: Long-Horizon Personalization with Evolving Preferences](https://mlsanity.com/paper/2604.17283/) (2026)
- [CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experiments](https://mlsanity.com/paper/2509.13588/) (2025)
- [PrefDisco: Benchmarking Proactive Personalized Reasoning](https://mlsanity.com/paper/2510.00177/) (2025)
- [Cold-Start Personalization via Training-Free Priors from Structured World Models](https://mlsanity.com/paper/2602.15012/) (2026)
- [Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interest](https://mlsanity.com/paper/2604.08525/) (2026)
- [Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling](https://mlsanity.com/paper/2606.04284/) (2026)
- [Interactive Task Alignment as a POMDP](https://mlsanity.com/paper/2607.16412/) (2026)
- [Which course? Discourse! Teaching Discourse and Generation in the Era of LLMs](https://mlsanity.com/paper/2602.02878/) (2026)
- [Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP](https://mlsanity.com/paper/2603.28351/) (2026)
- [Efficient Personalization of Generative User Interfaces](https://mlsanity.com/paper/2604.09876/) (2026)
- [PubSwap: Public-Data Off-Policy Coordination for Federated RLVR](https://mlsanity.com/paper/2604.12160/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [On the Role of Citations in Preference Data](https://mlsanity.com/paper/2608.21376/) (2026)
- [IsoVRBench: Video Reward Modeling Without Human Feedback](https://mlsanity.com/paper/2610.17234/) (2026)

## Abstract (arXiv)

Personalizing AI systems requires understanding not just what users prefer, but the reasons that underlie those preferences - yet current preference models typically treat human judgment as a black box. We introduce PrefPalette, a framework that decomposes preferences into attribute dimensions and tailors its preference prediction to distinct social community values in a human-interpretable manner. PrefPalette operationalizes a cognitive science principle known as multi-attribute decision making in two ways: (1) a scalable counterfactual attribute synthesis step that involves generating synthetic training data to isolate for individual attribute effects (e.g., formality, humor, cultural values), and (2) attention-based preference modeling that learns how different social communities dynamically weight these attributes. This approach moves beyond aggregate preference modeling to capture the diverse evaluation frameworks that drive human judgment. When evaluated on 45 social communities from the online platform Reddit, PrefPalette outperforms GPT-4o by 46.6% in average prediction accuracy. Beyond raw predictive improvements, PrefPalette also shed light on intuitive, community-specific profiles: scholarly communities prioritize verbosity and stimulation, conflict-oriented communities value sarcasm and directness, and support-based communities emphasize empathy. By modeling the attribute-mediated structure of human judgment, PrefPalette delivers both superior preference modeling and transparent, interpretable insights, and serves as a first step toward more trustworthy, value-aware personalized applications.