# SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs

arXiv: https://arxiv.org/abs/2506.05598 · HTML: https://mlsanity.com/paper/2506.05598/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 38 of its 67 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 38)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://mlsanity.com/paper/2406.11695/) (2024)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits](https://mlsanity.com/paper/2305.02547/) (2023)
- [Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment](https://mlsanity.com/paper/2402.10207/) (2024)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [Personalized Language Modeling from Personalized Human Feedback](https://mlsanity.com/paper/2402.05133/) (2024)
- [Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration](https://mlsanity.com/paper/2406.15951/) (2024)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [Aligning Language Models to User Opinions](https://mlsanity.com/paper/2305.14929/) (2023)
- [Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model](https://mlsanity.com/paper/2310.09520/) (2023)
- [Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM](https://mlsanity.com/paper/2404.12259/) (2024)
- [HYDRA: Model Factorization Framework for Black-Box LLM Personalization](https://mlsanity.com/paper/2406.02888/) (2024)
- [PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences](https://mlsanity.com/paper/2406.08469/) (2024)
- [Facilitating Self-Guided Mental Health Interventions Through Human-Language Model Interaction: A Case Study of Cognitive Restructuring](https://mlsanity.com/paper/2310.15461/) (2023)
- [On the steerability of large language models toward data-driven personas](https://mlsanity.com/paper/2311.04978/) (2023)
- [LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordination](https://mlsanity.com/paper/2309.11696/) (2023)

## Built on by (most cited first, 29 of 29)

- [Agent-as-a-Judge](https://mlsanity.com/paper/2601.05111/) (2026)
- [SteerX: Disentangled Steering for LLM Personalization](https://mlsanity.com/paper/2510.22256/) (2025)
- [PrefIx: Understand and Adapt to User Preference in Human-Agent Interaction](https://mlsanity.com/paper/2602.06714/) (2026)
- [Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization](https://mlsanity.com/paper/2604.07343/) (2026)
- [Learning User Simulators with Turing Rewards](https://mlsanity.com/paper/2606.19336/) (2026)
- [Personalization Increases Affective Alignment but Has Role-Dependent Effects on Epistemic Independence in LLMs](https://mlsanity.com/paper/2603.00024/) (2026)
- [Personalized Reward Modeling for Text-to-Image Generation](https://mlsanity.com/paper/2511.19458/) (2025)
- [P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling](https://mlsanity.com/paper/2602.12116/) (2026)
- [SLMJury: Can Small Language Models Judge as Well as Large Ones?](https://mlsanity.com/paper/2606.07810/) (2026)
- [ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation](https://mlsanity.com/paper/2507.12674/) (2025)
- [Fints: Efficient Inference-Time Personalization for LLMs with Fine-Grained Instance-Tailored Steering](https://mlsanity.com/paper/2510.27206/) (2025)
- [Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users](https://mlsanity.com/paper/2603.16120/) (2026)
- [Preference-Aware Rubric Learning for Personalized Evaluation](https://mlsanity.com/paper/2605.31545/) (2026)
- [Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling](https://mlsanity.com/paper/2606.04284/) (2026)
- [Statistical Priors for Implicit Preferences: Decoupling Skill Selection as a Local Harness in Personal Agents](https://mlsanity.com/paper/2606.05828/) (2026)
- [Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization](https://mlsanity.com/paper/2604.00997/) (2026)
- [ChildEval: When large language models meet children's personalities](https://mlsanity.com/paper/2605.27805/) (2026)
- [Personagram: Bridging Personas and Product Design for Creative Ideation with Multimodal LLMs](https://mlsanity.com/paper/2602.06197/) (2026)
- [The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footage](https://mlsanity.com/paper/2602.10339/) (2026)
- [EpiPersona: Persona Projection and Episode Coupling for Pluralistic Preference Modeling](https://mlsanity.com/paper/2603.28197/) (2026)
- [Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures](https://mlsanity.com/paper/2605.10991/) (2026)
- [In-Context Reward Adaptation for Robust Preference Modeling](https://mlsanity.com/paper/2605.30323/) (2026)
- [PAFO: Pareto Fairness Optimization for Personalized Reward Modeling](https://mlsanity.com/paper/2606.07988/) (2026)
- [From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMs](https://mlsanity.com/paper/2609.28942/) (2026)
- [Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling](https://mlsanity.com/paper/2609.35109/) (2026)
- [Gradient-Aligned Pair Selection for Personalized Preference Optimization](https://mlsanity.com/paper/2610.00061/) (2026)
- [Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulation](https://mlsanity.com/paper/2610.04676/) (2026)
- [FleetPilot: Fine-Grained Steering Empowers Real-World LLM Personalization](https://mlsanity.com/paper/2610.14970/) (2026)
- [Learning Individual Scientific Taste with Privileged Hindsight Distillation](https://mlsanity.com/paper/2610.19719/) (2026)

## Abstract (arXiv)

Recent calls for pluralistic alignment of Large Language Models (LLMs) encourage adapting models to diverse user preferences. However, most prior work on personalized reward models heavily rely on additional identity information, such as demographic details or a predefined set of preference categories. To this end, we introduce SynthesizeMe, an approach to inducing synthetic user personas from user interactions for personalized reward modeling. SynthesizeMe first generates and verifies reasoning to explain user preferences, then induces synthetic user personas from that reasoning, and finally filters to informative prior user interactions in order to build personalized prompts for a particular user. We show that using SynthesizeMe induced prompts improves personalized LLM-as-a-judge accuracy by 4.4% on Chatbot Arena. Combining SynthesizeMe derived prompts with a reward model achieves top performance on PersonalRewardBench: a new curation of user-stratified interactions with chatbots collected from 854 users of Chatbot Arena and PRISM.