mlsanityData to 14 Oct 2026

SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs

arXiv 2506.05598 ↗First posted 5 Jun 2025cs.CL cs.AIcited by 29

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Recent calls for pluralistic alignment of Large Language Models (LLMs) encourage adapting models to diverse user preferences. However, most prior work on personalized reward models heavily rely on additional identity information, such as demographic details or a predefined set of preference categories. To this end, we introduce SynthesizeMe, an approach to inducing synthetic user personas from user interactions for personalized reward modeling. SynthesizeMe first generates and verifies reasoning to explain user preferences, then induces synthetic user personas from that reasoning, and finally filters to informative prior user interactions in order to build personalized prompts for a particular user. We show that using SynthesizeMe induced prompts improves personalized LLM-as-a-judge accuracy by 4.4% on Chatbot Arena. Combining SynthesizeMe derived prompts with a reward model achieves top performance on PersonalRewardBench: a new curation of user-stratified interactions with chatbots collected from 854 users of Chatbot Arena and PRISM.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 67 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

38 of 67 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users26 Feb
  2. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration22 Jun
  3. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  5. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback2 Oct
  6. 2023Aligning Language Models to User Opinions24 May

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  6. 2023Whose Opinions Do Language Models Reflect?cited by 773
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  8. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  9. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  10. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  11. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  12. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
20 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  2. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  3. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  4. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  5. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  6. 2024Can LLM be a Personalized Judge?cited by 100
  7. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  8. 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 78
  9. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 60
  10. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 54
  11. 2023Facilitating Self-Guided Mental Health Interventions Through Human-Language Model Interaction: A Case Study of Cognitive Restructuringcited by 53
  12. 2023On the steerability of large language models toward data-driven personascited by 53
  13. 2023LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordinationcited by 46
  14. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  15. 2024Aligning Language Models with Demonstrated Feedbackcited by 34
  16. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  17. 2024Personalized Text Generation with Fine-Grained Linguistic Controlcited by 20
  18. 2024Small But Funny: A Feedback-Driven Approach to Humor Distillationcited by 12
  19. 2024Provably Efficient Interactive-Grounded Learning with Personalized Rewardcited by 3
  20. 2024Impacts of Anthropomorphizing Large Language Models in Learning Environmentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 25.

29 papers.

Sort:

By area:

  1. 2026Agent-as-a-Judgecited by 17
  2. 2025SteerX: Disentangled Steering for LLM Personalizationcited by 7
  3. 2026PrefIx: Understand and Adapt to User Preference in Human-Agent Interactioncited by 7
  4. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7
  5. 2026Learning User Simulators with Turing Rewardscited by 7
  6. 2026Personalization Increases Affective Alignment but Has Role-Dependent Effects on Epistemic Independence in LLMscited by 5
  7. 2025Personalized Reward Modeling for Text-to-Image Generationcited by 4
  8. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scalingcited by 4
  9. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?cited by 3
  10. 2025ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluationcited by 2
  11. 2025Fints: Efficient Inference-Time Personalization for LLMs with Fine-Grained Instance-Tailored Steeringcited by 2
  12. 2026Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Userscited by 2
17 more
  1. 2026Preference-Aware Rubric Learning for Personalized Evaluationcited by 2
  2. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modelingcited by 2
  3. 2026Statistical Priors for Implicit Preferences: Decoupling Skill Selection as a Local Harness in Personal Agentscited by 2
  4. 2026Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalizationcited by 1
  5. 2026ChildEval: When large language models meet children's personalitiescited by 1
  6. 2026Personagram: Bridging Personas and Product Design for Creative Ideation with Multimodal LLMscited by 0
  7. 2026The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footagecited by 0
  8. 2026EpiPersona: Persona Projection and Episode Coupling for Pluralistic Preference Modelingcited by 0
  9. 2026Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failurescited by 0
  10. 2026In-Context Reward Adaptation for Robust Preference Modelingcited by 0
  11. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  12. 2026From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMscited by 0
  13. 2026Using Context Is Not Enough: Test-Time Training for Personalized Reward Modelingcited by 0
  14. 2026Gradient-Aligned Pair Selection for Personalized Preference Optimizationcited by 0
  15. 2026Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulationcited by 0
  16. 2026FleetPilot: Fine-Grained Steering Empowers Real-World LLM Personalizationcited by 0
  17. 2026Learning Individual Scientific Taste with Privileged Hindsight Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Models
  2. 2024PersonalLLM: Tailoring LLMs to Individual Preferences
  3. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory
  4. 2025Language Model Personalization via Reward Factorization
  5. 2026Re-Centering Humans in LLM Personalization
  6. 2026One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
  7. 2025PersonaAgent: Bridging Memory and Action for Personalized LLM Agents
  8. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions