mlsanityData to 7 Oct 2026

PersonalLLM: Tailoring LLMs to Individual Preferences

arXiv 2409.20296 ↗First posted 30 Sep 2024cs.LG cs.CLcited by 56

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

As LLMs become capable of complex tasks, there is growing potential for personalized interactions tailored to the subtle and idiosyncratic preferences of the user. We present a public benchmark, PersonalLLM, focusing on adapting LLMs to provide maximal benefits for a particular user. Departing from existing alignment benchmarks that implicitly assume uniform preferences, we curate open-ended prompts paired with many high-quality answers over which users would be expected to display heterogeneous latent preferences. Instead of persona-prompting LLMs based on high-level attributes (e.g., user's race or response length), which yields homogeneous preferences relative to humans, we develop a method that can simulate a large user base with diverse preferences from a set of pre-trained reward models. Our dataset and generated personalities offer an innovative testbed for developing personalization algorithms that grapple with continual data sparsity--few relevant feedback from the particular user--by leveraging historical data from other (similar) users. We explore basic in-context learning and meta-learning baselines to illustrate the utility of PersonalLLM and highlight the need for future methodological development. Our dataset is available at https://huggingface.co/datasets/namkoong-lab/PersonalLLM

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 33 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

22 of 33 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignment24 Jul
  2. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  3. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  4. 2023Aligning Language Models to User Opinions24 May

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  11. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  12. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
6 more
  1. 2018Training Millions of Personalized Dialogue Agentscited by 126
  2. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  3. 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 91
  4. 2020Citation Recommendation: Approaches and Datasetscited by 36
  5. 2023KuaiSim: A Comprehensive Simulator for Recommender Systemscited by 30
  6. 2022Rethinking Personalized Ranking at Pinterest: An End-to-End Approachcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 22; 2026: 31.

56 papers.

Sort:

By area:

  1. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 163
  2. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
  3. 2025LLM Generated Persona is a Promise with a Catchcited by 77
  4. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  5. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  6. 2025Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducementscited by 31
  7. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 29
  8. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  9. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  10. 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignmentcited by 22
  11. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  12. 2025Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIcited by 19
38 more
  1. 2025What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commercecited by 18
  2. 2025Personalize Your LLM: Fake it then Align itcited by 15
  3. 2025PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalizationcited by 13
  4. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  5. 2026VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactionscited by 11
  6. 2024Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedbackcited by 10
  7. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  8. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  9. 2024Steering AI-Driven Personalization of Scientific Text for General Audiencescited by 7
  10. 2026AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignmentcited by 6
  11. 2024Towards Effective Discrimination Testing for Generative AIcited by 5
  12. 2025FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Datacited by 5
  13. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scalingcited by 4
  14. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  15. 2025LLM-Generated Ads: From Personalization Parity to Persuasion Superioritycited by 2
  16. 2025Benchmarking In-context Experiential Learning Through Repeated Product Recommendationscited by 1
  17. 2026The Mouth is Not the Brain: Bridging Energy-Based World Models and Language Generationcited by 1
  18. 2026Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalizationcited by 1
  19. 2026LATTE: Forecasting Peer Anchored Preference Trajectories for Personalized LLM Generationcited by 1
  20. 2026Self-supervised User Profile Generation for Personalizationcited by 1
  21. 2026PersonalBench: Measuring the Authorship Gap in LLM Personalizationcited by 1
  22. 2025Reflective Personalization Optimization: A Post-hoc Rewriting Framework for Black-Box Large Language Modelscited by 0
  23. 2026Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognitioncited by 0
  24. 2026Social Knowledge for Cross-Domain User Preference Modelingcited by 0
  25. 2026PowerLens: Taming LLM Agents for Safe and Personalized Mobile Power Managementcited by 0
  26. 2026EpiPersona: Persona Projection and Episode Coupling for Pluralistic Preference Modelingcited by 0
  27. 2026Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalizationcited by 0
  28. 2026CLIPer: Tailoring Diverse User Preference via Classifier-Guided Inference-Time Personalizationcited by 0
  29. 2026Personalized Alignment Revisited: The Necessity and Sufficiency of User Diversitycited by 0
  30. 2026Variance-aware Reward Modeling with Anchor Guidancecited by 0
  31. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  32. 2026MapSatisfyBench: Benchmarking Satisfaction-Aware Map Agents through Behavior-Grounded Implicit Decision Factorscited by 0
  33. 2026The Severance Problem: LLMs are Unaware of the Person Beyond the Promptcited by 0
  34. 2026Learning Dynamic User Personas from Implicit Interaction Streams via Iterative Refinementcited by 0
  35. 2026MIDAS: Multi-LLM Iterative Data-Adaptive Summarizationcited by 0
  36. 2026Cautious Context Steering for Language Model Personalizationcited by 0
  37. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimizationcited by 0
  38. 2026PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Taskscited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning LLMs with Individual Preferences via Interaction
  2. 2026Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions
  3. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Users
  4. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMs
  5. 2025CoPL: Collaborative Preference Learning for Personalizing LLMs
  6. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Models
  7. 2024Orchestrating LLMs with Different Personalizations
  8. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs