mlsanityData to 13 Oct 2026

From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment

arXiv 2503.15463 ↗First posted 19 Mar 2025cs.CL cs.AIcited by 30

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large language models (LLMs) have traditionally been aligned through one-size-fits-all approaches that assume uniform human preferences, fundamentally overlooking the diversity in user values and needs. This paper introduces a comprehensive framework for scalable personalized alignment of LLMs. We establish a systematic preference space characterizing psychological and behavioral dimensions, alongside diverse persona representations for robust preference inference in real-world scenarios. Building upon this foundation, we introduce textscAlignX, a large-scale dataset of over 1.3 million personalized preference examples, and develop two complementary alignment approaches: textitin-context alignment directly conditioning on persona representations and textitpreference-bridged alignment modeling intermediate preference distributions. Extensive experiments demonstrate substantial improvements over existing methods, with an average 17.06% accuracy gain across four benchmarks while exhibiting a strong adaptation capability to novel preferences, robustness to limited user data, and precise preference controllability. These results validate our approach toward user-adaptive AI systems.

Key facts. Cited by 30 papers in the map (data through 13 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 95 references that are in the map. First posted on arXiv 19 Mar 2025.

Builds on

41 of 95 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024ComPO: Community Preferences for Language Model Personalization21 Oct
  2. 2024PAD: Personalized Alignment of LLMs at Decoding-Time5 Oct
  3. 2024PersonalLLM: Tailoring LLMs to Individual Preferences30 Sep
  4. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  6. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct
  7. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback2 Oct
  8. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models23 Aug
  9. 2022In conversation with Artificial Intelligence: aligning language models with human values1 Sep

Other references · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2017Proximal Policy Optimization Algorithmscited by 14,592
  4. 2024The Llama 3 Herd of Modelscited by 14,486
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  8. 2023Mistral 7Bcited by 3,123
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,289
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,041
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 954
20 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 853
  2. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 477
  3. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 259
  4. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 253
  5. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 233
  6. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  7. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 179
  8. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 156
  9. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 152
  10. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 130
  11. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 99
  12. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 73
  13. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 73
  14. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  15. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 54
  16. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  17. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 44
  18. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 43
  19. 2024Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Modelscited by 20
  20. 2024Understanding the Role of User Profile in the Personalization of Large Language Modelscited by 11

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 9; 2026: 21.

30 papers.

Sort:

By area:

  1. 2025Memory OS of AI Agentcited by 144
  2. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 20
  3. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 13
  4. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  5. 2025LLM-Based Social Simulations Require a Boundarycited by 11
  6. 2026LifeSim: Long-Horizon User Life Simulator for Personalized Assistant Evaluationcited by 11
  7. 2026PersonaVLM: Long-Term Personalized Multimodal LLMscited by 11
  8. 2026OdysSim: Building Foundation Models for Human Behavior Simulationcited by 11
  9. 2026UserLM-R1: Modeling Human Reasoning in User Language Models with Multi-Reward Reinforcement Learningcited by 8
  10. 2026How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized Assistantscited by 8
  11. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 6
  12. 2025Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signalscited by 5
18 more
  1. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inferencecited by 5
  2. 2025EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Modelscited by 4
  3. 2026CURP: Codebook-based Continuous User Representation for Personalized Generation with LLMscited by 4
  4. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Modelscited by 4
  5. 2026Text as a Universal Interface for Transferable Personalizationcited by 2
  6. 2026CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Taskscited by 1
  7. 2025DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Valuescited by 0
  8. 2026PersonaDual: Balancing Personalization and Objectivity via Adaptive Reasoningcited by 0
  9. 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedbackcited by 0
  10. 2026EpiPersona: Persona Projection and Episode Coupling for Pluralistic Preference Modelingcited by 0
  11. 2026Many Preferences, Few Policies: Towards Scalable Language Model Personalizationcited by 0
  12. 2026Efficient Personalization of Generative User Interfacescited by 0
  13. 2026UserGPT Technical Reportcited by 0
  14. 2026Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMscited by 0
  15. 2026Co-design of LLM-based preference agents: participation may drive overtrustcited by 0
  16. 2026Learning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learningcited by 0
  17. 2026From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMscited by 0
  18. 2026Extracting Persona Subspaces Through Iterative Nullspace Projection For Modulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  2. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  3. 2024Aligning to Thousands of Preferences via System Message Generalization
  4. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
  5. 2025Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?
  6. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferences
  7. 2024Aligning LLMs with Individual Preferences via Interaction
  8. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation