mlsanityData to 14 Oct 2026

Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?

arXiv 2505.13257 ↗First posted 19 May 2025cs.CL cs.AI cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

A prominent issue in aligning language models (LMs) to personalized preferences is underspecification -- the lack of information from users about their preferences. A popular trend of injecting such specification is adding a prefix (e.g. prior relevant conversations) to the current user's conversation to steer preference distribution. Most methods passively model personal preferences with prior example preferences pairs. We ask whether models benefit from actively inferring preference descriptions, and address this question by creating a synthetic personalized alignment dataset based on famous people with known public preferences. We then test how effective finetuned 1-8B size models are at inferring and aligning to personal preferences. Results show that higher-quality active prefixes lead to better generalization, more contextually faithful models, and less systematic biases across different protected attributes. All our results suggest active alignment can lead to a more controllable and efficient path for personalized alignment.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 87 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

56 of 87 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Can Language Models Reason about Individualistic Human Values and Preferences?4 Oct
  2. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignment24 Jul
  3. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration22 Jun
  4. 2024What are human values, and how do we align AI to them?27 Mar
  5. 2024A Roadmap to Pluralistic Alignment7 Feb
  6. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  7. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  8. 2023Learning Personalized Alignment for Evaluating Open-ended Text Generation5 Oct
  9. 2023Aligning Language Models to User Opinions24 May
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  11. 2020Learning to summarize from human feedback2 Sep

Other references · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  10. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  11. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
33 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  2. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  3. 2023Whose Opinions Do Language Models Reflect?cited by 773
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  5. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  6. 2020Artificial Intelligence, Values and Alignmentcited by 448
  7. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  8. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  9. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  10. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  11. 2023Mitigating the Alignment Tax of RLHFcited by 183
  12. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  13. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  14. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  15. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  16. 2024Can LLM be a Personalized Judge?cited by 100
  17. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  18. 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 88
  19. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  20. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  21. 2024Large Language Models Assume People are More Rational than We Really arecited by 54
  22. 2023On the steerability of large language models toward data-driven personascited by 53
  23. 2024Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinementcited by 49
  24. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  25. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 44
  26. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 32
  27. 2024Few-shot Personalization of LLMs with Mis-aligned Responsescited by 32
  28. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
  29. 2017Should Robots be Obedient?cited by 20
  30. 2024ReasoningRec: Bridging Personalized Recommendations and Human-Interpretable Explanations through LLM Reasoningcited by 14
  31. 2024ValueScope: Unveiling Implicit Norms and Values via Return Potential Model of Social Interactionscited by 12
  32. 2023Social Contract AI: Aligning AI Assistants with Implicit Group Normscited by 9
  33. 2022AugCSE: Contrastive Sentence Embedding with Diverse Augmentationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Pragmatic Persona: Discovering LLM Persona through Bridging Inferencecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
  2. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  3. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  4. 2024Aligning LLMs with Individual Preferences via Interaction
  5. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment
  6. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation
  7. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferences
  8. 2024PersonalLLM: Tailoring LLMs to Individual Preferences