mlsanityData to 14 Oct 2026

Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback

arXiv 2303.05453 ↗First posted 9 Mar 2023cs.CL cs.CYcited by 87

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large language models (LLMs) are used to generate content for a wide range of tasks, and are set to reach a growing audience in coming years due to integration in product interfaces like ChatGPT or search engines like Bing. This intensifies the need to ensure that models are aligned with human preferences and do not produce unsafe, inaccurate or toxic outputs. While alignment techniques like reinforcement learning with human feedback (RLHF) and red-teaming can mitigate some safety concerns and improve model capabilities, it is unlikely that an aggregate fine-tuning process can adequately represent the full range of users' preferences and values. Different people may legitimately disagree on their preferences for language and conversational norms, as well as on values or ideologies which guide their communication. Personalising LLMs through micro-level preference learning processes may result in models that are better aligned with each user. However, there are several normative challenges in defining the bounds of a societally-acceptable and safe degree of personalisation. In this paper, we ask how, and in what ways, LLMs should be personalised. First, we review literature on current paradigms for aligning LLMs with human feedback, and identify issues including (i) a lack of clarity regarding what alignment means; (ii) a tendency of technology providers to prescribe definitions of inherently subjective preferences and values; and (iii) a 'tyranny of the crowdworker', exacerbated by a lack of documentation in who we are really aligning to. Second, we present a taxonomy of benefits and risks associated with personalised LLMs, for individuals and society at large. Finally, we propose a three-tiered policy framework that allows users to experience the benefits of personalised alignment, while restraining unsafe and undesirable LLM-behaviours within (supra-)national and organisational bounds.

Key facts. Cited by 87 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 101 of its 258 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

101 of 258 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits1 Jan
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
  5. 2022Enabling Classifiers to Make Judgements Explicitly Aligned with Human Values14 Oct
  6. 2022In conversation with Artificial Intelligence: aligning language models with human values1 Sep
  7. 2022Towards Boosting the Open-Domain Chatbot with Human Feedback30 Aug
  8. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback5 Aug
  9. 2022Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback12 Apr
  10. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Data15 Sep
  11. 2020Learning to summarize from human feedback2 Sep
  12. 2020Aligning AI With Shared Human Values5 Aug
2 more
  1. 2019Learning Norms from Stories: A Prior for Value Aligned Agents7 Dec
  2. 2017Learning Semantic Relatedness From Human Feedback Using Metric Learning21 May

Other references · 87

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
38 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  6. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  8. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  10. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  11. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  12. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  13. 2020Artificial Intelligence, Values and Alignmentcited by 448
  14. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  15. 2018AI safety via debatecited by 400
  16. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  17. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  18. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  19. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  20. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  21. 2016Recommendations as Treatments: Debiasing Learning and Evaluationcited by 248
  22. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  23. 2019Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Goodcited by 209
  24. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 207
  25. 2021Alignment of Language Agentscited by 168
  26. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  27. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  28. 2021Can Machines Learn Morality? The Delphi Experimentcited by 141
  29. 2022Power to the People? Opportunities and Challenges for Participatory AIcited by 138
  30. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 138
  31. 2023Auditing large language models: a three-layered approachcited by 137
  32. 2018Training Millions of Personalized Dialogue Agentscited by 126
  33. 2017How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utilitycited by 120
  34. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  35. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  36. 2022The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systemscited by 93
  37. 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
  38. 2019Personalizing Dialogue Agents via Meta-Learningcited by 83

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 20; 2024: 39; 2025: 17; 2026: 11.

87 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  3. 2023Tool Learning with Foundation Modelscited by 302
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  5. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  6. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  7. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  8. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  9. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 154
  10. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  11. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  12. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
38 more
  1. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  2. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 92
  3. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 82
  4. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 62
  5. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 60
  6. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 54
  7. 2023Multimodal LLMs for health grounded in individual-specific datacited by 49
  8. 2024Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinementcited by 49
  9. 2024NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Modelscited by 48
  10. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 47
  11. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  12. 2024Aligning Language Models with Demonstrated Feedbackcited by 34
  13. 2023Learning Personalized Alignment for Evaluating Open-ended Text Generationcited by 29
  14. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 27
  15. 2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
  16. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
  17. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  18. 2025Virtual Agent Economiescited by 21
  19. 2024The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youthcited by 20
  20. 2024Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Modelscited by 19
  21. 2024Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learningcited by 19
  22. 2024Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generatorscited by 18
  23. 2025The Future of Continual Learning in the Era of Foundation Models: Three Key Directionscited by 17
  24. 2025Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approachcited by 16
  25. 2025Personalize Your LLM: Fake it then Align itcited by 15
  26. 2023Modeling Empathic Similarity in Personal Narrativescited by 14
  27. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 14
  28. 2024Near to Mid-term Risks and Opportunities of Open-Source Generative AIcited by 13
  29. 2024Risks and Opportunities of Open-Source Generative AIcited by 13
  30. 2023Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?cited by 12
  31. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  32. 2024Understanding the Role of User Profile in the Personalization of Large Language Modelscited by 11
  33. 2023Decolonial AI Alignment: Openness, Vi'sedsa-Dharma, and Including Excluded Knowledgescited by 10
  34. 2024Alignment Studio: Aligning Large Language Models to Particular Contextual Regulationscited by 10
  35. 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 10
  36. 2024Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedbackcited by 10
  37. 2023Unpacking the Ethical Value Alignment in Big Modelscited by 9
  38. 2024Should agentic conversational AI change how we think about ethics? Characterising an interactional ethics centred on respectcited by 8

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
  2. 2024Aligning LLMs with Individual Preferences via Interaction
  3. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
  4. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  5. 2024Aligning to Thousands of Preferences via System Message Generalization
  6. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning
  7. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
  8. 2025Language Model Personalization via Reward Factorization