mlsanityData to 14 Oct 2026

ComPO: Community Preferences for Language Model Personalization

arXiv 2410.16027 ↗First posted 21 Oct 2024cs.CLcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Conventional algorithms for training language models (LMs) with human feedback rely on preferences that are assumed to account for an "average" user, disregarding subjectivity and finer-grained variations. Recent studies have raised concerns that aggregating such diverse and often contradictory human feedback to finetune models results in generic models that generate outputs not preferred by many user groups, as they tend to average out styles and norms. To address this issue, we draw inspiration from recommendation systems and propose ComPO, a method to personalize preference optimization in LMs by contextualizing the probability distribution of model outputs with the preference provider. Focusing on group-level preferences rather than individuals, we collect and release ComPRed, a question answering dataset with community-level preferences from Reddit. This dataset facilitates studying diversity in preferences without incurring privacy concerns associated with individual feedback. Our experiments reveal that conditioning language models on a community identifier (i.e., subreddit name) during preference tuning substantially enhances model performance. Conversely, replacing this context with random subreddit identifiers significantly diminishes performance, highlighting the effectiveness of our approach in tailoring responses to communities' preferences.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 56 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

33 of 56 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  2. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  3. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct
  4. 2022Fine-tuning language models to find agreement among humans with diverse preferences28 Nov
  5. 2022Enabling Classifiers to Make Judgements Explicitly Aligned with Human Values14 Oct
  6. 2020Learning to summarize from human feedback2 Sep

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  12. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
15 more
  1. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  2. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  3. 2020The Pushshift Reddit Datasetcited by 292
  4. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  5. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  6. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  7. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  8. 2024Can LLM be a Personalized Judge?cited by 100
  9. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  10. 2021UserIdentifier: Implicit User Representations for Simple and Effective Personalized Sentiment Analysiscited by 28
  11. 2023Embedding in Recommender Systems: A Surveycited by 20
  12. 2023Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptionscited by 9
  13. 2016Science, AskScience, and BadScience: On the Coexistence of Highly Related Communitiescited by 6
  14. 2023ChatGPT Is More Likely to Be Perceived as Male Than Femalecited by 5
  15. 2022No Community Can Do Everything: Why People Participate in Similar Online Communitiescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 7.

19 papers.

Sort:

By area:

  1. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 42
  2. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  3. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 30
  4. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  5. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 27
  6. 2025PrefPalette: Personalized Preference Modeling with Latent Attributescited by 16
  7. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  8. 2025Embedding-to-Prefix: Parameter-Efficient Personalization for Pre-Trained Large Language Modelscited by 10
  9. 2025STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Modelscited by 8
  10. 2025Extended Inductive Reasoning for Personalized Preference Inference from Behavioral Signalscited by 5
  11. 2025Who's Asking? Simulating Role-Based Questions for Conversational AI Evaluationcited by 2
  12. 2026LLUMI: Improving LLM Writing Assistance for Mental Health Support with Online Community Feedbackcited by 2
7 more
  1. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  2. 2026CommunityBench: Benchmarking Community-Level Alignment across Diverse Groups and Taskscited by 1
  3. 2025Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Modelscited by 0
  4. 2026DiscoTrace: Representing and Comparing Answering Strategies of Humans and LLMs in Information-Seeking Question Answeringcited by 0
  5. 2026Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maximscited by 0
  6. 2026What Do People Actually Want From AI? Mapping Preference Pluralitycited by 0
  7. 2026Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Postscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unsupervised Human Preference Learning
  2. 2026VASTU: Language Models Struggle to Recognize Online Community Values
  3. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models
  4. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
  5. 2025WorldPM: Scaling Human Preference Modeling
  6. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularization
  7. 2026Online LLM Personalization from Pairwise Preferences Across Users
  8. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization