mlsanityData to 7 Oct 2026

Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversations

arXiv 2503.08035 ↗First posted 11 Mar 2025cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

LLMs often fail to meet the specialized needs of distinct user groups due to their one-size-fits-all training paradigm citelucy-etal-2024-one and there is limited research on what personalization aspects each group expect. To address these limitations, we propose a group-aware personalization framework, Group Preference Alignment (GPA), that identifies context-specific variations in conversational preferences across user groups and then steers LLMs to address those preferences. Our approach consists of two steps: (1) Group-Aware Preference Extraction, where maximally divergent user-group preferences are extracted from real-world conversation logs and distilled into interpretable rubrics, and (2) Tailored Response Generation, which leverages these rubrics through two methods: a) Context-Tuned Inference (GAP-CT), that dynamically adjusts responses via context-dependent prompt instructions, and b) Rubric-Finetuning Inference (GPA-FT), which uses the rubrics to generate contrastive synthetic data for personalization of group-specific models via alignment. Experiments demonstrate that our framework significantly improves alignment of the output with respect to user preferences and outperforms baseline methods, while maintaining robust performance on standard benchmarks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 47 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

26 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas20 Jan
  2. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback28 Aug
  3. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration22 Jun
  4. 2024Aligning to Thousands of Preferences via System Message Generalization28 May
  5. 2024A Roadmap to Pluralistic Alignment7 Feb
  6. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  8. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  10. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  11. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  12. 2024Can LLM be a Personalized Judge?cited by 99
8 more
  1. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 98
  2. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  3. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  4. 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 78
  5. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  6. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
  7. 2024Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judgecited by 19
  8. 2023"One-Size-Fits-All"? Examining Expectations around What Constitute "Fair" or "Good" NLG System Behaviorscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemblecited by 2
  2. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning LLMs with Individual Preferences via Interaction
  2. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions
  3. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models
  4. 2025FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data
  5. 2024PersonalLLM: Tailoring LLMs to Individual Preferences
  6. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation
  7. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
  8. 2026GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization