mlsanityData to 14 Oct 2026

Group Preference Optimization: Few-Shot Alignment of Large Language Models

arXiv 2310.11523 ↗First posted 17 Oct 2023cs.LG cs.AI cs.CLcited by 63

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Many applications of large language models (LLMs), ranging from chatbots to creative writing, require nuanced subjective judgments that can differ significantly across different groups. Existing alignment algorithms can be expensive to align for each group, requiring prohibitive amounts of group-specific preference data and computation for real-world use cases. We introduce Group Preference Optimization (GPO), an alignment framework that steers language models to preferences of individual groups in a few-shot manner. In GPO, we augment the base LLM with an independent transformer module trained to predict the preferences of a group for the LLM generations. For few-shot learning, we parameterize this module as an in-context autoregressive transformer and train it via meta-learning on several groups. We empirically validate the efficacy of GPO through rigorous evaluations using LLMs with varied sizes on three human opinion adaptation tasks. These tasks involve adapting to the preferences of US demographic groups, global countries, and individual users. Our results demonstrate that GPO not only aligns models more accurately but also requires fewer group-specific preferences, and less training and inference computing resources, outperforming existing strategies such as in-context steering and fine-tuning methods.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 43 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

33 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Preference Ranking Optimization for Human Alignment30 Jun
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar
  4. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  12. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
17 more
  1. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  2. 2023Whose Opinions Do Language Models Reflect?cited by 773
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  5. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  7. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  8. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  9. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  10. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  11. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 127
  12. 2023Aligning Language Models to User Opinionscited by 99
  13. 2019Bias Correction of Learned Generative Models using Likelihood-Free Importance Weightingcited by 70
  14. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  15. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  16. 2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFcited by 20
  17. 2023TART: A plug-and-play Transformer module for task-agnostic reasoningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 20; 2026: 25.

63 papers.

Sort:

By area:

  1. 2024A Roadmap to Pluralistic Alignmentcited by 184
  2. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  3. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  4. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  5. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 73
  6. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 66
  7. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  8. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 38
  9. 2024Aligning Language Models with Demonstrated Feedbackcited by 34
  10. 2024Few-shot Personalization of LLMs with Mis-aligned Responsescited by 32
  11. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 31
  12. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
38 more
  1. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
  2. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  3. 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrationscited by 19
  4. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  5. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 18
  6. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  7. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimizationcited by 15
  8. 2025FSPO: Few-Shot Optimization of Synthetic Preferences Personalizes to Real Userscited by 15
  9. 2024Vector-ICL: In-context Learning with Continuous Vector Representationscited by 13
  10. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
  11. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  12. 2024The Strong Pull of Prior Knowledge in Large Language Models and Its Impact on Emotion Recognitioncited by 10
  13. 2025Learning to summarize user information for personalized reinforcement learning from human feedbackcited by 10
  14. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  15. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
  16. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  17. 2025Capturing Individual Human Preferences with Reward Featurescited by 8
  18. 2025Personalized Language Models via Privacy-Preserving Evolutionary Model Mergingcited by 8
  19. 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowdscited by 7
  20. 2026Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalizationcited by 7
  21. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inferencecited by 6
  22. 2026nabla-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Spacecited by 5
  23. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  24. 2025Personalized Reward Modeling for Text-to-Image Generationcited by 4
  25. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scalingcited by 4
  26. 2026AI Alignment Breaks at the Edgecited by 4
  27. 2026GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedbackcited by 4
  28. 2026When Can Digital Personas Reliably Approximate Human Survey Findings?cited by 4
  29. 2025EmboMatrix: A Scalable Training-Ground for Embodied Decision-Makingcited by 3
  30. 2025A Systematic Evaluation of Preference Aggregation in Federated RLHF for Pluralistic Alignment of LLMscited by 3
  31. 2025Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemblecited by 2
  32. 2025Aligning Deep Implicit Preferences by Learning to Reason Defensivelycited by 2
  33. 2024Securing Reliability: A Brief Overview on Enhancing In-Context Learning for Foundation Modelscited by 1
  34. 2026Autoregressive Direct Preference Optimizationcited by 1
  35. 2026APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMscited by 1
  36. 2026Steerable Cultural Preference Optimization of Reward Modelscited by 1
  37. 2026Constructive Alignment: Governing Preference Dynamics in Human-AI Interactioncited by 1
  38. 2026Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluationcited by 1

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2024No Preference Left Behind: Group Distributional Preference Optimization
  4. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMs
  5. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  6. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  7. 2025Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversations
  8. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments