Steerable Cultural Preference Optimization of Reward Models
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
It is essential for large language model (LLM) technology to serve many different cultural sub-communities in a manner that is acceptable to each community. However, research on LLM alignment has so far predominantly focused on predicting a unified response preference of annotators from certain regions. This paper aims to advance the development of alignment models with a more global outlook, that are able to accurately represent the preferences of subcommunities and do not exhibit excessive bias towards any of them. We focus on the development of reward models for this purpose and present a novel reward model training algorithm (SCPO) that can incorporate diverse cultural preferences in a balanced manner. Our method results in performance increases of the minority reward model of up to 7 points over the baseline model across two datasets, PRISM and GlobalOpinionQA, and across 7 countries. SCPO is up to 280% more training data-efficient than full-data finetuning of reward models. In addition, we perform analysis of bias by separately evaluating on the preference of subcommunities and show that excessive bias is mitigated via our weighting method. Our code is available at https://github.com/minsik-ai/Steerable-Cultural-Preference
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 23 references that are in the map. First posted on arXiv 17 Jun 2026.
Builds on
16 of 23 references.
Closest ancestors · 1
Same direction, newest first
Other references · 15
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
- 2023Whose Opinions Do Language Models Reflect?cited by 773
- 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
- 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
- 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
- 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
- 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 63
- 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
- 2024CultureLLM: Incorporating Cultural Differences into Large Language Modelscited by 50
3 more
- 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersionscited by 18
- 2023SuperHF: Supervised Iterative Learning from Human Feedbackcited by 10
- 2024OPTune: Efficient Online Preference Tuningcited by 10
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
- 2025Cultural Alignment in Large Language Models Using Soft Prompt Tuning
- 2023On Diversified Preferences of Large Language Model Alignment
- 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
- 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
- 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
- 2024CREAM: Consistency Regularized Self-Rewarding Language Models
- 2026Prompt Programming for Cultural Bias and Alignment of Large Language Models