mlsanityData to 7 Oct 2026

DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values

arXiv 2509.08022 ↗First posted 9 Sep 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Aligning large language models (LLMs) with diverse human values is essential for safe and effective deployment, yet existing benchmarks often overlook cultural and demographic variation. We introduce DiverValue-Bench, a population-aware benchmark for evaluating multi-dimensional value alignment across 74 countries/regions. It contains 23,763 quality-controlled instances derived from PRISM user feedback and audited through large-scale human validation, with fine-grained value labels, personalized questions, contrastive reference answers, and rich demographic metadata. Using DiverValue-Bench, we evaluate representative LLMs and reveal substantial geographic and demographic disparities that are masked by aggregate performance. We further show that lightweight preference-based fine-tuning with Low-Rank Adaptation (LoRA) and Direct Preference Optimization (DPO) substantially improves in-domain value alignment while yielding consistent out-of-domain gains. These results highlight the need for population-aware alignment evaluation and demonstrate the utility of DiverValue-Bench as a practical foundation for global alignment, personalized value modeling, and equitable AI development.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 37 references that are in the map. First posted on arXiv 9 Sep 2025.

Builds on

31 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment19 Mar
  2. 2024MAP: Multi-Human-Value Alignment Palette24 Oct
  3. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignment24 Jul
  4. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models24 Apr
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  6. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback9 Mar
  7. 2022In conversation with Artificial Intelligence: aligning language models with human values1 Sep
  8. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning30 Dec

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  12. 2023Large Language Model Alignment: A Surveycited by 241
11 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  3. 2023Self-Alignment with Instruction Backtranslationcited by 149
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  5. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  6. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  7. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  8. 2024The Political Preferences of LLMscited by 83
  9. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  10. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  11. 2023Large Language Models are Built-in Autoregressive Search Enginescited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models
  2. 2026VC-Soup: Value-Consistency Guided Multi-Value Alignment for Large Language Models
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  4. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models
  5. 2025Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation
  6. 2026VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment
  7. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMs
  8. 2025Cultural Alignment in Large Language Models Using Soft Prompt Tuning