mlsanityData to 14 Oct 2026

Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark

arXiv 2608.30065 ↗First posted 30 Aug 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Large language models (LLMs) demonstrate strong Helpfulness, Harmlessness, and Honesty (3H) alignment in English-centric settings, but these gains transfer poorly to low-resource languages due to cultural mismatches. Existing multilingual 3H benchmarks rely predominantly on automated translation or LLM based synthesis, propagating source-language biases while sacrificing local relevance. To address this gap, we introduce Pak3H1, the first human-validated, culturally contextualized Urdu benchmark suite for 3H alignment, comprising PakAlpaca (helpfulness), PakBeaverTails (harmlessness), and PakTruthfulQA (honesty). Our multi-stage pipeline integrates manual cultural adaptation and dictionary-guided post editing to prioritize native speaker judgment, ensuring both semantic fidelity and contextual authenticity. Zero-shot evaluations across multiple open and proprietary LLM architectures reveal systematic cross-lingual alignment gaps: helpfulness win rates decline under localized contexts, harmlessness guardrails break down against regional safety risks, and composite honesty metrics degrade substantially due to localized factual constraints. These findings expose structural limitations in current alignment approaches, underscoring the necessity of human-guided localization for equitable multilingual evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 11 of its 18 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

11 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr

Other references · 10

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  3. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  4. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  5. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  6. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  7. 2025Too Helpful, Too Harmless, Too Honest or Just Right?cited by 9
  8. 2025Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examplescited by 6
  9. 2025Aligning Large Language Models to Low-Resource Languages through LLM-Based Selective Translation: A Systematic Studycited by 4
  10. 2025PakBBQ: A Culturally Adapted Bias Benchmark for QAcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop
  2. 2023CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models
  3. 2025Alif: Advancing Urdu Large Language Models via Multilingual Synthetic Data Distillation
  4. 2025ELAB: Extensive LLM Alignment Benchmark in Persian Language
  5. 2026AlignCultura: Towards Culturally Aligned Large Language Models?
  6. 2026CoCoA: Context-Conditional Cultural Alignment for Large Language Models
  7. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  8. 2024Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning