mlsanityData to 14 Oct 2026

NepPlural: A Human-Anchored Benchmark for Persona Annotation in Nepali Migration Discourse

arXiv 2610.15732 ↗First posted 11 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pluralistic alignment asks that language models faithfully represent a society's diverse, competing values rather than collapse them into one dominant viewpoint, yet flagship pluralistic-alignment datasets remain heavily skewed toward Western respondents. We introduce NepPlural, a human-anchored benchmark for modelling competing socio-cultural personas in Nepali public discourse on youth migration and brain drain. We curate 2,148 code-mixed (Devanagari, Romanized Nepali, English) YouTube comments, annotate them along a four-axis persona taxonomy (Intent, Primary Driver, Value Orientation, Affect) plus an explicit No-Persona class, and build a 300-comment gold set labelled independently by three native Nepali speakers (mean Fleiss' kappa ≈ 0.55). Using this gold set to audit automatic annotation, we find that LLM annotation reliability is driven by protocol. Labels collected in batches through chat interfaces reach only near-chance inter-model agreement (kappa ≈ 0.17). Annotating one comment at a time raises agreement above the native-speaker level (kappa ≈ 0.72), but consistency is not validity: under a forced-choice schema, models agree on personas for the 39% of comments native speakers judge to express none, keeping macro-F1 against human gold near 0.50. Allowing abstention resolves this: models recover No-Persona comments with F1 of 0.79 to 0.87 and, against leave-one-annotator-out references, match or exceed individual native speakers (macro-F1 0.75 to 0.86 vs. 0.75). Encoders fine-tuned on the batched labels reach only 0.34. Finally, a human-judged steerability probe finds that three frontier models adopt single persona axes with 59 to 77% fidelity but compose full personas only 33% of the time, with no congruence gap detected at this scale between Western-congruent and Nepali-collectivist personas.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

23 of 38 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  5. 2023Whose Opinions Do Language Models Reflect?cited by 773
  6. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  7. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  8. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
  9. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 149
  10. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  11. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
  12. 2024The Ethics of Advanced AI Assistantscited by 107
11 more
  1. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  2. 2024How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensionscited by 52
  3. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 46
  4. 2025Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerabilitycited by 20
  5. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitationscited by 11
  6. 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 10
  7. 2025Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regressioncited by 7
  8. 2024Development of Pre-Trained Transformer-based Models for the Nepali Languagecited by 6
  9. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 6
  10. 2025Histoires Morales: A French Dataset for Assessing Moral Alignmentcited by 4
  11. 2024Consolidating and Developing Benchmarking Datasets for the Nepali Natural Language Understanding Taskscited by 3

Built on by

Not cited yet.