Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLP
Not on the map. Why?
Abstract (arXiv)
Humanitarian organizations face a critical choice: invest in costly commercial APIs or rely on free open-weight models for multilingual human rights monitoring. While commercial systems offer reliability, open-weight alternatives lack empirical validation -- especially for low-resource languages common in conflict zones. This paper presents the first systematic comparison of commercial and open-weight large language models (LLMs) for human-rights-violation detection across seven languages, quantifying the cost-reliability trade-off facing resource-constrained organizations. Across 78,000 multilingual inferences, we evaluate six models -- four instruction-aligned (Claude-Sonnet-4, DeepSeek-V3, Gemini-Flash-2.0, GPT-4.1-mini) and two open-weight (LLaMA-3-8B, Mistral-7B) -- using both standard classification metrics and new measures of cross-lingual reliability: Calibration Deviation (CD), Decision Bias (B), Language Robustness Score (LRS), and Language Stability Score (LSS). Results show that alignment, not scale, determines stability: aligned models maintain near-invariant accuracy and balanced calibration across typologically distant and low-resource languages (e.g., Lingala, Burmese), while open-weight models exhibit significant prompt-language sensitivity and calibration drift. These findings demonstrate that multilingual alignment enables language-agnostic reasoning and provide practical guidance for humanitarian organizations balancing budget constraints with reliability in multilingual deployment.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 26 Oct 2025.
Builds on
16 of 28 references.
References · 16
- 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
- 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 149
- 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 97
- 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
- 2020From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformerscited by 35
- 2021Unpacking the Interdependent Systems of Discrimination: Ableist Bias in NLP Systems through an Intersectional Lenscited by 30
- 2023Mitigating Biases for Instruction-following Language Models via Bias Neurons Eliminationcited by 23
- 2025Multiple LLM Agents Debate for Equitable Cultural Alignmentcited by 21
- 2024On Large Language Models in National Security Applicationscited by 19
- 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
- 2024Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Modelscited by 11
- 2024Soft Prompt Tuning for Cross-Lingual Transfer: When Less is Morecited by 7
4 more
- 2023Optimizing National Security Strategies through LLM-Driven Artificial Intelligence Integrationcited by 6
- 2023A New Task and Dataset on Detecting Attacks on Human Rights Defenderscited by 4
- 2025Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecastingcited by 3
- 2025Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reportscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.