# Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLP

arXiv: https://arxiv.org/abs/2510.22823 · HTML: https://mlsanity.com/paper/2510.22823/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 26 Oct 2025.

## Builds on (most cited first, 16 of 16)

- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs](https://mlsanity.com/paper/2410.12405/) (2024)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [The Missing Ingredient in Zero-Shot Neural Machine Translation](https://mlsanity.com/paper/1903.07091/) (2019)
- [From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers](https://mlsanity.com/paper/2005.00633/) (2020)
- [Unpacking the Interdependent Systems of Discrimination: Ableist Bias in NLP Systems through an Intersectional Lens](https://mlsanity.com/paper/2110.00521/) (2021)
- [Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination](https://mlsanity.com/paper/2311.09627/) (2023)
- [Multiple LLM Agents Debate for Equitable Cultural Alignment](https://mlsanity.com/paper/2505.24671/) (2025)
- [On Large Language Models in National Security Applications](https://mlsanity.com/paper/2407.03453/) (2024)
- [Cross-lingual Transfer of Reward Models in Multilingual Alignment](https://mlsanity.com/paper/2410.18027/) (2024)
- [Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models](https://mlsanity.com/paper/2402.02099/) (2024)
- [Soft Prompt Tuning for Cross-Lingual Transfer: When Less is More](https://mlsanity.com/paper/2402.03782/) (2024)
- [Optimizing National Security Strategies through LLM-Driven Artificial Intelligence Integration](https://mlsanity.com/paper/2305.13927/) (2023)
- [A New Task and Dataset on Detecting Attacks on Human Rights Defenders](https://mlsanity.com/paper/2306.17695/) (2023)
- [Do Large Language Models Know Conflict? Investigating Parametric vs. Non-Parametric Knowledge of LLMs for Conflict Forecasting](https://mlsanity.com/paper/2505.09852/) (2025)
- [Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reports](https://mlsanity.com/paper/2505.10586/) (2025)

## Built on by (most cited first, 1 of 1)

- [SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses](https://mlsanity.com/paper/2607.02049/) (2026)

## Abstract (arXiv)

Humanitarian organizations face a critical choice: invest in costly commercial APIs or rely on free open-weight models for multilingual human rights monitoring. While commercial systems offer reliability, open-weight alternatives lack empirical validation -- especially for low-resource languages common in conflict zones. This paper presents the first systematic comparison of commercial and open-weight large language models (LLMs) for human-rights-violation detection across seven languages, quantifying the cost-reliability trade-off facing resource-constrained organizations. Across 78,000 multilingual inferences, we evaluate six models -- four instruction-aligned (Claude-Sonnet-4, DeepSeek-V3, Gemini-Flash-2.0, GPT-4.1-mini) and two open-weight (LLaMA-3-8B, Mistral-7B) -- using both standard classification metrics and new measures of cross-lingual reliability: Calibration Deviation (CD), Decision Bias (B), Language Robustness Score (LRS), and Language Stability Score (LSS). Results show that alignment, not scale, determines stability: aligned models maintain near-invariant accuracy and balanced calibration across typologically distant and low-resource languages (e.g., Lingala, Burmese), while open-weight models exhibit significant prompt-language sensitivity and calibration drift. These findings demonstrate that multilingual alignment enables language-agnostic reasoning and provide practical guidance for humanitarian organizations balancing budget constraints with reliability in multilingual deployment.