# Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMs

arXiv: https://arxiv.org/abs/2510.14565 · HTML: https://mlsanity.com/paper/2510.14565/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 62 references that are in the map.
- First posted on arXiv 16 Oct 2025.

## Builds on (most cited first, 30 of 37)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [On Protecting the Data Privacy of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2403.05156/) (2024)
- [KMMLU: Measuring Massive Multitask Language Understanding in Korean](https://mlsanity.com/paper/2402.11548/) (2024)
- [What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers](https://mlsanity.com/paper/2109.04650/) (2021)
- [AceGPT, Localizing Large Language Models in Arabic](https://mlsanity.com/paper/2309.12053/) (2023)
- [SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models](https://mlsanity.com/paper/2305.11840/) (2023)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation](https://mlsanity.com/paper/2010.01717/) (2020)
- [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022)
- [Multilingual Language Models are not Multicultural: A Case Study in Emotion](https://mlsanity.com/paper/2307.01370/) (2023)
- [EXAONE 3.5: Series of Large Language Models for Real-world Use Cases](https://mlsanity.com/paper/2412.04862/) (2024)
- [Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense](https://mlsanity.com/paper/2405.04655/) (2024)
- [NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly](https://mlsanity.com/paper/2210.08604/) (2022)
- [Multi-lingual and Multi-cultural Figurative Language Understanding](https://mlsanity.com/paper/2305.16171/) (2023)
- [Distract Large Language Models for Automatic Jailbreak Attack](https://mlsanity.com/paper/2403.08424/) (2024)
- [A Skeleton-Based Model for Promoting Coherence Among Sentences in Narrative Story Generation](https://mlsanity.com/paper/1808.06945/) (2018)
- [COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances](https://mlsanity.com/paper/2311.01012/) (2023)
- [CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge](https://mlsanity.com/paper/2404.06664/) (2024)
- [Global Voices, Local Biases: Socio-Cultural Prejudices across Languages](https://mlsanity.com/paper/2310.17586/) (2023)

## Built on by (most cited first, 2 of 2)

- [SEATauBench: Progressively Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages](https://mlsanity.com/paper/2606.28715/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

Recent trends in LLMs development clearly show growing interest in the use and application of sovereign LLMs. The global debate over sovereign LLMs highlights the need for governments to develop their LLMs, tailored to their unique socio-cultural and historical contexts. However, there remains a shortage of frameworks and datasets to verify two critical questions: (1) how well these models align with users' socio-cultural backgrounds, and (2) whether they maintain safety and technical robustness without exposing users to potential harms and risks. To address this gap, we construct a new dataset and introduce an analytic framework for extracting and evaluating the socio-cultural elements of sovereign LLMs, alongside assessments of their technical robustness. Our experimental results demonstrate that while sovereign LLMs play a meaningful role in supporting low-resource languages, they do not always meet the popular claim that these models serve their target users well. We also show that pursuing this untested claim may lead to underestimating critical quality attributes such as safety. Our study suggests that advancing sovereign LLMs requires a more extensive evaluation that incorporates a broader range of well-grounded and practical criteria.