# Qorgau: Evaluating LLM Safety in Kazakh-Russian Bilingual Contexts

arXiv: https://arxiv.org/abs/2502.13640 · HTML: https://mlsanity.com/paper/2502.13640/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 6 of 6)

- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?](https://mlsanity.com/paper/2404.14397/) (2024)
- [MERA: A Comprehensive LLM Evaluation in Russian](https://mlsanity.com/paper/2401.04531/) (2024)
- [Arabic Dataset for LLM Safeguard Evaluation](https://mlsanity.com/paper/2410.17040/) (2024)
- [Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling](https://mlsanity.com/paper/2311.18711/) (2023)
- [KazSAnDRA: Kazakh Sentiment Analysis Dataset of Reviews and Attitudes](https://mlsanity.com/paper/2403.19335/) (2024)

## Built on by (most cited first, 2 of 2)

- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [To Lie or Not to Lie? Investigating The Biased Spread of Global Lies by LLMs](https://mlsanity.com/paper/2604.06552/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are known to have the potential to generate harmful content, posing risks to users. While significant progress has been made in developing taxonomies for LLM risks and safety evaluation prompts, most studies have focused on monolingual contexts, primarily in English. However, language- and region-specific risks in bilingual contexts are often overlooked, and core findings can diverge from those in monolingual settings. In this paper, we introduce Qorgau, a novel dataset specifically designed for safety evaluation in Kazakh and Russian, reflecting the unique bilingual context in Kazakhstan, where both Kazakh (a low-resource language) and Russian (a high-resource language) are spoken. Experiments with both multilingual and language-specific LLMs reveal notable differences in safety performance, emphasizing the need for tailored, region-specific datasets to ensure the responsible and safe deployment of LLMs in countries like Kazakhstan. Warning: this paper contains example data that may be offensive, harmful, or biased.