mlsanityData to 14 Oct 2026

SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models

arXiv 2605.25420 ↗First posted 25 May 2026cs.CL cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language model safety evaluation remains heavily English-centered, leaving low-resource languages under-measured even when models are deployed globally. We evaluate four open-weight instruction-tuned models on SomaliBench v0, a native-author-verified benchmark of 100 harmful-intent prompts paired across English and Somali. Each of Llama-3.1-8B-Instruct, Gemma-2-9B-Instruct, Qwen-2.5-7B-Instruct, and Aya-23-8B is run locally with temperature 0 and the same English "helpful, harmless, and honest" (HHH) system prompt. We find large English-to-Somali refusal gaps for all four models, ranging from 0.40 to 0.93, all strictly positive under a paired bootstrap and significant by exact McNemar tests. For three models, the dominant Somali non-refusal mode is not fluent harmful compliance but unclear output: wrong-language, incoherent, or off-topic generations. A pinned Claude Sonnet snapshot (claude-sonnet-4-5-20250929) classifies each response as refused, complied, or unclear; its safety layer declined 34 of 800 classifications, which the native author labeled manually. A native-author spot-check achieves 100% agreement with the judge (Cohen's kappa=1.00) on 74 comparable rows. We report aggregate refusal rates, category gaps, and reliability statistics only; raw model generations are retained locally and are not released because some may contain harmful content.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 24 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

11 of 24 references.

References · 11

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  8. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  9. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  10. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  11. 2026SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmarkcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Arabic Safety Alignment as Selective Refusal: An Empirical Study of SFT, DPO, and Guard Calibration
  2. 2026LSR: Linguistic Safety Robustness Benchmark for Low-Resource West African Languages
  3. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
  4. 2026Latent Space Refusal Anchoring for Low-Resource African Languages: Mechanistic Safety Recovery Without Retraining
  5. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models
  6. 2026The Illusion of Cross-Lingual Safety in Low-Resource Languages
  7. 2026Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language Models
  8. 2026TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages