# PL-Guard: Benchmarking Language Model Safety for Polish

arXiv: https://arxiv.org/abs/2506.16322 · HTML: https://mlsanity.com/paper/2506.16322/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 19 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts](https://mlsanity.com/paper/2402.16822/) (2024)
- [AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts](https://mlsanity.com/paper/2404.05993/) (2024)
- [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://mlsanity.com/paper/2005.00630/) (2020)
- [HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish](https://mlsanity.com/paper/2105.01735/) (2021)
- [PL-MTEB: Polish Massive Text Embedding Benchmark](https://mlsanity.com/paper/2405.10138/) (2024)
- [Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluation](https://mlsanity.com/paper/2410.18565/) (2024)
- [This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish](https://mlsanity.com/paper/2211.13112/) (2022)
- [LLMzSzL: a comprehensive LLM benchmark for Polish](https://mlsanity.com/paper/2501.02266/) (2025)
- [BAN-PL: a Novel Polish Dataset of Banned Harmful and Offensive Content from Wykop.pl web service](https://mlsanity.com/paper/2308.10592/) (2023)
- [Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models](https://mlsanity.com/paper/2506.07645/) (2025)
- [Evaluation of Few-Shot Learning for Classification Tasks in the Polish Language](https://mlsanity.com/paper/2404.17832/) (2024)
- [MultiSlav: Using Cross-Lingual Knowledge Transfer to Combat the Curse of Multilinguality](https://mlsanity.com/paper/2502.14509/) (2025)

## Built on by (most cited first, 2 of 2)

- [Bielik Guard: Efficient Polish Language Safety Classifiers for LLM Content Moderation](https://mlsanity.com/paper/2602.07954/) (2026)
- [Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier](https://mlsanity.com/paper/2609.29266/) (2026)

## Abstract (arXiv)

Despite increasing efforts to ensure the safety of large language models (LLMs), most existing safety assessments and moderation tools remain heavily biased toward English and other high-resource languages, leaving majority of global languages underexamined. To address this gap, we introduce a manually annotated benchmark dataset for language model safety classification in Polish. We also create adversarially perturbed variants of these samples designed to challenge model robustness. We conduct a series of experiments to evaluate LLM-based and classifier-based models of varying sizes and architectures. Specifically, we fine-tune three models: Llama-Guard-3-8B, a HerBERT-based classifier (a Polish BERT derivative), and PLLuM, a Polish-adapted Llama-8B model. We train these models using different combinations of annotated data and evaluate their performance, comparing it against publicly available guard models. Results demonstrate that the HerBERT-based classifier achieves the highest overall performance, particularly under adversarial conditions.