# Baszta: Data-Centric Fine-Tuning of a Polish Multi-Label Safety Classifier

arXiv: https://arxiv.org/abs/2609.29266 · HTML: https://mlsanity.com/paper/2609.29266/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Qwen3Guard Technical Report](https://mlsanity.com/paper/2510.14276/) (2025)
- [PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages](https://mlsanity.com/paper/2504.04377/) (2025)
- [KLEJ: Comprehensive Benchmark for Polish Language Understanding](https://mlsanity.com/paper/2005.00630/) (2020)
- [HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish](https://mlsanity.com/paper/2105.01735/) (2021)
- [Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models](https://mlsanity.com/paper/2206.09917/) (2022)
- [PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods](https://mlsanity.com/paper/2402.13350/) (2024)
- [Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval](https://mlsanity.com/paper/2208.00511/) (2022)
- [BAN-PL: a Novel Polish Dataset of Banned Harmful and Offensive Content from Wykop.pl web service](https://mlsanity.com/paper/2308.10592/) (2023)
- [PL-Guard: Benchmarking Language Model Safety for Polish](https://mlsanity.com/paper/2506.16322/) (2025)
- [Bielik Guard: Efficient Polish Language Safety Classifiers for LLM Content Moderation](https://mlsanity.com/paper/2602.07954/) (2026)
- [Multilingual Refusal Alignment for Safer Large Language Models](https://mlsanity.com/paper/2606.07535/) (2026)
- [Detection of Criminal Texts for the Polish State Border Guard](https://mlsanity.com/paper/2108.10580/) (2021)
- [Behind Closed Words: Creating and Investigating the forePLay Annotated Dataset for Polish Erotic Discourse](https://mlsanity.com/paper/2412.17533/) (2024)

## Abstract (arXiv)

We develop a multi-label Polish content-safety classifier by fine-tuning allegro/herbert-base-cased (124M) across five categories (hate, vulgarity, sexual content, crime, self-harm) using a Focal + R-Drop objective, and evaluate the resulting model against Bielik Guard (S'ojka) on the shared out-of-distribution Gadzi Jkezyk benchmark. Both systems are given per-category threshold tuning on the same calibration split. Under that matched protocol our model holds a small but statistically significant lead in micro F1, while an apparent macro-F1 lead does not survive: it was an artifact of comparing a tuned model against an untuned one. We also report what that micro figure is worth. Because Gadzi Jkezyk is 97% crime-positive, a classifier that flags crime on every input and nothing else already scores 0.910 micro F1 on the same test split, so micro separates neither system from a degenerate strategy and macro is the column that does. Per-category and per-protocol figures are reported in Section 4. The residual out-of-distribution gap is one of calibration rather than discrimination. Ranking quality stays high while positive probabilities collapse, and per-category temperature scaling recovers the loss where Platt scaling and isotonic regression do not. That recovery turns out to be conditional on the calibration set containing safe text. Gadzi Jkezyk contains almost none, so thresholds fitted on it flag crime on every safe input, and a balanced refit buys a deployable operating point at the cost of adversarial recall. We report both operating points rather than only the flattering one. Two changes that are standard practice, per-class cost-sensitive weighting and mean pooling, each raise in-distribution macro F1 while lowering the out-of-distribution figure, which indicates that robustness has to be selected for directly rather than inherited from in-distribution accuracy.