# Slovene SuperGLUE Benchmark: Translation and Evaluation

arXiv: https://arxiv.org/abs/2202.04994 · HTML: https://mlsanity.com/paper/2202.04994/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 10 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2010.15925/) (2020)
- [jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models](https://mlsanity.com/paper/2003.02249/) (2020)
- [Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand](https://mlsanity.com/paper/2112.04139/) (2021)
- [FinEst BERT and CroSloEngual BERT: less is more in multilingual models](https://mlsanity.com/paper/2006.07890/) (2020)
- [How not to Lie with a Benchmark: Rearranging NLP Leaderboards](https://mlsanity.com/paper/2112.01342/) (2021)

## Built on by (most cited first, 7 of 7)

- [Sequence to sequence pretraining for a less-resourced Slovenian language](https://mlsanity.com/paper/2207.13988/) (2022)
- [PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguese](https://mlsanity.com/paper/2404.05333/) (2024)
- [Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction](https://mlsanity.com/paper/2507.09834/) (2025)
- [bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2306.02349/) (2023)
- [skLEP: A Slovak General Language Understanding Benchmark](https://mlsanity.com/paper/2506.21508/) (2025)
- [Unified Question Answering in Slovene](https://mlsanity.com/paper/2211.09159/) (2022)
- [Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation](https://mlsanity.com/paper/2511.17290/) (2025)

## Abstract (arXiv)

We present a Slovene combined machine-human translated SuperGLUE benchmark. We describe the translation process and problems arising due to differences in morphology and grammar. We evaluate the translated datasets in several modes: monolingual, cross-lingual, and multilingual, taking into account differences between machine and human translated training sets. The results show that the monolingual Slovene SloBERTa model is superior to massively multilingual and trilingual BERT models, but these also show a good cross-lingual performance on certain tasks. The performance of Slovene models still lags behind the best English models.