# GREEK-BERT: The Greeks visiting Sesame Street

arXiv: https://arxiv.org/abs/2008.12014 · HTML: https://mlsanity.com/paper/2008.12014/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2020.

## Built on by (most cited first, 30 of 38)

- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses](https://mlsanity.com/paper/2104.14694/) (2021)
- [KinyaBERT: a Morphology-aware Kinyarwanda Language Model](https://mlsanity.com/paper/2203.08459/) (2022)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [Multi-granular Legal Topic Classification on Greek Legislation](https://mlsanity.com/paper/2109.15298/) (2021)
- [Bringing order into the realm of Transformer-based language models for artificial intelligence and law](https://mlsanity.com/paper/2308.05502/) (2023)
- [SsciBERT: A Pre-trained Language Model for Social Science Texts](https://mlsanity.com/paper/2206.04510/) (2022)
- [SlovakBERT: Slovak Masked Language Model](https://mlsanity.com/paper/2109.15254/) (2021)
- [Exploring Large Language Models for Classical Philology](https://mlsanity.com/paper/2305.13698/) (2023)
- [GreekBART: The First Pretrained Greek Sequence-to-Sequence Model](https://mlsanity.com/paper/2304.00869/) (2023)
- [Bertinho: Galician BERT Representations](https://mlsanity.com/paper/2103.13799/) (2021)
- [Don't Touch My Diacritics](https://mlsanity.com/paper/2410.24140/) (2024)
- [Scalar Adjective Identification and Multilingual Ranking](https://mlsanity.com/paper/2105.01180/) (2021)
- [PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors](https://mlsanity.com/paper/2108.01139/) (2021)
- [An Analysis of Social Biases Present in BERT Variants Across Multiple Languages](https://mlsanity.com/paper/2211.14402/) (2022)
- [Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages](https://mlsanity.com/paper/2305.02215/) (2023)
- [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024)
- [Zero-shot Large Language Models for Automatic Readability Assessment](https://mlsanity.com/paper/2604.24470/) (2026)
- [Sort by Structure: Language Model Ranking as Dependency Probing](https://mlsanity.com/paper/2206.04935/) (2022)
- [Cross-Lingual Knowledge Transfer for Clinical Phenotyping](https://mlsanity.com/paper/2208.01912/) (2022)
- [GreekT5: A Series of Greek Sequence-to-Sequence Models for News Summarization](https://mlsanity.com/paper/2311.07767/) (2023)
- [Dialect Normalization using Large Language Models and Morphological Rules](https://mlsanity.com/paper/2506.08907/) (2025)
- [Linguistic Cues of Deception in a Multilingual April Fools' Day Context](https://mlsanity.com/paper/2111.03913/) (2021)
- [Multilingual Name Entity Recognition and Intent Classification Employing Deep Learning Architectures](https://mlsanity.com/paper/2211.02415/) (2022)
- [OYXOY: A Modern NLP Test Suite for Modern Greek](https://mlsanity.com/paper/2309.07009/) (2023)
- [A Systematic Survey of Natural Language Processing for the Greek Language](https://mlsanity.com/paper/2407.09861/) (2024)
- [Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation](https://mlsanity.com/paper/2510.20002/) (2025)
- [Analysing the Greek Parliament Records with Emotion Classification](https://mlsanity.com/paper/2205.12012/) (2022)

## Abstract (arXiv)

Transformer-based language models, such as BERT and its variants, have achieved state-of-the-art performance in several downstream natural language processing (NLP) tasks on generic benchmark datasets (e.g., GLUE, SQUAD, RACE). However, these models have mostly been applied to the resource-rich English language. In this paper, we present GREEK-BERT, a monolingual BERT-based language model for modern Greek. We evaluate its performance in three NLP tasks, i.e., part-of-speech tagging, named entity recognition, and natural language inference, obtaining state-of-the-art performance. Interestingly, in two of the benchmarks GREEK-BERT outperforms two multilingual Transformer-based models (M-BERT, XLM-R), as well as shallower neural baselines operating on pre-trained word embeddings, by a large margin (5%-10%). Most importantly, we make both GREEK-BERT and our training code publicly available, along with code illustrating how GREEK-BERT can be fine-tuned for downstream NLP tasks. We expect these resources to boost NLP research and applications for modern Greek.