# Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese arXiv: https://arxiv.org/abs/2205.10517 ยท HTML: https://mlsanity.com/paper/2205.10517/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 41 references that are in the map. - First posted on arXiv 21 May 2022. ## Builds on (most cited first, 25 of 25) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019) - [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019) - [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020) - [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019) - [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016) - [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019) - [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020) - [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016) - [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020) - [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019) - [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021) - [Are All Languages Created Equal in Multilingual BERT?](https://mlsanity.com/paper/2005.09093/) (2020) - [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://mlsanity.com/paper/2010.12858/) (2020) - [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019) - [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://mlsanity.com/paper/2012.15562/) (2020) - [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019) - [75 Languages, 1 Model: Parsing Universal Dependencies Universally](https://mlsanity.com/paper/1904.02099/) (2019) - [Extending Multilingual BERT to Low-Resource Languages](https://mlsanity.com/paper/2004.13640/) (2020) - [Changing the World by Changing the Data](https://mlsanity.com/paper/2105.13947/) (2021) - [Give your Text Representation Models some Love: the Case for Basque](https://mlsanity.com/paper/2004.00033/) (2020) - [Specializing Multilingual Language Models: An Empirical Study](https://mlsanity.com/paper/2106.09063/) (2021) - [Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank](https://mlsanity.com/paper/2009.14124/) (2020) ## Built on by (most cited first, 14 of 14) - [Lost in Translation: Large Language Models in Non-English Content Analysis](https://mlsanity.com/paper/2306.07377/) (2023) - [MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning](https://mlsanity.com/paper/2212.12510/) (2022) - [A Survey of Large Language Models for European Languages](https://mlsanity.com/paper/2408.15040/) (2024) - [Cross-Lingual Transfer from Related Languages: Treating Low-Resource Maltese as Multilingual Code-Switching](https://mlsanity.com/paper/2401.16895/) (2024) - [Disentangling Singlish Discourse Particles with Task-Driven Representation](https://mlsanity.com/paper/2409.20366/) (2024) - [Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?](https://mlsanity.com/paper/2310.17271/) (2023) - [MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP](https://mlsanity.com/paper/2506.04385/) (2025) - [XDoGE: Multilingual Data Reweighting to Enhance Language Inclusivity in LLMs](https://mlsanity.com/paper/2512.10545/) (2025) - [Data Processing for the OpenGPT-X Model Family](https://mlsanity.com/paper/2410.08800/) (2024) - [From Measurement to Mitigation: Exploring the Transferability of Debiasing Approaches to Gender Bias in Maltese Language Models](https://mlsanity.com/paper/2507.03142/) (2025) - [Data Augmentation for Maltese NLP using Transliterated and Machine Translated Arabic Data](https://mlsanity.com/paper/2509.12853/) (2025) - [Enhancing BERT Fine-Tuning for Sentiment Analysis in Lower-Resourced Languages](https://mlsanity.com/paper/2512.01460/) (2025) - [MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation](https://mlsanity.com/paper/2602.21379/) (2026) - [LV-ROVER-MLT: Low-Resource Maltese OCR by Synthetic Fine-Tuning and Multi-Stream Arbitration](https://mlsanity.com/paper/2607.00250/) (2026) ## Abstract (arXiv) Multilingual language models such as mBERT have seen impressive cross-lingual transfer to a variety of languages, but many languages remain excluded from these models. In this paper, we analyse the effect of pre-training with monolingual data for a low-resource language that is not included in mBERT -- Maltese -- with a range of pre-training set ups. We conduct evaluations with the newly pre-trained models on three morphosyntactic tasks -- dependency parsing, part-of-speech tagging, and named-entity recognition -- and one semantic classification task -- sentiment analysis. We also present a newly created corpus for Maltese, and determine the effect that the pre-training data size and domain have on the downstream performance. Our results show that using a mixture of pre-training domains is often superior to using Wikipedia text only. We also find that a fraction of this corpus is enough to make significant leaps in performance over Wikipedia-trained models. We pre-train and compare two models on the new corpus: a monolingual BERT model trained from scratch (BERTu), and a further pre-trained multilingual BERT (mBERTu). The models achieve state-of-the-art performance on these tasks, despite the new corpus being considerably smaller than typically used corpora for high-resourced languages. On average, BERTu outperforms or performs competitively with mBERTu, and the largest gains are observed for higher-level tasks.