# Give your Text Representation Models some Love: the Case for Basque

arXiv: https://arxiv.org/abs/2004.00033 · HTML: https://mlsanity.com/paper/2004.00033/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2020.

## Built on by (most cited first, 28 of 28)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Testing the Predictions of Surprisal Theory in 11 Languages](https://mlsanity.com/paper/2307.03667/) (2023)
- [A Primer on Pretrained Multilingual Language Models](https://mlsanity.com/paper/2107.00676/) (2021)
- [Latxa: An Open Language Model and Evaluation Suite for Basque](https://mlsanity.com/paper/2403.20266/) (2024)
- [Lost in Translation: Large Language Models in Non-English Content Analysis](https://mlsanity.com/paper/2306.07377/) (2023)
- [Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan](https://mlsanity.com/paper/2107.07903/) (2021)
- [Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltese](https://mlsanity.com/paper/2205.10517/) (2022)
- [A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models](https://mlsanity.com/paper/2201.05601/) (2022)
- [Does Corpus Quality Really Matter for Low-Resource Languages?](https://mlsanity.com/paper/2203.08111/) (2022)
- [Semi-automatic Generation of Multilingual Datasets for Stance Detection in Twitter](https://mlsanity.com/paper/2101.11978/) (2021)
- [Multilingual Counter Narrative Type Classification](https://mlsanity.com/paper/2109.13664/) (2021)
- [Lessons learned from the evaluation of Spanish Language Models](https://mlsanity.com/paper/2212.08390/) (2022)
- [Bertinho: Galician BERT Representations](https://mlsanity.com/paper/2103.13799/) (2021)
- [Linguistically inspired roadmap for building biologically reliable protein language models](https://mlsanity.com/paper/2207.00982/) (2022)
- [Social Analysis of Young Basque Speaking Communities in Twitter](https://mlsanity.com/paper/2109.03487/) (2021)
- [Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations](https://mlsanity.com/paper/2307.13776/) (2023)
- [On the Role of Morphological Information for Contextual Lemmatization](https://mlsanity.com/paper/2302.00407/) (2023)
- [XNLIeu: a dataset for cross-lingual NLI in Basque](https://mlsanity.com/paper/2404.06996/) (2024)
- [COMBO: State-of-the-Art Morphosyntactic Analysis](https://mlsanity.com/paper/2109.05361/) (2021)
- [BasqueParl: A Bilingual Corpus of Basque Parliamentary Transcriptions](https://mlsanity.com/paper/2205.01506/) (2022)
- [Assessment of Pre-Trained Models Across Languages and Grammars](https://mlsanity.com/paper/2309.11165/) (2023)
- [Lost in Variation? Evaluating NLI Performance in Basque and Spanish Geographical Variants](https://mlsanity.com/paper/2506.15239/) (2025)
- [Evaluating Shortest Edit Script Methods for Contextual Lemmatization](https://mlsanity.com/paper/2403.16968/) (2024)
- [LySACoru~na at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing](https://mlsanity.com/paper/2204.12820/) (2022)
- [EuSQuAD: Automatically Translated and Aligned SQuAD2.0 for Basque](https://mlsanity.com/paper/2404.12177/) (2024)
- [Euskarazko lehen C1 ebaluatzaile automatikoa](https://mlsanity.com/paper/2503.01851/) (2025)
- [BERnaT: Basque Encoders for Representing Natural Textual Diversity](https://mlsanity.com/paper/2512.03903/) (2025)
- [A Catalog of Basque Dialectal Resources: Online Collections and Standard-to-Dialectal Adaptations](https://mlsanity.com/paper/2603.25189/) (2026)

## Abstract (arXiv)

Word embeddings and pre-trained language models allow to build rich representations of text and have enabled improvements across most NLP tasks. Unfortunately they are very expensive to train, and many small companies and research groups tend to use models that have been pre-trained and made available by third parties, rather than building their own. This is suboptimal as, for many languages, the models have been trained on smaller (or lower quality) corpora. In addition, monolingual pre-trained models for non-English languages are not always available. At best, models for those languages are included in multilingual versions, where each language shares the quota of substrings and parameters with the rest of the languages. This is particularly true for smaller languages such as Basque. In this paper we show that a number of monolingual models (FastText word embeddings, FLAIR and BERT language models) trained with larger Basque corpora produce much better results than publicly available versions in downstream NLP tasks, including topic classification, sentiment classification, PoS tagging and NER. This work sets a new state-of-the-art in those tasks for Basque. All benchmarks and models used in this work are publicly available.