# The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews

arXiv: https://arxiv.org/abs/1910.00896 · HTML: https://mlsanity.com/paper/1910.00896/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2019.

## Built on by (most cited first, 14 of 14)

- [SICKNL: A Dataset for Dutch Natural Language Inference](https://mlsanity.com/paper/2101.05716/) (2021)
- [Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation](https://mlsanity.com/paper/2310.03477/) (2023)
- [FairDistillation: Mitigating Stereotyping in Language Models](https://mlsanity.com/paper/2207.04546/) (2022)
- [DUMB: A Benchmark for Smart Evaluation of Dutch Models](https://mlsanity.com/paper/2305.13026/) (2023)
- [MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch](https://mlsanity.com/paper/2509.12340/) (2025)
- [Improving BERT Pretraining with Syntactic Supervision](https://mlsanity.com/paper/2104.10516/) (2021)
- [RobBERTje: a Distilled Dutch BERT Model](https://mlsanity.com/paper/2204.13511/) (2022)
- [Discontinuous Constituency and BERT: A Case Study of Dutch](https://mlsanity.com/paper/2203.01063/) (2022)
- [Machine Learning for Violence Risk Assessment Using Dutch Clinical Notes](https://mlsanity.com/paper/2204.13535/) (2022)
- [RobBERT-2022: Updating a Dutch Language Model to Account for Evolving Language Use](https://mlsanity.com/paper/2211.08192/) (2022)
- [Translating the Grievance Dictionary: a psychometric evaluation of Dutch, German, and Italian versions](https://mlsanity.com/paper/2505.07495/) (2025)
- [belabBERT: a Dutch RoBERTa-based language model applied to psychiatric classification](https://mlsanity.com/paper/2106.01091/) (2021)
- [Connecting Humanities and Social Sciences: Applying Language and Speech Technology to Online Panel Surveys](https://mlsanity.com/paper/2302.10593/) (2023)
- [Translation as a Computationally Efficient Bridge: Feasibility of English BERT for Low-Resource Languages](https://mlsanity.com/paper/2607.12612/) (2026)

## Abstract (arXiv)

We evaluated the effectiveness of using language models, that were pre-trained in one domain, as the basis for a classification model in another domain: Dutch book reviews. Pre-trained language models have opened up new possibilities for classification tasks with limited labelled data, because representation can be learned in an unsupervised fashion. In our experiments we have studied the effects of training set size (100-1600 items) on the prediction accuracy of a ULMFiT classifier, based on a language models that we pre-trained on the Dutch Wikipedia. We also compared ULMFiT to Support Vector Machines, which is traditionally considered suitable for small collections. We found that ULMFiT outperforms SVM for all training set sizes and that satisfactory results (~90%) can be achieved using training sets that can be manually annotated within a few hours. We deliver both our new benchmark collection of Dutch book reviews for sentiment classification as well as the pre-trained Dutch language model to the community.