# GEITje 7B Ultra: A Conversational Model for Dutch

arXiv: https://arxiv.org/abs/2412.04092 · HTML: https://mlsanity.com/paper/2412.04092/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 9 of 9)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks](https://mlsanity.com/paper/2406.13469/) (2024)
- [ScandEval: A Benchmark for Scandinavian Natural Language Processing](https://mlsanity.com/paper/2304.00906/) (2023)
- [Language Resources for Dutch Large Language Modelling](https://mlsanity.com/paper/2312.12852/) (2023)

## Built on by (most cited first, 4 of 4)

- [Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructure](https://mlsanity.com/paper/2601.01244/) (2026)
- [LLMs vs. Traditional Sentiment Tools in Psychology: An Evaluation on Belgian-Dutch Narratives](https://mlsanity.com/paper/2511.07641/) (2025)
- [When Context Misleads: Surprisal, Energy and Attention Entropy as Metrics of Coherence Illusions in LLMs](https://mlsanity.com/paper/2606.21203/) (2026)
- [From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch](https://mlsanity.com/paper/2608.09925/) (2026)

## Abstract (arXiv)

Language models have rapidly evolved, predominantly focusing on English while often neglecting extensive pretraining in other languages. This approach has required initiatives to adapt powerful, English-centric models to other linguistic contexts through finetuning. For Dutch, such a recent endeavour is ``GEITje'' a model originally derived from the English-based Mistral 7B. Building on this fundamental work, the current research extends the capabilities of GEITje by supervised finetuning on newly created high-quality synthetic conversational datasets, along with an additional preference alignment procedure on a synthetic feedback dataset. Both the developed models and the created datasets are openly available.