# Pretraining Finnish ModernBERTs

arXiv: https://arxiv.org/abs/2511.09213 · HTML: https://mlsanity.com/paper/2511.09213/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2025.

## Built on by (most cited first, 1 of 1)

- [Looking for Affect in Spontaneous Finnish Speech through Linguistic Interpretability](https://mlsanity.com/paper/2607.24155/) (2026)

## Abstract (arXiv)

This paper reports on pretraining ModernBERT encoder models in six different sizes, ranging from 51M to 475M parameters, with a focus on limited multilingualism, emphasizing languages relevant to Finland. Our models are competitive with, or superior to, existing multilingual models. They outperform monolingual models on tasks that require a context longer than 512 tokens. We present empirical results on using different data in the final stage of training. The code and models are publicly released.