# NorwAI's Large Language Models: Technical Report

arXiv: https://arxiv.org/abs/2601.03034 · HTML: https://mlsanity.com/paper/2601.03034/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Exploring Human-Like Translation Strategy with Large Language Models](https://mlsanity.com/paper/2305.04118/) (2023)
- [EPIC: An Energy-Efficient, High-Performance GPGPU Computing Research Infrastructure](https://mlsanity.com/paper/1912.05848/) (2019)
- [Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model](https://mlsanity.com/paper/2104.09617/) (2021)
- [NorBench -- A Benchmark for Norwegian Language Models](https://mlsanity.com/paper/2305.03880/) (2023)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian](https://mlsanity.com/paper/2312.01314/) (2023)
- [The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective](https://mlsanity.com/paper/2412.09460/) (2024)
- [NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark](https://mlsanity.com/paper/2504.07749/) (2025)

## Abstract (arXiv)

Norwegian, spoken by approximately five million people, remains underrepresented in many of the most significant breakthroughs in Natural Language Processing (NLP). To address this gap, the NorLLM team at NorwAI has developed a family of models specifically tailored to Norwegian and other Scandinavian languages, building on diverse Transformer-based architectures such as GPT, Mistral, Llama2, Mixtral and Magistral. These models are either pretrained from scratch or continually pretrained on 25B - 88.45B tokens, using a Norwegian-extended tokenizer and advanced post-training strategies to optimize performance, enhance robustness, and improve adaptability across various real-world tasks. Notably, instruction-tuned variants (e.g., Mistral-7B-Instruct and Mixtral-8x7B-Instruct) showcase strong assistant-style capabilities, underscoring their potential for practical deployment in interactive and domain-specific applications. The NorwAI large language models are openly available to Nordic organizations, companies and students for both research and experimental use. This report provides detailed documentation of the model architectures, training data, tokenizer design, fine-tuning strategies, deployment, and evaluations.