mlsanityData to 14 Oct 2026

Small Languages, Big Models: A Study of Continual Training on Languages of Norway

arXiv 2412.06484 ↗First posted 9 Dec 2024cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages like Northern S'ami. To address this issue, we present a novel three-stage continual training approach that substantially improves the downstream performance together with the inference efficiency for the target languages. Based on our findings, we train, evaluate, and openly release a new generative language model for Norwegian Bokmral, Nynorsk, and Northern S'ami with 11.4 billion parameters: NorMistral-11B.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 47 of its 75 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

47 of 75 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
35 more
  1. 2020GLU Variants Improve Transformercited by 1,804
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  8. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  9. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  10. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  11. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  12. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  13. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 128
  14. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  15. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  16. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  17. 2018Universal Neural Machine Translation for Extremely Low Resource Languagescited by 66
  18. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  19. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  20. 2023FinGPT: Large Generative Models for a Small Languagecited by 49
  21. 2023How are Prompts Different in Terms of Sensitivity?cited by 35
  22. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 28
  23. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  24. 2024GPT or BERT: why not both?cited by 27
  25. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  26. 2021Operationalizing a National Digital Library: The Case for a Norwegian Transformer Modelcited by 23
  27. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  28. 2021Large-Scale Contextualised Language Modelling for Norwegiancited by 21
  29. 2024Poro 34B and the Blessing of Multilingualitycited by 18
  30. 2017NoReC: The Norwegian Review Corpuscited by 17
  31. 2024BERTs are Generative In-Context Learnerscited by 12
  32. 2023NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegiancited by 9
  33. 2023NorQuAD: Norwegian Question Answering Datasetcited by 7
  34. 2024ENTP: Encoder-only Next Token Predictioncited by 5
  35. 2024Towards a More Inclusive AI: Progress and Perspectives in Large Language Model Training for the S'ami Languagecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 4.

12 papers.

  1. 2024The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspectivecited by 8
  2. 2025NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmarkcited by 8
  3. 2025Grow Up and Merge: Scaling Strategies for Efficient Language Adaptationcited by 3
  4. 2026Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedishcited by 3
  5. 2025Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articlescited by 2
  6. 2025A Collection of Question Answering Datasets for Norwegiancited by 2
  7. 2025Multi-label Scandinavian Language Identification (SLIDE)cited by 2
  8. 2025Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Modelscited by 2
  9. 2026EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Trainingcited by 1
  10. 2025Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languagescited by 0
  11. 2026Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Trainingcited by 0
  12. 2026To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMscited by 0