mlsanityData to 7 Oct 2026

GPT-SW3: An Autoregressive Language Model for the Nordic Languages

arXiv 2305.12987 ↗First posted 22 May 2023cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper details the process of developing the first native large generative language model for the Nordic languages, GPT-SW3. We cover all parts of the development process, from data collection and processing, training configuration and instruction finetuning, to evaluation and considerations for release strategies. We hope that this paper can serve as a guide and reference for other researchers that undertake the development of large generative models for smaller languages.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

34 of 57 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
22 more
  1. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  2. 2022Galactica: A Large Language Model for Sciencecited by 736
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  9. 2016FastText.zip: Compressing text classification modelscited by 331
  10. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  11. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  12. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  13. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  14. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  15. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  16. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  17. 2023The Gradient of Generative AI Release: Methods and Considerationscited by 63
  18. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  19. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  20. 2021Operationalizing a National Digital Library: The Case for a Norwegian Transformer Modelcited by 23
  21. 2020The Danish Gigaword Projectcited by 17
  22. 2023The Nordic Pile: A 1.2TB Nordic Dataset for Language Modelingcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 3.

8 papers.

  1. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  2. 2024How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLMs?cited by 13
  3. 2023Continual Learning Under Language Shiftcited by 9
  4. 2023The Less the Merrier? Investigating Language Representation in Multilingual Modelscited by 2
  5. 2024Open Llama2 Model for the Lithuanian Languagecited by 2
  6. 2025Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian Historycited by 1
  7. 2025The American Ghost in the Machine: How language models align culturally and the effects of cultural promptingcited by 1
  8. 2025DeepSeek's WEIRD Behavior: The cultural alignment of Large Language Models and the effects of prompt language and cultural promptingcited by 0