mlsanityData to 7 Oct 2026

Training and Evaluation of a Multilingual Tokenizer for GPT-SW3

arXiv 2304.14780 ↗First posted 28 Apr 2023cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper provides a detailed discussion of the multilingual tokenizer used for GPT-SW3. It was trained on the Nordic Pile using the SentencePiece library and the BPE algorithm. We outline the tokenizer's most important features and share details on its learned vocabulary. In addition, we systematically analyze the properties and evaluate the performance of the tokenizer with regard to the different languages present in the data.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 7 of its 9 references that are in the map. First posted on arXiv 28 Apr 2023.

Builds on

7 of 9 references.

References · 7

  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  6. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  7. 2023The Nordic Pile: A 1.2TB Nordic Dataset for Language Modelingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 0; 2025: 2; 2026: 1.

7 papers.

  1. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  2. 2023Efficiently Adapting Pretrained Language Models To New Languagescited by 28
  3. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  4. 2023Continual Learning Under Language Shiftcited by 9
  5. 2023Core Building Blocks: Next Gen Geo Spatial GPT Applicationcited by 4
  6. 2025How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysiscited by 3
  7. 2026TREX: Tokenizer Regression for Optimal Data Mixturecited by 0