mlsanityData to 7 Oct 2026

Poro 34B and the Blessing of Multilinguality

arXiv 2404.01856 ↗First posted 2 Apr 2024cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

The pretraining of state-of-the-art large language models now requires trillions of words of text, which is orders of magnitude more than available for the vast majority of languages. While including text in more than one language is an obvious way to acquire more pretraining data, multilinguality is often seen as a curse, and most model training efforts continue to focus near-exclusively on individual large languages. We believe that multilinguality can be a blessing: when the lack of training data is a constraint for effectively training larger models for a target language, augmenting the dataset with other languages can offer a way to improve over the capabilities of monolingual models for that language. In this study, we introduce Poro 34B, a 34 billion parameter model trained for 1 trillion tokens of Finnish, English, and programming languages, and demonstrate that a multilingual training approach can produce a model that substantially advances over the capabilities of existing models for Finnish and excels in translation, while also achieving competitive performance in its class for English and programming languages. We release the model parameters, scripts, and data under open licenses at https://huggingface.co/LumiOpen/Poro-34B.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 47 of its 72 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

47 of 72 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
35 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  11. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  12. 2023The Falcon Series of Open Language Modelscited by 526
  13. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  14. 2023Scaling Data-Constrained Language Modelscited by 391
  15. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  16. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  17. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  18. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  19. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  20. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  21. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  22. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  23. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  24. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  25. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  26. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  27. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  28. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  29. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  30. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  31. 2023FinGPT: Large Generative Models for a Small Languagecited by 49
  32. 2023Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?cited by 39
  33. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  34. 2020WikiBERT models: deep transfer learning for many languagescited by 20
  35. 2023Position Interpolation Improves ALiBi Extrapolationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 5; 2026: 5.

18 papers.

Sort:

By area:

  1. 2025SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakescited by 27
  2. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 16
  3. 2024LiCoEval: Evaluating LLMs on License Compliance in Code Generationcited by 13
  4. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  5. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  6. 2025OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunchescited by 10
  7. 2024A Survey of Large Language Models for European Languagescited by 6
  8. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  9. 2024LLMs for Extremely Low-Resource Finno-Ugric Languagescited by 3
  10. 2025Got Compute, but No Data: Lessons From Post-training a Finnish LLMcited by 3
  11. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  12. 2024LLMs' morphological analyses of complex FST-generated Finnish wordscited by 1
6 more
  1. 2026EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Trainingcited by 1
  2. 2024GECKO: Generative Language Model for English, Code and Koreancited by 0
  3. 2025FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Modelscited by 0
  4. 2026BYOL: Bring Your Own Language Into LLMscited by 0
  5. 2026Towards AI Evaluation in Domain-Specific RAG Systems: The AgriHubi Case Studycited by 0
  6. 2026On the limited utility of parallel data for learning shared multilingual representationscited by 0