mlsanityData to 7 Oct 2026

A Survey of Large Language Models for European Languages

arXiv 2408.15040 ↗First posted 27 Aug 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have gained significant attention due to their high performance on a wide range of natural language tasks since the release of ChatGPT. The LLMs learn to understand and generate language by training billions of model parameters on vast volumes of text data. Despite being a relatively new field, LLM research is rapidly advancing in various directions. In this paper, we present an overview of LLM families, including LLaMA, PaLM, GPT, and MoE, and the methods developed to create and enhance LLMs for official European Union (EU) languages. We provide a comprehensive summary of common monolingual and multilingual datasets used for pretraining large language models.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 100 of its 156 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

100 of 156 references.

References · 100

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  11. 2024Mixtral of Expertscited by 1,785
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  13. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  14. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  15. 2023PaLM 2 Technical Reportcited by 1,154
  16. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  17. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  18. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  19. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  20. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  21. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  22. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  23. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  24. 2023A Comprehensive Overview of Large Language Modelscited by 639
  25. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  26. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  27. 2023The Falcon Series of Open Language Modelscited by 526
  28. 2024Large Language Models: A Surveycited by 506
  29. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  30. 2019CamemBERT: a Tasty French Language Modelcited by 368
  31. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  32. 2023Explainability for Large Language Models: A Surveycited by 331
  33. 2022UL2: Unifying Language Learning Paradigmscited by 292
  34. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  35. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  36. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  37. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  38. 2020German's Next Language Modelcited by 144

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 19
  2. 2024Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysiscited by 11
  3. 2026Racka: Efficient Hungarian LLM Adaptation on Academic Infrastructurecited by 1
  4. 2025Cross-Cultural Fashion Design via Interactive Large Language Models and Diffusion Modelscited by 0
  5. 2026Towards AI Evaluation in Domain-Specific RAG Systems: The AgriHubi Case Studycited by 0
  6. 2026Language Models for Portuguese: A Systematic Mapping Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
  2. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performance
  3. 2025Multilingual Language Model Pretraining using Machine-translated Data
  4. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
  5. 2024Exploring Advanced Large Language Models with LLMsuite
  6. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
  7. 2024Multilingual Large Language Models: A Systematic Survey
  8. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages