mlsanityData to 7 Oct 2026

EuroBERT: Scaling Multilingual Encoders for European Languages

arXiv 2503.05500 ↗First posted 7 Mar 2025cs.CL cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 49 of its 71 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

49 of 71 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2020GLU Variants Improve Transformercited by 1,724
37 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,425
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  6. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  12. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  13. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  14. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  15. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  16. 2023Llemma: An Open Language Model For Mathematicscited by 385
  17. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  18. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
  19. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  20. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  21. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  22. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  23. 2020The Multilingual Amazon Reviews Corpuscited by 142
  24. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  25. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  26. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  27. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  28. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
  29. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51
  30. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  31. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  32. 2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 26
  33. 2021PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretrainingcited by 21
  34. 2022Learning Better Masking for Better Language Model Pre-trainingcited by 14
  35. 2022On the Role of Parallel Data in Cross-lingual Transfer Learningcited by 11
  36. 2023Dynamic Masking Rate Schedules for MLM Pretrainingcited by 11
  37. 2025MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Trainingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 21.

28 papers.

Sort:

By area:

  1. 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 39
  2. 2026jina-embeddings-v5-text: Task-Targeted Embedding Distillationcited by 19
  3. 2025Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoringcited by 6
  4. 2025New Encoders for German Trained from Scratch: Comparing ModernGBERT with Converted LLM2Vec Modelscited by 5
  5. 2025SindBERT, the Sailor: Charting the Seas of Turkish NLPcited by 4
  6. 2026Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptationcited by 4
  7. 2025Multi-Lingual Implicit Discourse Relation Recognition with Multi-Label Hierarchical Learningcited by 1
  8. 2025TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkishcited by 1
  9. 2026One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Modelscited by 1
  10. 2026Understanding or Memorizing? A Case Study of German Definite Articles in Language Modelscited by 1
  11. 2026Long-Context Encoder Models for Polish Language Understandingcited by 1
  12. 2026When Do Concepts Become Functionally Sufficient During Language-Model Training?cited by 1
16 more
  1. 2026Scaling Unsupervised Word Alignment to Documents via Structural Constraintscited by 1
  2. 2025GHaLIB: A Multilingual Framework for Hope Speech Detection in Low-Resource Languagescited by 0
  3. 2026Large-Scale Aspect-Based Sentiment Analysis with Reasoning-Infused LLMscited by 0
  4. 2026EmbeddingRWKV: State-Centric Retrieval with Reusable Statescited by 0
  5. 2026Curate-Train-Refine: A Closed-Loop Agentic Framework for Zero Shot Classificationcited by 0
  6. 2026Leveraging LLMs For Turkish Skill Extractioncited by 0
  7. 2026FEAST: Retrieval-Augmented Multi-Hierarchical Food Classification for the FoodEx2 Systemcited by 0
  8. 2026Is She Even Relevant? When BERT Ignores Explicit Gender Cuescited by 0
  9. 2026ML-Embed: Inclusive and Efficient Embeddings for a Multilingual Worldcited by 0
  10. 2026LEDGER: A Long-Context Benchmark of Corporate Annual Reports for Grounded Financial Retrieval and Extractioncited by 0
  11. 2026Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretrainingcited by 0
  12. 2026moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERTcited by 0
  13. 2026ToxiREX: A Dataset on Toxic REasoning in ConteXtcited by 0
  14. 2026Polish ModernBERT: The Long and Short of Polish Language Understandingcited by 0
  15. 2026Online Language Adaptive Sampling for Better Distributed Cross-lingual Gainscited by 0
  16. 2026N"urnberg NLP at ChildSafeAds 2026: Structurally Dissimilar Voter Ensembles under Four Levels of Data Accesscited by 0