MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation
Not on the map. Why?
Abstract (arXiv)
We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 55 of its 96 references that are in the map. First posted on arXiv 24 Feb 2026.
Builds on
55 of 96 references.
References · 55
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
38 more
- 2023StarCoder: may the source be with you!cited by 790
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
- 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
- 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
- 2022Matryoshka Representation Learningcited by 268
- 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
- 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
- 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
- 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 134
- 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
- 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
- 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
- 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
- 2023MathPile: A Billion-Token-Scale Pretraining Corpus for Mathcited by 67
- 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56
- 2024Flextron: Many-in-One Flexible Large Language Modelcited by 42
- 2025mmBERT: A Modern Multilingual Encoder with Annealed Language Learningcited by 39
- 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
- 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 33
- 2021Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalancited by 26
- 2024Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Textscited by 26
- 2025BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLPcited by 26
- 2023Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*cited by 24
- 2021Operationalizing a National Digital Library: The Case for a Norwegian Transformer Modelcited by 23
- 2018Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabularycited by 18
- 2020Towards an Open Platform for Legal Informationcited by 18
- 2020The Danish Gigaword Projectcited by 17
- 2025PyLate: Flexible Training and Retrieval for Late Interaction Modelscited by 17
- 2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformerscited by 16
- 2021Multi-granular Legal Topic Classification on Greek Legislationcited by 15
- 2022Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltesecited by 14
- 2025SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compressioncited by 13
- 2025Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledgecited by 10
- 2024HydraViT: Stacking Heads for a Scalable ViTcited by 9
- 2018Word Embeddings from Large-Scale Greek Web Contentcited by 6
- 2025ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inferencecited by 6
50 most cited of 55 All 55 (JSON)
Built on by
Not cited yet.