mlsanityData to 14 Oct 2026

New Encoders for German Trained from Scratch: Comparing ModernGBERT with Converted LLM2Vec Models

arXiv 2505.13136 ↗First posted 19 May 2025cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Encoders remain essential for efficient German NLP and NLU scenarios despite the rise of decoder-only LLMs. This work studies two routes to high-quality German encoders under identical data and training constraints: 1) training from scratch and 2) converting decoders via LLM2Vec. We introduce two resources: ModernGBERT (134M, 1B), fully transparent German encoders in the ModernBERT style, and LL"aMmleinVec (120M, 1B, 7B), decoder-to-encoder conversions trained with masked next-token prediction, both undergoing a context extension to 8.192 tokens. Across SuperGLEBer, ModernGBERT 1B sets a new state of the art (avg 0.808), surpassing GBERT Large (+4%) and the seven-times larger converted 7B model (0.787). On German MTEB after supervised fine-tuning, ModernGBERT 1B (0.551) approaches the converted 7B model (0.557). We release all models, checkpoints, datasets, and full training records, and introduce an encoder-adapted QA-NIAH evaluation. All in all, our results provide actionable guidance: when parameter efficiency and latency matter, from-scratch encoders dominate. When a pre-trained decoder exists and compute is a limited, conversion offers an effective alternative. ModernGBERT and LL"aMmleinVec, including all code, data and intermediary checkpoints are published under a research-only RAIL license.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 30 of its 55 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

30 of 55 references.

References · 30

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  7. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  8. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  9. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  11. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  12. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
18 more
  1. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
  2. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  3. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  4. 2020German's Next Language Modelcited by 144
  5. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 87
  6. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  7. 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 45
  8. 2021GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrievalcited by 39
  9. 2023BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddingscited by 32
  10. 2025EuroBERT: Scaling Multilingual Encoders for European Languagescited by 31
  11. 2024Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labelingcited by 24
  12. 2021Cross-Market Product Recommendationcited by 22
  13. 2025DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrieverscited by 19
  14. 2024CamemBERT 2.0: A Smarter French Language Model Aged to Perfectioncited by 14
  15. 2025ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performancecited by 12
  16. 2023On the Impact of Cross-Domain Data on German Language Modelscited by 5
  17. 2025MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilitiescited by 5
  18. 2025llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Lengthcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkishcited by 1
  2. 2026Understanding or Memorizing? A Case Study of German Definite Articles in Language Modelscited by 1
  3. 2026Embedded Conditional Independence Tests for Large Language Model Generated Text with an Application to German Parliament Speechescited by 1
  4. 2026The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLPcited by 0
  5. 2026moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERTcited by 0