Data to 7 Oct 2026

German's Next Language Model

arXiv 2010.10906 ↗First posted 21 Oct 2020cs.CL cs.LGcited by 144

Not on the map. Why?

Abstract (arXiv)

In this work we present the experiments which lead to the creation of our BERT and ELECTRA based German language models, GBERT and GELECTRA. By varying the input training data, model size, and the presence of Whole Word Masking (WWM) we were able to attain SoTA performance across a set of document classification and named entity recognition (NER) tasks for both models of base and large size. We adopt an evaluation driven approach in training these models and our results indicate that both adding more data and utilizing WWM improve model performance. By benchmarking against existing German models, we show that these models are the best German models to date. Our trained models will be made publicly available to the research community.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 31; 2023: 27; 2024: 32; 2025: 19; 2026: 14.

144 papers.

Sort:

By area:

2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 89
2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
2021GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrievalcited by 39
2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
2022KinyaBERT: a Morphology-aware Kinyarwanda Language Modelcited by 27
2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26
2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24

38 more

2022Sign Language Video Retrieval with Free-Form Textual Queriescited by 21
2022Discovering Salient Neurons in Deep NLP Modelscited by 21
2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
2023Sabi'a: Portuguese Large Language Modelscited by 19
2023Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languagescited by 19
2021Small-Text: Active Learning for Text Classification in Pythoncited by 18
2022Do Multilingual Language Models Capture Differing Moral Norms?cited by 15
2022Changing the Representation: Examining Language Representation for Neural Sign Language Productioncited by 15
2021Automated Essay Scoring Using Transformer Modelscited by 14
2023ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domaincited by 14
2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13
2022Evaluation of Transfer Learning for Polish with a Text-to-Text Modelcited by 13
2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
2024German Text Embedding Clustering Benchmarkcited by 13
2024Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Datasetcited by 13
2022You Are What You Write: Preserving Privacy in the Era of Large Language Modelscited by 11
2023Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptationcited by 11
2023MEDBERT.de: A Comprehensive German BERT Model for the Medical Domaincited by 10
2024LL"aMmlein: Transparent, Compact and Competitive German-Only Language Models from Scratchcited by 9
2022'UFAL CorPipe at CRAC 2022: Effectivity of Multilingual Models for Coreference Resolutioncited by 8
2023SwissBERT: The Multilingual Language Model for Switzerlandcited by 8
2024Sebastian, Basti, Wastl?! Recognizing Named Entities in Bavarian Dialectal Datacited by 8
2024VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understandingcited by 8
2021FHAC at GermEval 2021: Identifying German toxic, engaging, and fact-claiming comments with ensemble learningcited by 7
2022Automatic Readability Assessment of German Sentences with Transformer Ensemblescited by 7
2023Conversational Tree Search: A New Hybrid Dialog Taskcited by 7
2024MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebankcited by 7
2024Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategycited by 7
2021MDAPT: Multilingual Domain Adaptive Pretraining in a Single Modelcited by 6
2021Introducing an Abusive Language Classification Framework for Telegram to Investigate the German Hater Communitycited by 6
2022Bias at a Second Glance: A Deep Dive into Bias for German Educational Peer-Review Data Modelingcited by 6
2022Transformers Go for the LOLs: Generating (Humourous) Titles from Scientific Abstracts End-to-Endcited by 6
2023German BERT Model for Legal Named Entity Recognitioncited by 6
2024Comparing Explanation Faithfulness between Multilingual and Monolingual Fine-tuned Language Modelscited by 6
2024A Survey of Large Language Models for European Languagescited by 6
2021Can BERT Dig It? -- Named Entity Recognition for Information Retrieval in the Archaeology Domaincited by 5
2021On the ability of monolingual models to learn language-agnostic representationscited by 5
2022RigoBERTa: A State-of-the-Art Language Model For Spanishcited by 5

50 most cited of 144 All 144 (JSON)