mlsanityData to 14 Oct 2026

MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning

arXiv 2212.12510 ↗First posted 23 Dec 2022cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Transformer language models (TLMs) are critical for most NLP tasks, but they are difficult to create for low-resource languages because of how much pretraining data they require. In this work, we investigate two techniques for training monolingual TLMs in a low-resource setting: greatly reducing TLM size, and complementing the masked language modeling objective with two linguistically rich supervised tasks (part-of-speech tagging and dependency parsing). Results from 7 diverse languages indicate that our model, MicroBERT, is able to produce marked improvements in downstream task evaluations relative to a typical monolingual TLM pretraining approach. Specifically, we find that monolingual MicroBERT models achieve gains of up to 18% for parser LAS and 11% for NER F1 compared to a multilingual baseline, mBERT, while having less than 1% of its parameter count. We conclude reducing TLM parameter count and using labeled data for pretraining low-resource TLMs can yield large quality benefits and in some cases produce models that outperform multilingual approaches.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 23 Dec 2022.

Builds on

33 of 48 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2016Enriching Word Vectors with Subword Informationcited by 1,583
21 more
  1. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  2. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  3. 2019How multilingual is Multilingual BERT?cited by 645
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  5. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  6. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  7. 2019CamemBERT: a Tasty French Language Modelcited by 368
  8. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  9. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  10. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  11. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  12. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  13. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  14. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  15. 2017Transfer Learning across Low-Resource, Related Languages for Neural Machine Translationcited by 58
  16. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  17. 2020On the importance of pre-training data volume for compact language modelscited by 21
  18. 2021Specializing Multilingual Language Models: An Empirical Studycited by 21
  19. 2020Contextual Embeddings: When Are They Worth It?cited by 15
  20. 2022Pre-training Data Quality and Quantity for a Low-Resource Language: New Corpus and BERT Models for Maltesecited by 14
  21. 2021Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?cited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 3; 2025: 1; 2026: 2.

12 papers.

  1. 2024Natural Language Processing RELIES on Linguisticscited by 9
  2. 2023Robust Generalization Strategies for Morpheme Glossing in an Endangered Language Documentation Contextcited by 4
  3. 2023SIGMORPHON 2023 Shared Task of Interlinear Glossing: Baseline Modelcited by 3
  4. 2023PrOnto: Language Model Evaluations for 859 Languagescited by 1
  5. 2023FonMTL: Towards Multitask Learning for the Fon Languagecited by 1
  6. 2025A UD Treebank for Bohairic Copticcited by 1
  7. 2026AdNanny: One Reasoning LLM for All Offline Ads Recommendation Taskscited by 1
  8. 2023Taxonomic Loss for Morphological Glossing of Low-Resource Languagescited by 0
  9. 2023Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?cited by 0
  10. 2024Transferring BERT Capabilities from High-Resource to Low-Resource Languages Using Vocabulary Matchingcited by 0
  11. 2024Classifier identification in Ancient Egyptian as a low-resource sequence-labelling taskcited by 0
  12. 2026Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languagescited by 0