mlsanityData to 13 Oct 2026

Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?

arXiv 2311.00268 ↗First posted 1 Nov 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

A line of work on Transformer-based language models such as BERT has attempted to use syntactic inductive bias to enhance the pretraining process, on the theory that building syntactic structure into the training process should reduce the amount of data needed for training. But such methods are often tested for high-resource languages such as English. In this work, we investigate whether these methods can compensate for data sparseness in low-resource languages, hypothesizing that they ought to be more effective for low-resource languages. We experiment with five low-resource languages: Uyghur, Wolof, Maltese, Coptic, and Ancient Greek. We find that these syntactic inductive bias methods produce uneven results in low-resource settings, and provide surprisingly little benefit in most cases.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

28 of 48 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,360
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
  7. 2018Deep contextualized word representationscited by 2,286
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,581
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 999
  10. 2020A Primer in BERTology: What we know about how BERT workscited by 809
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 792
  12. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 720
16 more
  1. 2019How multilingual is Multilingual BERT?cited by 643
  2. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 577
  3. 2019CamemBERT: a Tasty French Language Modelcited by 368
  4. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  5. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 88
  6. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  7. 2021How to Adapt Your Pretrained Multilingual Model to 1600 Languagescited by 54
  8. 2020Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Inductioncited by 47
  9. 2020Syntax-Enhanced Pre-trained Modelcited by 29
  10. 2021Pre-train or Annotate? Domain Adaptation with a Constrained Budgetcited by 27
  11. 2020Improving BERT with Syntax-aware Local Attentioncited by 26
  12. 2020On the importance of pre-training data volume for compact language modelscited by 21
  13. 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
  14. 2019LIMIT-BERT : Linguistic Informed Multi-Task BERTcited by 10
  15. 2020AMALGUM -- A Free, Balanced, Multilayer English Web Corpuscited by 9
  16. 2023PrOnto: Language Model Evaluations for 859 Languagescited by 1

Built on by

Not cited yet.