mlsanityData to 14 Oct 2026

Syntax-augmented Multilingual BERT for Cross-lingual Transfer

arXiv 2106.02134 ↗First posted 3 Jun 2021cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In recent years, we have seen a colossal effort in pre-training multilingual text encoders using large-scale corpora in many languages to facilitate cross-lingual transfer learning. However, due to typological differences across languages, the cross-lingual transfer is challenging. Nevertheless, language syntax, e.g., syntactic dependencies, can bridge the typological gap. Previous works have shown that pre-trained multilingual encoders, such as mBERT citedevlin-etal-2019-bert, capture language syntax, helping cross-lingual transfer. This work shows that explicitly providing language syntax and training mBERT using an auxiliary objective to encode the universal dependency tree structure helps cross-lingual transfer. We perform rigorous experiments on four NLP tasks, including text classification, question answering, named entity recognition, and task-oriented semantic parsing. The experiment results show that syntax-augmented mBERT improves cross-lingual transfer on popular benchmarks, such as PAWS-X and MLQA, by 1.4 and 1.6 points on average across all languages. In the generalized transfer setting, the performance boosted significantly, with 3.9 and 3.1 points on average in PAWS-X and MLQA.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2017Graph Attention Networkscited by 8,923
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  8. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  10. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  11. 2019How multilingual is Multilingual BERT?cited by 645
  12. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
20 more
  1. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  2. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  3. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  4. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  5. 2019BERT for Joint Intent Classification and Slot Fillingcited by 196
  6. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  7. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
  8. 2017Simple and Effective Multi-Paragraph Reading Comprehensioncited by 120
  9. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 110
  10. 2020End-to-End Slot Alignment and Recognition for Cross-Lingual NLUcited by 83
  11. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  12. 2019SG-Net: Syntax-Guided Machine Reading Comprehensioncited by 52
  13. 2020Do Syntax Trees Help Pre-trained Transformers Extract Information?cited by 48
  14. 2020GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extractioncited by 43
  15. 2019Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networkscited by 40
  16. 2019Enhancing Machine Translation with Dependency-Aware Self-Attentioncited by 24
  17. 2019Self-Attention with Structural Position Representationscited by 21
  18. 2017Improved Neural Machine Translation with a Syntax-Aware Encoder and Decodercited by 16
  19. 2019Syntax-Enhanced Self-Attention-Based Semantic Role Labelingcited by 8
  20. 2019Source Dependency-Aware Transformer with Supervised Self-Attentioncited by 2

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 5; 2024: 2.

11 papers.

  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  2. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  3. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  4. 2023Contextual Label Projection for Cross-Lingual Structured Predictioncited by 9
  5. 2023Syntax and Domain Aware Model for Unsupervised Program Translationcited by 7
  6. 2023Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizercited by 7
  7. 2022A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learningcited by 4
  8. 2023Self-Augmentation Improves Zero-Shot Cross-Lingual Transfercited by 3
  9. 2023Cross-Lingual Transfer of Cognitive Processing Complexitycited by 2
  10. 2024Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfercited by 1
  11. 2022A Knowledge-Enhanced Adversarial Model for Cross-lingual Structured Sentiment Analysiscited by 0