# Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding

arXiv: https://arxiv.org/abs/1911.06156 · HTML: https://mlsanity.com/paper/1911.06156/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 20 of 20)

- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees](https://mlsanity.com/paper/2103.04350/) (2021)
- [Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale](https://mlsanity.com/paper/2203.00633/) (2022)
- [Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models](https://mlsanity.com/paper/2104.07578/) (2021)
- [Controlled Text Reduction](https://mlsanity.com/paper/2210.13449/) (2022)
- [Better Neural Machine Translation by Extracting Linguistic Information from BERT](https://mlsanity.com/paper/2104.02831/) (2021)
- [What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Study](https://mlsanity.com/paper/2109.09105/) (2021)
- [Semantics-aware Attention Improves Neural Machine Translation](https://mlsanity.com/paper/2110.06920/) (2021)
- [FCDS: Fusing Constituency and Dependency Syntax into Document-Level Relation Extraction](https://mlsanity.com/paper/2403.01886/) (2024)
- [Not Just Plain Text! Fuel Document-Level Relation Extraction with Explicit Syntax Refinement and Subsentence Modeling](https://mlsanity.com/paper/2211.05343/) (2022)
- [Semi-Supervised Few-Shot Intent Classification and Slot Filling](https://mlsanity.com/paper/2109.08754/) (2021)
- [Does Entity Abstraction Help Generative Transformers Reason?](https://mlsanity.com/paper/2201.01787/) (2022)
- [How do lexical semantics affect translation? An empirical study](https://mlsanity.com/paper/2201.00075/) (2021)
- [Debiasing Gender Bias in Information Retrieval Models](https://mlsanity.com/paper/2208.01755/) (2022)
- [Fine-Grained Causality Extraction From Natural Language Requirements Using Recursive Neural Tensor Networks](https://mlsanity.com/paper/2107.09980/) (2021)
- [Improving Downstream Task Performance by Treating Numbers as Entities](https://mlsanity.com/paper/2205.03559/) (2022)
- [Leveraging Grammar Induction for Language Understanding and Generation](https://mlsanity.com/paper/2410.04878/) (2024)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [A Syntax Aware BERT for Identifying Well-Formed Queries in a Curriculum Framework](https://mlsanity.com/paper/2208.09912/) (2022)
- [Pseudo-OOD training for robust language models](https://mlsanity.com/paper/2210.09132/) (2022)

## Abstract (arXiv)

Attention-based models have shown significant improvement over traditional algorithms in several NLP tasks. The Transformer, for instance, is an illustrative example that generates abstract representations of tokens inputted to an encoder based on their relationships to all tokens in a sequence. Recent studies have shown that although such models are capable of learning syntactic features purely by seeing examples, explicitly feeding this information to deep learning models can significantly enhance their performance. Leveraging syntactic information like part of speech (POS) may be particularly beneficial in limited training data settings for complex models such as the Transformer. We show that the syntax-infused Transformer with multiple features achieves an improvement of 0.7 BLEU when trained on the full WMT 14 English to German translation dataset and a maximum improvement of 1.99 BLEU points when trained on a fraction of the dataset. In addition, we find that the incorporation of syntax into BERT fine-tuning outperforms baseline on a number of downstream tasks from the GLUE benchmark.