# Syntactic Knowledge via Graph Attention with BERT in Machine Translation

arXiv: https://arxiv.org/abs/2305.13413 · HTML: https://mlsanity.com/paper/2305.13413/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [Towards Making the Most of BERT in Neural Machine Translation](https://mlsanity.com/paper/1908.05672/) (2019)
- [SG-Net: Syntax Guided Transformer for Language Representation](https://mlsanity.com/paper/2012.13915/) (2020)

## Abstract (arXiv)

Although the Transformer model can effectively acquire context features via a self-attention mechanism, deeper syntactic knowledge is still not effectively modeled. To alleviate the above problem, we propose Syntactic knowledge via Graph attention with BERT (SGB) in Machine Translation (MT) scenarios. Graph Attention Network (GAT) and BERT jointly represent syntactic dependency feature as explicit knowledge of the source language to enrich source language representations and guide target language generation. Our experiments use gold syntax-annotation sentences and Quality Estimation (QE) model to obtain interpretability of translation quality improvement regarding syntactic knowledge without being limited to a BLEU score. Experiments show that the proposed SGB engines improve translation quality across the three MT tasks without sacrificing BLEU scores. We investigate what length of source sentences benefits the most and what dependencies are better identified by the SGB engines. We also find that learning of specific dependency relations by GAT can be reflected in the translation quality containing such relations and that syntax on the graph leads to new modeling of syntactic aspects of source sentences in the middle and bottom layers of BERT.