# Syntax-Directed Attention for Neural Machine Translation

arXiv: https://arxiv.org/abs/1711.04231 · HTML: https://mlsanity.com/paper/1711.04231/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2017.

## Built on by (most cited first, 10 of 10)

- [Discriminative Reasoning for Document-level Relation Extraction](https://mlsanity.com/paper/2106.01562/) (2021)
- [Structural Guidance for Transformer Language Models](https://mlsanity.com/paper/2108.00104/) (2021)
- [Hybrid Routing Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2203.15310/) (2022)
- [Improving Long Context Document-Level Machine Translation](https://mlsanity.com/paper/2306.05183/) (2023)
- [Explicit Syntactic Guidance for Neural Text Generation](https://mlsanity.com/paper/2306.11485/) (2023)
- [On the Relation between Syntactic Divergence and Zero-Shot Performance](https://mlsanity.com/paper/2110.04644/) (2021)
- [Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-sentence Dependency Graph](https://mlsanity.com/paper/2112.00503/) (2021)
- [Enhancing the Transformer Decoder with Transition-based Syntax](https://mlsanity.com/paper/2101.12640/) (2021)
- [Deps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network](https://mlsanity.com/paper/2111.11707/) (2021)
- [Synergizing Machine Learning & Symbolic Methods: A Survey on Hybrid Approaches to Natural Language Processing](https://mlsanity.com/paper/2401.11972/) (2024)

## Abstract (arXiv)

Attention mechanism, including global attention and local attention, plays a key role in neural machine translation (NMT). Global attention attends to all source words for word prediction. In comparison, local attention selectively looks at fixed-window source words. However, alignment weights for the current target word often decrease to the left and right by linear distance centering on the aligned source position and neglect syntax-directed distance constraints. In this paper, we extend local attention with syntax-distance constraint, to focus on syntactically related source words with the predicted target word, thus learning a more effective context vector for word prediction. Moreover, we further propose a double context NMT architecture, which consists of a global context vector and a syntax-directed context vector over the global attention, to provide more translation performance for NMT from source representation. The experiments on the large-scale Chinese-to-English and English-to-Germen translation tasks show that the proposed approach achieves a substantial and significant improvement over the baseline system.