# Enhancing Machine Translation with Dependency-Aware Self-Attention

arXiv: https://arxiv.org/abs/1909.03149 · HTML: https://mlsanity.com/paper/1909.03149/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2019.

## Built on by (most cited first, 24 of 24)

- [CTC Alignments Improve Autoregressive Translation](https://mlsanity.com/paper/2210.05200/) (2022)
- [CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Tasks](https://mlsanity.com/paper/2310.14623/) (2023)
- [Does injecting linguistic structure into language models lead to better alignment with brain recordings?](https://mlsanity.com/paper/2101.12608/) (2021)
- [Syntax-augmented Multilingual BERT for Cross-lingual Transfer](https://mlsanity.com/paper/2106.02134/) (2021)
- [Headed-Span-Based Projective Dependency Parsing](https://mlsanity.com/paper/2108.04750/) (2021)
- [RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarization](https://mlsanity.com/paper/2405.00657/) (2024)
- [Controlled Text Reduction](https://mlsanity.com/paper/2210.13449/) (2022)
- [Semantics-aware Attention Improves Neural Machine Translation](https://mlsanity.com/paper/2110.06920/) (2021)
- [Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages](https://mlsanity.com/paper/2208.06061/) (2022)
- [ChatGPT is a Potential Zero-Shot Dependency Parser](https://mlsanity.com/paper/2310.16654/) (2023)
- [On the Relation between Syntactic Divergence and Zero-Shot Performance](https://mlsanity.com/paper/2110.04644/) (2021)
- [Tree-Planted Transformers: Unidirectional Transformer Language Models with Implicit Syntactic Supervision](https://mlsanity.com/paper/2402.12691/) (2024)
- [Syntax-guided Localized Self-attention by Constituency Syntactic Distance](https://mlsanity.com/paper/2210.11759/) (2022)
- [Context-Aware Machine Translation with Source Coreference Explanation](https://mlsanity.com/paper/2404.19505/) (2024)
- [Enhancing the Transformer Decoder with Transition-based Syntax](https://mlsanity.com/paper/2101.12640/) (2021)
- [Deps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network](https://mlsanity.com/paper/2111.11707/) (2021)
- [An Effective Method using Phrase Mechanism in Neural Machine Translation](https://mlsanity.com/paper/2308.10482/) (2023)
- [Leveraging Grammar Induction for Language Understanding and Generation](https://mlsanity.com/paper/2410.04878/) (2024)
- [Combining Improvements for Exploiting Dependency Trees in Neural Semantic Parsing](https://mlsanity.com/paper/2112.13179/) (2021)
- [Schema-Free Dependency Parsing via Sequence Generation](https://mlsanity.com/paper/2201.12407/) (2022)
- [Rethinking Relation Classification with Graph Meaning Representations](https://mlsanity.com/paper/2310.09772/) (2023)
- [Advancing Bangla Machine Translation Through Informal Datasets](https://mlsanity.com/paper/2512.13487/) (2025)
- [Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs](https://mlsanity.com/paper/2602.15846/) (2026)
- [Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillation](https://mlsanity.com/paper/2603.11342/) (2026)

## Abstract (arXiv)

Most neural machine translation models only rely on pairs of parallel sentences, assuming syntactic information is automatically learned by an attention mechanism. In this work, we investigate different approaches to incorporate syntactic knowledge in the Transformer model and also propose a novel, parameter-free, dependency-aware self-attention mechanism that improves its translation quality, especially for long sentences and in low-resource scenarios. We show the efficacy of each approach on WMT English-German and English-Turkish, and WAT English-Japanese translation tasks.