mlsanityData to 7 Oct 2026

Semantics-aware Attention Improves Neural Machine Translation

arXiv 2110.06920 ↗First posted 13 Oct 2021cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic structures. In this work we propose two novel parameter-free methods for injecting semantic information into Transformers, both rely on semantics-aware masking of (some of) the attention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs. We further show an additional gain when using both semantic and syntactic structures in some language pairs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

22 of 45 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  6. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  7. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  8. 2019Semantic Neural Machine Translation using AMRcited by 37
  9. 2018Deep RNNs Encode Soft Hierarchical Syntaxcited by 34
  10. 2018Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networkscited by 30
  11. 2018Semantic Structural Evaluation for Text Simplificationcited by 30
  12. 2019Enhancing Machine Translation with Dependency-Aware Self-Attentioncited by 24
10 more
  1. 2018Simple and Effective Text Simplification Using Semantic and Neural Methodscited by 20
  2. 2019Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understandingcited by 20
  3. 2019Syntactically Supervised Transformers for Faster Neural Machine Translationcited by 19
  4. 2017Nematus: a Toolkit for Neural Machine Translationcited by 16
  5. 2015Building Subject-aligned Comparable Corpora and Mining it for Truly Parallel Sentence Pairscited by 15
  6. 2019Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translationcited by 15
  7. 2021Mediators in Determining what Processing BERT Performs Firstcited by 14
  8. 2017A Transition-Based Directed Acyclic Graph Parser for UCCAcited by 13
  9. 2020Fine-Grained Analysis of Cross-Linguistic Syntactic Divergencescited by 9
  10. 2021On the Difficulty of Translating Free-Order Case-Marking Languagescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 1; 2024: 0; 2025: 0; 2026: 1.

5 papers.

  1. 2021Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modelingcited by 10
  2. 2022Controlled Text Reductioncited by 10
  3. 2022Cross-domain Generalization for AMR Parsingcited by 2
  4. 2023Enhancing Neural Machine Translation with Semantic Unitscited by 2
  5. 2026Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillationcited by 0