# Self-Attention with Structural Position Representations

arXiv: https://arxiv.org/abs/1909.00383 · HTML: https://mlsanity.com/paper/1909.00383/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2019.

## Built on by (most cited first, 21 of 21)

- [Pure Transformers are Powerful Graph Learners](https://mlsanity.com/paper/2207.02505/) (2022)
- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [Transformer for Graphs: An Overview from Architecture Perspective](https://mlsanity.com/paper/2202.08455/) (2022)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding](https://mlsanity.com/paper/2106.02795/) (2021)
- [Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction](https://mlsanity.com/paper/2110.09681/) (2021)
- [Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model](https://mlsanity.com/paper/2109.05244/) (2021)
- [Graph Attention Networks with Positional Embeddings](https://mlsanity.com/paper/2105.04037/) (2021)
- [STable: Table Generation Framework for Encoder-Decoder Models](https://mlsanity.com/paper/2206.04045/) (2022)
- [HyperExpan: Taxonomy Expansion with Hyperbolic Representation Learning](https://mlsanity.com/paper/2109.10500/) (2021)
- [CaEGCN: Cross-Attention Fusion based Enhanced Graph Convolutional Network for Clustering](https://mlsanity.com/paper/2101.06883/) (2021)
- [Syntax-augmented Multilingual BERT for Cross-lingual Transfer](https://mlsanity.com/paper/2106.02134/) (2021)
- [Mitigating the Position Bias of Transformer Models in Passage Re-Ranking](https://mlsanity.com/paper/2101.06980/) (2021)
- [Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation](https://mlsanity.com/paper/2210.17027/) (2022)
- [The Impact of Positional Encodings on Multilingual Compression](https://mlsanity.com/paper/2109.05388/) (2021)
- [P-Transformer: Towards Better Document-to-Document Neural Machine Translation](https://mlsanity.com/paper/2212.05830/) (2022)
- [Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models](https://mlsanity.com/paper/2310.16240/) (2023)
- [Equipping Sketch Patches with Context-Aware Positional Encoding for Graphic Sketch Representation](https://mlsanity.com/paper/2403.17525/) (2024)
- [Trees in transformers: a theoretical analysis of the Transformer's ability to represent trees](https://mlsanity.com/paper/2112.11913/) (2021)
- [Negation Triplet Extraction with Syntactic Dependency and Semantic Consistency](https://mlsanity.com/paper/2404.09830/) (2024)
- [Paragraph Boundaries Are Not White Space:Compression Depth as the Signature of Hierarchical Structure](https://mlsanity.com/paper/2609.23551/) (2026)

## Abstract (arXiv)

Although self-attention networks (SANs) have advanced the state-of-the-art on various NLP tasks, one criticism of SANs is their ability of encoding positions of input words (Shaw et al., 2018). In this work, we propose to augment SANs with structural position representations to model the latent structure of the input sentence, which is complementary to the standard sequential positional representations. Specifically, we use dependency tree to represent the grammatical structure of a sentence, and propose two strategies to encode the positional relationships among words in the dependency tree. Experimental results on NIST Chinese-to-English and WMT14 English-to-German translation tasks show that the proposed approach consistently boosts performance over both the absolute and relative sequential position representations.