# DA-Transformer: Distance-aware Transformer

arXiv: https://arxiv.org/abs/2010.06925 · HTML: https://mlsanity.com/paper/2010.06925/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2020.

## Built on by (most cited first, 23 of 23)

- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Position Information in Transformers: An Overview](https://mlsanity.com/paper/2102.11090/) (2021)
- [KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation](https://mlsanity.com/paper/2205.09921/) (2022)
- [SHAPE: Shifted Absolute Position Embedding for Transformers](https://mlsanity.com/paper/2109.05644/) (2021)
- [Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling](https://mlsanity.com/paper/2106.01040/) (2021)
- [Transformer-based Entity Typing in Knowledge Graphs](https://mlsanity.com/paper/2210.11151/) (2022)
- [Towards Real-Time Panoptic Narrative Grounding by an End-to-End Grounding Network](https://mlsanity.com/paper/2301.03160/) (2023)
- [Algebraic Positional Encodings](https://mlsanity.com/paper/2312.16045/) (2023)
- [Time-distance vision transformers in lung cancer diagnosis from longitudinal computed tomography](https://mlsanity.com/paper/2209.01676/) (2022)
- [Monotonic Location Attention for Length Generalization](https://mlsanity.com/paper/2305.20019/) (2023)
- [Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification](https://mlsanity.com/paper/2304.02836/) (2023)
- [Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decoder](https://mlsanity.com/paper/2102.09206/) (2021)
- [Quantformer: from attention to profit with a quantitative transformer trading strategy](https://mlsanity.com/paper/2404.00424/) (2024)
- [Reference Neural Operators: Learning the Smooth Dependence of Solutions of PDEs on Geometric Deformations](https://mlsanity.com/paper/2405.17509/) (2024)
- [Group Representational Position Encoding](https://mlsanity.com/paper/2512.07805/) (2025)
- [CycleTransGAN-EVC: A CycleGAN-based Emotional Voice Conversion Model with Transformer](https://mlsanity.com/paper/2111.15159/) (2021)
- [LEA: Improving Sentence Similarity Robustness to Typos Using Lexical Attention Bias](https://mlsanity.com/paper/2307.02912/) (2023)
- [DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks](https://mlsanity.com/paper/2204.08688/) (2022)
- [Enhancing Multivariate Time Series Classifiers through Self-Attention and Relative Positioning Infusion](https://mlsanity.com/paper/2302.06683/) (2023)
- [PIPE: Physics-Informed Position Encoding for Alignment of Satellite Images and Time Series](https://mlsanity.com/paper/2506.14786/) (2025)
- [User recommendation system based on MIND dataset](https://mlsanity.com/paper/2209.06131/) (2022)
- [TAU: Modeling Temporal Consistency Through Temporal Attentive U-Net for PPG Peak Detection](https://mlsanity.com/paper/2503.10733/) (2025)
- [Spatially-informed transformers: Injecting geostatistical covariance biases into self-attention for spatio-temporal forecasting](https://mlsanity.com/paper/2512.17696/) (2025)

## Abstract (arXiv)

Transformer has achieved great success in the NLP field by composing various advanced models like BERT and GPT. However, Transformer and its existing variants may not be optimal in capturing token distances because the position or distance embeddings used by these methods usually cannot keep the precise information of real distances, which may not be beneficial for modeling the orders and relations of contexts. In this paper, we propose DA-Transformer, which is a distance-aware Transformer that can exploit the real distance. We propose to incorporate the real distances between tokens to re-scale the raw self-attention weights, which are computed by the relevance between attention query and key. Concretely, in different self-attention heads the relative distance between each pair of tokens is weighted by different learnable parameters, which control the different preferences on long- or short-term information of these heads. Since the raw weighted real distances may not be optimal for adjusting self-attention weights, we propose a learnable sigmoid function to map them into re-scaled coefficients that have proper ranges. We first clip the raw self-attention weights via the ReLU function to keep non-negativity and introduce sparsity, and then multiply them with the re-scaled coefficients to encode real distance information into self-attention. Extensive experiments on five benchmark datasets show that DA-Transformer can effectively improve the performance of many tasks and outperform the vanilla Transformer and its several variants.