# Attention Link: An Efficient Attention-Based Low Resource Machine Translation Architecture

arXiv: https://arxiv.org/abs/2302.00340 · HTML: https://mlsanity.com/paper/2302.00340/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 1 Feb 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?](https://mlsanity.com/paper/1804.06323/) (2018)
- [Long-Short Transformer: Efficient Transformers for Language and Vision](https://mlsanity.com/paper/2107.02192/) (2021)
- [A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation](https://mlsanity.com/paper/2009.13818/) (2020)
- [Poolingformer: Long Document Modeling with Pooling Attention](https://mlsanity.com/paper/2105.04371/) (2021)
- [Unsupervised Statistical Machine Translation](https://mlsanity.com/paper/1809.01272/) (2018)
- [Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation](https://mlsanity.com/paper/2002.10260/) (2020)
- [Hard-Coded Gaussian Attention for Neural Machine Translation](https://mlsanity.com/paper/2005.00742/) (2020)
- [Optimizing Transformer for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2011.02266/) (2020)
- [On Optimal Transformer Depth for Low-Resource Language Translation](https://mlsanity.com/paper/2004.04418/) (2020)

## Abstract (arXiv)

Transformers have achieved great success in machine translation, but transformer-based NMT models often require millions of bilingual parallel corpus for training. In this paper, we propose a novel architecture named as attention link (AL) to help improve transformer models' performance, especially in low training resources. We theoretically demonstrate the superiority of our attention link architecture in low training resources. Besides, we have done a large number of experiments, including en-de, de-en, en-fr, en-it, it-en, en-ro translation tasks on the IWSLT14 dataset as well as real low resources scene on bn-gu and gu-ta translation tasks on the CVIT PIB dataset. All the experiment results show our attention link is powerful and can lead to a significant improvement. In addition, we achieve a 37.9 BLEU score, a new sota, on the IWSLT14 de-en task by combining our attention link and other advanced methods.