# Multi-Pass Transformer for Machine Translation

arXiv: https://arxiv.org/abs/2009.11382 · HTML: https://mlsanity.com/paper/2009.11382/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2020.

## Built on by (most cited first, 5 of 5)

- [The Closeness of In-Context Learning and Weight Shifting for Softmax Regression](https://mlsanity.com/paper/2304.13276/) (2023)
- [Oriented Object Detection with Transformer](https://mlsanity.com/paper/2106.03146/) (2021)
- [RomeBERT: Robust Training of Multi-Exit BERT](https://mlsanity.com/paper/2101.09755/) (2021)
- [Scalable Transformers for Neural Machine Translation](https://mlsanity.com/paper/2106.02242/) (2021)
- [CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers](https://mlsanity.com/paper/2404.06709/) (2024)

## Abstract (arXiv)

In contrast with previous approaches where information flows only towards deeper layers of a stack, we consider a multi-pass transformer (MPT) architecture in which earlier layers are allowed to process information in light of the output of later layers. To maintain a directed acyclic graph structure, the encoder stack of a transformer is repeated along a new multi-pass dimension, keeping the parameters tied, and information is allowed to proceed unidirectionally both towards deeper layers within an encoder stack and towards any layer of subsequent stacks. We consider both soft (i.e., continuous) and hard (i.e., discrete) connections between parallel encoder stacks, relying on a neural architecture search to find the best connection pattern in the hard case. We perform an extensive ablation study of the proposed MPT architecture and compare it with other state-of-the-art transformer architectures. Surprisingly, Base Transformer equipped with MPT can surpass the performance of Large Transformer on the challenging machine translation En-De and En-Fr datasets. In the hard connection case, the optimal connection pattern found for En-De also leads to improved performance for En-Fr.