# TranSFormer: Slow-Fast Transformer for Machine Translation

arXiv: https://arxiv.org/abs/2305.16982 · HTML: https://mlsanity.com/paper/2305.16982/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018)
- [Context-Aware Neural Machine Translation Learns Anaphora Resolution](https://mlsanity.com/paper/1805.10163/) (2018)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://mlsanity.com/paper/1610.03017/) (2016)
- [Modeling Localness for Self-Attention Networks](https://mlsanity.com/paper/1810.10182/) (2018)
- [Convolutional Self-Attention Networks](https://mlsanity.com/paper/1904.03107/) (2019)
- [Revisiting Character-Based Neural Machine Translation with Capacity and Compression](https://mlsanity.com/paper/1808.09943/) (2018)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning](https://mlsanity.com/paper/1911.09483/) (2019)
- [DeLighT: Deep and Light-weight Transformer](https://mlsanity.com/paper/2008.00623/) (2020)
- [Weighted Transformer Network for Machine Translation](https://mlsanity.com/paper/1711.02132/) (2017)
- [Multi-Scale Self-Attention for Text Classification](https://mlsanity.com/paper/1912.00544/) (2019)
- [Multi-Unit Transformers for Neural Machine Translation](https://mlsanity.com/paper/2010.10743/) (2020)
- [Character-Level Translation with Self-attention](https://mlsanity.com/paper/2004.14788/) (2020)
- [Multi-Granularity Self-Attention for Neural Machine Translation](https://mlsanity.com/paper/1909.02222/) (2019)
- [Multi-branch Attentive Transformer](https://mlsanity.com/paper/2006.10270/) (2020)

## Built on by (most cited first, 4 of 4)

- [PanoSwin: a Pano-style Swin Transformer for Panorama Understanding](https://mlsanity.com/paper/2308.14726/) (2023)
- [VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models](https://mlsanity.com/paper/2311.18837/) (2023)
- [Core Building Blocks: Next Gen Geo Spatial GPT Application](https://mlsanity.com/paper/2310.11029/) (2023)
- [Enhancing Neural Machine Translation with Semantic Units](https://mlsanity.com/paper/2310.11360/) (2023)

## Abstract (arXiv)

Learning multiscale Transformer models has been evidenced as a viable approach to augmenting machine translation systems. Prior research has primarily focused on treating subwords as basic units in developing such systems. However, the incorporation of fine-grained character-level features into multiscale Transformer has not yet been explored. In this work, we present a Slow-Fast two-stream learning model, referred to as TranSFormer, which utilizes a ``slow'' branch to deal with subword sequences and a ``fast'' branch to deal with longer character sequences. This model is efficient since the fast branch is very lightweight by reducing the model width, and yet provides useful fine-grained features for the slow branch. Our TranSFormer shows consistent BLEU improvements (larger than 1 BLEU point) on several machine translation benchmarks.