# Convolutional Sequence to Sequence Learning

arXiv: https://arxiv.org/abs/1705.03122 · HTML: https://mlsanity.com/paper/1705.03122/

## Key facts

- Cited by 552 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2017.

## Built on by (most cited first, 30 of 552)

- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion](https://mlsanity.com/paper/2111.11326/) (2021)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [Transformer Quality in Linear Time](https://mlsanity.com/paper/2202.10447/) (2022)
- [Mega: Moving Average Equipped Gated Attention](https://mlsanity.com/paper/2209.10655/) (2022)
- [Transformer Language Models without Positional Encodings Still Learn Positional Information](https://mlsanity.com/paper/2203.16634/) (2022)
- [SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields](https://mlsanity.com/paper/2211.12254/) (2022)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation](https://mlsanity.com/paper/2209.09002/) (2022)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [Randomized Positional Encodings Boost Length Generalization of Transformers](https://mlsanity.com/paper/2305.16843/) (2023)
- [EdgeViTs: Competing Light-weight CNNs on Mobile Devices with Vision Transformers](https://mlsanity.com/paper/2205.03436/) (2022)
- [Overcoming a Theoretical Limitation of Self-Attention](https://mlsanity.com/paper/2202.12172/) (2022)
- [Fully Hyperbolic Neural Networks](https://mlsanity.com/paper/2105.14686/) (2021)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)

## Abstract (arXiv)

The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT'14 English-French translation at an order of magnitude faster speed, both on GPU and CPU.