# Depthwise Separable Convolutions for Neural Machine Translation

arXiv: https://arxiv.org/abs/1706.03059 · HTML: https://mlsanity.com/paper/1706.03059/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2017.

## Built on by (most cited first, 27 of 27)

- [S2-MLP: Spatial-Shift MLP Architecture for Vision](https://mlsanity.com/paper/2106.07477/) (2021)
- [NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search](https://mlsanity.com/paper/2105.14444/) (2021)
- [Container: Context Aggregation Network](https://mlsanity.com/paper/2106.01401/) (2021)
- [LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search](https://mlsanity.com/paper/2102.04040/) (2021)
- [Are Pre-trained Convolutions Better than Pre-trained Transformers?](https://mlsanity.com/paper/2105.03322/) (2021)
- [Analyzing and Mitigating Interference in Neural Architecture Search](https://mlsanity.com/paper/2108.12821/) (2021)
- [Sequence Modeling with Multiresolution Convolutional Memory](https://mlsanity.com/paper/2305.01638/) (2023)
- [Group Equivariant Conditional Neural Processes](https://mlsanity.com/paper/2102.08759/) (2021)
- [Rethinking Token-Mixing MLP for MLP-based Vision Backbone](https://mlsanity.com/paper/2106.14882/) (2021)
- [SiamMask: A Framework for Fast Online Object Tracking and Segmentation](https://mlsanity.com/paper/2207.02088/) (2022)
- [Security Vulnerability Detection Using Deep Learning Natural Language Processing](https://mlsanity.com/paper/2105.02388/) (2021)
- [A Comprehensive Study of Real-Time Object Detection Networks Across Multiple Domains: A Survey](https://mlsanity.com/paper/2208.10895/) (2022)
- [DeepLSS: breaking parameter degeneracies in large scale structure with deep learning analysis of combined probes](https://mlsanity.com/paper/2203.09616/) (2022)
- [Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability](https://mlsanity.com/paper/2311.04449/) (2023)
- [Learning on tree architectures outperforms a convolutional feedforward network](https://mlsanity.com/paper/2211.11378/) (2022)
- [MSA-CNN: A Lightweight Multi-Scale CNN with Attention for Sleep Stage Classification](https://mlsanity.com/paper/2501.02949/) (2025)
- [THINK: A Novel Conversation Model for Generating Grammatically Correct and Coherent Responses](https://mlsanity.com/paper/2105.13630/) (2021)
- [Scalable Transformers for Neural Machine Translation](https://mlsanity.com/paper/2106.02242/) (2021)
- [TinyM2Net: A Flexible System Algorithm Co-designed Multimodal Learning Framework for Tiny Devices](https://mlsanity.com/paper/2202.04303/) (2022)
- [PS-ARM: An End-to-End Attention-aware Relation Mixer Network for Person Search](https://mlsanity.com/paper/2210.03433/) (2022)
- [Dual Atrous Separable Convolution for Improving Agricultural Semantic Segmentation](https://mlsanity.com/paper/2506.22570/) (2025)
- [SmartSplit: Latency-Energy-Memory Optimisation for CNN Splitting on Smartphone Environment](https://mlsanity.com/paper/2111.01077/) (2021)
- [COVID-Net Assistant: A Deep Learning-Driven Virtual Assistant for COVID-19 Symptom Prediction and Recommendation](https://mlsanity.com/paper/2211.11944/) (2022)
- [A Pervasive Framework for Human Detection and Tracking](https://mlsanity.com/paper/2303.11170/) (2023)
- [Pluggable Style Representation Learning for Multi-Style Transfer](https://mlsanity.com/paper/2503.20368/) (2025)
- [Multi-refined Feature Enhanced Sentiment Analysis Using Contextual Instruction](https://mlsanity.com/paper/2511.00537/) (2025)
- [How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions](https://mlsanity.com/paper/2605.29448/) (2026)

## Abstract (arXiv)

Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency. They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures). Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results. In this work, we study how depthwise separable convolutions can be applied to neural machine translation. We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves new state-of-the-art results. In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation. We also introduce a new "super-separable" convolution operation that further reduces the number of parameters and computational cost for obtaining state-of-the-art results.