# Syntactically Supervised Transformers for Faster Neural Machine Translation

arXiv: https://arxiv.org/abs/1906.02780 · HTML: https://mlsanity.com/paper/1906.02780/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2019.

## Built on by (most cited first, 19 of 19)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Non-Autoregressive Translation by Learning Target Categorical Codes](https://mlsanity.com/paper/2103.11405/) (2021)
- [AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate](https://mlsanity.com/paper/2109.06481/) (2021)
- [textitlatent-GLAT: Glancing at Latent Variables for Parallel Text Generation](https://mlsanity.com/paper/2204.02030/) (2022)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [Controlled Text Reduction](https://mlsanity.com/paper/2210.13449/) (2022)
- [Improved Latent Tree Induction with Distant Supervision via Span Constraints](https://mlsanity.com/paper/2109.05112/) (2021)
- [Semantics-aware Attention Improves Neural Machine Translation](https://mlsanity.com/paper/2110.06920/) (2021)
- [A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models](https://mlsanity.com/paper/2405.13019/) (2024)
- [Data-to-text Generation by Splicing Together Nearest Neighbors](https://mlsanity.com/paper/2101.08248/) (2021)
- [Modeling Coverage for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2104.11897/) (2021)
- [Non-autoregressive Machine Translation with Probabilistic Context-free Grammar](https://mlsanity.com/paper/2311.07941/) (2023)
- [Hybrid-Regressive Neural Machine Translation](https://mlsanity.com/paper/2210.10416/) (2022)
- [Rephrasing the Reference for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2211.16863/) (2022)
- [Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2303.17910/) (2023)
- [RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer](https://mlsanity.com/paper/2303.07665/) (2023)
- [Shared Latent Space by Both Languages in Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2305.03511/) (2023)
- [Faith: An Efficient Framework for Transformer Verification on GPUs](https://mlsanity.com/paper/2209.12708/) (2022)

## Abstract (arXiv)

Standard decoders for neural machine translation autoregressively generate a single target token per time step, which slows inference especially for long outputs. While architectural advances such as the Transformer fully parallelize the decoder computations at training time, inference still proceeds sequentially. Recent developments in non- and semi- autoregressive decoding produce multiple tokens per time step independently of the others, which improves inference speed but deteriorates translation quality. In this work, we propose the syntactically supervised Transformer (SynST), which first autoregressively predicts a chunked parse tree before generating all of the target tokens in one shot conditioned on the predicted parse. A series of controlled experiments demonstrates that SynST decodes sentences ~ 5x faster than the baseline autoregressive Transformer while achieving higher BLEU scores than most competing methods on En-De and En-Fr datasets.