# Non-Autoregressive Translation by Learning Target Categorical Codes

arXiv: https://arxiv.org/abs/2103.11405 · HTML: https://mlsanity.com/paper/2103.11405/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 21 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [Fast Structured Decoding for Sequence Models](https://mlsanity.com/paper/1910.11555/) (2019)
- [Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior](https://mlsanity.com/paper/1908.07181/) (2019)
- [Non-Autoregressive Machine Translation with Auxiliary Regularization](https://mlsanity.com/paper/1902.10245/) (2019)
- [Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input](https://mlsanity.com/paper/1812.09664/) (2018)
- [Imitation Learning for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1906.02041/) (2019)
- [Syntactically Supervised Transformers for Faster Neural Machine Translation](https://mlsanity.com/paper/1906.02780/) (2019)
- [Non-autoregressive Transformer by Position Learning](https://mlsanity.com/paper/1911.10677/) (2019)

## Built on by (most cited first, 25 of 25)

- [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022)
- [AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation](https://mlsanity.com/paper/2305.09515/) (2023)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers](https://mlsanity.com/paper/2212.10325/) (2022)
- [Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2205.07459/) (2022)
- [Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision](https://mlsanity.com/paper/2110.07515/) (2021)
- [On the Learning of Non-Autoregressive Transformers](https://mlsanity.com/paper/2206.05975/) (2022)
- [textitlatent-GLAT: Glancing at Latent Variables for Parallel Text Generation](https://mlsanity.com/paper/2204.02030/) (2022)
- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.03953/) (2022)
- [DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer](https://mlsanity.com/paper/2110.05999/) (2021)
- [Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.05193/) (2022)
- [Multi-Granularity Optimization for Non-Autoregressive Translation](https://mlsanity.com/paper/2210.11017/) (2022)
- [Non-Autoregressive Neural Machine Translation: A Call for Clarity](https://mlsanity.com/paper/2205.10577/) (2022)
- [Non-autoregressive Machine Translation with Probabilistic Context-free Grammar](https://mlsanity.com/paper/2311.07941/) (2023)
- [Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing](https://mlsanity.com/paper/2506.13485/) (2025)
- [Rephrasing the Reference for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2211.16863/) (2022)
- [Translating Images to Road Network: A Sequence-to-Sequence Perspective](https://mlsanity.com/paper/2402.08207/) (2024)
- [Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2303.17910/) (2023)
- [Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators](https://mlsanity.com/paper/2211.06075/) (2022)
- [Accelerating Antimicrobial Peptide Discovery with Latent Structure](https://mlsanity.com/paper/2212.09450/) (2022)
- [What Have We Achieved on Non-autoregressive Translation?](https://mlsanity.com/paper/2405.12788/) (2024)
- [Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularization](https://mlsanity.com/paper/2402.09725/) (2024)
- [Diffusion Is Your Friend in Show, Suggest and Tell](https://mlsanity.com/paper/2512.10038/) (2025)
- [Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation](https://mlsanity.com/paper/2601.22546/) (2026)

## Abstract (arXiv)

Non-autoregressive Transformer is a promising text generation model. However, current non-autoregressive models still fall behind their autoregressive counterparts in translation quality. We attribute this accuracy gap to the lack of dependency modeling among decoder inputs. In this paper, we propose CNAT, which learns implicitly categorical codes as latent variables into the non-autoregressive decoding. The interaction among these categorical codes remedies the missing dependencies and improves the model capacity. Experiment results show that our model achieves comparable or better performance in machine translation tasks, compared with several strong baselines.