# Hint-Based Training for Non-Autoregressive Machine Translation

arXiv: https://arxiv.org/abs/1909.06708 · HTML: https://mlsanity.com/paper/1909.06708/

## Key facts

- Cited by 28 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 15 Sep 2019.

## Built on by (most cited first, 28 of 28)

- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2106.05093/) (2021)
- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition](https://mlsanity.com/paper/2105.03842/) (2021)
- [AlphaNet: Improved Training of Supernets with Alpha-Divergence](https://mlsanity.com/paper/2102.07954/) (2021)
- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [Don't Take It Literally: An Edit-Invariant Sequence Loss for Text Generation](https://mlsanity.com/paper/2106.15078/) (2021)
- [Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation](https://mlsanity.com/paper/2101.08942/) (2021)
- [How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?](https://mlsanity.com/paper/2105.12900/) (2021)
- [Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference](https://mlsanity.com/paper/2407.09722/) (2024)
- [Duplex Sequence-to-Sequence Learning for Reversible Machine Translation](https://mlsanity.com/paper/2105.03458/) (2021)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [ngram-OAXE: Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.03999/) (2022)
- [Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation](https://mlsanity.com/paper/2304.11791/) (2023)
- [Non-Autoregressive Math Word Problem Solver with Unified Tree Structure](https://mlsanity.com/paper/2305.04556/) (2023)
- [MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2108.08447/) (2021)
- [Non-Autoregressive Neural Machine Translation: A Call for Clarity](https://mlsanity.com/paper/2205.10577/) (2022)
- [Modeling Coverage for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2104.11897/) (2021)
- [Topic Shift Detection in Chinese Dialogues: Corpus and Benchmark](https://mlsanity.com/paper/2305.01195/) (2023)
- [Hybrid-Regressive Neural Machine Translation](https://mlsanity.com/paper/2210.10416/) (2022)
- [Attentive Multi-Layer Perceptron for Non-autoregressive Generation](https://mlsanity.com/paper/2310.09512/) (2023)
- [Non-Autoregressive Sentence Ordering](https://mlsanity.com/paper/2310.12640/) (2023)
- [AMLNet: Adversarial Mutual Learning Neural Network for Non-AutoRegressive Multi-Horizon Time Series Forecasting](https://mlsanity.com/paper/2310.19289/) (2023)
- [NARRepair: Non-Autoregressive Code Generation Model for Automatic Program Repair](https://mlsanity.com/paper/2406.16526/) (2024)

## Abstract (arXiv)

Due to the unparallelizable nature of the autoregressive factorization, AutoRegressive Translation (ART) models have to generate tokens sequentially during decoding and thus suffer from high inference latency. Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time, but could only achieve inferior translation accuracy. In this paper, we proposed a novel approach to leveraging the hints from hidden states and word alignments to help the training of NART models. The results achieve significant improvement over previous NART models for the WMT14 En-De and De-En datasets and are even comparable to a strong LSTM-based ART baseline but one order of magnitude faster in inference.