# Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation arXiv: https://arxiv.org/abs/2106.06751 ยท HTML: https://mlsanity.com/paper/2106.06751/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. - Builds on 36 of its 49 references that are in the map. - First posted on arXiv 12 Jun 2021. ## Placed in - Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/) ## Builds on (most cited first, 30 of 36) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018) - [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017) - [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016) - [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020) - [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018) - [An Actor-Critic Algorithm for Sequence Prediction](https://mlsanity.com/paper/1607.07086/) (2016) - [Bridging the Gap between Training and Inference for Neural Machine Translation](https://mlsanity.com/paper/1906.02448/) (2019) - [Achieving Human Parity on Automatic Chinese to English News Translation](https://mlsanity.com/paper/1803.05567/) (2018) - [A Study of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/1808.08866/) (2018) - [Regularizing Neural Machine Translation by Target-bidirectional Agreement](https://mlsanity.com/paper/1808.04064/) (2018) - [Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.09320/) (2019) - [Translationese in Machine Translation Evaluation](https://mlsanity.com/paper/1906.09833/) (2019) - [Future-Guided Incremental Transformer for Simultaneous Translation](https://mlsanity.com/paper/2012.12465/) (2020) - [Synchronous Bidirectional Neural Machine Translation](https://mlsanity.com/paper/1905.04847/) (2019) - [Asynchronous Bidirectional Decoding for Neural Machine Translation](https://mlsanity.com/paper/1801.05122/) (2018) - [Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets](https://mlsanity.com/paper/1703.04887/) (2017) - [Learning to Decode for Future Success](https://mlsanity.com/paper/1701.06549/) (2017) - [Towards Decoding as Continuous Optimization in Neural Machine Translation](https://mlsanity.com/paper/1701.02854/) (2017) - [Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1906.09444/) (2019) - [Modeling Fluency and Faithfulness for Diverse Neural Machine Translation](https://mlsanity.com/paper/1912.00178/) (2019) - [Synchronous Bidirectional Inference for Neural Sequence Generation](https://mlsanity.com/paper/1902.08955/) (2019) - [Pre-Translation for Neural Machine Translation](https://mlsanity.com/paper/1610.05243/) (2016) - [Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation](https://mlsanity.com/paper/1809.03132/) (2018) ## Built on by (most cited first, 8 of 8) - [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021) - [Reducing Position Bias in Simultaneous Machine Translation with Length-Aware Framework](https://mlsanity.com/paper/2203.09053/) (2022) - [Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2305.08096/) (2023) - [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026) - [EM-Network: Oracle Guided Self-distillation for Sequence Learning](https://mlsanity.com/paper/2306.10058/) (2023) - [Enhancing Low-Resource NMT with a Multilingual Encoder and Knowledge Distillation: A Case Study](https://mlsanity.com/paper/2407.06538/) (2024) - [Improving Autoregressive Training with Dynamic Oracles](https://mlsanity.com/paper/2406.09393/) (2024) - [RetroMPA: A Molecular Property-Aware Auxiliary Framework for Enhancing Retrosynthesis Prediction](https://mlsanity.com/paper/2608.16111/) (2026) ## Abstract (arXiv) Although teacher forcing has become the main training paradigm for neural machine translation, it usually makes predictions only conditioned on past information, and hence lacks global planning for the future. To address this problem, we introduce another decoder, called seer decoder, into the encoder-decoder framework during training, which involves future information in target predictions. Meanwhile, we force the conventional decoder to simulate the behaviors of the seer decoder via knowledge distillation. In this way, at test the conventional decoder can perform like the seer decoder without the attendance of it. Experiment results on the Chinese-English, English-German and English-Romanian translation tasks show our method can outperform competitive baselines significantly and achieves greater improvements on the bigger data sets. Besides, the experiments also prove knowledge distillation the best way to transfer knowledge from the seer decoder to the conventional decoder compared to adversarial learning and L2 regularization.