# TeaForN: Teacher-Forcing with N-grams

arXiv: https://arxiv.org/abs/2010.03494 · HTML: https://mlsanity.com/paper/2010.03494/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Generative Regression Based Watch Time Prediction for Short-Video Recommendation](https://mlsanity.com/paper/2412.20211/) (2024)
- [Scheduled Sampling Based on Decoding Steps for Neural Machine Translation](https://mlsanity.com/paper/2108.12963/) (2021)
- [Confidence-Aware Scheduled Sampling for Neural Machine Translation](https://mlsanity.com/paper/2107.10427/) (2021)
- [Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2305.08096/) (2023)
- [Adaptive Bridge between Training and Inference for Dialogue](https://mlsanity.com/paper/2110.11560/) (2021)
- [Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More](https://mlsanity.com/paper/2503.10542/) (2025)
- [Dynamic Scheduled Sampling with Imitation Loss for Neural Text Generation](https://mlsanity.com/paper/2301.13753/) (2023)
- [Bilevel Scheduled Sampling for Dialogue Generation](https://mlsanity.com/paper/2309.01953/) (2023)

## Abstract (arXiv)

Sequence generation models trained with teacher-forcing suffer from issues related to exposure bias and lack of differentiability across timesteps. Our proposed method, Teacher-Forcing with N-grams (TeaForN), addresses both these problems directly, through the use of a stack of N decoders trained to decode along a secondary time axis that allows model parameter updates based on N prediction steps. TeaForN can be used with a wide class of decoder architectures and requires minimal modifications from a standard teacher-forcing setup. Empirically, we show that TeaForN boosts generation quality on one Machine Translation benchmark, WMT 2014 English-French, and two News Summarization benchmarks, CNN/Dailymail and Gigaword.