# Parallel Scheduled Sampling

arXiv: https://arxiv.org/abs/1906.04331 · HTML: https://mlsanity.com/paper/1906.04331/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 17 of 17)

- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [Learning to Generate Better Than Your LLM](https://mlsanity.com/paper/2306.11816/) (2023)
- [Rethinking Perturbations in Encoder-Decoders for Fast Training](https://mlsanity.com/paper/2104.01853/) (2021)
- [WeChat Neural Machine Translation Systems for WMT21](https://mlsanity.com/paper/2108.02401/) (2021)
- [Scheduled Sampling Based on Decoding Steps for Neural Machine Translation](https://mlsanity.com/paper/2108.12963/) (2021)
- [Learning Coupled Policies for Simultaneous Machine Translation using Imitation Learning](https://mlsanity.com/paper/2002.04306/) (2020)
- [Confidence-Aware Scheduled Sampling for Neural Machine Translation](https://mlsanity.com/paper/2107.10427/) (2021)
- [Mixed Cross Entropy Loss for Neural Machine Translation](https://mlsanity.com/paper/2106.15880/) (2021)
- [Cross-Lingual Text-to-Speech Using Multi-Task Learning and Speaker Classifier Joint Training](https://mlsanity.com/paper/2201.08124/) (2022)
- [Translating away Translationese without Parallel Data](https://mlsanity.com/paper/2310.18830/) (2023)
- [Attention Forcing for Machine Translation](https://mlsanity.com/paper/2104.01264/) (2021)
- [Sequence-level Large Language Model Training with Contrastive Preference Optimization](https://mlsanity.com/paper/2502.16433/) (2025)
- [Deliberation Networks and How to Train Them](https://mlsanity.com/paper/2211.03217/) (2022)
- [Parallel Attention Forcing for Machine Translation](https://mlsanity.com/paper/2211.03237/) (2022)
- [Dynamic Scheduled Sampling with Imitation Loss for Neural Text Generation](https://mlsanity.com/paper/2301.13753/) (2023)
- [Improving Autoregressive Training with Dynamic Oracles](https://mlsanity.com/paper/2406.09393/) (2024)
- [Breaking Chains with Trees: Model-Parallel Deep Learning with O(log N) Time Complexity](https://mlsanity.com/paper/2606.21497/) (2026)

## Abstract (arXiv)

Auto-regressive models are widely used in sequence generation problems. The output sequence is typically generated in a predetermined order, one discrete unit (pixel or word or character) at a time. The models are trained by teacher-forcing where ground-truth history is fed to the model as input, which at test time is replaced by the model prediction. Scheduled Sampling aims to mitigate this discrepancy between train and test time by randomly replacing some discrete units in the history with the model's prediction. While teacher-forced training works well with ML accelerators as the computation can be parallelized across time, Scheduled Sampling involves undesirable sequential processing. In this paper, we introduce a simple technique to parallelize Scheduled Sampling across time. Experimentally, we find the proposed technique leads to equivalent or better performance on image generation, summarization, dialog generation, and translation compared to teacher-forced training. In dialog response generation task, Parallel Scheduled Sampling achieves 1.6 BLEU score (11.5%) improvement over teacher-forcing while in image generation it achieves 20% and 13.8% improvement in Frechet Inception Distance (FID) and Inception Score (IS) respectively. Further, we discuss the effects of different hyper-parameters associated with Scheduled Sampling on the model performance.