# Scheduled Sampling for Transformers

arXiv: https://arxiv.org/abs/1906.07651 · HTML: https://mlsanity.com/paper/1906.07651/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2019.

## Built on by (most cited first, 30 of 48)

- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [End-to-End Training for Autoregressive Video Diffusion via Self-Resampling](https://mlsanity.com/paper/2512.15702/) (2025)
- [Learning to Generate Better Than Your LLM](https://mlsanity.com/paper/2306.11816/) (2023)
- [LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior](https://mlsanity.com/paper/2410.21264/) (2024)
- [Towards Scalable and Stable Parallelization of Nonlinear RNNs](https://mlsanity.com/paper/2407.19115/) (2024)
- [Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objects](https://mlsanity.com/paper/2205.05960/) (2022)
- [Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models](https://mlsanity.com/paper/2403.02178/) (2024)
- [PASTA: Controllable Part-Aware Shape Generation with Autoregressive Transformers](https://mlsanity.com/paper/2407.13677/) (2024)
- [Neural Inertial Localization](https://mlsanity.com/paper/2203.15851/) (2022)
- [Flipped Classroom: Effective Teaching for Time Series Forecasting](https://mlsanity.com/paper/2210.08959/) (2022)
- [Semi-Offline Reinforcement Learning for Optimized Text Generation](https://mlsanity.com/paper/2306.09712/) (2023)
- [WeChat Neural Machine Translation Systems for WMT21](https://mlsanity.com/paper/2108.02401/) (2021)
- [Scheduled Sampling Based on Decoding Steps for Neural Machine Translation](https://mlsanity.com/paper/2108.12963/) (2021)
- [Confidence-Aware Scheduled Sampling for Neural Machine Translation](https://mlsanity.com/paper/2107.10427/) (2021)
- [Auto-regressive Image Synthesis with Integrated Quantization](https://mlsanity.com/paper/2207.10776/) (2022)
- [ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention](https://mlsanity.com/paper/2311.16682/) (2023)
- [Mixed Cross Entropy Loss for Neural Machine Translation](https://mlsanity.com/paper/2106.15880/) (2021)
- [Generative World Modelling for Humanoids: 1X World Model Challenge Technical Report](https://mlsanity.com/paper/2510.07092/) (2025)
- [Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens](https://mlsanity.com/paper/2410.14655/) (2024)
- [Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders](https://mlsanity.com/paper/2304.01016/) (2023)
- [Translating away Translationese without Parallel Data](https://mlsanity.com/paper/2310.18830/) (2023)
- [Your Autoregressive Generative Model Can be Better If You Treat It as an Energy-Based One](https://mlsanity.com/paper/2206.12840/) (2022)
- [Global memory transformer for processing long documents](https://mlsanity.com/paper/2212.01650/) (2022)
- [Beyond MLE: Convex Learning for Text Generation](https://mlsanity.com/paper/2310.17217/) (2023)
- [NEURAL MARIONETTE: A Transformer-based Multi-action Human Motion Synthesis System](https://mlsanity.com/paper/2209.13204/) (2022)
- [MLP: Motion Label Prior for Temporal Sentence Localization in Untrimmed 3D Human Motions](https://mlsanity.com/paper/2404.13657/) (2024)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)
- [Adversarial Machine Learning in Text Analysis and Generation](https://mlsanity.com/paper/2101.08675/) (2021)
- [One Source, Two Targets: Challenges and Rewards of Dual Decoding](https://mlsanity.com/paper/2109.10197/) (2021)
- [An Comparative Analysis of Different Pitch and Metrical Grid Encoding Methods in the Task of Sequential Music Generation](https://mlsanity.com/paper/2301.13383/) (2023)

## Abstract (arXiv)

Scheduled sampling is a technique for avoiding one of the known problems in sequence-to-sequence generation: exposure bias. It consists of feeding the model a mix of the teacher forced embeddings and the model predictions from the previous step in training time. The technique has been used for improving the model performance with recurrent neural networks (RNN). In the Transformer model, unlike the RNN, the generation of a new word attends to the full sentence generated so far, not only to the last word, and it is not straightforward to apply the scheduled sampling technique. We propose some structural changes to allow scheduled sampling to be applied to Transformer architecture, via a two-pass decoding strategy. Experiments on two language pairs achieve performance close to a teacher-forcing baseline and show that this technique is promising for further exploration.