# How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?

arXiv: https://arxiv.org/abs/1511.05101 · HTML: https://mlsanity.com/paper/1511.05101/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2015.

## Built on by (most cited first, 30 of 78)

- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [Aligning Language Models with Preferences through f-divergence Minimization](https://mlsanity.com/paper/2302.08215/) (2023)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [Autoregressive Knowledge Distillation through Imitation Learning](https://mlsanity.com/paper/2009.07253/) (2020)
- [Text Generation by Learning from Demonstrations](https://mlsanity.com/paper/2009.07839/) (2020)
- [FakeInversion: Learning to Detect Images from Unseen Text-to-Image Models by Inverting Stable Diffusion](https://mlsanity.com/paper/2406.08603/) (2024)
- [Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application](https://mlsanity.com/paper/2407.01885/) (2024)
- [f-Domain-Adversarial Learning: Theory and Algorithms](https://mlsanity.com/paper/2106.11344/) (2021)
- [Introduction to Normalizing Flows for Lattice Field Theory](https://mlsanity.com/paper/2101.08176/) (2021)
- [Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding](https://mlsanity.com/paper/2605.29707/) (2026)
- [Reward Gaming in Conditional Text Generation](https://mlsanity.com/paper/2211.08714/) (2022)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Tailoring Language Generation Models under Total Variation Distance](https://mlsanity.com/paper/2302.13344/) (2023)
- [Learning to Generate 3D Shapes with Generative Cellular Automata](https://mlsanity.com/paper/2103.04130/) (2021)
- [Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition](https://mlsanity.com/paper/2303.04291/) (2023)
- [Relating Neural Text Degeneration to Exposure Bias](https://mlsanity.com/paper/2109.08705/) (2021)
- [On the Efficacy of Sampling Adapters](https://mlsanity.com/paper/2307.03749/) (2023)
- [MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies](https://mlsanity.com/paper/2305.16958/) (2023)
- [Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation](https://mlsanity.com/paper/2603.13260/) (2026)
- [TextBox: A Unified, Modularized, and Extensible Framework for Text Generation](https://mlsanity.com/paper/2101.02046/) (2021)
- [Free-Editor: Zero-shot Text-driven 3D Scene Editing](https://mlsanity.com/paper/2312.13663/) (2023)
- [Fundamentals of Generative Large Language Models and Perspectives in Cyber-Defense](https://mlsanity.com/paper/2303.12132/) (2023)
- [Can Diffusion Model Achieve Better Performance in Text Generation? Bridging the Gap between Training and Inference!](https://mlsanity.com/paper/2305.04465/) (2023)
- [Efficient Inference for Large Language Model-based Generative Recommendation](https://mlsanity.com/paper/2410.05165/) (2024)
- [Realizing GANs via a Tunable Loss Function](https://mlsanity.com/paper/2106.05232/) (2021)
- [Time-series Generation by Contrastive Imitation](https://mlsanity.com/paper/2311.01388/) (2023)

## Abstract (arXiv)

Modern applications and progress in deep learning research have created renewed interest for generative models of text and of images. However, even today it is unclear what objective functions one should use to train and evaluate these models. In this paper we present two contributions. Firstly, we present a critique of scheduled sampling, a state-of-the-art training method that contributed to the winning entry to the MSCOCO image captioning benchmark in 2015. Here we show that despite this impressive empirical performance, the objective function underlying scheduled sampling is improper and leads to an inconsistent learning algorithm. Secondly, we revisit the problems that scheduled sampling was meant to address, and present an alternative interpretation. We argue that maximum likelihood is an inappropriate training objective when the end-goal is to generate natural-looking samples. We go on to derive an ideal objective function to use in this situation instead. We introduce a generalisation of adversarial training, and show how such method can interpolate between maximum likelihood training and our ideal training objective. To our knowledge this is the first theoretical analysis that explains why adversarial training tends to produce samples with higher perceived quality.