# Connecting the Dots Between MLE and RL for Sequence Prediction

arXiv: https://arxiv.org/abs/1811.09740 · HTML: https://mlsanity.com/paper/1811.09740/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2018.

## Built on by (most cited first, 3 of 3)

- [Semi-Offline Reinforcement Learning for Optimized Text Generation](https://mlsanity.com/paper/2306.09712/) (2023)
- [Toward a `Standard Model' of Machine Learning](https://mlsanity.com/paper/2108.07783/) (2021)
- [Nested-Wasserstein Self-Imitation Learning for Sequence Generation](https://mlsanity.com/paper/2001.06944/) (2020)

## Abstract (arXiv)

Sequence prediction models can be learned from example sequences with a variety of training algorithms. Maximum likelihood learning is simple and efficient, yet can suffer from compounding error at test time. Reinforcement learning such as policy gradient addresses the issue but can have prohibitively poor exploration efficiency. A rich set of other algorithms such as RAML, SPG, and data noising, have also been developed from different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation, and show that the apparently distinct algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of a reward function and a couple of hyperparameters. The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency. Besides, inspired from the framework, we present a new algorithm that dynamically interpolates among the family of algorithms for scheduled sequence model learning. Experiments on machine translation, text summarization, and game imitation learning demonstrate the superiority of the proposed algorithm.