# Optimal Completion Distillation for Sequence Learning

arXiv: https://arxiv.org/abs/1810.01398 · HTML: https://mlsanity.com/paper/1810.01398/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2018.

## Built on by (most cited first, 14 of 14)

- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network](https://mlsanity.com/paper/2104.02133/) (2021)
- [Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models](https://mlsanity.com/paper/2107.07675/) (2021)
- [R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents](https://mlsanity.com/paper/2303.00732/) (2023)
- [Do End-to-End Speech Recognition Models Care About Context?](https://mlsanity.com/paper/2102.09928/) (2021)
- [Speaker conditioning of acoustic models using affine transformation for multi-speaker speech recognition](https://mlsanity.com/paper/2111.00320/) (2021)
- [Ranking-based Client Selection with Imitation Learning for Efficient Federated Learning](https://mlsanity.com/paper/2405.04122/) (2024)
- [SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation](https://mlsanity.com/paper/2608.09271/) (2026)
- [Edit Distance based RL for RNNT decoding](https://mlsanity.com/paper/2306.01789/) (2023)
- [O-1: Self-training with Oracle and 1-best Hypothesis](https://mlsanity.com/paper/2308.07486/) (2023)
- [Efficient Sample-Specific Encoder Perturbations](https://mlsanity.com/paper/2405.01601/) (2024)
- [Improving Autoregressive Training with Dynamic Oracles](https://mlsanity.com/paper/2406.09393/) (2024)
- [BEFL: Balancing Energy Consumption in Federated Learning for Mobile Edge IoT](https://mlsanity.com/paper/2412.03950/) (2024)

## Abstract (arXiv)

We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not require pretraining or joint optimization with conditional log-likelihood. Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm. Then, for each position of the generated sequence, we use a target distribution that puts equal probability on the first token of all the optimal suffixes. OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving 9.3% WER and 4.5% WER respectively.