# A Stable and Effective Learning Strategy for Trainable Greedy Decoding

arXiv: https://arxiv.org/abs/1804.07915 · HTML: https://mlsanity.com/paper/1804.07915/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2018.

## Built on by (most cited first, 12 of 12)

- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks](https://mlsanity.com/paper/2504.11358/) (2025)
- [Token Dropping for Efficient BERT Pretraining](https://mlsanity.com/paper/2203.13240/) (2022)
- [The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models](https://mlsanity.com/paper/2402.07658/) (2024)
- [A Mixture-of-Expert Approach to RL-based Dialogue Management](https://mlsanity.com/paper/2206.00059/) (2022)
- [R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents](https://mlsanity.com/paper/2303.00732/) (2023)
- [CFL: Causally Fair Language Models Through Token-level Attribute Controlled Generation](https://mlsanity.com/paper/2306.00374/) (2023)
- [Amortized Noisy Channel Neural Machine Translation](https://mlsanity.com/paper/2112.08670/) (2021)
- [Consistency Is the Key: Detecting Hallucinations in LLM Generated Text By Checking Inconsistencies About Key Facts](https://mlsanity.com/paper/2511.12236/) (2025)
- [Plug and Play Counterfactual Text Generation for Model Robustness](https://mlsanity.com/paper/2206.10429/) (2022)
- [Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP](https://mlsanity.com/paper/2108.12522/) (2021)
- [Beyond Distribution Sharpening: The Importance of Task Rewards](https://mlsanity.com/paper/2604.16259/) (2026)

## Abstract (arXiv)

Beam search is a widely used approximate search strategy for neural network decoders, and it generally outperforms simple greedy decoding on tasks like machine translation. However, this improvement comes at substantial computational cost. In this paper, we propose a flexible new method that allows us to reap nearly the full benefits of beam search with nearly no additional computational cost. The method revolves around a small neural network actor that is trained to observe and manipulate the hidden state of a previously-trained decoder. To train this actor network, we introduce the use of a pseudo-parallel corpus built using the output of beam search on a base model, ranked by a target quality metric like BLEU. Our method is inspired by earlier work on this problem, but requires no reinforcement learning, and can be trained reliably on a range of models. Experiments on three parallel corpora and three architectures show that the method yields substantial improvements in translation quality and speed over each base system.