# Language Model Decoding as Direct Metrics Optimization

arXiv: https://arxiv.org/abs/2310.01041 · HTML: https://mlsanity.com/paper/2310.01041/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 73 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 30 of 46)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [A Contrastive Framework for Neural Text Generation](https://mlsanity.com/paper/2202.06417/) (2022)
- [A Distributional Approach to Controlled Text Generation](https://mlsanity.com/paper/2012.11635/) (2020)
- [Minimum Risk Training for Neural Machine Translation](https://mlsanity.com/paper/1512.02433/) (2015)
- [Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation](https://mlsanity.com/paper/2204.01171/) (2022)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)
- [Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness](https://mlsanity.com/paper/1905.13472/) (2019)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [Long Text Generation via Adversarial Training with Leaked Information](https://mlsanity.com/paper/1709.08624/) (2017)
- [Language GANs Falling Short](https://mlsanity.com/paper/1811.02549/) (2018)
- [Trading Off Diversity and Quality in Natural Language Generation](https://mlsanity.com/paper/2004.10450/) (2020)
- [Residual Energy-Based Models for Text Generation](https://mlsanity.com/paper/2004.11714/) (2020)
- [Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](https://mlsanity.com/paper/2206.02369/) (2022)
- [Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text](https://mlsanity.com/paper/2107.01294/) (2021)
- [A Theoretical Analysis of the Repetition Problem in Text Generation](https://mlsanity.com/paper/2012.14660/) (2020)
- [How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?](https://mlsanity.com/paper/1511.05101/) (2015)

## Built on by (most cited first, 8 of 8)

- [Towards Efficient Exact Optimization of Language Model Alignment](https://mlsanity.com/paper/2402.00856/) (2024)
- [Non-myopic Generation of Language Models for Reasoning and Planning](https://mlsanity.com/paper/2410.17195/) (2024)
- [Antidistillation Sampling](https://mlsanity.com/paper/2504.13146/) (2025)
- [MAP: Multi-Human-Value Alignment Palette](https://mlsanity.com/paper/2410.19198/) (2024)
- [Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability](https://mlsanity.com/paper/2605.09214/) (2026)
- [Synthesize, Partition, then Adapt: Eliciting Diverse Samples from Foundation Models](https://mlsanity.com/paper/2411.06722/) (2024)
- [Waste Not, Want Not; Recycled Gumbel Noise Improves Consistency in Natural Language Generation](https://mlsanity.com/paper/2503.00831/) (2025)
- [Composable Decoding on the Probability Simplex: Theory and Implementation](https://mlsanity.com/paper/2609.34992/) (2026)

## Abstract (arXiv)

Despite the remarkable advances in language modeling, current mainstream decoding methods still struggle to generate texts that align with human texts across different aspects. In particular, sampling-based methods produce less-repetitive texts which are often disjunctive in discourse, while search-based methods maintain topic coherence at the cost of increased repetition. Overall, these methods fall short in achieving holistic alignment across a broad range of aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts measured by multiple metrics of desired aspects simultaneously. The resulting decoding distribution enjoys an analytical solution that scales the input language model distribution via a sequence-level energy function defined by these metrics. And most importantly, we prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts. To facilitate tractable sampling from this globally normalized distribution, we adopt the Sampling-Importance-Resampling technique. Experiments on various domains and model scales demonstrate the superiority of our method in metrics alignment with human texts and human evaluation over strong baselines.