# Tailoring Language Generation Models under Total Variation Distance

arXiv: https://arxiv.org/abs/2302.13344 · HTML: https://mlsanity.com/paper/2302.13344/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 57 references that are in the map.
- First posted on arXiv 26 Feb 2023.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization](https://mlsanity.com/paper/1911.03437/) (2019)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://mlsanity.com/paper/2103.15025/) (2021)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation](https://mlsanity.com/paper/2204.01171/) (2022)
- [Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness](https://mlsanity.com/paper/1905.13472/) (2019)
- [Long Text Generation via Adversarial Training with Leaked Information](https://mlsanity.com/paper/1709.08624/) (2017)
- [Language GANs Falling Short](https://mlsanity.com/paper/1811.02549/) (2018)
- [Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text](https://mlsanity.com/paper/2107.01294/) (2021)
- [On the Weaknesses of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/1907.01752/) (2019)
- [Reward Augmented Maximum Likelihood for Neural Structured Prediction](https://mlsanity.com/paper/1609.00150/) (2016)
- [How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?](https://mlsanity.com/paper/1511.05101/) (2015)
- [Improved Natural Language Generation via Loss Truncation](https://mlsanity.com/paper/2004.14589/) (2020)
- [A Study of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/1808.08866/) (2018)
- [Multi-Reward Reinforced Summarization with Saliency and Entailment](https://mlsanity.com/paper/1804.06451/) (2018)
- [Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting](https://mlsanity.com/paper/1906.09531/) (2019)
- [Text Generation by Learning from Demonstrations](https://mlsanity.com/paper/2009.07839/) (2020)
- [Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence](https://mlsanity.com/paper/2105.08963/) (2021)

## Built on by (most cited first, 27 of 27)

- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024)
- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation](https://mlsanity.com/paper/2410.13640/) (2024)
- [Towards Efficient Exact Optimization of Language Model Alignment](https://mlsanity.com/paper/2402.00856/) (2024)
- [SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters](https://mlsanity.com/paper/2502.00883/) (2025)
- [LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding](https://mlsanity.com/paper/2602.23881/) (2026)
- [MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies](https://mlsanity.com/paper/2305.16958/) (2023)
- [Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs](https://mlsanity.com/paper/2503.16870/) (2025)
- [Sinkhorn Distance Minimization for Knowledge Distillation](https://mlsanity.com/paper/2402.17110/) (2024)
- [Language Model Decoding as Direct Metrics Optimization](https://mlsanity.com/paper/2310.01041/) (2023)
- [Language Generation with Strictly Proper Scoring Rules](https://mlsanity.com/paper/2405.18906/) (2024)
- [EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling](https://mlsanity.com/paper/2310.04691/) (2023)
- [Improving Diversity in Language Models: When Temperature Fails, Change the Loss](https://mlsanity.com/paper/2508.09654/) (2025)
- [Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models](https://mlsanity.com/paper/2310.00840/) (2023)
- [Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understanding](https://mlsanity.com/paper/2409.19532/) (2024)
- [Predict the Next Word: Humans exhibit uncertainty in this task and language models _____](https://mlsanity.com/paper/2402.17527/) (2024)
- [Diversity in Large Language Models under Supervised Fine-Tuning](https://mlsanity.com/paper/2605.00195/) (2026)
- [Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models](https://mlsanity.com/paper/2409.12512/) (2024)
- [ReviewScore: Misinformed Peer Review Detection with Large Language Models](https://mlsanity.com/paper/2509.21679/) (2025)
- [Perturbation is All You Need for Extrapolating Language Models](https://mlsanity.com/paper/2605.04344/) (2026)
- [Mitigation of Gender and Ethnicity Bias in AI-Generated Stories through Model Explanations](https://mlsanity.com/paper/2509.04515/) (2025)
- [Short-Context Dominance: How Much Local Context Natural Language Actually Needs?](https://mlsanity.com/paper/2512.08082/) (2025)
- [Equalized Generative Treatment: Matching f-divergences for Fairness in Generative Models](https://mlsanity.com/paper/2602.08660/) (2026)
- [Learning Perturbations to Extrapolate Your LLM](https://mlsanity.com/paper/2605.13284/) (2026)
- [DEL: Digit Entropy Loss for Numerical Learning of Large Language Models](https://mlsanity.com/paper/2605.20369/) (2026)
- [Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models](https://mlsanity.com/paper/2608.29590/) (2026)
- [How Divergence Becomes Decision Flips in Compressed Language Models](https://mlsanity.com/paper/2610.00694/) (2026)

## Abstract (arXiv)

The standard paradigm of neural language generation adopts maximum likelihood estimation (MLE) as the optimizing method. From a distributional view, MLE in fact minimizes the Kullback-Leibler divergence (KLD) between the distribution of the real data and that of the model. However, this approach forces the model to distribute non-zero (sometimes large) probability mass to all training samples regardless of their quality. Moreover, in the attempt to cover the low-probability regions in the data distribution, the model systematically overestimates the probability of corrupted text sequences, which we conjecture is one of the main reasons for text degeneration during autoregressive decoding. To remedy this problem, we leverage the total variation distance (TVD) with its robustness to outliers, and develop practical bounds to apply it to language generation. Then, we introduce the TaiLr objective that balances the tradeoff of estimating TVD. Intuitively, TaiLr downweights real data samples that have low model probabilities with tunable penalization intensity. Experimental results show that our method alleviates the overestimation of degenerated sequences without sacrificing diversity and improves generation quality on a wide range of text generation tasks.