# Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generation

arXiv: https://arxiv.org/abs/2106.07207 · HTML: https://mlsanity.com/paper/2106.07207/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement](https://mlsanity.com/paper/1903.06059/) (2019)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)
- [Modeling Coverage for Neural Machine Translation](https://mlsanity.com/paper/1601.04811/) (2016)
- [Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training](https://mlsanity.com/paper/1911.03860/) (2019)
- [A Hierarchical Approach for Generating Descriptive Image Paragraphs](https://mlsanity.com/paper/1611.06607/) (2016)
- [Describing like humans: on diversity in image captioning](https://mlsanity.com/paper/1903.12020/) (2019)
- [Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints](https://mlsanity.com/paper/1603.08887/) (2016)
- [Contextual Temperature for Language Modeling](https://mlsanity.com/paper/2012.13575/) (2020)
- [TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation](https://mlsanity.com/paper/2003.11963/) (2020)
- [Improving Diversity of Neural Text Generation via Inverse Probability Weighting](https://mlsanity.com/paper/2103.07649/) (2021)

## Built on by (most cited first, 19 of 19)

- [Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation](https://mlsanity.com/paper/2206.02369/) (2022)
- [Jointly Reinforcing Diversity and Quality in Language Model Generations](https://mlsanity.com/paper/2509.02534/) (2025)
- [SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization](https://mlsanity.com/paper/2203.06569/) (2022)
- [Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective](https://mlsanity.com/paper/2310.10226/) (2023)
- [From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers](https://mlsanity.com/paper/2402.13512/) (2024)
- [Understanding In-Context Learning from Repetitions](https://mlsanity.com/paper/2310.00297/) (2023)
- [Unveiling Attractor Cycles in Large Language Models: A Dynamical Systems View of Successive Paraphrasing](https://mlsanity.com/paper/2502.15208/) (2025)
- [Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models](https://mlsanity.com/paper/2404.13752/) (2024)
- [Nearest Neighbor Knowledge Distillation for Neural Machine Translation](https://mlsanity.com/paper/2205.00479/) (2022)
- [R2D2: Robust Data-to-Text with Replacement Detection](https://mlsanity.com/paper/2205.12467/) (2022)
- [Language Model Decoding as Direct Metrics Optimization](https://mlsanity.com/paper/2310.01041/) (2023)
- [Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models](https://mlsanity.com/paper/2310.00840/) (2023)
- [Multi-level Adaptive Contrastive Learning for Knowledge Internalization in Dialogue Generation](https://mlsanity.com/paper/2310.08943/) (2023)
- [Decouple Non-parametric Knowledge Distillation For End-to-end Speech Translation](https://mlsanity.com/paper/2304.10295/) (2023)
- [Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation](https://mlsanity.com/paper/2307.01542/) (2023)
- [Resource Consumption Threats in Large Language Models](https://mlsanity.com/paper/2603.16068/) (2026)
- [Dynamic Scheduled Sampling with Imitation Loss for Neural Text Generation](https://mlsanity.com/paper/2301.13753/) (2023)
- [Exploring Automatic Text Simplification of German Narrative Documents](https://mlsanity.com/paper/2312.09907/) (2023)
- [FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models](https://mlsanity.com/paper/2608.26676/) (2026)

## Abstract (arXiv)

Advanced large-scale neural language models have led to significant success in many language generation tasks. However, the most commonly used training objective, Maximum Likelihood Estimation (MLE), has been shown problematic, where the trained model prefers using dull and repetitive phrases. In this work, we introduce ScaleGrad, a modification straight to the gradient of the loss function, to remedy the degeneration issue of the standard MLE objective. By directly maneuvering the gradient information, ScaleGrad makes the model learn to use novel tokens. Empirical results show the effectiveness of our method not only in open-ended generation, but also in directed generation tasks. With the simplicity in architecture, our method can serve as a general training objective that is applicable to most of the neural text generation tasks.