# Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings

arXiv: https://arxiv.org/abs/2109.03127 · HTML: https://mlsanity.com/paper/2109.03127/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Regularizing and Optimizing LSTM Language Models](https://mlsanity.com/paper/1708.02182/) (2017)
- [Breaking the Softmax Bottleneck: A High-Rank RNN Language Model](https://mlsanity.com/paper/1711.03953/) (2017)
- [Analyzing Uncertainty in Neural Machine Translation](https://mlsanity.com/paper/1803.00047/) (2018)
- [Improving Neural Language Modeling via Adversarial Training](https://mlsanity.com/paper/1906.03805/) (2019)
- [FRAGE: Frequency-Agnostic Word Representation](https://mlsanity.com/paper/1809.06858/) (2018)
- [A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space](https://mlsanity.com/paper/2106.01183/) (2021)
- [Stolen Probability: A Structural Weakness of Neural Language Models](https://mlsanity.com/paper/2005.02433/) (2020)
- [Unsupervised Post-processing of Word Vectors via Conceptor Negation](https://mlsanity.com/paper/1811.11001/) (2018)

## Built on by (most cited first, 27 of 27)

- [Anisotropy Is Inherent to Self-Attention in Transformers](https://mlsanity.com/paper/2401.12143/) (2024)
- [Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier](https://mlsanity.com/paper/2504.00178/) (2025)
- [Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond](https://mlsanity.com/paper/2310.05317/) (2023)
- [Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity](https://mlsanity.com/paper/2306.00458/) (2023)
- [AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation](https://mlsanity.com/paper/2503.19693/) (2025)
- [HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation](https://mlsanity.com/paper/2212.08853/) (2022)
- [Token Imbalance Adaptation for Radiology Report Generation](https://mlsanity.com/paper/2304.09185/) (2023)
- [Metis: Training LLMs with FP4 Quantization](https://mlsanity.com/paper/2509.00404/) (2025)
- [Isotropic Representation Can Improve Dense Retrieval](https://mlsanity.com/paper/2209.00218/) (2022)
- [No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media](https://mlsanity.com/paper/2211.03634/) (2022)
- [Is Anisotropy Inherent to Transformers?](https://mlsanity.com/paper/2306.07656/) (2023)
- [Headless Language Models: Learning without Predicting with Contrastive Weight Tying](https://mlsanity.com/paper/2309.08351/) (2023)
- [Tokenization for Molecular Foundation Models](https://mlsanity.com/paper/2409.15370/) (2024)
- [InvGC: Robust Cross-Modal Retrieval by Inverse Graph Convolution](https://mlsanity.com/paper/2310.13276/) (2023)
- [Better Embeddings with Coupled Adam](https://mlsanity.com/paper/2502.08441/) (2025)
- [Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention](https://mlsanity.com/paper/2211.07164/) (2022)
- [Disentangling Geometry, Performance, and Training in Language Models](https://mlsanity.com/paper/2602.20433/) (2026)
- [Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics](https://mlsanity.com/paper/2604.08764/) (2026)
- [Token-Level Fitting Issues of Seq2seq Models](https://mlsanity.com/paper/2305.04493/) (2023)
- [BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition](https://mlsanity.com/paper/2308.08625/) (2023)
- [Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion](https://mlsanity.com/paper/2505.17793/) (2025)
- [Revealing the Numeracy Gap: An Empirical Investigation of Text Embedding Models](https://mlsanity.com/paper/2509.05691/) (2025)
- [Sequential Recommendation with Controllable Diversification: Representation Degeneration and Diversity](https://mlsanity.com/paper/2306.11986/) (2023)
- [Norm of Mean Contextualized Embeddings Determines their Variance](https://mlsanity.com/paper/2409.11253/) (2024)
- [Exploring Representational Disparities Between Multilingual and Bilingual Translation Models](https://mlsanity.com/paper/2305.14230/) (2023)
- [Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models](https://mlsanity.com/paper/2408.01308/) (2024)
- [Stop Jostling: Adaptive Negative Sampling Reduces the Marginalization of Low-Resource Language Tokens by Cross-Entropy Loss](https://mlsanity.com/paper/2601.22439/) (2026)

## Abstract (arXiv)

Recent studies have determined that the learned token embeddings of large-scale neural language models are degenerated to be anisotropic with a narrow-cone shape. This phenomenon, called the representation degeneration problem, facilitates an increase in the overall similarity between token embeddings that negatively affect the performance of the models. Although the existing methods that address the degeneration problem based on observations of the phenomenon triggered by the problem improves the performance of the text generation, the training dynamics of token embeddings behind the degeneration problem are still not explored. In this study, we analyze the training dynamics of the token embeddings focusing on rare token embedding. We demonstrate that the specific part of the gradient for rare token embeddings is the key cause of the degeneration problem for all tokens during training stage. Based on the analysis, we propose a novel method called, adaptive gradient gating (AGG). AGG addresses the degeneration problem by gating the specific part of the gradient for rare token embeddings. Experimental results from language modeling, word similarity, and machine translation tasks quantitatively and qualitatively verify the effectiveness of AGG.