Data to 7 Oct 2026

Representation Degeneration Problem in Training Natural Language Generation Models

arXiv 1907.12009 ↗First posted 28 Jul 2019cs.CLcited by 250

Not on the map. Why?

Abstract (arXiv)

We study an interesting problem in training neural network-based models for natural language generation tasks, which we call the representation degeneration problem. We observe that when training a model for natural language generation tasks through likelihood maximization with the weight tying trick, especially with big training datasets, most of the learnt word embeddings tend to degenerate and be distributed into a narrow cone, which largely limits the representation power of word embeddings. We analyze the conditions and causes of this problem and propose a novel regularization method to address it. Experiments on language modeling and machine translation show that our method can largely mitigate the representation degeneration problem and achieve better performance than baseline algorithms.

Key facts. Cited by 250 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 31; 2023: 51; 2024: 33; 2025: 30; 2026: 83.

250 papers.

Sort:

By area:

2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
2022Analyzing Transformers in Embedding Spacecited by 131
2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
2021A Survey of Visual Transformerscited by 117

38 more

2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
2022Self-supervised Trajectory Representation Learning with Temporal Regularities and Travel Semanticscited by 58
2025On the Emergence of Position Bias in Transformerscited by 57
2025PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selectioncited by 57
2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
2021Vision Transformers with Patch Diversificationcited by 45
2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
2022Empowering Diffusion Models on the Embedding Space for Text Generationcited by 40
2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
2023The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Modelscited by 37
2022The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancycited by 35
2021Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translationcited by 34
2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
2021An Isotropy Analysis in the Multilingual BERT Embedding Spacecited by 25
2022Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embeddingcited by 25
2024Towards High-resolution 3D Anomaly Detection via Group-Level Feature Contrastive Learningcited by 25
2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
2024Hyperbolic Fine-Tuning for Large Language Modelscited by 24
2021Learning to Remove: Towards Isotropic Pre-trained BERT Embeddingcited by 23
2021How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropycited by 23
2023Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfercited by 22
2023Text Rendering Strategies for Pixel Language Modelscited by 22
2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
2022CoCoSoDa: Effective Contrastive Learning for Code Searchcited by 21
2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
2023Byte Pair Encoding for Symbolic Musiccited by 19
2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 19
2025Scaling Language-Centric Omnimodal Representation Learningcited by 19

50 most cited of 250 All 250 (JSON)