mlsanityData to 7 Oct 2026

A Contrastive Framework for Neural Text Generation

arXiv 2202.06417 ↗First posted 13 Feb 2022cs.CLcited by 224

Not on the map. Why?

Abstract (arXiv)

Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model's representation space, and (ii) a decoding method -- contrastive search -- to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.

Key facts. Cited by 224 papers in the map (data through 7 Oct 2026). Builds on 52 of its 93 references that are in the map. First posted on arXiv 13 Feb 2022.

Builds on

52 of 93 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
38 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2018Hierarchical Neural Story Generationcited by 1,182
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  9. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  11. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  12. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  13. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  14. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  15. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  16. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  17. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  18. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  19. 2017Challenges in Data-to-Document Generationcited by 226
  20. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 205
  21. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  22. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  23. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  24. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  25. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  26. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  27. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  28. 2021Adaptive Semiparametric Language Modelscited by 83
  29. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  30. 2021CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learningcited by 64
  31. 2019UER: An Open-Source Toolkit for Pre-training Modelscited by 61
  32. 2021Plan-then-Generate: Controlled Data-to-Text Generation via Planningcited by 52
  33. 2021Non-Autoregressive Text Generation with Pre-trained Language Modelscited by 36
  34. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  35. 2020Dialogue Response Selection with Hierarchical Curriculum Learningcited by 27
  36. 2020SongNet: Rigid Formats Controlled Text Generationcited by 25
  37. 2020Prototype-to-Style: Dialogue Generation with Style-Aware Editing on Retrieval Memorycited by 22
  38. 2021Few-Shot Table-to-Text Generation with Prototype Memorycited by 21

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 29; 2023: 62; 2024: 55; 2025: 44; 2026: 34.

224 papers.

Sort:

By area:

  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  3. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  4. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  5. 2025ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generationcited by 177
  6. 2023On the Reliability of Watermarks for Large Language Modelscited by 170
  7. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 112
  8. 2022Latent Diffusion for Language Generationcited by 110
  9. 2023On the Exploitability of Instruction Tuningcited by 110
  10. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  11. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 90
  12. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
38 more
  1. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  2. 2022RankGen: Improving Text Generation with Large Ranking Modelscited by 72
  3. 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 68
  4. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  5. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  6. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  7. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 58
  8. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
  9. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  10. 2022Contrastive language and vision learning of general fashion conceptscited by 41
  11. 2023Click: Controllable Text Generation with Sequence Likelihood Contrastive Learningcited by 41
  12. 2023Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectivecited by 39
  13. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  14. 2025Understanding the Repeat Curse in Large Language Models from a Feature Perspectivecited by 36
  15. 2023OLaLa: Ontology Matching with Large Language Modelscited by 35
  16. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  17. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  18. 2024AI and the Problem of Knowledge Collapsecited by 34
  19. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  20. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  21. 2022Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimizationcited by 32
  22. 2022Visualize Before You Write: Imagination-Guided Open-Ended Text Generationcited by 30
  23. 2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 28
  24. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  25. 2024Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generationcited by 27
  26. 2024Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generationcited by 26
  27. 2023Copy Is All You Needcited by 25
  28. 2023TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wildcited by 22
  29. 2025Sparks of Science: Hypothesis Generation Using Structured Paper Datacited by 22
  30. 2022CoNT: Contrastive Neural Text Generationcited by 21
  31. 2024Improving Open-Ended Text Generation via Adaptive Decodingcited by 21
  32. 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrastcited by 19
  33. 2022A Simple Contrastive Learning Objective for Alleviating Neural Text Degenerationcited by 18
  34. 2022ContraCLM: Contrastive Learning For Causal Language Modelcited by 18
  35. 2023Specialist or Generalist? Instruction Tuning for Specific NLP Taskscited by 17
  36. 2023A Frustratingly Simple Decoding Method for Neural Text Generationcited by 16
  37. 2023MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropiescited by 16
  38. 2024Causal Inference with Generative Artificial Intelligence: Application to Texts as Treatmentscited by 16

50 most cited of 224 All 224 (JSON)