mlsanityData to 14 Oct 2026

A Frustratingly Simple Decoding Method for Neural Text Generation

arXiv 2305.12675 ↗First posted 22 May 2023cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

We introduce a frustratingly simple, super efficient and surprisingly effective decoding method, which we call Frustratingly Simple Decoding (FSD), for neural text generation. The idea behind FSD is straightforward: we build an anti-LM based on previously generated text and use this anti-LM to penalize future generation of what has been generated. The anti-LM can be implemented as simple as an n-gram language model or a vectorized variant. In this way, FSD introduces no extra model parameters and negligible computational overhead (FSD can be as fast as greedy search). Despite the simplicity, FSD is surprisingly effective; Experiments show that FSD can outperform the canonical methods to date (i.e., nucleus sampling) as well as several strong baselines that were proposed recently.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 29 of its 47 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

29 of 47 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
17 more
  1. 2018Hierarchical Neural Story Generationcited by 1,193
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  4. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  5. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  6. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  7. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 271
  8. 2022A Contrastive Framework for Neural Text Generationcited by 225
  9. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  10. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
  11. 2022On Decoding Strategies for Neural Text Generatorscited by 90
  12. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  13. 2022COSPLAY: Concept Set Guided Personalized Dialogue Generation Across Both Party Personascited by 20
  14. 2021Relating Neural Text Degeneration to Exposure Biascited by 19
  15. 2021Implicit Unlikelihood Training: Improving Neural Text Generation with Reinforcement Learningcited by 16
  16. 2022An Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generationcited by 14
  17. 2022Momentum Decoding: Open-ended Text Generation As Graph Explorationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 3; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024A Thorough Examination of Decoding Methods in the Era of LLMscited by 114
  2. 2023Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspectivecited by 39
  3. 2024Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generationcited by 26
  4. 2023Specialist or Generalist? Instruction Tuning for Specific NLP Taskscited by 17
  5. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16
  6. 2024Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LMcited by 11
  7. 2026Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspectivecited by 6
  8. 2024Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Modelscited by 4
  9. 2025Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Modelscited by 3
  10. 2023Anti-LM Decoding for Zero-shot In-context Machine Translationcited by 2
  11. 2023TPPoet: Transformer-Based Persian Poem Generation using Minimal Data and Advanced Decoding Techniquescited by 2
  12. 2026Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinationscited by 2
4 more
  1. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
  2. 2025LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generationcited by 0
  3. 2026Temporal Guidance for Large Language Modelscited by 0
  4. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0