mlsanityData to 7 Oct 2026

Look-back Decoding for Open-Ended Text Generation

arXiv 2305.13477 ↗First posted 22 May 2023cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Given a prefix (context), open-ended generation aims to decode texts that are coherent, which do not abruptly drift from previous topics, and informative, which do not suffer from undesired repetitions. In this paper, we propose Look-back, an improved decoding algorithm that leverages the Kullback-Leibler divergence to track the distribution distance between current and historical decoding steps. Thus Look-back can automatically predict potential repetitive phrase and topic drift, and remove tokens that may cause the failure modes, restricting the next token probability distribution within a plausible distance to the history. We perform decoding experiments on document continuation and story generation, and demonstrate that Look-back is able to generate more fluent and coherent text, outperforming other strong decoding methods significantly in both automatic and human evaluations.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

22 of 28 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2018Hierarchical Neural Story Generationcited by 1,182
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  10. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
10 more
  1. 2022A Contrastive Framework for Neural Text Generationcited by 224
  2. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  3. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  4. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  5. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  6. 2018Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translationcited by 29
  7. 2020A Systematic Characterization of Sampling Algorithms for Open-ended Language Generationcited by 28
  8. 2022A Simple Contrastive Learning Objective for Alleviating Neural Text Degenerationcited by 18
  9. 2022An Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generationcited by 14
  10. 2022An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 1; 2026: 2.

10 papers.

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  2. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  3. 2024Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMscited by 33
  4. 2024Improving Open-Ended Text Generation via Adaptive Decodingcited by 21
  5. 2024Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decodingcited by 11
  6. 2024Monotonic Paraphrasing Improves Generalization of Language Model Promptingcited by 10
  7. 2024Towards Chapter-to-Chapter Context-Aware Literary Translation via Large Language Modelscited by 2
  8. 2026Interrupting the Loop: Periodic Subject Changes Raise Judged Surprise and Connection in Base Language Modelscited by 2
  9. 2026Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decodingcited by 1
  10. 2024Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Predictioncited by 0