mlsanityData to 7 Oct 2026

Coherence boosting: When your pretrained language model is not paying enough attention

arXiv 2110.08294 ↗First posted 15 Oct 2021cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

Long-range semantic coherence remains a challenge in automatic language generation and understanding. We demonstrate that large language models have insufficiently learned the effect of distant words on next-token prediction. We present coherence boosting, an inference procedure that increases a LM's focus on a long context. We show the benefits of coherence boosting with pretrained models by distributional analyses of generated ordinary text and dialog responses. It is also found that coherence boosting with state-of-the-art models for various zero-shot NLP tasks yields performance gains with no additional training.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 45 of its 72 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

45 of 72 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
33 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  13. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  14. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  15. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  16. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  17. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  18. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  19. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  20. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  21. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  22. 2020Extractive Summarization as Text Matchingcited by 189
  23. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  24. 2018Generating Informative and Diverse Conversational Responses via Adversarial Information Maximizationcited by 114
  25. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  26. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  27. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93
  28. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  29. 2021Language Model Evaluation Beyond Perplexitycited by 77
  30. 2019Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Studycited by 47
  31. 2020GPT-too: A language-model-first approach for AMR-to-text generationcited by 38
  32. 2021Learning to Reason for Text Generation from Scientific Tablescited by 13
  33. 2021Studying word order through iterative shufflingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 14; 2024: 10; 2025: 8; 2026: 2.

38 papers.

Sort:

By area:

  1. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  2. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 226
  3. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  4. 2023Amortizing intractable inference in large language modelscited by 94
  5. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  6. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  7. 2023GPT-RE: In-context Learning for Relation Extraction using Large Language Modelscited by 62
  8. 2024Enhancing Contextual Understanding in Large Language Models through Contrastive Decodingcited by 49
  9. 2022AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languagescited by 46
  10. 2022CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learningcited by 32
  11. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  12. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
26 more
  1. 2025Parallel Scaling Law for Language Modelscited by 29
  2. 2023Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructionscited by 24
  3. 2022N-Best Hypotheses Reranking for Text-To-SQL Systemscited by 19
  4. 2024Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contextscited by 19
  5. 2023GPT is becoming a Turing machine: Here are some ways to program itcited by 16
  6. 2024Selective Prompt Anchoring for Code Generationcited by 16
  7. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactionscited by 14
  8. 2023Increasing Probability Mass on Answer Choices Does Not Always Improve Accuracycited by 11
  9. 2024Latent Logic Tree Extraction for Event Sequence Explanation from LLMscited by 9
  10. 2025Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detectioncited by 8
  11. 2023POE: Process of Elimination for Multiple Choice Reasoningcited by 7
  12. 2024The Queen of England is not England's Queen: On the Lack of Factual Coherency in PLMscited by 7
  13. 2023David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMscited by 6
  14. 2024MM-PoE: Multiple Choice Reasoning via. Process of Elimination using Multi-Modal Modelscited by 5
  15. 2024LLMSat: A Large Language Model-Based Goal-Oriented Agent for Autonomous Space Explorationcited by 3
  16. 2026Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretrainingcited by 3
  17. 2023Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generationcited by 2
  18. 2024GPT-3 Powered Information Extraction for Building Robust Knowledge Basescited by 1
  19. 2024Enhancing Fact Retrieval in PLMs through Truthfulnesscited by 1
  20. 2025Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMscited by 1
  21. 2025Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Modelscited by 1
  22. 2025Short-Context Dominance: How Much Local Context Natural Language Actually Needs?cited by 1
  23. 2026From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decodingcited by 1
  24. 2024Mitigating the Influence of Distractor Tasks in LMs with Prior-Aware Decodingcited by 0
  25. 2025Topological constraints on self-organisation in locally interacting systemscited by 0
  26. 2025Enhancing Instruction-Following Capabilities in Seq2Seq Models: DoLA Adaptations for T5cited by 0