mlsanityData to 7 Oct 2026

Evaluating Document Coherence Modelling

arXiv 2103.10133 ↗First posted 18 Mar 2021cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

While pretrained language models ("LM") have driven impressive gains over morpho-syntactic and semantic tasks, their ability to model discourse and pragmatic phenomena is less clear. As a step towards a better understanding of their discourse modelling capabilities, we propose a sentence intrusion detection task. We examine the performance of a broad range of pretrained LMs on this detection task for English. Lacking a dataset for the task, we introduce INSteD, a novel intruder sentence detection dataset, containing 170,000+ documents constructed from English Wikipedia and CNN news articles. Our experiments show that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalisation capacity. Further results over a novel linguistic probe dataset show that there is substantial room for improvement, especially in the cross-domain setting.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 46 of its 97 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

46 of 97 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2015Teaching Machines to Read and Comprehendcited by 1,039
  10. 2018Neural Network Acceptability Judgmentscited by 952
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  12. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
34 more
  1. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  2. 2019Designing and Interpreting Probes with Control Taskscited by 608
  3. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  4. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  5. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  6. 2015Skip-Thought Vectorscited by 321
  7. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  8. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  9. 2018A Simple Method for Commonsense Reasoningcited by 223
  10. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  11. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  12. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  13. 2019Strategies for Structuring Story Generationcited by 124
  14. 2016Technical Report on the CleverHans v2.1.0 Adversarial Examples Librarycited by 123
  15. 2018Learning to Write with Cooperative Discriminatorscited by 119
  16. 2017Deep Text Classification Can be Fooledcited by 105
  17. 2018Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examplescited by 93
  18. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  19. 2018Interpretable Adversarial Perturbation in Input Embedding Space for Textcited by 67
  20. 2017Towards Crafting Text Adversarial Samplescited by 55
  21. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  22. 2018Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Datacited by 37
  23. 2018Deep RNNs Encode Soft Hierarchical Syntaxcited by 34
  24. 2018Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architecturescited by 30
  25. 2019What Makes A Good Story? Designing Composite Rewards for Visual Storytellingcited by 29
  26. 2017Neural Discourse Structure for Text Categorizationcited by 24
  27. 2016Neural Net Models for Open-Domain Discourse Coherencecited by 23
  28. 2017SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoringcited by 22
  29. 2019Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representationscited by 21
  30. 2020Topological Sort for Sentence Orderingcited by 21
  31. 2018Discourse Coherence in the Wild: A Dataset, Evaluation and Methodscited by 20
  32. 2019A Cross-Domain Transferable Neural Coherence Modelcited by 19
  33. 2018Disfluency Detection using Auto-Correlational Neural Networkscited by 15
  34. 2019CITE: A Corpus of Image-Text Discourse Relationscited by 12

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 0; 2024: 2; 2025: 2.

8 papers.

  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  2. 2022SNaC: Coherence Error Detection for Narrative Summarizationcited by 19
  3. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  4. 2021Rethinking Self-Supervision Objectives for Generalizable Coherence Modelingcited by 8
  5. 2022Towards Unification of Discourse Annotation Frameworkscited by 8
  6. 2024CoUDA: Coherence Evaluation via Unified Data Augmentationcited by 0
  7. 2024Converging Dimensions: Information Extraction and Summarization through Multisource, Multimodal, and Multilingual Fusioncited by 0
  8. 2025Coreference as an indicator of context scope in multimodal narrativecited by 0