mlsanityData to 7 Oct 2026

DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence

arXiv 2201.11176 ↗First posted 26 Jan 2022cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Recently, there has been a growing interest in designing text generation systems from a discourse coherence perspective, e.g., modeling the interdependence between sentences. Still, recent BERT-based evaluation metrics are weak in recognizing coherence, and thus are not reliable in a way to spot the discourse-level improvements of those text generation systems. In this work, we introduce DiscoScore, a parametrized discourse metric, which uses BERT to model discourse coherence from different perspectives, driven by Centering theory. Our experiments encompass 16 non-discourse and discourse metrics, including DiscoScore and popular coherence models, evaluated on summarization and document-level machine translation (MT). We find that (i) the majority of BERT-based metrics correlate much worse with human rated coherence than early discourse metrics, invented a decade ago; (ii) the recent state-of-the-art BARTScore is weak when operated at system level -- which is particularly problematic as systems are typically compared in this manner. DiscoScore, in contrast, achieves strong system-level correlation with human ratings, not only in coherence but also in factual consistency and other aspects, and surpasses BARTScore by over 10 correlation points on average. Further, aiming to understand DiscoScore, we provide justifications to the importance of discourse coherence for evaluation metrics, and explain the superiority of one variant over another. Our code is available at urlhttps://github.com/AIPHES/DiscoScore.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 45 of its 87 references that are in the map. First posted on arXiv 26 Jan 2022.

Builds on

45 of 87 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  11. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
33 more
  1. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  2. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  3. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  4. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  5. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  6. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  7. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  8. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  9. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 104
  10. 2018Context-Aware Neural Machine Translation Learns Anaphora Resolutioncited by 97
  11. 2021Learning Compact Metrics for MTcited by 93
  12. 2017Evaluating Discourse Phenomena in Neural Machine Translationcited by 87
  13. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  14. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  15. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  16. 2020Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Modelscited by 49
  17. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  18. 2021Discourse Probing of Pretrained Language Modelscited by 38
  19. 2020GRUEN for Evaluating Linguistic Quality of Generated Textcited by 37
  20. 2020On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluationcited by 36
  21. 2021Coreference-Aware Dialogue Summarizationcited by 34
  22. 2021Better than Average: Paired Evaluation of NLP Systemscited by 30
  23. 2021The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Resultscited by 27
  24. 2021Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factorscited by 25
  25. 2019Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representationscited by 21
  26. 2021A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancycited by 21
  27. 2018Discourse Coherence in the Wild: A Dataset, Evaluation and Methodscited by 20
  28. 2019A Unified Neural Coherence Modelcited by 19
  29. 2021Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Testcited by 17
  30. 2018Modeling Coherence for Discourse Neural Machine Translationcited by 15
  31. 2021Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Modelscited by 15
  32. 2017Discourse Structure in Machine Translation Evaluationcited by 7
  33. 2020Using Context in Neural Machine Translation Training Objectivescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 13; 2025: 7; 2026: 6.

35 papers.

Sort:

By area:

  1. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  2. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  3. 2022Model Criticism for Long-Form Text Generationcited by 21
  4. 2022MENLI: Robust Evaluation Metrics from Natural Language Inferencecited by 18
  5. 2024SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similaritycited by 18
  6. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  7. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  8. 2023Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluationcited by 14
  9. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  10. 2023Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluationcited by 8
  11. 2024Pap2Pat: Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairscited by 8
  12. 2024Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherencecited by 7
23 more
  1. 2024Creativity in AI: Progresses and Challengescited by 7
  2. 2024Parse Trees Guided LLM Prompt Compressioncited by 6
  3. 2022How to Find Strong Summary Coherence Measures? A Toolbox and a Comparative Study for Summary Coherence Measure Evaluationcited by 4
  4. 2025DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translationcited by 4
  5. 2024Presence or Absence: Are Unknown Word Usages in Dictionaries?cited by 3
  6. 2023AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Contentcited by 2
  7. 2025Joint Modeling of Entities and Discourse Relations for Coherence Assessmentcited by 2
  8. 2024LUNA: A Framework for Language Understanding and Naturalness Assessmentcited by 1
  9. 2024Legally Binding but Unfair? Towards Assessing Fairness of Privacy Policiescited by 1
  10. 2024What Would You Ask When You First Saw a2+b2=c2? Evaluating LLM on Curiosity-Driven Questioningcited by 1
  11. 2025Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluationcited by 1
  12. 2026SteuerLLM: Local specialized large language model for German tax law analysiscited by 1
  13. 2023Nine-year-old children outperformed ChatGPT in emotion: Evidence from Chinese writingcited by 0
  14. 2024`Keep it Together': Enforcing Cohesion in Extractive Summaries by Simulating Human Memorycited by 0
  15. 2024CoUDA: Coherence Evaluation via Unified Data Augmentationcited by 0
  16. 2025Synthetic News Generation for Fake News Classificationcited by 0
  17. 2025Data-driven Methods of Extracting Text Structure and Information Transfercited by 0
  18. 2025Unlocking Latent Discourse Translation in LLMs Through Quality-Aware Decodingcited by 0
  19. 2026Which course? Discourse! Teaching Discourse and Generation in the Era of LLMscited by 0
  20. 2026NLP for Local Governance Meeting Records: A Focus Article on Tasks, Datasets, Metrics and Benchmarkcited by 0
  21. 2026Reference-Based Analysis of Coherence and Diversity in Open-Ended Text Generationcited by 0
  22. 2026LogicTree-RAG: Logic Tree-guided Retrieval-Augmented Generation for Long-form Patent Draftingcited by 0
  23. 2026Coherence-Aware Distributional Evaluation of Open-Ended Text Generationcited by 0