# Unlocking Structure Measuring: Introducing PDD, an Automatic Metric for Positional Discourse Coherence

arXiv: https://arxiv.org/abs/2402.10175 · HTML: https://mlsanity.com/paper/2402.10175/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation](https://mlsanity.com/paper/1904.03396/) (2019)
- [Better Summarization Evaluation with Word Embeddings for ROUGE](https://mlsanity.com/paper/1508.06034/) (2015)
- [Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence](https://mlsanity.com/paper/2105.08963/) (2021)
- [Discourse Probing of Pretrained Language Models](https://mlsanity.com/paper/2104.05882/) (2021)
- [GRUEN for Evaluating Linguistic Quality of Generated Text](https://mlsanity.com/paper/2010.02498/) (2020)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://mlsanity.com/paper/2201.11176/) (2022)
- [Model Criticism for Long-Form Text Generation](https://mlsanity.com/paper/2210.08444/) (2022)
- [How Do We Answer Complex Questions: Discourse Structure of Long-form Answers](https://mlsanity.com/paper/2203.11048/) (2022)
- [Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test](https://mlsanity.com/paper/2107.03448/) (2021)
- [Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models](https://mlsanity.com/paper/2105.03495/) (2021)
- [Plug-and-Play Recipe Generation with Content Planning](https://mlsanity.com/paper/2212.05093/) (2022)
- [Instruct-SCTG: Guiding Sequential Controlled Text Generation through Instructions](https://mlsanity.com/paper/2312.12299/) (2023)

## Built on by (most cited first, 7 of 7)

- [Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators](https://mlsanity.com/paper/2403.16950/) (2024)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [QUDsim: Quantifying Discourse Similarities in LLM-Generated Text](https://mlsanity.com/paper/2504.09373/) (2025)
- [Measuring Psychological Depth in Language Models](https://mlsanity.com/paper/2406.12680/) (2024)
- [Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization](https://mlsanity.com/paper/2502.06185/) (2025)
- [MCP-Orchestrated Multi-Agent System for Automated Disinformation Detection](https://mlsanity.com/paper/2508.10143/) (2025)
- [Dial HEALTHDIAL for Advice: A Multilingual and Multi-Parallel Spoken Dialogue Dataset for Knowledge-Grounded Information Seeking](https://mlsanity.com/paper/2605.30107/) (2026)

## Abstract (arXiv)

Recent large language models (LLMs) have shown remarkable performance in aligning generated text with user intentions across various tasks. When it comes to long-form text generation, there has been a growing interest in generation from a discourse coherence perspective. However, existing lexical or semantic metrics such as BLEU, ROUGE, BertScore cannot effectively capture the discourse coherence. The development of discourse-specific automatic evaluation methods for assessing the output of LLMs warrants greater focus and exploration. In this paper, we present a novel automatic metric designed to quantify the discourse divergence between two long-form articles. Extensive experiments on three datasets from representative domains demonstrate that our metric aligns more closely with human preferences and GPT-4 coherence evaluation, outperforming existing evaluation methods.