# Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs

arXiv: https://arxiv.org/abs/2407.04855 · HTML: https://mlsanity.com/paper/2407.04855/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 5 Jul 2024.

## Builds on (most cited first, 18 of 18)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [MeetingBank: A Benchmark Dataset for Meeting Summarization](https://mlsanity.com/paper/2305.17529/) (2023)
- [Learning to Extract Coherent Summary via Deep Reinforcement Learning](https://mlsanity.com/paper/1804.07036/) (2018)
- [Extractive Summarization via ChatGPT for Faithful Summary Generation](https://mlsanity.com/paper/2304.04193/) (2023)
- [DebateSum: A large-scale argument mining and summarization dataset](https://mlsanity.com/paper/2011.07251/) (2020)
- [Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control](https://mlsanity.com/paper/2207.02263/) (2022)
- [Enhancing Coherence of Extractive Summarization with Multitask Learning](https://mlsanity.com/paper/2305.12851/) (2023)

## Built on by (most cited first, 3 of 3)

- [Joint Modeling of Entities and Discourse Relations for Coherence Assessment](https://mlsanity.com/paper/2509.04182/) (2025)
- [Not Worth Mentioning? A Pilot Study on Salient Proposition Annotation](https://mlsanity.com/paper/2603.27358/) (2026)
- [A Unifying Scheme for Extractive Content Selection Tasks](https://mlsanity.com/paper/2507.16922/) (2025)

## Abstract (arXiv)

Extractive summarization plays a pivotal role in natural language processing due to its wide-range applications in summarizing diverse content efficiently, while also being faithful to the original content. Despite significant advancement achieved in extractive summarization by Large Language Models (LLMs), these summaries frequently exhibit incoherence. An important aspect of the coherent summary is its readability for intended users. Although there have been many datasets and benchmarks proposed for creating coherent extractive summaries, none of them currently incorporate user intent to improve coherence in extractive summarization. Motivated by this, we propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback, offering valuable insights into how to improve coherence in extractive summaries. We utilize this dataset for aligning LLMs through supervised fine-tuning with natural language human feedback to enhance the coherence of their generated summaries. Preliminary experiments with Falcon-40B and Llama-2-13B show significant performance improvements (~10% Rouge-L) in terms of producing coherent summaries. We further utilize human feedback to benchmark results over instruction-tuned models such as FLAN-T5 which resulted in several interesting findings. Data and source code are available at https://github.com/Mihir3009/Extract-AI.