# OpenAsp: A Benchmark for Multi-document Open Aspect-based Summarization

arXiv: https://arxiv.org/abs/2312.04440 · HTML: https://mlsanity.com/paper/2312.04440/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 21 of 21)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [Extractive Opinion Summarization in Quantized Transformer Spaces](https://mlsanity.com/paper/2012.04443/) (2020)
- [Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised](https://mlsanity.com/paper/1808.08858/) (2018)
- [WikiAsp: A Dataset for Multi-domain Aspect-based Summarization](https://mlsanity.com/paper/2011.07832/) (2020)
- [SummIt: Iterative Text Summarization via ChatGPT](https://mlsanity.com/paper/2305.14835/) (2023)
- [Aspect-Controllable Opinion Summarization](https://mlsanity.com/paper/2109.03171/) (2021)
- [Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization](https://mlsanity.com/paper/1808.06218/) (2018)
- [AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarization](https://mlsanity.com/paper/2010.12694/) (2020)
- [ASPECTNEWS: Aspect-Oriented Summarization of News Documents](https://mlsanity.com/paper/2110.08296/) (2021)
- [Controlled Crowdsourcing for High-Quality QA-SRL Annotation](https://mlsanity.com/paper/1911.03243/) (2019)
- [Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documents](https://mlsanity.com/paper/2106.00130/) (2021)
- [OASum: Large-Scale Open Domain Aspect-based Summarization](https://mlsanity.com/paper/2212.09233/) (2022)
- [Summarizing Text on Any Aspects: A Knowledge-Informed Weakly-Supervised Approach](https://mlsanity.com/paper/2010.06792/) (2020)
- [Massive Multi-Document Summarization of Product Reviews with Weak Supervision](https://mlsanity.com/paper/2007.11348/) (2020)

## Built on by (most cited first, 15 of 15)

- [Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP](https://mlsanity.com/paper/2407.00402/) (2024)
- [MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows](https://mlsanity.com/paper/2406.06357/) (2024)
- [From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization](https://mlsanity.com/paper/2410.13961/) (2024)
- [Multi2: Multi-Agent Test-Time Scalable Framework for Multi-Document Processing](https://mlsanity.com/paper/2502.20592/) (2025)
- [GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization](https://mlsanity.com/paper/2410.04087/) (2024)
- [SEAM: A Stochastic Benchmark for Multi-Document Tasks](https://mlsanity.com/paper/2406.16086/) (2024)
- [Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts](https://mlsanity.com/paper/2402.10554/) (2024)
- [MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections](https://mlsanity.com/paper/2502.00322/) (2025)
- [MODABS: Multi-Objective Learning for Dynamic Aspect-Based Summarization](https://mlsanity.com/paper/2406.03479/) (2024)
- [IDEAL: Leveraging Infinite and Dynamic Characterizations of Large Language Models for Query-focused Summarization](https://mlsanity.com/paper/2407.10486/) (2024)
- [Language Models Should be Used to Surface the Unwritten Code of Science and Society](https://mlsanity.com/paper/2505.18942/) (2025)
- [A Unifying Scheme for Extractive Content Selection Tasks](https://mlsanity.com/paper/2507.16922/) (2025)
- [MARS: Multilingual Aspect-centric Review Summarisation](https://mlsanity.com/paper/2410.09991/) (2024)
- [Topic-Guided Reinforcement Learning with LLMs for Enhancing Multi-Document Summarization](https://mlsanity.com/paper/2509.09852/) (2025)
- [CARPAS: Towards Content-Aware Refinement of Provided Aspects for Summarization in Large Language Models](https://mlsanity.com/paper/2510.07177/) (2025)

## Abstract (arXiv)

The performance of automatic summarization models has improved dramatically in recent years. Yet, there is still a gap in meeting specific information needs of users in real-world scenarios, particularly when a targeted summary is sought, such as in the useful aspect-based summarization setting targeted in this paper. Previous datasets and studies for this setting have predominantly concentrated on a limited set of pre-defined aspects, focused solely on single document inputs, or relied on synthetic data. To advance research on more realistic scenarios, we introduce OpenAsp, a benchmark for multi-document textitopen aspect-based summarization. This benchmark is created using a novel and cost-effective annotation protocol, by which an open aspect dataset is derived from existing generic multi-document summarization datasets. We analyze the properties of OpenAsp showcasing its high-quality content. Further, we show that the realistic open-aspect setting realized in OpenAsp poses a challenge for current state-of-the-art summarization models, as well as for large language models.