# USB: A Unified Summarization Benchmark Across Tasks and Domains

arXiv: https://arxiv.org/abs/2305.14296 · HTML: https://mlsanity.com/paper/2305.14296/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 33)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021)
- [Entity-level Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/2102.09130/) (2021)
- [Abstractive Summarization of Reddit Posts with Multi-level Memory Networks](https://mlsanity.com/paper/1811.00783/) (2018)
- [Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques](https://mlsanity.com/paper/2005.01795/) (2020)
- [WikiAsp: A Dataset for Multi-domain Aspect-based Summarization](https://mlsanity.com/paper/2011.07832/) (2020)
- [Neural Network-Based Abstract Generation for Opinions and Arguments](https://mlsanity.com/paper/1606.02785/) (2016)
- [Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling](https://mlsanity.com/paper/2210.12378/) (2022)
- [On Improving Summarization Factual Consistency from Natural Language Feedback](https://mlsanity.com/paper/2212.09968/) (2022)
- [Scoring Sentence Singletons and Pairs for Abstractive Summarization](https://mlsanity.com/paper/1906.00077/) (2019)
- [Revisiting Summarization Evaluation for Scientific Articles](https://mlsanity.com/paper/1604.00400/) (2016)
- [Visual Interaction with Deep Learning Models through Collaborative Semantic Inference](https://mlsanity.com/paper/1907.10739/) (2019)

## Built on by (most cited first, 5 of 5)

- [GenAudit: Fixing Factual Errors in Language Model Outputs with Evidence](https://mlsanity.com/paper/2402.12566/) (2024)
- [The Power of Summary-Source Alignments](https://mlsanity.com/paper/2406.00842/) (2024)
- [Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generation](https://mlsanity.com/paper/2504.13054/) (2025)
- [How Performance Pressure Influences AI-Assisted Decision Making](https://mlsanity.com/paper/2410.16560/) (2024)
- [A Unifying Scheme for Extractive Content Selection Tasks](https://mlsanity.com/paper/2507.16922/) (2025)

## Abstract (arXiv)

While the NLP community has produced numerous summarization benchmarks, none provide the rich annotations required to simultaneously address many important problems related to control and reliability. We introduce a Wikipedia-derived benchmark, complemented by a rich set of crowd-sourced annotations, that supports 8 interrelated tasks: (i) extractive summarization; (ii) abstractive summarization; (iii) topic-based summarization; (iv) compressing selected sentences into a one-line summary; (v) surfacing evidence for a summary sentence; (vi) predicting the factual accuracy of a summary sentence; (vii) identifying unsubstantiated spans in a summary sentence; (viii) correcting factual errors in summaries. We compare various methods on this benchmark and discover that on multiple tasks, moderately-sized fine-tuned models consistently outperform much larger few-shot prompted language models. For factuality-related tasks, we also evaluate existing heuristics to create training data and find that training on them results in worse performance than training on 20× less human-labeled data. Our articles draw from 6 domains, facilitating cross-domain analysis. On some tasks, the amount of training data matters more than the domain where it comes from, while for other tasks training specifically on data from the target domain, even if limited, is more beneficial.