# SUMIE: A Synthetic Benchmark for Incremental Entity Summarization

arXiv: https://arxiv.org/abs/2406.05079 · HTML: https://mlsanity.com/paper/2406.05079/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 11 of 11)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [Resolving Knowledge Conflicts in Large Language Models](https://mlsanity.com/paper/2310.00935/) (2023)
- [Entity Summarization: State of the Art and Future Challenges](https://mlsanity.com/paper/1910.08252/) (2019)
- [Incremental Extractive Opinion Summarization Using Cover Trees](https://mlsanity.com/paper/2401.08047/) (2024)
- [ESA: Entity Summarization with Attention](https://mlsanity.com/paper/1905.10625/) (2019)
- [ESBM: An Entity Summarization BenchMark](https://mlsanity.com/paper/2003.03734/) (2020)

## Built on by (most cited first, 3 of 3)

- [Enhancing Incremental Summarization with Structured Representations](https://mlsanity.com/paper/2407.15021/) (2024)
- [PRISM: Efficient Long-Range Reasoning With Short-Context LLMs](https://mlsanity.com/paper/2412.18914/) (2024)
- [Leveraging LLM For Synchronizing Information Across Multilingual Tables](https://mlsanity.com/paper/2504.02559/) (2025)

## Abstract (arXiv)

No existing dataset adequately tests how well language models can incrementally update entity summaries - a crucial ability as these models rapidly advance. The Incremental Entity Summarization (IES) task is vital for maintaining accurate, up-to-date knowledge. To address this, we introduce SUMIE, a fully synthetic dataset designed to expose real-world IES challenges. This dataset effectively highlights problems like incorrect entity association and incomplete information presentation. Unlike common synthetic datasets, ours captures the complexity and nuances found in real-world data. We generate informative and diverse attributes, summaries, and unstructured paragraphs in sequence, ensuring high quality. The alignment between generated summaries and paragraphs exceeds 96%, confirming the dataset's quality. Extensive experiments demonstrate the dataset's difficulty - state-of-the-art LLMs struggle to update summaries with an F1 higher than 80.4%. We will open source the benchmark and the evaluation metrics to help the community make progress on IES tasks.