# SummVis: Interactive Visual Analysis of Models, Data, and Evaluation for Text Summarization

arXiv: https://arxiv.org/abs/2104.07605 · HTML: https://mlsanity.com/paper/2104.07605/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://mlsanity.com/paper/1804.11283/) (2018)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Neural Text Summarization: A Critical Evaluation](https://mlsanity.com/paper/1908.08960/) (2019)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models](https://mlsanity.com/paper/1804.09299/) (2018)
- [Content Selection in Deep Learning Models of Summarization](https://mlsanity.com/paper/1810.12343/) (2018)
- [Earlier Isn't Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarization](https://mlsanity.com/paper/1908.11723/) (2019)
- [Visual Interaction with Deep Learning Models through Collaborative Semantic Inference](https://mlsanity.com/paper/1907.10739/) (2019)
- [A Closer Look at Data Bias in Neural Extractive Summarization Models](https://mlsanity.com/paper/1909.13705/) (2019)
- [How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation](https://mlsanity.com/paper/2101.11298/) (2021)

## Built on by (most cited first, 13 of 13)

- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization](https://mlsanity.com/paper/2301.13298/) (2023)
- [Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents](https://mlsanity.com/paper/2110.10150/) (2021)
- [Measuring Data](https://mlsanity.com/paper/2212.05129/) (2022)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Text-Blueprint: An Interactive Platform for Plan-based Conditional Generation](https://mlsanity.com/paper/2305.00034/) (2023)
- [Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges](https://mlsanity.com/paper/2303.05392/) (2023)
- [SummerTime: Text Summarization Toolkit for Non-experts](https://mlsanity.com/paper/2108.12738/) (2021)
- [Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Study](https://mlsanity.com/paper/2306.01169/) (2023)
- [Summary Explorer: Visualizing the State of the Art in Text Summarization](https://mlsanity.com/paper/2108.01879/) (2021)
- [Seven challenges for harmonizing explainability requirements](https://mlsanity.com/paper/2108.05390/) (2021)
- [Source Identification in Abstractive Summarization](https://mlsanity.com/paper/2402.04677/) (2024)
- [Summary Workbench: Unifying Application and Evaluation of Text Summarization Models](https://mlsanity.com/paper/2210.09587/) (2022)

## Abstract (arXiv)

Novel neural architectures, training strategies, and the availability of large-scale corpora haven been the driving force behind recent progress in abstractive text summarization. However, due to the black-box nature of neural models, uninformative evaluation metrics, and scarce tooling for model and data analysis, the true performance and failure modes of summarization models remain largely unknown. To address this limitation, we introduce SummVis, an open-source tool for visualizing abstractive summaries that enables fine-grained analysis of the models, data, and evaluation metrics associated with text summarization. Through its lexical and semantic visualizations, the tools offers an easy entry point for in-depth model prediction exploration across important dimensions such as factual consistency or abstractiveness. The tool together with several pre-computed model outputs is available at https://github.com/robustness-gym/summvis.