# Summaformers @ LaySumm 20, LongSumm 20

arXiv: https://arxiv.org/abs/2101.03553 · HTML: https://mlsanity.com/paper/2101.03553/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 10 Jan 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://mlsanity.com/paper/1804.05685/) (2018)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [Fine-tune BERT for Extractive Summarization](https://mlsanity.com/paper/1903.10318/) (2019)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [On Extractive and Abstractive Neural Document Summarization with Transformer Language Models](https://mlsanity.com/paper/1909.03186/) (2019)
- [Text Summarization Techniques: A Brief Survey](https://mlsanity.com/paper/1707.02268/) (2017)
- [Extractive Summarization of Long Documents by Combining Global and Local Context](https://mlsanity.com/paper/1909.08089/) (2019)
- [A Supervised Approach to Extractive Summarisation of Scientific Papers](https://mlsanity.com/paper/1706.03946/) (2017)
- [Adapting Language Models for Non-Parallel Author-Stylized Rewriting](https://mlsanity.com/paper/1909.09962/) (2019)
- [TalkSumm: A Dataset and Scalable Annotation Method for Scientific Paper Summarization Based on Conference Talks](https://mlsanity.com/paper/1906.01351/) (2019)
- [A Summarization System for Scientific Documents](https://mlsanity.com/paper/1908.11152/) (2019)
- [Hybrid MemNet for Extractive Summarization](https://mlsanity.com/paper/1912.11701/) (2019)
- [Unity in Diversity: Learning Distributed Heterogeneous Sentence Representation for Extractive Summarization](https://mlsanity.com/paper/1912.11688/) (2019)

## Built on by (most cited first, 4 of 4)

- [Cross-Domain Multi-Task Learning for Sequential Sentence Classification in Research Papers](https://mlsanity.com/paper/2102.06008/) (2021)
- [TSTR: Too Short to Represent, Summarize with Details! Intro-Guided Extended Summary Generation](https://mlsanity.com/paper/2206.00847/) (2022)
- [Lay Text Summarisation Using Natural Language Processing: A Narrative Literature Review](https://mlsanity.com/paper/2303.14222/) (2023)
- [Advancements in Natural Language Processing for Automatic Text Summarization](https://mlsanity.com/paper/2502.19773/) (2025)

## Abstract (arXiv)

Automatic text summarization has been widely studied as an important task in natural language processing. Traditionally, various feature engineering and machine learning based systems have been proposed for extractive as well as abstractive text summarization. Recently, deep learning based, specifically Transformer-based systems have been immensely popular. Summarization is a cognitively challenging task - extracting summary worthy sentences is laborious, and expressing semantics in brief when doing abstractive summarization is complicated. In this paper, we specifically look at the problem of summarizing scientific research papers from multiple domains. We differentiate between two types of summaries, namely, (a) LaySumm: A very short summary that captures the essence of the research paper in layman terms restricting overtly specific technical jargon and (b) LongSumm: A much longer detailed summary aimed at providing specific insights into various ideas touched upon in the paper. While leveraging latest Transformer-based models, our systems are simple, intuitive and based on how specific paper sections contribute to human summaries of the two types described above. Evaluations against gold standard summaries using ROUGE metrics prove the effectiveness of our approach. On blind test corpora, our system ranks first and third for the LongSumm and LaySumm tasks respectively.