# Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarization

arXiv: https://arxiv.org/abs/2110.04741 · HTML: https://mlsanity.com/paper/2110.04741/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 10 Oct 2021.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents](https://mlsanity.com/paper/1611.04230/) (2016)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Learned in Translation: Contextualized Word Vectors](https://mlsanity.com/paper/1708.00107/) (2017)
- [Adversarial Multi-task Learning for Text Classification](https://mlsanity.com/paper/1704.05742/) (2017)
- [Dynamic Coattention Networks For Question Answering](https://mlsanity.com/paper/1611.01604/) (2016)
- [Ask the Right Questions: Active Question Reformulation with Reinforcement Learning](https://mlsanity.com/paper/1705.07830/) (2017)
- [Attention-over-Attention Neural Networks for Reading Comprehension](https://mlsanity.com/paper/1607.04423/) (2016)
- [Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation](https://mlsanity.com/paper/1805.11004/) (2018)
- [Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering](https://mlsanity.com/paper/1711.05116/) (2017)
- [Sentence-State LSTM for Text Representation](https://mlsanity.com/paper/1805.02474/) (2018)
- [Meta Multi-Task Learning for Sequence Modeling](https://mlsanity.com/paper/1802.08969/) (2018)
- [Efficient Summarization with Read-Again and Copy Mechanism](https://mlsanity.com/paper/1611.03382/) (2016)
- [Words or Characters? Fine-grained Gating for Reading Comprehension](https://mlsanity.com/paper/1611.01724/) (2016)

## Built on by (most cited first, 6 of 6)

- [Uncovering and Quantifying Social Biases in Code Generation](https://mlsanity.com/paper/2305.15377/) (2023)
- [The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models](https://mlsanity.com/paper/2406.10130/) (2024)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Uncovering and Categorizing Social Biases in Text-to-SQL](https://mlsanity.com/paper/2305.16253/) (2023)
- [Parallel Sentence-Level Explanation Generation for Real-World Low-Resource Scenarios](https://mlsanity.com/paper/2302.10707/) (2023)
- [Real-time 3D Semantic Scene Completion Via Feature Aggregation and Conditioned Prediction](https://mlsanity.com/paper/2303.10967/) (2023)

## Abstract (arXiv)

Long text understanding is important yet challenging in natural language processing. A long article or essay usually contains many redundant words that are not pertinent to its gist and sometimes can be regarded as noise. In this paper, we consider the problem of how to disentangle the gist-relevant and irrelevant information for long text understanding. With distillation mechanism, we transfer the knowledge about how to focus the salient parts from the abstractive summarization model and further integrate the distilled model, named Gist Detector, into existing models as a supplementary component to augment the long text understanding. Experiments on document classification, distantly supervised open-domain question answering (DS-QA) and non-parallel text style transfer show that our method can significantly improve the performance of the baseline models, and achieves state-of-the-art overall results for document classification.