# Improving Long Text Understanding with Knowledge Distilled from Summarization Model

arXiv: https://arxiv.org/abs/2405.04955 · HTML: https://mlsanity.com/paper/2405.04955/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 41 references that are in the map.
- First posted on arXiv 8 May 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision](https://mlsanity.com/paper/2106.01226/) (2021)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Context Autoencoder for Self-Supervised Representation Learning](https://mlsanity.com/paper/2202.03026/) (2022)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://mlsanity.com/paper/1704.05179/) (2017)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior](https://mlsanity.com/paper/2003.14052/) (2020)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)
- [Entity-level Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/2102.09130/) (2021)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [Compressible-composable NeRF via Rank-residual Decomposition](https://mlsanity.com/paper/2205.14870/) (2022)
- [Adversarial Multi-task Learning for Text Classification](https://mlsanity.com/paper/1704.05742/) (2017)
- [Efficient Long-Text Understanding with Short-Text Models](https://mlsanity.com/paper/2208.00748/) (2022)
- [When Language Model Meets Private Library](https://mlsanity.com/paper/2210.17236/) (2022)
- [AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization](https://mlsanity.com/paper/2103.11332/) (2021)
- [Ask the Right Questions: Active Question Reformulation with Reinforcement Learning](https://mlsanity.com/paper/1705.07830/) (2017)
- [Uncovering and Quantifying Social Biases in Code Generation](https://mlsanity.com/paper/2305.15377/) (2023)
- [Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering](https://mlsanity.com/paper/1711.05116/) (2017)
- [Sentence-State LSTM for Text Representation](https://mlsanity.com/paper/1805.02474/) (2018)
- [Not All Voxels Are Equal: Semantic Scene Completion from the Point-Voxel Perspective](https://mlsanity.com/paper/2112.12925/) (2021)
- [Conditional DETR V2: Efficient Detection Transformer with Box Queries](https://mlsanity.com/paper/2207.08914/) (2022)
- [Point Scene Understanding via Disentangled Instance Mesh Reconstruction](https://mlsanity.com/paper/2203.16832/) (2022)
- [Meta Multi-Task Learning for Sequence Modeling](https://mlsanity.com/paper/1802.08969/) (2018)

## Built on by (most cited first, 3 of 3)

- [A Multi-Level Benchmark for Causal Language Understanding in Social Media Discourse](https://mlsanity.com/paper/2509.16722/) (2025)
- [98× Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router](https://mlsanity.com/paper/2603.12646/) (2026)
- [Improving Expert Radiology Report Summarization by Prompting Large Language Models with a Layperson Summary](https://mlsanity.com/paper/2406.14500/) (2024)

## Abstract (arXiv)

Long text understanding is important yet challenging for natural language processing. A long article or document usually contains many redundant words that are not pertinent to its gist and sometimes can be regarded as noise. With recent advances of abstractive summarization, we propose our Gist Detector to leverage the gist detection ability of a summarization model and integrate the extracted gist into downstream models to enhance their long text understanding ability. Specifically, Gist Detector first learns the gist detection knowledge distilled from a summarization model, and then produces gist-aware representations to augment downstream models. We evaluate our method on three different tasks: long document classification, distantly supervised open-domain question answering, and non-parallel text style transfer. The experimental results show that our method can significantly improve the performance of baseline models on all tasks.