# Towards robust long-context understanding of large language model via active recap learning

arXiv: https://arxiv.org/abs/2601.13734 · HTML: https://mlsanity.com/paper/2601.13734/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 18 references that are in the map.
- First posted on arXiv 20 Jan 2026.

## Builds on (most cited first, 16 of 16)

- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Augmenting Language Models with Long-Term Memory](https://mlsanity.com/paper/2306.07174/) (2023)
- [MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations](https://mlsanity.com/paper/2407.01523/) (2024)
- [How to Train Long-Context Language Models (Effectively)](https://mlsanity.com/paper/2410.02660/) (2024)
- [VCSUM: A Versatile Chinese Meeting Summarization Dataset](https://mlsanity.com/paper/2305.05280/) (2023)
- [LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding](https://mlsanity.com/paper/2503.04359/) (2025)
- [Leveraging Long-Context Large Language Models for Multi-Document Understanding and Summarization in Enterprise Applications](https://mlsanity.com/paper/2409.18454/) (2024)

## Abstract (arXiv)

In this paper, we propose active recap learning (ARL), a framework for enhancing large language model (LLM) in understanding long contexts. ARL enables models to revisit and summarize earlier content through targeted sequence construction during contined pretraining and retrospective summarization at inference. First, we identify key tokens in prepared long context based on loss gaps between long and short forward contexts and find most revant preceding paragraphs, then summarize them using an LLM. Second, ARL equips models with the ability to autonomously generate and utilize these retrospective summaries during inference, thereby establishing a recursive memory mechanism across paragraphs. Experimental results show substantial gains, with ARL achieving a 26.8% improvement on RULER and a 9.44% improvement on LongBench. Overall, ARL offers a simple yet effective continued pretraining-based approach to strengthen long-context understanding, advancing scalable memory augmentation in LLM