# FocusLLM: Precise Understanding of Long Context by Dynamic Condensing

arXiv: https://arxiv.org/abs/2408.11745 · HTML: https://mlsanity.com/paper/2408.11745/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2024.

## Built on by (most cited first, 10 of 10)

- [MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent](https://mlsanity.com/paper/2507.02259/) (2025)
- [Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets](https://mlsanity.com/paper/2604.02460/) (2026)
- [Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Tasks](https://mlsanity.com/paper/2409.06338/) (2024)
- [HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs](https://mlsanity.com/paper/2507.19823/) (2025)
- [Periodic RoPE for Infinite Context LLMs](https://mlsanity.com/paper/2605.27980/) (2026)
- [PEVLM: Parallel Encoding for Vision-Language Models](https://mlsanity.com/paper/2506.19651/) (2025)
- [LLM as Graph Kernel: Rethinking Message Passing on Text-Rich Graphs](https://mlsanity.com/paper/2603.14937/) (2026)
- [UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inference](https://mlsanity.com/paper/2603.18446/) (2026)
- [CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.11101/) (2025)
- [Semantic Drift and the Stability of Operator Control in Reasoning-Class Decision Support Systems](https://mlsanity.com/paper/2607.09790/) (2026)

## Abstract (arXiv)

Empowering LLMs with the ability to precisely understand long contexts is crucial for many downstream applications. However, handling long contexts with conventional transformer architecture requires substantial training and inference resources. Existing context condensing methods cannot accurately understand the full context, as there is a considerable amount of information loss in the condensing process. To address these issues, we present FocusLLM, a framework designed to extend the fixed context length of any decoder-only LLM, allowing the model to focus on relevant information from very long sequences. FocusLLM first divides long text input into chunks based on the model's original context length. It then employs the dynamic condensing process to distill crucial information from each chunk. Ultimately, through the novel parallel decoding mechanism, FocusLLM can integrate the extracted information into its local context. FocusLLM stands out for great training efficiency and versatility: trained with an 8K input length and with much less training cost than previous methods, FocusLLM exhibits superior performance across downstream tasks and maintains strong language modeling ability when handling extensive long texts, even up to 400K tokens. Our code is available at https://github.com/leezythu/FocusLLM.