# Are Long-LLMs A Necessity For Long-Context Tasks?

arXiv: https://arxiv.org/abs/2405.15318 · HTML: https://mlsanity.com/paper/2405.15318/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](https://mlsanity.com/paper/2306.02224/) (2023)
- [Distilling Knowledge from Reader to Retriever for Question Answering](https://mlsanity.com/paper/2012.04584/) (2020)
- [Retrieval meets Long Context Large Language Models](https://mlsanity.com/paper/2310.03025/) (2023)
- [LongCoder: A Long-Range Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2306.14893/) (2023)

## Built on by (most cited first, 22 of 22)

- [MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation](https://mlsanity.com/paper/2409.05591/) (2024)
- [CompAct: Compressing Retrieved Documents Actively for Question Answering](https://mlsanity.com/paper/2407.09014/) (2024)
- [Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Example](https://mlsanity.com/paper/2408.06318/) (2024)
- [Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k](https://mlsanity.com/paper/2506.08479/) (2025)
- [Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foraging](https://mlsanity.com/paper/2505.09316/) (2025)
- [LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Models](https://mlsanity.com/paper/2410.09342/) (2024)
- [Evaluating the Goal-Directedness of Large Language Models](https://mlsanity.com/paper/2504.11844/) (2025)
- [LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards](https://mlsanity.com/paper/2603.02146/) (2026)
- [From Perceptions to Decisions: Wildfire Evacuation Decision Prediction with Behavioral Theory-informed LLMs](https://mlsanity.com/paper/2502.17701/) (2025)
- [Practical Considerations for Agentic LLM Systems](https://mlsanity.com/paper/2412.04093/) (2024)
- [When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework](https://mlsanity.com/paper/2506.16411/) (2025)
- [Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning](https://mlsanity.com/paper/2502.13127/) (2025)
- [Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration](https://mlsanity.com/paper/2505.21471/) (2025)
- [Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Tasks](https://mlsanity.com/paper/2409.06338/) (2024)
- [PRISM: Efficient Long-Range Reasoning With Short-Context LLMs](https://mlsanity.com/paper/2412.18914/) (2024)
- [Replace, Don't Expand: Mitigating Context Dilution in Multi-Hop RAG via Fixed-Budget Evidence Assembly](https://mlsanity.com/paper/2512.10787/) (2025)
- [Boosting Long-Context Management via Query-Guided Activation Refilling](https://mlsanity.com/paper/2412.12486/) (2024)
- [Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks](https://mlsanity.com/paper/2507.19353/) (2025)
- [PL-CA: A Parametric Legal Case Augmentation Framework](https://mlsanity.com/paper/2509.06356/) (2025)
- [Context-Driven Performance Modeling for Causal Inference Operators on Neural Processing Units](https://mlsanity.com/paper/2509.25155/) (2025)
- [Towards Infinite Length Extrapolation: A Unified Approach](https://mlsanity.com/paper/2601.06113/) (2026)
- [DISCO: Distributed Long Context Scaling with Grounding-Reasoning Disaggregation](https://mlsanity.com/paper/2609.33485/) (2026)

## Abstract (arXiv)

The learning and deployment of long-LLMs remains a challenging problem despite recent progresses. In this work, we argue that the long-LLMs are not a necessity to solve long-context tasks, as common long-context tasks are short-context solvable, i.e. they can be solved by purely working with oracle short-contexts within the long-context tasks' inputs. On top of this argument, we propose a framework called LC-Boost (Long-Context Bootstrapper), which enables a short-LLM to address the long-context tasks in a bootstrapping manner. In our framework, the short-LLM prompts itself to reason for two critical decisions: 1) how to access to the appropriate part of context within the input, 2) how to make effective use of the accessed context. By adaptively accessing and utilizing the context based on the presented tasks, LC-Boost can serve as a general framework to handle diversified long-context processing problems. We comprehensively evaluate different types of tasks from popular long-context benchmarks, where LC-Boost is able to achieve a substantially improved performance with a much smaller consumption of resource.