# ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs

arXiv: https://arxiv.org/abs/2602.03226 · HTML: https://mlsanity.com/paper/2602.03226/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning](https://mlsanity.com/paper/2401.06805/) (2024)
- [xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token](https://mlsanity.com/paper/2405.13792/) (2024)
- [PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training](https://mlsanity.com/paper/2309.10400/) (2023)
- [Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization](https://mlsanity.com/paper/2406.16008/) (2024)
- [How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study](https://mlsanity.com/paper/2402.16061/) (2024)
- [Probing the Emergence of Cross-lingual Alignment during LLM Training](https://mlsanity.com/paper/2406.13229/) (2024)
- [Generalized Huber Loss for Robust Learning and its Efficient Minimization for a Robust Statistics](https://mlsanity.com/paper/2108.12627/) (2021)
- [Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation](https://mlsanity.com/paper/2311.00306/) (2023)
- [Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs](https://mlsanity.com/paper/2406.02376/) (2024)
- [Beyond-RAG: Question Identification and Answer Generation in Real-Time Conversations](https://mlsanity.com/paper/2410.10136/) (2024)
- [Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generation](https://mlsanity.com/paper/2409.15699/) (2024)
- [Evaluating LLMs with Multiple Problems at once](https://mlsanity.com/paper/2406.10786/) (2024)

## Built on by (most cited first, 5 of 5)

- [Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR](https://mlsanity.com/paper/2603.26246/) (2026)
- [Multi-Layered Memory Architectures for LLM Agents: An Experimental Evaluation of Long-Term Context Retention](https://mlsanity.com/paper/2603.29194/) (2026)
- [Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Models](https://mlsanity.com/paper/2606.12203/) (2026)
- [Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference](https://mlsanity.com/paper/2609.25537/) (2026)
- [Developing Adaptive Context Compression Techniques for Large Language Models (LLMs) in Long-Running Interactions](https://mlsanity.com/paper/2603.29193/) (2026)

## Abstract (arXiv)

Long-context inputs in large language models (LLMs) often suffer from the "lost in the middle" problem, where critical information becomes diluted or ignored due to excessive length. Context compression methods aim to address this by reducing input size, but existing approaches struggle with balancing information preservation and compression efficiency. We propose Adaptive Task-Aware Compressor (ATACompressor), which dynamically adjusts compression based on the specific requirements of the task. ATACompressor employs a selective encoder that compresses only the task-relevant portions of long contexts, ensuring that essential information is preserved while reducing unnecessary content. Its adaptive allocation controller perceives the length of relevant content and adjusts the compression rate accordingly, optimizing resource utilization. We evaluate ATACompressor on three QA datasets: HotpotQA, MSMARCO, and SQUAD-showing that it outperforms existing methods in terms of both compression efficiency and task performance. Our approach provides a scalable solution for long-context processing in LLMs. Furthermore, we perform a range of ablation studies and analysis experiments to gain deeper insights into the key components of ATACompressor.