mlsanityData to 14 Oct 2026

ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs

arXiv 2602.03226 ↗First posted 3 Feb 2026cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Long-context inputs in large language models (LLMs) often suffer from the "lost in the middle" problem, where critical information becomes diluted or ignored due to excessive length. Context compression methods aim to address this by reducing input size, but existing approaches struggle with balancing information preservation and compression efficiency. We propose Adaptive Task-Aware Compressor (ATACompressor), which dynamically adjusts compression based on the specific requirements of the task. ATACompressor employs a selective encoder that compresses only the task-relevant portions of long contexts, ensuring that essential information is preserved while reducing unnecessary content. Its adaptive allocation controller perceives the length of relevant content and adjusts the compression rate accordingly, optimizing resource utilization. We evaluate ATACompressor on three QA datasets: HotpotQA, MSMARCO, and SQUAD-showing that it outperforms existing methods in terms of both compression efficiency and task performance. Our approach provides a scalable solution for long-context processing in LLMs. Furthermore, we perform a range of ablation studies and analysis experiments to gain deeper insights into the key components of ATACompressor.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

24 of 34 references.

References · 24

  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  7. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  8. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  9. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  10. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
  11. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  12. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
12 more
  1. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  2. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 125
  3. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  4. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
  5. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  6. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  7. 2021Generalized Huber Loss for Robust Learning and its Efficient Minimization for a Robust Statisticscited by 32
  8. 2023Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generationcited by 31
  9. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMscited by 24
  10. 2024Beyond-RAG: Question Identification and Answer Generation in Real-Time Conversationscited by 9
  11. 2024Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generationcited by 6
  12. 2024Evaluating LLMs with Multiple Problems at oncecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASRcited by 2
  2. 2026Multi-Layered Memory Architectures for LLM Agents: An Experimental Evaluation of Long-Term Context Retentioncited by 2
  3. 2026Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Modelscited by 1
  4. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inferencecited by 1
  5. 2026Developing Adaptive Context Compression Techniques for Large Language Models (LLMs) in Long-Running Interactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  2. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  3. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  4. 2024Long Context Compression with Activation Beacon
  5. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  6. 2025Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention
  7. 2025Sentence-Anchored Gist Compression for Long-Context LLMs
  8. 2026End-to-End Context Compression at Scale