# Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs

arXiv: https://arxiv.org/abs/2406.02376 · HTML: https://mlsanity.com/paper/2406.02376/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Placed in

- Efficient LLM inference and serving / [Prompt and context compression](https://mlsanity.com/inference/prompt-and-context-compression/)

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](https://mlsanity.com/paper/2110.01786/) (2021)
- [Unlimiformer: Long-Range Transformers with Unlimited Length Input](https://mlsanity.com/paper/2305.01625/) (2023)
- [Learning to Filter Context for Retrieval-Augmented Generation](https://mlsanity.com/paper/2311.08377/) (2023)
- [Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study](https://mlsanity.com/paper/2304.06762/) (2023)
- [RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models](https://mlsanity.com/paper/2308.07922/) (2023)
- [Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering](https://mlsanity.com/paper/2101.00294/) (2021)
- [Simplifying Neural Machine Translation with Addition-Subtraction Twin-Gated Recurrent Networks](https://mlsanity.com/paper/1810.12546/) (2018)

## Built on by (most cited first, 24 of 24)

- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [CompAct: Compressing Retrieved Documents Actively for Question Answering](https://mlsanity.com/paper/2407.09014/) (2024)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [A Survey of Small Language Models](https://mlsanity.com/paper/2410.20011/) (2024)
- [FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.08938/) (2025)
- [Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios](https://mlsanity.com/paper/2409.19272/) (2024)
- [Long Context In-Context Compression by Getting to the Gist of Gisting](https://mlsanity.com/paper/2504.08934/) (2025)
- [COMI: Coarse-to-fine Context Compression via Marginal Information Gain](https://mlsanity.com/paper/2602.01719/) (2026)
- [Read As Human: Compressing Context via Parallelizable Close Reading and Skimming](https://mlsanity.com/paper/2602.01840/) (2026)
- [CompLLM: Compression for Long Context Q&A](https://mlsanity.com/paper/2509.19228/) (2025)
- [ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMs](https://mlsanity.com/paper/2602.03226/) (2026)
- [Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12468/) (2024)
- [PoC: Performance-oriented Context Compression for Large Language Models via Performance Prediction](https://mlsanity.com/paper/2603.19733/) (2026)
- [Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven](https://mlsanity.com/paper/2605.09463/) (2026)
- [Lost in the Passage: Passage-level In-context Learning Does Not Necessarily Need a "Passage"](https://mlsanity.com/paper/2502.10634/) (2025)
- [Less Is More: Elevating RAG via Performance-Driven Context Compression](https://mlsanity.com/paper/2508.19282/) (2025)
- [Beyond Black-Box Interventions: Latent Probing for Faithful Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.12460/) (2025)
- [MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents](https://mlsanity.com/paper/2608.07068/) (2026)
- [Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Model](https://mlsanity.com/paper/2602.01778/) (2026)
- [IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answering](https://mlsanity.com/paper/2608.13588/) (2026)
- [Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference](https://mlsanity.com/paper/2609.25537/) (2026)
- [Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation](https://mlsanity.com/paper/2505.19529/) (2025)
- [Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2512.08943/) (2025)
- [MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression](https://mlsanity.com/paper/2609.27380/) (2026)

## Abstract (arXiv)

The growing popularity of Large Language Models has sparked interest in context compression for Large Language Models (LLMs). However, the performance of previous methods degrades dramatically as compression ratios increase, sometimes even falling to the closed-book level. This decline can be attributed to the loss of key information during the compression process. Our preliminary study supports this hypothesis, emphasizing the significance of retaining key information to maintain model performance under high compression ratios. As a result, we introduce Query-Guided Compressor (QGC), which leverages queries to guide the context compression process, effectively preserving key information within the compressed context. Additionally, we employ a dynamic compression strategy. We validate the effectiveness of our proposed QGC on the Question Answering task, including NaturalQuestions, TriviaQA, and HotpotQA datasets. Experimental results show that QGC can consistently perform well even at high compression ratios, which also offers significant benefits in terms of inference cost and throughput.