mlsanityData to 7 Oct 2026

Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs

arXiv 2406.02376 ↗First posted 4 Jun 2024cs.CLcited by 24

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

The growing popularity of Large Language Models has sparked interest in context compression for Large Language Models (LLMs). However, the performance of previous methods degrades dramatically as compression ratios increase, sometimes even falling to the closed-book level. This decline can be attributed to the loss of key information during the compression process. Our preliminary study supports this hypothesis, emphasizing the significance of retaining key information to maintain model performance under high compression ratios. As a result, we introduce Query-Guided Compressor (QGC), which leverages queries to guide the context compression process, effectively preserving key information within the compressed context. Additionally, we employ a dynamic compression strategy. We validate the effectiveness of our proposed QGC on the Question Answering task, including NaturalQuestions, TriviaQA, and HotpotQA datasets. Experimental results show that QGC can consistently perform well even at high compression ratios, which also offers significant benefits in terms of inference cost and throughput.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 32 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

23 of 32 references.

References · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  11. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
11 more
  1. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  2. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  3. 2023Adapting Language Models to Compress Contextscited by 350
  4. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  5. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  6. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  7. 2023Learning to Filter Context for Retrieval-Augmented Generationcited by 82
  8. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  9. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 36
  10. 2021Rider: Reader-Guided Passage Reranking for Open-Domain Question Answeringcited by 28
  11. 2018Simplifying Neural Machine Translation with Addition-Subtraction Twin-Gated Recurrent Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 9; 2026: 10.

24 papers.

Sort:

By area:

  1. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  2. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  3. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  4. 2024A Survey of Small Language Modelscited by 31
  5. 2025FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generationcited by 27
  6. 2024Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarioscited by 22
  7. 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
  8. 2025Long Context In-Context Compression by Getting to the Gist of Gistingcited by 12
  9. 2026Read As Human: Compressing Context via Parallelizable Close Reading and Skimmingcited by 10
  10. 2025CompLLM: Compression for Long Context Q&Acited by 5
  11. 2026ATACompressor: Adaptive Task-Aware Compression for Efficient Long-Context Processing in LLMscited by 5
  12. 2024Familiarity-Aware Evidence Compression for Retrieval-Augmented Generationcited by 4
12 more
  1. 2026PoC: Performance-oriented Context Compression for Large Language Models via Performance Predictioncited by 3
  2. 2026Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Drivencited by 3
  3. 2025Lost in the Passage: Passage-level In-context Learning Does Not Necessarily Need a "Passage"cited by 2
  4. 2025Less Is More: Elevating RAG via Performance-Driven Context Compressioncited by 2
  5. 2025Beyond Black-Box Interventions: Latent Probing for Faithful Retrieval-Augmented Generationcited by 2
  6. 2026Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Modelcited by 1
  7. 2026IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answeringcited by 1
  8. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inferencecited by 1
  9. 2025Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptationcited by 0
  10. 2025Noise-Robust Abstractive Compression in Retrieval-Augmented Language Modelscited by 0
  11. 2026MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agentscited by 0
  12. 2026MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
  2. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  3. 2026EnComp: Lightweight Encoder-Only Context Compression for Retrieval-Augmented Question Answering
  4. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
  5. 2026SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
  6. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  7. 2026TopoCompress: Long Context Compression via Graph-Wired Semantic Trajectories
  8. 2025From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition