mlsanityData to 14 Oct 2026

From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compression

arXiv 2410.04139 ↗First posted 5 Oct 2024cs.CL cs.AIcited by 7

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Large language models (LLMs) have achieved significant performance gains using advanced prompting techniques over various tasks. However, the increasing length of prompts leads to high computational costs and often obscures crucial information. Prompt compression has been proposed to alleviate these issues, but it faces challenges in (i) capturing the global context and (ii) training the compressor effectively. To tackle these challenges, we introduce a novel prompt compression method, namely Reading To Compressing (R2C), utilizing the Fusion-in-Decoder (FiD) architecture to identify the important information in the prompt. Specifically, the cross-attention scores of the FiD are used to discern essential chunks and sentences from the prompt. R2C effectively captures the global context without compromising semantic consistency while detouring the necessity of pseudo-labels for training the compressor. Empirical results show that R2C retains key contexts, enhancing the LLM performance by 6% in out-of-domain evaluations while reducing the prompt length by 80%.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 30 references that are in the map. First posted on arXiv 5 Oct 2024.

Builds on

21 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token22 May
  2. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression30 Mar
  3. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar
  4. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct
  5. 2023Discrete Prompt Compression with Reinforcement Learning17 Aug

Other references · 16

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  11. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  12. 2023Adapting Language Models to Compress Contextscited by 356
4 more
  1. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  2. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  3. 2024Learning to Compress Prompt in Natural Language Formatscited by 55
  4. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMscited by 15
  2. 2025Draft-based Approximate Inference for LLMscited by 5
  3. 2025Conflict-Aware Soft Prompting for Retrieval-Augmented Generationcited by 5
  4. 2025Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressorcited by 1
  5. 2026IterCOMP: Reasoning-aware Adaptive Prompt Compression for Multi-hop Question Answeringcited by 1
  6. 2026Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressorscited by 0
  7. 2026Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompt Compression for Large Language Models: A Survey
  2. 2024Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles
  3. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Models
  4. 2025An Empirical Study on Prompt Compression for Large Language Models
  5. 2026Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
  6. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
  7. 2025Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptor
  8. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference