mlsanityData to 7 Oct 2026

Context Embeddings for Efficient Answer Generation in RAG

arXiv 2407.09252 ↗First posted 12 Jul 2024cs.CL cs.IRcited by 26

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

Retrieval-Augmented Generation (RAG) allows overcoming the limited knowledge of LLMs by extending the input with external information. As a consequence, the contextual inputs to the model become much longer which slows down decoding time directly translating to the time a user has to wait for an answer. We address this challenge by presenting COCOM, an effective context compression method, reducing long contexts to only a handful of Context Embeddings speeding up the generation time by a large margin. Our method allows for different compression rates trading off decoding time for answer quality. Compared to earlier methods, COCOM allows for handling multiple contexts more effectively, significantly reducing decoding time for long inputs. Our method demonstrates a speed-up of up to 5.69 × while achieving higher performance compared to existing efficient context compression methods.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 21 of its 36 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

21 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection25 May
  2. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token22 May
  3. 2024Reliable, Adaptable, and Attributable Language Models with Retrieval5 Mar
  4. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation6 Oct
  5. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug

Other references · 16

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  8. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  9. 2023Adapting Language Models to Compress Contextscited by 350
  10. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  11. 2024Generative Representational Instruction Tuningcited by 227
  12. 2024SPLADE-v3: New baselines for SPLADEcited by 69
4 more
  1. 2023Unlocking Context Constraints of LLMs: Enhancing Context Efficiency of LLMs with Self-Information-Based Content Filteringcited by 62
  2. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  3. 2024RAGGED: Towards Informed Design of Scalable and Stable RAG Systemscited by 24
  4. 2024LLoCO: Learning Long Contexts Offlinecited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 11.

26 papers.

Sort:

By area:

  1. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  2. 2024500xCompressor: Generalized Prompt Compression for Large Language Modelscited by 45
  3. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  4. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  5. 2025Provence: efficient and robust context pruning for retrieval-augmented generationcited by 25
  6. 2024Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarioscited by 22
  7. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generationcited by 20
  8. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  9. 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
  10. 2026Read As Human: Compressing Context via Parallelizable Close Reading and Skimmingcited by 10
  11. 2025Enhancing RAG Efficiency with Adaptive Context Compressioncited by 7
  12. 2025OSCAR: Online Soft Compression And Rerankingcited by 4
14 more
  1. 2025A Survey of LLM × DATAcited by 4
  2. 2026Rethinking Soft Compression in Retrieval-Augmented Generation: A Query-Conditioned Selector Perspectivecited by 4
  3. 2025On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Surveycited by 3
  4. 2025Less Is More: Elevating RAG via Performance-Driven Context Compressioncited by 2
  5. 2026Decoupled Reasoning with Implicit Fact Tokens (DRIFT): A Dual-Model Framework for Efficient Long-Context Inferencecited by 1
  6. 2025WindVE: Collaborative CPU-NPU Vector Embeddingcited by 0
  7. 2025Osiris: A Lightweight Open-Source Hallucination Detection Systemcited by 0
  8. 2026RegGuard: AI-Powered Retrieval-Enhanced Assistant for Pharmaceutical Regulatory Compliancecited by 0
  9. 2026The Virtuous Cycle: AI-Powered Vector Search and Vector Search-Augmented AIcited by 0
  10. 2026A Unified Model and Document Representation for On-Device Retrieval-Augmented Generationcited by 0
  11. 2026LeakDojo: Decoding the Leakage Threats of RAG Systemscited by 0
  12. 2026Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systemscited by 0
  13. 2026Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAGcited by 0
  14. 2026Beyond Raw Context Transfer: Representation-based Federated Retrieval-Augmented Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024In Defense of RAG in the Era of Long-Context Language Models
  2. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey
  3. 2024Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generation
  4. 2024Introducing a new hyper-parameter for RAG: Context Window Utilization
  5. 2024Context Awareness Gate For Retrieval Augmented Generation
  6. 2025REFRAG: Rethinking RAG based Decoding
  7. 2025Dynamic Context Selection for Retrieval-Augmented Generation: Mitigating Distractors and Positional Bias
  8. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems