mlsanityData to 14 Oct 2026

OSCAR: Online Soft Compression And Reranking

arXiv 2504.07109 ↗First posted 17 Mar 2025cs.IR cs.AI cs.CLcited by 4

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge, leading to improved accuracy and relevance. However, scaling RAG pipelines remains computationally expensive as retrieval sizes grow. To address this, we introduce OSCAR, a novel query-dependent online soft compression method that reduces computational overhead while preserving performance. Unlike traditional hard compression methods, which shorten retrieved texts, or soft compression approaches, which map documents to continuous embeddings offline, OSCAR dynamically compresses retrieved information at inference time, eliminating storage overhead and enabling higher compression rates. Additionally, we extend OSCAR to simultaneously perform reranking, further optimizing the efficiency of the RAG pipeline. Our experiments demonstrate state-of-the-art performance with a 2-5x speed-up in inference and minimal to no loss in accuracy for LLMs ranging from 1B to 24B parameters. The models are available at: https://huggingface.co/collections/naver/oscar-67d446a8e3a2551f57464295.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 42 of its 52 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

42 of 52 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Provence: efficient and robust context pruning for retrieval-augmented generation27 Jan
  2. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generation27 Jan
  3. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text10 Oct
  4. 2024Finch: Prompt-guided Key-Value Cache Compression31 Jul
  5. 2024Context Embeddings for Efficient Answer Generation in RAG12 Jul
  6. 2024CompAct: Compressing Retrieved Documents Actively for Question Answering12 Jul
  7. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token22 May
  8. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation6 Oct
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories20 Dec
  10. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  10. 2019Passage Re-ranking with BERTcited by 645
  11. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  12. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
19 more
  1. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  2. 2023Adapting Language Models to Compress Contextscited by 356
  3. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 286
  4. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  5. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  6. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  7. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 155
  8. 2020Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillationcited by 145
  9. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  10. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  11. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  12. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 86
  13. 2023Learning to Filter Context for Retrieval-Augmented Generationcited by 83
  14. 2024SPLADE-v3: New baselines for SPLADEcited by 72
  15. 2022FiD-Light: Efficient and Effective Retrieval-Augmented Text Generationcited by 63
  16. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  17. 2025Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacitycited by 28
  18. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  19. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Fixed RAG Compression Collapses Measured Reader Scalingcited by 1
  2. 2026RAGOCR: Optical Compression of Retrieval-Augmented Text via Visual Representationcited by 0
  3. 2026From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankerscited by 0
  4. 2026Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Soft Compression in Retrieval-Augmented Generation: A Query-Conditioned Selector Perspective
  2. 2025Enhancing RAG Efficiency with Adaptive Context Compression
  3. 2026AdaMem: Adaptive Memory Token Allocation for Soft Compression in Retrieval-Augmented Generation
  4. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey
  5. 2026Efficient Listwise Reranking with Compressed Document Representations
  6. 2025SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression
  7. 2025Reranking with Compressed Document Representation ↗
  8. 2025Less Is More: Elevating RAG via Performance-Driven Context Compression