mlsanityData to 14 Oct 2026

RARe: Retrieval Augmented Retrieval with In-Context Examples

arXiv 2410.20088 ↗First posted 26 Oct 2024cs.CL cs.AI cs.IRcited by 2

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

While in-context learning is well-studied with decoder-only language models (LLMs), its utility for encoder-only models remains underexplored. We study in-context learning for encoder-only models for text retrieval tasks. Can incorporating in-context examples (query-document pairs) to the target query enhance retriever performance? Our approach, RARe, finetunes a pre-trained model with in-context examples whose query is semantically similar to the target query. This approach achieves performance gains of up to +2.72% nDCG across open-domain retrieval datasets (BeIR, RAR-b) compared to using the target query only as an input. In particular, we find RARe exhibits stronger out-of-domain generalization compared to models using queries without in-context examples, similar to what is seen for in-context learning in LLMs. We further provide analysis on the design choices of in-context example augmentation for retrievers and lay the foundation for future work.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 59 of its 74 references that are in the map. First posted on arXiv 26 Oct 2024.

Builds on

59 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024NV-Retriever: Improving text embedding models with effective hard-negative mining22 Jul
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 57

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  12. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
38 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  9. 2019ELI5: Long Form Question Answeringcited by 575
  10. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  11. 2021MetaICL: Learning to Learn In Contextcited by 454
  12. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  13. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  14. 2023Improving Text Embeddings with Large Language Modelscited by 303
  15. 2019Abductive Commonsense Reasoningcited by 301
  16. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  17. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 235
  18. 2024Generative Representational Instruction Tuningcited by 233
  19. 2022Active Example Selection for In-Context Learningcited by 196
  20. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  21. 2021Meta-learning via Language Model In-context Tuningcited by 164
  22. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  23. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  24. 2023Query2doc: Query Expansion with Large Language Modelscited by 149
  25. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  26. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
  27. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 112
  28. 2022Task-aware Retrieval with Instructionscited by 107
  29. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 89
  30. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 83
  31. 2024Making Text Embedders Few-Shot Learnerscited by 79
  32. 2024Repetition Improves Language Model Embeddingscited by 72
  33. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 69
  34. 2023Query Expansion by Prompting Large Language Modelscited by 57
  35. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 43
  36. 2023Pre-Training to Learn in Contextcited by 42
  37. 2024Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Modelscited by 40
  38. 2023Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Modelscited by 37

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  2. 2024FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answeringcited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds
  2. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models
  3. 2023Learning to Retrieve In-Context Examples for Large Language Models
  4. 2023In-Context Retrieval-Augmented Language Models
  5. 2024Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning
  6. 2024Context Awareness Gate For Retrieval Augmented Generation
  7. 2024Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts
  8. 2026In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspective