# "In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"

arXiv: https://arxiv.org/abs/2405.01116 · HTML: https://mlsanity.com/paper/2405.01116/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2024.

## Built on by (most cited first, 6 of 6)

- [The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers](https://mlsanity.com/paper/2506.20844/) (2025)
- [Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model](https://mlsanity.com/paper/2409.17745/) (2024)
- [HF-RAG: Hierarchical Fusion-based RAG with Multiple Sources and Rankers](https://mlsanity.com/paper/2509.02837/) (2025)
- [The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content](https://mlsanity.com/paper/2606.11198/) (2026)
- [RAQG-QPP: Query Performance Prediction with Retrieved Query Variants and Retrieval Augmented Query Generation](https://mlsanity.com/paper/2604.27244/) (2026)
- [One Size Does Not Fit All! Dynamic Retriever and Generator Selection for RAG](https://mlsanity.com/paper/2609.17709/) (2026)

## Abstract (arXiv)

With the increasing ability of large language models (LLMs), in-context learning (ICL) has evolved as a new paradigm for natural language processing (NLP), where instead of fine-tuning the parameters of an LLM specific to a downstream task with labeled examples, a small number of such examples is appended to a prompt instruction for controlling the decoder's generation process. ICL, thus, is conceptually similar to a non-parametric approach, such as k-NN, where the prediction for each instance essentially depends on the local topology, i.e., on a localised set of similar instances and their labels (called few-shot examples). This suggests that a test instance in ICL is analogous to a query in IR, and similar examples in ICL retrieved from a training set relate to a set of documents retrieved from a collection in IR. While standard unsupervised ranking models can be used to retrieve these few-shot examples from a training set, the effectiveness of the examples can potentially be improved by re-defining the notion of relevance specific to its utility for the downstream task, i.e., considering an example to be relevant if including it in the prompt instruction leads to a correct prediction. With this task-specific notion of relevance, it is possible to train a supervised ranking model (e.g., a bi-encoder or cross-encoder), which potentially learns to optimally select the few-shot examples. We believe that the recent advances in neural rankers can potentially find a use case for this task of optimally choosing examples for more effective downstream ICL predictions.