# RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning

arXiv: https://arxiv.org/abs/2305.14502 · HTML: https://mlsanity.com/paper/2305.14502/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://mlsanity.com/paper/2209.14610/) (2022)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [Active Example Selection for In-Context Learning](https://mlsanity.com/paper/2211.04486/) (2022)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)

## Built on by (most cited first, 24 of 24)

- [In-Context Learning with Iterative Demonstration Selection](https://mlsanity.com/paper/2310.09881/) (2023)
- [In-context Learning with Retrieved Demonstrations for Language Models: A Survey](https://mlsanity.com/paper/2401.11624/) (2024)
- [A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoning](https://mlsanity.com/paper/2501.02497/) (2025)
- [Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?](https://mlsanity.com/paper/2311.18021/) (2023)
- [Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning](https://mlsanity.com/paper/2312.08901/) (2023)
- [DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task](https://mlsanity.com/paper/2406.16332/) (2024)
- [Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Models](https://mlsanity.com/paper/2402.15637/) (2024)
- [CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharing](https://mlsanity.com/paper/2410.16670/) (2024)
- [In-Context Learning Demonstration Selection via Influence Analysis](https://mlsanity.com/paper/2402.11750/) (2024)
- [Math Multiple Choice Question Generation via Human-Large Language Model Collaboration](https://mlsanity.com/paper/2405.00864/) (2024)
- [Demonstration Selection for In-Context Learning via Reinforcement Learning](https://mlsanity.com/paper/2412.03966/) (2024)
- [Knowledge Tagging System on Math Questions via LLMs with Flexible Demonstration Retriever](https://mlsanity.com/paper/2406.13885/) (2024)
- [The Role of Diversity in In-Context Learning for Large Language Models](https://mlsanity.com/paper/2505.19426/) (2025)
- [Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples](https://mlsanity.com/paper/2506.16502/) (2025)
- [Automatic Combination of Sample Selection Strategies for Few-Shot Learning](https://mlsanity.com/paper/2402.03038/) (2024)
- [PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks](https://mlsanity.com/paper/2412.05710/) (2024)
- [Post-Incorporating Code Structural Knowledge into Pretrained Models via ICL for Code Translation](https://mlsanity.com/paper/2503.22776/) (2025)
- [Order Matters: Rethinking Prompt Construction in In-Context Learning](https://mlsanity.com/paper/2511.09700/) (2025)
- [GraphIC: A Graph-Based In-Context Example Retrieval Model for Multi-Step Reasoning](https://mlsanity.com/paper/2410.02203/) (2024)
- [Augmenting In-Context-Learning in LLMs via Automatic Data Labeling and Refinement](https://mlsanity.com/paper/2410.10348/) (2024)
- [PerkwE_COQA: Enhanced Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Models](https://mlsanity.com/paper/2404.05406/) (2024)
- [Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learning](https://mlsanity.com/paper/2602.12123/) (2026)
- [Downstream-Aware Context Selection for Online In-Context Reinforcement Learning](https://mlsanity.com/paper/2609.33166/) (2026)
- [Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptations](https://mlsanity.com/paper/2610.00928/) (2026)

## Abstract (arXiv)

Recent developments in large pre-trained language models have enabled unprecedented performance on a variety of downstream tasks. Achieving best performance with these models often leverages in-context learning, where a model performs a (possibly new) task given one or more examples. However, recent work has shown that the choice of examples can have a large impact on task performance and that finding an optimal set of examples is non-trivial. While there are many existing methods for selecting in-context examples, they generally score examples independently, ignoring the dependency between them and the order in which they are provided to the model. In this work, we propose Retrieval for In-Context Learning (RetICL), a learnable method for modeling and optimally selecting examples sequentially for in-context learning. We frame the problem of sequential example selection as a Markov decision process and train an example retriever using reinforcement learning. We evaluate RetICL on math word problem solving and scientific question answering tasks and show that it consistently outperforms or matches heuristic and learnable baselines. We also use case studies to show that RetICL implicitly learns representations of problem solving strategies.