# In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting

arXiv: https://arxiv.org/abs/2408.13028 · HTML: https://mlsanity.com/paper/2408.13028/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 23 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension](https://mlsanity.com/paper/1902.00164/) (2019)
- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)
- [Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator](https://mlsanity.com/paper/2206.08082/) (2022)
- [Coverage-based Example Selection for In-Context Learning](https://mlsanity.com/paper/2305.14907/) (2023)
- [Improving Multi-turn Dialogue Modelling with Utterance ReWriter](https://mlsanity.com/paper/1906.07004/) (2019)
- [GECOR: An End-to-End Generative Ellipsis and Co-reference Resolution Model for Task-Oriented Dialogue](https://mlsanity.com/paper/1909.12086/) (2019)
- [Incomplete Utterance Rewriting as Semantic Segmentation](https://mlsanity.com/paper/2009.13166/) (2020)
- [SARG: A Novel Semi Autoregressive Generator for Multi-turn Incomplete Utterance Restoration](https://mlsanity.com/paper/2008.01474/) (2020)
- [Enhance Incomplete Utterance Restoration by Joint Learning Token Extraction and Text Generation](https://mlsanity.com/paper/2204.03958/) (2022)

## Abstract (arXiv)

In-context learning (ICL) of large language models (LLMs) has attracted increasing attention in the community where LLMs make predictions only based on instructions augmented with a few examples. Existing example selection methods for ICL utilize sparse or dense retrievers and derive effective performance. However, these methods do not utilize direct feedback of LLM to train the retriever and the examples selected can not necessarily improve the analogy ability of LLM. To tackle this, we propose our policy-based reinforcement learning framework for example selection (RLS), which consists of a language model (LM) selector and an LLM generator. The LM selector encodes the candidate examples into dense representations and selects the top-k examples into the demonstration for LLM. The outputs of LLM are adopted to compute the reward and policy gradient to optimize the LM selector. We conduct experiments on different datasets and significantly outperform existing example selection methods. Moreover, our approach shows advantages over supervised finetuning (SFT) models in few shot setting. Further experiments show the balance of abundance and the similarity with the test case of examples is important for ICL performance of LLM.