# Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learning

arXiv: https://arxiv.org/abs/2310.08309 · HTML: https://mlsanity.com/paper/2310.08309/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [Active Example Selection for In-Context Learning](https://mlsanity.com/paper/2211.04486/) (2022)
- [Efficient Large Scale Language Modeling with Mixtures of Experts](https://mlsanity.com/paper/2112.10684/) (2021)
- [Noisy Channel Language Model Prompting for Few-Shot Text Classification](https://mlsanity.com/paper/2108.04106/) (2021)
- [Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations](https://mlsanity.com/paper/2205.12685/) (2022)
- [Finding Support Examples for In-Context Learning](https://mlsanity.com/paper/2302.13539/) (2023)
- [Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations](https://mlsanity.com/paper/2212.09865/) (2022)
- [FLEX: Unifying Evaluation for Few-Shot NLP](https://mlsanity.com/paper/2107.07170/) (2021)
- [True Few-Shot Learning with Prompts -- A Real-World Perspective](https://mlsanity.com/paper/2111.13440/) (2021)

## Built on by (most cited first, 9 of 9)

- [More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives](https://mlsanity.com/paper/2501.04070/) (2025)
- [Mitigating Copy Bias in In-Context Learning through Neuron Pruning](https://mlsanity.com/paper/2410.01288/) (2024)
- [Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?](https://mlsanity.com/paper/2406.12809/) (2024)
- [Vocabulary-Defined Semantics: Latent Space Clustering for Improving In-Context Learning](https://mlsanity.com/paper/2401.16184/) (2024)
- [ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization](https://mlsanity.com/paper/2402.09320/) (2024)
- [Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science](https://mlsanity.com/paper/2409.14673/) (2024)
- [Experience Transfer for Multimodal LLM Agents in Minecraft Game](https://mlsanity.com/paper/2604.05533/) (2026)
- [Prompting Large Language Models with Partial Knowledge for Answering Questions with Unseen Entities](https://mlsanity.com/paper/2508.01290/) (2025)
- [Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Models](https://mlsanity.com/paper/2410.19195/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) have recently gained the In-Context Learning (ICL) ability with the models scaling up, allowing them to quickly adapt to downstream tasks with only a few demonstration examples prepended in the input sequence. Nonetheless, the current practice of ICL treats all demonstration examples equally, which still warrants improvement, as the quality of examples is usually uneven. In this paper, we investigate how to determine approximately optimal weights for demonstration examples and how to apply them during ICL. To assess the quality of weights in the absence of additional validation data, we design a masked self-prediction (MSP) score that exhibits a strong correlation with the final ICL performance. To expedite the weight-searching process, we discretize the continuous weight space and adopt beam search. With approximately optimal weights obtained, we further propose two strategies to apply them to demonstrations at different model positions. Experimental results on 8 text classification tasks show that our approach outperforms conventional ICL by a large margin. Our code are publicly available at https:github.com/Zhe-Young/WICL.