mlsanityData to 7 Oct 2026

Monte Carlo Sampling for Analyzing In-Context Examples

arXiv 2503.22002 ↗First posted 27 Mar 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Prior works have shown that in-context learning is brittle to presentation factors such as the order, number, and choice of selected examples. However, ablation-based guidance on selecting the number of examples may ignore the interplay between different presentation factors. In this work we develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples. We find that previous guidance on how many in-context examples to select does not always generalize across different sets of selected examples and orderings, and whether one-shot settings outperform zero-shot settings is highly dependent on the selected example. Additionally, inspired by data valuation, we apply our sampling method to in-context example selection to select examples that perform well across different orderings. We find a negative result, that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

22 of 36 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  12. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
10 more
  1. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  2. 2022Active Example Selection for In-Context Learningcited by 195
  3. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  4. 2023Compositional Exemplars for In-context Learningcited by 143
  5. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  6. 2023In-context Example Selection with Influencescited by 60
  7. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  8. 2013Bounding the Estimation Error of Sampling-based Shapley Value Approximationcited by 56
  9. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  10. 2023Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Valuescited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Activation-Based Active Learning for In-Context Learning: Challenges and Insightscited by 0
  2. 2026Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classificationcited by 0