mlsanityData to 7 Oct 2026

Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection

arXiv 2310.20046 ↗First posted 30 Oct 2023cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) can adapt to new tasks via in-context learning (ICL). ICL is efficient as it does not require any parameter updates to the trained LLM, but only few annotated examples as input for the LLM. In this work, we investigate an active learning approach for ICL, where there is a limited budget for annotating examples. We propose a model-adaptive optimization-free algorithm, termed AdaICL, which identifies examples that the model is uncertain about, and performs semantic diversity-based example selection. Diversity-based sampling improves overall effectiveness, while uncertainty sampling improves budget efficiency and helps the LLM learn new information. Moreover, AdaICL poses its sampling strategy as a Maximum Coverage problem, that dynamically adapts based on the model's feedback and can be approximately solved via greedy algorithms. Extensive experiments on nine datasets and seven LLMs show that AdaICL improves performance by 4.4% accuracy points over SOTA (7.7% relative improvement), is up to 3x more budget-efficient than performing annotations uniformly at random, while it outperforms SOTA with 2x fewer ICL examples.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 54 of its 74 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

54 of 74 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2017On Calibration of Modern Neural Networkscited by 4,860
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  10. 2022Transformers learn in-context by gradient descentcited by 714
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  12. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  13. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  14. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  15. 2021MetaICL: Learning to Learn In Contextcited by 450
  16. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  17. 2023Larger language models do in-context learning differentlycited by 386
  18. 2021True Few-Shot Learning with Language Modelscited by 343
  19. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  20. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  21. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  22. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  23. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  24. 2022Active Example Selection for In-Context Learningcited by 195
  25. 2021Meta-learning via Language Model In-context Tuningcited by 163
  26. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  27. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  28. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  29. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  30. 2023Finding Support Examples for In-Context Learningcited by 104
  31. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  32. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  33. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  34. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  35. 2023In-context Example Selection with Influencescited by 60
  36. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 59
  37. 2023Fairness-guided Few-shot Prompting for Large Language Modelscited by 53
  38. 2020A Survey of Active Learning for Text Classification using Deep Neural Networkscited by 50

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 15; 2025: 14; 2026: 5.

36 papers.

Sort:

By area:

  1. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  2. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  3. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 28
  4. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 15
  5. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  6. 2024Misconfidence-based Demonstration Selection for LLM In-Context Learningcited by 14
  7. 2025MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learningcited by 13
  8. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  9. 2025Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistancecited by 11
  10. 2025Towards Fair In-Context Learning with Tabular Foundation Modelscited by 8
  11. 2024Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalitiescited by 7
  12. 2024Demonstration Selection for In-Context Learning via Reinforcement Learningcited by 7
24 more
  1. 2024Revisiting In-Context Learning with Long Context Language Modelscited by 7
  2. 2024Automate Knowledge Concept Tagging on Math Questions with LLMscited by 6
  3. 2025Enhancing Multimodal In-Context Learning for Image Classification through Coreset Optimizationcited by 6
  4. 2024Automatic Combination of Sample Selection Strategies for Few-Shot Learningcited by 4
  5. 2024Efficient In-Context Medical Segmentation with Meta-driven Visual Prompt Selectioncited by 4
  6. 2025Unraveling Indirect In-Context Learning Using Influence Functionscited by 3
  7. 2025ConText: Driving In-context Learning for Text Removal and Segmentationcited by 3
  8. 2025ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinkingcited by 3
  9. 2023Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learningcited by 2
  10. 2025StaICC: Standardized Evaluation for Classification Task in In-context Learningcited by 2
  11. 2025Implicit In-Context Learning: Evidence from Artificial Language Experimentscited by 2
  12. 2025Incomplete In-context Learningcited by 2
  13. 2025On Theoretical Interpretations of Concept-Based In-Context Learningcited by 2
  14. 2024On Sensitivity of Learning with Limited Labelled Data to the Effects of Randomness: Impact of Interactions and Systematic Choicescited by 1
  15. 2025InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversitycited by 1
  16. 2025Protocode: Prototype-Driven Interpretability for Code Generation in LLMscited by 1
  17. 2024Sparse Linear Bandits with Blocking Constraintscited by 0
  18. 2024The Power of Adaptation: Boosting In-Context Learning through Adaptive Promptingcited by 0
  19. 2025Exploring the Heterogeneity of Tabular Data: A Diversity-aware Data Generator via LLMscited by 0
  20. 2026Geometry-Aware Hallucination Detection in Large Language Modelscited by 0
  21. 2026Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learningcited by 0
  22. 2026AsynDBT: Asynchronous Distributed Bilevel Tuning for efficient In-Context Learning with Large Language Modelscited by 0
  23. 2026Activation-Based Active Learning for In-Context Learning: Challenges and Insightscited by 0
  24. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0