mlsanityData to 7 Oct 2026

More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineering

arXiv 2311.09782 ↗First posted 16 Nov 2023cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

While most existing works on LLM prompting techniques focus only on how to select a better set of data samples inside one single prompt input (In-Context Learning or ICL), why can not we design and leverage multiple prompts together to further improve the LLM's performance? In this work, we propose In-Context Sampling (ICS), a low-resource LLM prompting technique to produce confident predictions by optimizing the construction of multiple ICL prompt inputs. Extensive experiments with three open-source LLMs (FlanT5-XL, Mistral-7B, and Mixtral-8x7B) on four NLI datasets (e-SNLI, Multi-NLI, ANLI, and Contract-NLI) and one QA dataset (CommonsenseQA) illustrate that ICS can consistently enhance LLMs' performance. An in-depth evaluation with three data similarity-based ICS strategies suggests that these strategies can further elevate LLM's performance, which sheds light on a new yet promising future research direction.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 39 of its 83 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

39 of 83 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
27 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  4. 2022A Survey on In-context Learningcited by 717
  5. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  6. 2020A Survey of Deep Active Learningcited by 547
  7. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  8. 2019Variational Adversarial Active Learningcited by 284
  9. 2022Active Example Selection for In-Context Learningcited by 195
  10. 2021Active Learning by Acquiring Contrastive Examplescited by 154
  11. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 145
  12. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  13. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  14. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  15. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generatorcited by 68
  16. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  17. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  18. 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 59
  19. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  20. 2020A Survey of Active Learning for Text Classification using Deep Neural Networkscited by 50
  21. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38
  22. 2019Sampling Bias in Deep Active Classification: An Empirical Studycited by 36
  23. 2023Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosiscited by 30
  24. 2022ALLSH: Active Learning Guided by Local Sensitivity and Hardnesscited by 23
  25. 2023Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanationscited by 13
  26. 2023Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecturecited by 10
  27. 2020Learning Structured Representations of Entity Names using Active Learning and Weak Supervisioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 3.

13 papers.

Sort:

By area:

  1. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 26
  2. 2024BeHonest: Benchmarking Honesty in Large Language Modelscited by 20
  3. 2024UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs' Memorizationcited by 17
  4. 2025Understanding and Improving Information Preservation in Prompt Compression for LLMscited by 13
  5. 2024Secret Use of Large Language Model (LLM)cited by 9
  6. 2024Automate Knowledge Concept Tagging on Math Questions with LLMscited by 6
  7. 2025Generative Artificial Intelligence: Evolving Technology, Growing Societal Impact, and Opportunities for Information Systems Researchcited by 4
  8. 2024Incorporating Metabolic Information into LLMs for Anomaly Detection in Clinical Time-Seriescited by 2
  9. 2025Exploring Imbalanced Annotations for Effective In-Context Learningcited by 0
  10. 2025Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Modelscited by 0
  11. 2026AutoScreen-FW: An LLM-based Framework for Resume Screeningcited by 0
  12. 2026Activation-Based Active Learning for In-Context Learning: Challenges and Insightscited by 0
1 more
  1. 2026ClusterFewshot: Improving Few-shot Optimization for LLMs workflowcited by 0