mlsanityData to 14 Oct 2026

Ambiguity-Aware In-Context Learning with Large Language Models

arXiv 2309.07900 ↗First posted 14 Sep 2023cs.CL cs.IRcited by 11

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) i.e. showing LLMs only a few task-specific demonstrations has led to downstream gains with no task-specific fine-tuning required. However, LLMs are sensitive to the choice of prompts, and therefore a crucial research question is how to select good demonstrations for ICL. One effective strategy is leveraging semantic similarity between the ICL demonstrations and test inputs by using a text retriever, which however is sub-optimal as that does not consider the LLM's existing knowledge about that task. From prior work (Lyu et al., 2023), we already know that labels paired with the demonstrations bias the model predictions. This leads us to our hypothesis whether considering LLM's existing knowledge about the task, especially with respect to the output label space can help in a better demonstration selection strategy. Through extensive experimentation on three text classification tasks, we find that it is beneficial to not only choose semantically similar ICL demonstrations but also to choose those demonstrations that help resolve the inherent label ambiguity surrounding the test example. Interestingly, we find that including demonstrations that the LLM previously mis-classified and also fall on the test example's decision boundary, brings the most performance gain.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

34 of 44 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
22 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  5. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  6. 2023Larger language models do in-context learning differentlycited by 394
  7. 2022Active Example Selection for In-Context Learningcited by 196
  8. 2022In-context Examples Selection for Machine Translationcited by 172
  9. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  10. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  11. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  12. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  13. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 75
  14. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 72
  15. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  16. 2019Mis-classified Vector Guided Softmax Loss for Face Recognitioncited by 60
  17. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 54
  18. 2021Reordering Examples Helps during Priming-based Few-Shot Learningcited by 50
  19. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  20. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  21. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 33
  22. 2023Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 3; 2026: 1.

11 papers.

  1. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  2. 2024Learning or Self-aligning? Rethinking Instruction Fine-tuningcited by 43
  3. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  4. 2023Cost-Effective In-Context Learning for Entity Resolution: A Design Space Explorationcited by 15
  5. 2024IntentGPT: Few-shot Intent Discovery with Large Language Modelscited by 8
  6. 2025Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learningcited by 7
  7. 2024VernaCopter: Disambiguated Natural-Language-Driven Robot via Formal Specificationscited by 4
  8. 2023Enhancing Machine Translation through Advanced In-Context Learning: A Methodological Strategy for GPT-4 Improvementcited by 1
  9. 2025Large Language Models are Demonstration Pre-Selectors for Themselvescited by 1
  10. 2026Generative AI for Managerial Decision-Making under Ambiguity and Sycophancycited by 1
  11. 2025Refract ICL: Rethinking Example Selection in the Era of Million-Token Modelscited by 0