mlsanityData to 7 Oct 2026

Why does in-context learning fail sometimes? Evaluating in-context learning on open and closed questions

arXiv 2407.02028 ↗First posted 2 Jul 2024cs.CL cs.AI cs.IRcited by 2

Not on the map. Why?

Abstract (arXiv)

We measure the performance of in-context learning as a function of task novelty and difficulty for open and closed questions. For that purpose, we created a novel benchmark consisting of hard scientific questions, each paired with a context of various relevancy. We show that counter-intuitively, a context that is more aligned with the topic does not always help more than a less relevant context. This effect is especially visible for open questions and questions of high difficulty or novelty. This result reveals a fundamental difference between the treatment of close-form and open-form questions by large-language models and shows a need for a more robust evaluation of in-context learning on the variety of different types of questions. It also poses a new question of how to optimally select a context for large language models, especially in the context of Retrieval Augmented Generation (RAG) systems. Our results suggest that the answer to this question can be highly application-dependent and might be contingent on factors including the format of the question, the perceived difficulty level of the questions, and the novelty or popularity of the information we seek.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

33 of 42 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
21 more
  1. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  3. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  4. 2021Domain Generalization: A Surveycited by 760
  5. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  6. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  8. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  9. 2021MetaICL: Learning to Learn In Contextcited by 450
  10. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  11. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  12. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  13. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 130
  14. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  15. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  16. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  17. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  18. 2022POTATO: The Portable Text Annotation Toolcited by 64
  19. 2023SALLM: Security Assessment of Generated Codecited by 35
  20. 2023A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrologycited by 26
  21. 2023Ask Again, Then Fail: Large Language Models' Vacillations in Judgmentcited by 8

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspectivecited by 9
  2. 2024Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented In-Context Learning for Long-Tail Knowledge in LLMscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging In-Context Learning and Retrieval-Augmented Generation for Automatic Question Generation in Educational Domains
  2. 2024Context Quality Matters in Training Fusion-in-Decoder for Extractive Open-Domain Question Answering
  3. 2024Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Style
  4. 2026Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
  5. 2025On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems
  6. 2024Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts
  7. 2025Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
  8. 2025Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation