mlsanityData to 14 Oct 2026

In-Context Learning Learns Label Relationships but Is Not Conventional Learning

arXiv 2307.12375 ↗First posted 23 Jul 2023cs.CL cs.AI cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

The predictions of Large Language Models (LLMs) on downstream tasks often improve significantly when including examples of the input--label relationship in the context. However, there is currently no consensus about how this in-context learning (ICL) ability of LLMs works. For example, while Xie et al. (2021) liken ICL to a general-purpose learning algorithm, Min et al. (2022) argue ICL does not even learn label relationships from in-context examples. In this paper, we provide novel insights into how ICL leverages label information, revealing both capabilities and limitations. To ensure we obtain a comprehensive picture of ICL behavior, we study probabilistic aspects of ICL predictions and thoroughly examine the dynamics of ICL as more examples are provided. Our experiments show that ICL predictions almost always depend on in-context labels and that ICL can learn truly novel tasks in-context. However, we also find that ICL struggles to fully overcome prediction preferences acquired from pre-training data and, further, that ICL does not consider all in-context information equally.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 23 Jul 2023.

Builds on

49 of 66 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
37 more
  1. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  8. 2022Transformers learn in-context by gradient descentcited by 732
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  10. 2021MetaICL: Learning to Learn In Contextcited by 454
  11. 2023Larger language models do in-context learning differentlycited by 394
  12. 2021Transformers Can Do Bayesian Inferencecited by 360
  13. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  14. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  15. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  16. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  17. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  18. 2018Hate Speech Dataset from a White Supremacy Forumcited by 198
  19. 2022Active Example Selection for In-Context Learningcited by 196
  20. 2022In-context Examples Selection for Machine Translationcited by 172
  21. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  22. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  23. 2023The Learnability of In-Context Learningcited by 127
  24. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  25. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  26. 2021Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learningcited by 113
  27. 2018Meta-Learning Probabilistic Inference For Predictioncited by 110
  28. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  29. 2023Finding Support Examples for In-Context Learningcited by 104
  30. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  31. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
  32. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  33. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  34. 2020Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQscited by 48
  35. 2023A Latent Space Theory for Emergent Abilities in Large Language Modelscited by 43
  36. 2019On the marginal likelihood and cross-validationcited by 37
  37. 2021Distinguishing rule- and exemplar-based generalization in learning systemscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 20; 2025: 18; 2026: 10.

51 papers.

Sort:

By area:

  1. 2024Many-Shot In-Context Learningcited by 206
  2. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  3. 2024Large Language Model Safety: A Holistic Surveycited by 59
  4. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 57
  5. 2024UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AIcited by 51
  6. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 45
  7. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  8. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 29
  9. 2025BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Designcited by 24
  10. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  11. 2024Estimating the Hallucination Rate of Generative AIcited by 20
  12. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 17
38 more
  1. 2024Learning to Poison Large Language Models for Downstream Manipulationcited by 16
  2. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 16
  3. 2024What Do Language Models Learn in Context? The Structured Task Hypothesiscited by 13
  4. 2025Scaling Up Active Testing to Large Language Modelscited by 13
  5. 2024An Empirical Study of In-context Learning in LLMs for Machine Translationcited by 9
  6. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Modelscited by 9
  7. 2024Relevant or Random: Can LLMs Truly Perform Analogical Reasoning?cited by 8
  8. 2024Trust-Oriented Adaptive Guardrails for Large Language Modelscited by 8
  9. 2024Shortcut Learning in In-Context Learning: A Surveycited by 8
  10. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 6
  11. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learningcited by 6
  12. 2023Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learningcited by 5
  13. 2025Out-of-Distribution Detection using Synthetic Data Generationcited by 4
  14. 2025Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decodingcited by 4
  15. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  16. 2024Disentangling Latent Shifts of In-Context Learning with Weak Supervisioncited by 3
  17. 2025Unraveling Indirect In-Context Learning Using Influence Functionscited by 3
  18. 2025StaICC: Standardized Evaluation for Classification Task in In-context Learningcited by 2
  19. 2025Incomplete In-context Learningcited by 2
  20. 2025Surprise Calibration for Better In-Context Learningcited by 2
  21. 2025Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labelscited by 2
  22. 2025Fine-Tuned In-Context Learners for Efficient Adaptationcited by 2
  23. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  24. 2026SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Trainingcited by 2
  25. 2026Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottleneckscited by 2
  26. 2025Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusionscited by 1
  27. 2025Data-Efficient Biomedical In-Context Learning: A Diversity-Enhanced Submodular Perspectivecited by 1
  28. 2026The Effect of Training Task Diversity on In-Context Learning through the Lens of Low-Dimensional Subspacescited by 1
  29. 2024KPC-cF: Aspect-Based Sentiment Analysis via Implicit-Feature Alignment with Corpus Filteringcited by 0
  30. 2024DSAI: Unbiased and Interpretable Latent Feature Extraction for Data-Centric AIcited by 0
  31. 2025Mockingbird: How does LLM perform in general machine learning tasks?cited by 0
  32. 2025Generative KI f"ur TAcited by 0
  33. 2025Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspectivecited by 0
  34. 2026When Looks Do Not Lie: Discourse Structure Guided In-Context Learning for Faithful Diagram Generationcited by 0
  35. 2026Reconciling In-Context and In-Weight Learning via Dual Representation Space Encodingcited by 0
  36. 2026In-Context Examples Suppress Scientific Knowledge Recall in LLMscited by 0
  37. 2026In-Context Fixation: When Demonstrated Labels Override Semantics in Few-Shot Classificationcited by 0
  38. 2026Do LLMs Understand Sequential Structure? A Controlled Study of Inference and Generationcited by 0

50 most cited of 51 All 51 (JSON)