mlsanityData to 7 Oct 2026

What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning

arXiv 2305.09731 ↗First posted 16 May 2023cs.CL cs.LGcited by 149

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) exploit in-context learning (ICL) to solve tasks with only a few demonstrations, but its mechanisms are not yet well-understood. Some works suggest that LLMs only recall already learned concepts from pre-training, while others hint that ICL performs implicit learning over demonstrations. We characterize two ways through which ICL leverages demonstrations. Task recognition (TR) captures the extent to which LLMs can recognize a task through demonstrations -- even without ground-truth labels -- and apply their pre-trained priors, whereas task learning (TL) is the ability to capture new input-label mappings unseen in pre-training. Using a wide range of classification datasets and three LLM families (GPT-3, LLaMA and OPT), we design controlled experiments to disentangle the roles of TR and TL in ICL. We show that (1) models can achieve non-trivial performance with only TR, and TR does not further improve with larger models or more demonstrations; (2) LLMs acquire TL as the model scales, and TL's performance consistently improves with more demonstrations in context. Our findings unravel two different forces behind ICL and we advocate for discriminating them in future ICL research due to their distinct nature.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Builds on 16 of its 32 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

16 of 32 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2022Transformers learn in-context by gradient descentcited by 714
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2023Larger language models do in-context learning differentlycited by 386
  10. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  11. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  12. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
4 more
  1. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  2. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  3. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  4. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 53; 2025: 40; 2026: 22.

149 papers.

Sort:

By area:

  1. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  2. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  3. 2023Function Vectors in Large Language Modelscited by 292
  4. 2024Many-Shot In-Context Learningcited by 203
  5. 2023Large Language Models as General Pattern Machinescited by 196
  6. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  7. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  8. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  9. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  10. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  11. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  12. 2023In-Context Learning through the Bayesian Prismcited by 78
38 more
  1. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  2. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  3. 2023Efficient Benchmarking of Language Modelscited by 71
  4. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  5. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  6. 2024Attention Heads of Large Language Models: A Surveycited by 65
  7. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 60
  8. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  9. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  10. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 53
  11. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  12. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  13. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  14. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  15. 2023LLM4DyG: Can Large Language Models Solve Spatial-Temporal Problems on Dynamic Graphs?cited by 40
  16. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  17. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  18. 2024The broader spectrum of in-context learningcited by 33
  19. 2023How to Configure Good In-Context Sequence for Visual Question Answeringcited by 31
  20. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  21. 2023Understanding In-Context Learning from Repetitionscited by 27
  22. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 27
  23. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  24. 2025TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configurationcited by 23
  25. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntaxcited by 22
  26. 2025Large Language Models are Good Relational Learnerscited by 22
  27. 2023A Study on the Calibration of In-context Learningcited by 21
  28. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  29. 2023Generative Calibration for In-context Learningcited by 20
  30. 2023Supervised Knowledge Makes Large Language Models Better In-context Learnerscited by 20
  31. 2024Estimating the Hallucination Rate of Generative AIcited by 20
  32. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  33. 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 19
  34. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  35. 2024How do Large Language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learningcited by 18
  36. 2025GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Explorationcited by 18
  37. 2023Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Datasetcited by 17
  38. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Modelscited by 16

50 most cited of 149 All 149 (JSON)