mlsanityData to 14 Oct 2026

The Alchemy of Thought: Understanding In-Context Learning Through Supervised Classification

arXiv 2601.01290 ↗First posted 3 Jan 2026cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In-context learning (ICL) has become a prominent paradigm to rapidly customize LLMs to new tasks without fine-tuning. However, despite the empirical evidence of its usefulness, we still do not truly understand how ICL works. In this paper, we compare the behavior of in-context learning with supervised classifiers trained on ICL demonstrations to investigate three research questions: (1) Do LLMs with ICL behave similarly to classifiers trained on the same examples? (2) If so, which classifiers are closer, those based on gradient descent (GD) or those based on k-nearest neighbors (kNN)? (3) When they do not behave similarly, what conditions are associated with differences in behavior? Using text classification as a use case, with six datasets and three LLMs, we observe that LLMs behave similarly to these classifiers when the relevance of demonstrations is high. On average, ICL is closer to kNN than logistic regression, giving empirical evidence that the attention mechanism behaves more similarly to kNN than GD. However, when demonstration relevance is low, LLMs perform better than these classifiers, likely because LLMs can back off to their parametric memory, a luxury these classifiers do not have.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 22 of its 38 references that are in the map. First posted on arXiv 3 Jan 2026.

Builds on

22 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?25 Feb

Other references · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  7. 2019Designing and Interpreting Probes with Control Taskscited by 640
  8. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  9. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  10. 2022Active Example Selection for In-Context Learningcited by 196
  11. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  12. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
9 more
  1. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  2. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 54
  3. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 49
  4. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  5. 2022News Category Datasetcited by 43
  6. 2023Overview of AuTexTification at IberLEF 2023: Detection and Attribution of Machine-Generated Text in Multiple Domainscited by 31
  7. 2023In-context Learning and Gradient Descent Revisitedcited by 20
  8. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
  9. 2023In-Context Probing: Toward Building Robust Classifiers via Probing Large Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Relationship Between the Choice of Representation and In-Context Learning
  2. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  3. 2024Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning
  4. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
  5. 2025From Compression to Expression: A Layerwise Analysis of In-Context Learning
  6. 2023Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps
  7. 2025Is In-Context Learning Learning?
  8. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator