mlsanityData to 7 Oct 2026

Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learning

arXiv 2307.15411 ↗First posted 28 Jul 2023cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown remarkable capacity for in-context learning (ICL), where learning a new task from just a few training examples is done without being explicitly pre-trained. However, despite the success of LLMs, there has been little understanding of how ICL learns the knowledge from the given prompts. In this paper, to make progress toward understanding the learning behaviour of ICL, we train the same LLMs with the same demonstration examples via ICL and supervised learning (SL), respectively, and investigate their performance under label perturbations (i.e., noisy labels and label imbalance) on a range of classification tasks. First, via extensive experiments, we find that gold labels have significant impacts on the downstream in-context performance, especially for large language models; however, imbalanced labels matter little to ICL across all model sizes. Second, when comparing with SL, we show empirically that ICL is less sensitive to label perturbations than SL, and ICL gradually attains comparable performance to SL as the model size increases.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

29 of 49 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  11. 2017Focal Loss for Dense Object Detectioncited by 749
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
17 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  3. 2019Attention is not not Explanationcited by 629
  4. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  5. 2021MetaICL: Learning to Learn In Contextcited by 450
  6. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  7. 2018Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noisecited by 216
  8. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  9. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  10. 2022Active Example Selection for In-Context Learningcited by 195
  11. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  12. 2021Meta-learning via Language Model In-context Tuningcited by 163
  13. 2017Learning from Noisy Labels with Distillationcited by 160
  14. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  15. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  16. 2019Robust Inference via Generative Classifiers for Handling Noisy Labelscited by 67
  17. 2017Learning Deep Networks from Noisy Labels with Dropout Regularizationcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 2; 2026: 1.

11 papers.

  1. 2023Function Vectors in Large Language Modelscited by 292
  2. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  3. 2023Cost-Effective In-Context Learning for Entity Resolution: A Design Space Explorationcited by 15
  4. 2023How Robust are LLMs to In-Context Majority Label Bias?cited by 13
  5. 2023Ambiguity-Aware In-Context Learning with Large Language Modelscited by 11
  6. 2024Enhancing In-Context Learning via Implicit Demonstration Augmentationcited by 6
  7. 2025IA2: Alignment with ICL Activations Improves Supervised Fine-Tuningcited by 1
  8. 2026SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Trainingcited by 1
  9. 2024Leveraging Prompts in LLMs to Overcome Imbalances in Complex Educational Text Datacited by 0
  10. 2024In-Context Learning with Noisy Labelscited by 0
  11. 2025Multimodal Function Vectors for Visual Relationscited by 0