mlsanityData to 14 Oct 2026

Larger language models do in-context learning differently

arXiv 2303.03846 ↗First posted 7 Mar 2023cs.CLcited by 394

Not on the map. Why?

Cited by 46 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study how in-context learning (ICL) in language models is affected by semantic priors versus input-label mappings. We investigate two setups-ICL with flipped labels and ICL with semantically-unrelated labels-across various model families (GPT-3, InstructGPT, Codex, PaLM, and Flan-PaLM). First, experiments on ICL with flipped labels show that overriding semantic priors is an emergent ability of model scale. While small language models ignore flipped labels presented in-context and thus rely primarily on semantic priors from pretraining, large models can override semantic priors when presented with in-context exemplars that contradict priors, despite the stronger semantic priors that larger models may hold. We next study semantically-unrelated label ICL (SUL-ICL), in which labels are semantically unrelated to their inputs (e.g., foo/bar instead of negative/positive), thereby forcing language models to learn the input-label mappings shown in in-context exemplars in order to perform the task. The ability to do SUL-ICL also emerges primarily with scale, and large-enough language models can even perform linear classification in a SUL-ICL setting. Finally, we evaluate instruction-tuned models and find that instruction tuning strengthens both the use of semantic priors and the capacity to learn input-label mappings, but more of the former.

Key facts. Cited by 394 papers in the map (data through 14 Oct 2026). Builds on 34 of its 48 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

34 of 48 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
22 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  8. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  10. 2022Transformers learn in-context by gradient descentcited by 732
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  13. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  14. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  15. 2021MetaICL: Learning to Learn In Contextcited by 454
  16. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  17. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  18. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  19. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  20. 2022Predictability and Surprise in Large Generative Modelscited by 210
  21. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  22. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 115; 2024: 152; 2025: 71; 2026: 56.

394 papers.

Sort:

By area:

  1. 2023Large Language Models as Optimizerscited by 864
  2. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  3. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  4. 2023Explainability for Large Language Models: A Surveycited by 333
  5. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  6. 2023Function Vectors in Large Language Modelscited by 308
  7. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  9. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  10. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  11. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  12. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
38 more
  1. 2024Many-Shot In-Context Learningcited by 206
  2. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  3. 2023Personality Traits in Large Language Modelscited by 169
  4. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  5. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  6. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  7. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  8. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  9. 2024Large Language Models to Enhance Bayesian Optimizationcited by 135
  10. 2023The Learnability of In-Context Learningcited by 127
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  12. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  13. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  14. 2023In-Context Convergence of Transformerscited by 115
  15. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  16. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  17. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  18. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  19. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 90
  20. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 90
  21. 2023Mitigating Label Biases for In-context Learningcited by 89
  22. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  23. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  24. 2023In-Context Learning through the Bayesian Prismcited by 83
  25. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 83
  26. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  27. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 81
  28. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 79
  29. 2024Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Basescited by 77
  30. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 76
  31. 2023Symbol tuning improves in-context learning in language modelscited by 75
  32. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  33. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  34. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  35. 2023Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewritingcited by 69
  36. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 69
  37. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  38. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 64

50 most cited of 394 All 394 (JSON)