mlsanityData to 14 Oct 2026

A Survey on In-context Learning

arXiv 2301.00234 ↗First posted 31 Dec 2022cs.CL cs.AIcited by 721

Not on the map. Why?

Cited by 109 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the increasing capabilities of large language models (LLMs), in-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP), where LLMs make predictions based on contexts augmented with a few examples. It has been a significant trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis. Additionally, we explore various ICL application scenarios, such as data engineering and knowledge updating. Finally, we address the challenges of ICL and suggest potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.

Key facts. Cited by 721 papers in the map (data through 14 Oct 2026). Builds on 74 of its 187 references that are in the map. First posted on arXiv 31 Dec 2022.

Builds on

74 of 187 references.

References · 74

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  12. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  13. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  14. 2022In-context Learning and Induction Headscited by 964
  15. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  16. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  17. 2022Transformers learn in-context by gradient descentcited by 732
  18. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  19. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  20. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  21. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  22. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  23. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  24. 2021Editing Factual Knowledge in Language Modelscited by 553
  25. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  26. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  27. 2021MetaICL: Learning to Learn In Contextcited by 454
  28. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  29. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  30. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  31. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 318
  32. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  33. 2019Knowledge Enhanced Contextual Word Representationscited by 296
  34. 2022Teaching Small Language Models to Reasoncited by 284
  35. 2022Prompting GPT-3 To Be Reliablecited by 260
  36. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  37. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  38. 2022Visual Prompting via Image Inpaintingcited by 250

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 64; 2024: 285; 2025: 276; 2026: 96.

721 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  4. 2024A Survey on Large Language Models for Code Generationcited by 658
  5. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  7. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  8. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  9. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  10. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 223
  11. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  12. 2023Efficient Large Language Models: A Surveycited by 186
38 more
  1. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  4. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 152
  5. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  6. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  7. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  8. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  9. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  10. 2025Agentic Large Language Models, a surveycited by 108
  11. 2024Are self-explanations from Large Language Models faithful?cited by 101
  12. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 99
  13. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 97
  14. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  15. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  16. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  17. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 82
  18. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
  19. 2024Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Basescited by 77
  20. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 76
  21. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 68
  22. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 67
  23. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  24. 2024Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answeringcited by 60
  25. 2023Adaptive Machine Translation with Large Language Modelscited by 58
  26. 2024Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Explorationcited by 58
  27. 2023On the Creativity of Large Language Modelscited by 57
  28. 2023On the (In)Effectiveness of Large Language Models for Chinese Text Correctioncited by 54
  29. 2024Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Reviewcited by 53
  30. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inferencecited by 53
  31. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 53
  32. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  33. 2023From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Designcited by 51
  34. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  35. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 48
  36. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  37. 2025In-depth Analysis of Graph-based RAG in a Unified Frameworkcited by 46
  38. 2023Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategiescited by 45

50 most cited of 721 All 721 (JSON)