mlsanityData to 7 Oct 2026

Meta-in-context learning in large language models

arXiv 2305.12907 ↗First posted 22 May 2023cs.CL cs.AI cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

Large language models have shown tremendous performance in a variety of tasks. In-context learning -- the ability to improve at a task after being provided with a number of demonstrations -- is seen as one of the main contributors to their success. In the present paper, we demonstrate that the in-context learning abilities of large language models can be recursively improved via in-context learning itself. We coin this phenomenon meta-in-context learning. Looking at two idealized domains, a one-dimensional regression task and a two-armed bandit task, we show that meta-in-context learning adaptively reshapes a large language model's priors over expected tasks. Furthermore, we find that meta-in-context learning modifies the in-context learning strategies of such models. Finally, we extend our approach to a benchmark of real-world regression problems where we observe competitive performance to traditional learning algorithms. Taken together, our work improves our understanding of in-context learning and paves the way toward adapting large language models to the environment they are applied purely through meta-in-context learning rather than traditional finetuning.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

21 of 38 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2022Transformers learn in-context by gradient descentcited by 714
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  8. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  9. 2021MetaICL: Learning to Learn In Contextcited by 450
  10. 2022Using cognitive psychology to understand GPT-3cited by 389
  11. 2016Learning to reinforcement learncited by 369
  12. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
9 more
  1. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  2. 2022Prompting GPT-3 To Be Reliablecited by 258
  3. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  4. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 174
  5. 2022Language models show human-like content effects on reasoning taskscited by 171
  6. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  7. 2019Meta-learning of Sequential Strategiescited by 68
  8. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  9. 2022Modeling Human Exploration Through Resource-Rational Reinforcement Learningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 18; 2025: 20; 2026: 11.

59 papers.

Sort:

By area:

  1. 2023Playing repeated games with Large Language Modelscited by 213
  2. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  3. 2024Can large language models explore in-context?cited by 75
  4. 2024CogBench: a large language model walks into a psychology labcited by 60
  5. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 43
  6. 2023Visual cognition in multimodal large language modelscited by 38
  7. 2024The broader spectrum of in-context learningcited by 33
  8. 2023Generative Calibration for In-context Learningcited by 20
  9. 2025True Zero-Shot Inference of Dynamical Systems Preserving Long-Term Statisticscited by 20
  10. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  11. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devicescited by 17
  12. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 17
38 more
  1. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 15
  2. 2024Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Modelscited by 15
  3. 2025Toward Efficient Exploration by Large Language Model Agentscited by 15
  4. 2024In-context learning agents are asymmetric belief updaterscited by 13
  5. 2023Design of Chain-of-Thought in Math Problem Solvingcited by 12
  6. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  7. 2025Should You Use Your Large Language Model to Explore or Exploit?cited by 8
  8. 2023Decoding In-Context Learning: Neuroscience-inspired Analysis of Representations in Large Language Modelscited by 7
  9. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 7
  10. 2023Metalearning Continual Learning Algorithmscited by 6
  11. 2024Recall, Retrieve and Reason: Towards Better In-Context Relation Extractioncited by 6
  12. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  13. 2024SegICL: A Multimodal In-context Learning Framework for Enhanced Segmentation in Medical Imagingcited by 5
  14. 2025LLM-First Search: Self-Guided Exploration of the Solution Spacecited by 5
  15. 2023Task-Distributionally Robust Data-Free Meta-Learningcited by 4
  16. 2025Multimodal-to-Text Prompt Engineering in Large Language Models Using Feature Embeddings for GNSS Interference Characterizationcited by 4
  17. 2025Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worldscited by 4
  18. 2025Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortexcited by 4
  19. 2023In-context Interference in Chat-based Large Language Modelscited by 3
  20. 2024Human-like Category Learning by Injecting Ecological Priors from Large Language Models into Neural Networkscited by 3
  21. 2024Metalic: Meta-Learning In-Context with Protein Language Modelscited by 3
  22. 2025Are Large Language Models Good Temporal Graph Learners?cited by 3
  23. 2025On the Performance of LLMs for Real Estate Appraisalcited by 3
  24. 2025When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Trainingcited by 3
  25. 2025LLMs as In-Context Meta-Learners for Model and Hyperparameter Selectioncited by 3
  26. 2023Causal Intersectionality and Dual Form of Gradient Descent for Multimodal Analysis: a Case Study on Hateful Memescited by 2
  27. 2025Incomplete In-context Learningcited by 2
  28. 2026<SOG_k>: One LLM Token for Explicit Graph Structural Understandingcited by 2
  29. 2026Understanding the Dynamics of Demonstration Conflict in In-Context Learningcited by 2
  30. 2024ViDAS: Vision-based Danger Assessment and Scoringcited by 1
  31. 2024RWKV-edge: Deeply Compressed RWKV for Resource-Constrained Devicescited by 1
  32. 2025Preliminary Use of Vision Language Model Driven Extraction of Mouse Behavior Towards Understanding Fear Expressioncited by 1
  33. 2025Foundation of Intelligence: Review of Math Word Problems from Human Cognition Perspectivecited by 1
  34. 2026Meta-learning In-Context Enables Training-Free Cross Subject Brain Decodingcited by 1
  35. 2026Brick-DICL: Dynamic In-Context Learning for Automated Brick Schema Classificationcited by 1
  36. 2024VARS: Vision-based Assessment of Risk in Security Systemscited by 0
  37. 2024Can a Large Language Model Learn Matrix Functions In Context?cited by 0
  38. 2025Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answeringcited by 0

50 most cited of 59 All 59 (JSON)