mlsanityData to 14 Oct 2026

A Theoretical Interpretation of In-Context Learning via Probabilistic Modeling

arXiv 2606.28926 ↗First posted 27 Jun 2026cs.IT cs.LG math.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) is an emerging paradigm that employs the semantic information inherent in large language models (LLMs) for generating answers to user queries. While the remarkable performance of ICL has been widely known, a general modeling and a rigorous theoretical analysis of this paradigm are still lacking. This work presents a probabilistic model for ICL and derives the performance of ICL for both general parametric distributions and exponential families. Based on the derived results, the work explains the impact of multiple factors such as the number of demonstrations, the sensitivity of the probabilistic model to the variation of its parameters, as well as the similarity between the demonstrations and the query on the performance of ICL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 19 references that are in the map. First posted on arXiv 27 Jun 2026.

Builds on

15 of 19 references.

References · 15

  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  2. 2022Transformers learn in-context by gradient descentcited by 732
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  4. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  5. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  6. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  7. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  8. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  9. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  10. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  11. 2024Large Language Models to Enhance Bayesian Optimizationcited by 135
  12. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
3 more
  1. 2024When Large Language Model Agents Meet 6G Networks: Perception, Grounding, and Alignmentcited by 43
  2. 2025CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memorycited by 32
  3. 2025On Theoretical Interpretations of Concept-Based In-Context Learningcited by 3

Built on by

Not cited yet.