mlsanityData to 14 Oct 2026

On Theoretical Interpretations of Concept-Based In-Context Learning

arXiv 2509.20882 ↗First posted 25 Sep 2025cs.IT cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

In-Context Learning (ICL) has emerged as an important new paradigm in natural language processing and large language model (LLM) applications. However, the theoretical understanding of the ICL mechanism remains limited. This paper aims to investigate this issue by studying a particular ICL approach, called concept-based ICL (CB-ICL). In particular, we propose theoretical analyses on applying CB-ICL to ICL tasks, which explains why and when the CB-ICL performs well for predicting query labels in prompts with only a few demonstrations. In addition, the proposed theory quantifies the knowledge that can be leveraged by the LLMs to the prompt tasks, and leads to a similarity measure between the prompt demonstrations and the query input, which provides important insights and guidance for model pre-training and prompt engineering in ICL. Moreover, the impact of the prompt demonstration size and the dimension of the LLM embeddings in ICL are also explored based on the proposed theory. Finally, several real-data experiments are conducted to validate the practical usefulness of CB-ICL and the corresponding theory.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 47 of its 52 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

47 of 52 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
35 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  5. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  6. 2022Transformers learn in-context by gradient descentcited by 732
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  9. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  10. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  11. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  12. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  13. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  14. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  15. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  16. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  17. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  18. 2022Active Example Selection for In-Context Learningcited by 196
  19. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  20. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  21. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  22. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  23. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  24. 2023Finding Support Examples for In-Context Learningcited by 104
  25. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 102
  26. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 81
  27. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  28. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  29. 2024Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learningcited by 47
  30. 2023Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscited by 45
  31. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  32. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  33. 2023Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selectioncited by 37
  34. 2024MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learningcited by 15
  35. 2023ICL Markup: Structuring In-Context Learning using Soft-Token Tagscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  2. 2026A Theoretical Interpretation of In-Context Learning via Probabilistic Modelingcited by 0
  3. 2026CB-Agent: Concept-Based In-Context Learning for Sequential Action Selectioncited by 0