mlsanityData to 7 Oct 2026

A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Tasks

arXiv 2305.17040 ↗First posted 26 May 2023cs.LG cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

We study the phenomenon of textitin-context learning (ICL) exhibited by large language models, where they can adapt to a new learning task, given a handful of labeled examples, without any explicit parameter optimization. Our goal is to explain how a pre-trained transformer model is able to perform ICL under reasonable assumptions on the pre-training process and the downstream tasks. We posit a mechanism whereby a transformer can achieve the following: (a) receive an i.i.d. sequence of examples which have been converted into a prompt using potentially-ambiguous delimiters, (b) correctly segment the prompt into examples and labels, (c) infer from the data a textitsparse linear regressor hypothesis, and finally (d) apply this hypothesis on the given test example and return a predicted label. We establish that this entire procedure is implementable using the transformer mechanism, and we give sample complexity guarantees for this learning framework. Our empirical findings validate the challenge of segmentation, and we show a correspondence between our posited mechanisms and observed attention maps for step (c).

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

19 of 24 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  9. 2022Transformers learn in-context by gradient descentcited by 714
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  12. 2016Learning to learn by gradient descent by gradient descentcited by 616
7 more
  1. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  2. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  3. 2019Online Meta-Learningcited by 137
  4. 2023The Learnability of In-Context Learningcited by 126
  5. 2020A survey on domain adaptation theory: learning bounds and theoretical guaranteescited by 109
  6. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  7. 2022An Analysis of Attention via the Lens of Exchangeability and Latent Variable Modelscited by 15

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 10; 2025: 2.

17 papers.

Sort:

By area:

  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  2. 2023Challenges and Applications of Large Language Modelscited by 251
  3. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  4. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  5. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 97
  6. 2024Can large language models explore in-context?cited by 75
  7. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  8. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  9. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  10. 2024In-context Exploration-Exploitation for Reinforcement Learningcited by 16
  11. 2024One-Layer Transformer Provably Learns One-Nearest Neighbor In Contextcited by 14
  12. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 13
5 more
  1. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
  2. 2024How Far Can In-Context Alignment Go? Exploring the State of In-Context Alignmentcited by 6
  3. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  4. 2025Transformers versus the EM Algorithm in Multi-class Clusteringcited by 3
  5. 2025Optimization-Inspired Few-Shot Adaptation for Large Language Modelscited by 3