mlsanityData to 7 Oct 2026

In-Context Learning with Hypothesis-Class Guidance

arXiv 2502.19787 ↗First posted 27 Feb 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent research has investigated the underlying mechanisms of in-context learning (ICL) both theoretically and empirically, often using data generated from simple function classes. However, the existing work often focuses on the sequence consisting solely of labeled examples, while in practice, labeled examples are typically accompanied by an instruction, providing some side information about the task. In this work, we propose ICL with hypothesis-class guidance (ICL-HCG), a novel synthetic data model for ICL where the input context consists of the literal description of a (finite) hypothesis class H and (x,y) pairs from a hypothesis chosen from H. Under our framework ICL-HCG, we conduct extensive experiments to explore: (i) a variety of generalization abilities to new hypothesis classes; (ii) different model architectures; (iii) sample complexity; (iv) in-context data imbalance; (v) the role of instruction; and (vi) the effect of pretraining hypothesis diversity. As a result, we show that (a) Transformers can successfully learn ICL-HCG and generalize to unseen hypotheses and unseen hypothesis classes, and (b) compared with ICL without instruction, ICL-HCG achieves significantly higher accuracy, demonstrating the role of instructions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

28 of 36 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2022Transformers learn in-context by gradient descentcited by 714
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  10. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  12. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
16 more
  1. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
  2. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  3. 2023In-Context Convergence of Transformerscited by 112
  4. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  5. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  6. 2023In-Context Learning through the Bayesian Prismcited by 78
  7. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  8. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  9. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  10. 2024Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?cited by 60
  11. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  12. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  13. 2024Dual Operating Modes of In-Context Learningcited by 42
  14. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  15. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  16. 2024Task Diversity Shortens the ICL Plateaucited by 5

Built on by

Not cited yet.