mlsanityData to 7 Oct 2026

Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment

arXiv 2407.19346 ↗First posted 27 Jul 2024cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 8 of its 13 references that are in the map. First posted on arXiv 27 Jul 2024.

Builds on

8 of 13 references.

References · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  7. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  8. 2020When Do Curricula Work?cited by 105

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizercited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Linear Transformers with Learnable Kernel Functions are Better In-Context Models
  2. 2025Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency
  3. 2024In-Context Language Learning: Architectures and Algorithms
  4. 2023Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability
  5. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
  6. 2025Symbolic Regression with Multimodal Large Language Models and Kolmogorov Arnold Networks
  7. 2023Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning
  8. 2025How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalization