mlsanityData to 14 Oct 2026

Assessing the Limits of In-Context Learning beyond Functions using Partially Ordered Relation

arXiv 2506.13608 ↗First posted 16 Jun 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Generating rational and generally accurate responses to tasks, often accompanied by example demonstrations, highlights Large Language Model's (LLM's) remarkable In-Context Learning (ICL) capabilities without requiring updates to the model's parameter space. Despite having an ongoing exploration focused on the inference from a document-level concept, its behavior in learning well-defined functions or relations in context needs a careful investigation. In this article, we present the performance of ICL on partially ordered relation by introducing the notion of inductively increasing complexity in prompts. In most cases, the saturated performance of the chosen metric indicates that while ICL offers some benefits, its effectiveness remains constrained as we increase the complexity in the prompts even in presence of sufficient demonstrative examples. The behavior is evident from our empirical findings and has further been theoretically justified in term of its implicit optimization process. The code is available hrefhttps://anonymous.4open.science/r/ICLonPartiallyOrderSethere.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

22 of 31 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  12. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
10 more
  1. 2023In-Context Learning Creates Task Vectorscited by 312
  2. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  3. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  4. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 106
  5. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  6. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  7. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  8. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  9. 2024Transformers are Universal In-context Learnerscited by 43
  10. 2024Ask, and it shall be given: On the Turing completeness of promptingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Identifiability and Order-Dimension Limits of In-Context Learning on Partial Orderscited by 0