mlsanityData to 14 Oct 2026

Rethinking Invariance in In-context Learning

arXiv 2505.04994 ↗First posted 8 May 2025cs.CL cs.AIcited by 8

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In-Context Learning (ICL) has emerged as a pivotal capability of auto-regressive large language models, yet it is hindered by a notable sensitivity to the ordering of context examples regardless of their mutual independence. To address this issue, recent studies have introduced several variant algorithms of ICL that achieve permutation invariance. However, many of these do not exhibit comparable performance with the standard auto-regressive ICL algorithm. In this work, we identify two crucial elements in the design of an invariant ICL algorithm: information non-leakage and context interdependence, which are not simultaneously achieved by any of the existing methods. These investigations lead us to the proposed Invariant ICL (InvICL), a methodology designed to achieve invariance in ICL while ensuring the two properties. Empirically, our findings reveal that InvICL surpasses previous models, both invariant and non-invariant, in most benchmark datasets, showcasing superior generalization capabilities across varying input lengths. Code is available at https://github.com/PKU-ML/InvICL.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 14 of its 43 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

14 of 43 references.

References · 14

  1. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  2. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  3. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  4. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  5. 2022In-context Examples Selection for Machine Translationcited by 172
  6. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  7. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  8. 2022Parallel Context Windows for Large Language Modelscited by 81
  9. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
  10. 2023CausalLM is not optimal for in-context learningcited by 26
  11. 2024Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learningcited by 20
  12. 2023Do pretrained Transformers Learn In-Context by Gradient Descent?cited by 18
2 more
  1. 2024Addressing Order Sensitivity of In-Context Demonstration Examples in Causal Language Modelscited by 12
  2. 2023Positional Information Matters for Invariant In-Context Learning: A Case Study of Simple Function Classescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 6
  2. 2025Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understandingcited by 2
  3. 2025Stability of In-Context Learning: A Spectral Coverage Perspectivecited by 2
  4. 2026How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalizationcited by 2
  5. 2025In-Context Learning for Gradient-Free Receiver Adaptation: Principles, Applications, and Theorycited by 1
  6. 2025Optimizing In-Context Learning for Efficient Full Conformal Predictioncited by 1
  7. 2026PLR: Plackett-Luce for Reordering In-Context Learning Examplescited by 1
  8. 2026On the Runway Cascade of Transformers for Language Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
  2. 2024In-Context Language Learning: Architectures and Algorithms
  3. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  4. 2025Is In-Context Learning Learning?
  5. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
  6. 2025On the Relationship Between the Choice of Representation and In-Context Learning
  7. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment
  8. 2026Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning