mlsanityData to 14 Oct 2026

Stability of In-Context Learning: A Spectral Coverage Perspective

arXiv 2509.20677 ↗First posted 25 Sep 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) is a pivotal capability for the practical deployment of large-scale language models, yet its reliability can vary substantially with the number of demonstrations provided in the prompt. A central obstacle is that the target notion, distributional stability under demonstration resampling, is expensive to measure directly at scale, making prompt-length selection largely heuristic. We therefore study a computable sufficient condition based on a spectral-coverage proxy: the lower tail of the spectrum of a regularized empirical second-moment matrix formed from demonstration representations. Under sub-Gaussian representation assumptions, we derive a non-asymptotic sample-size requirement (a lower bound on K) that guarantees this proxy event with prescribed failure probability, yielding a conservative prompt-length recommendation produced by an observable two-stage estimator. In large-scale experiments, the resulting estimates consistently upper-bound empirical accuracy knee-points, which we treat only as a practical surrogate for the prompt-length transition rather than a definition of stability. On a smaller held-out subset, direct resampling-based distributional stability measurements further validate the intended stability interpretation. Finally, a validation-only calibration step tightens the conservatism (typically to about 1.03--1.20×) while preserving conservative ordering, providing practical and verifiable guidance for ICL prompt design.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

34 of 45 references.

References · 34

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  12. 2025Kimi K2: Open Agentic Intelligencecited by 567
22 more
  1. 2023In-Context Learning Creates Task Vectorscited by 312
  2. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  3. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  4. 2023Finding Support Examples for In-Context Learningcited by 104
  5. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 72
  6. 2023Multilingual LLMs are Better Cross-lingual In-context Learners with Alignmentcited by 71
  7. 2022Data Curation Alone Can Stabilize In-context Learningcited by 59
  8. 2024An Information-Theoretic Analysis of In-Context Learningcited by 40
  9. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  10. 2024Enhancing In-context Learning via Linear Probe Calibrationcited by 16
  11. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 15
  12. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  13. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  14. 2025Rethinking Invariance in In-context Learningcited by 8
  15. 2025Stable Diffusion Models are Secretly Good at Visual In-Context Learningcited by 8
  16. 2025Transformers Meet In-Context Learning: A Universal Approximation Theorycited by 6
  17. 2025SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learningcited by 5
  18. 2025Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classificationcited by 5
  19. 2025Provable Benefits of Task-Specific Prompts for In-context Learningcited by 4
  20. 2024Disentangling Latent Shifts of In-Context Learning with Weak Supervisioncited by 3
  21. 2025Surprise Calibration for Better In-Context Learningcited by 2
  22. 2025Boosting In-Context Learning in LLMs Through the Lens of Classical Supervised Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrievalcited by 15
  2. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12