# Stability of In-Context Learning: A Spectral Coverage Perspective

arXiv: https://arxiv.org/abs/2509.20677 · HTML: https://mlsanity.com/paper/2509.20677/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 30 of 34)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection](https://mlsanity.com/paper/2306.04637/) (2023)
- [MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers](https://mlsanity.com/paper/2012.15828/) (2020)
- [Finding Support Examples for In-Context Learning](https://mlsanity.com/paper/2302.13539/) (2023)
- [On the Relation between Sensitivity and Accuracy in In-context Learning](https://mlsanity.com/paper/2209.07661/) (2022)
- [Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment](https://mlsanity.com/paper/2305.05940/) (2023)
- [Data Curation Alone Can Stabilize In-context Learning](https://mlsanity.com/paper/2212.10378/) (2022)
- [An Information-Theoretic Analysis of In-Context Learning](https://mlsanity.com/paper/2401.15530/) (2024)
- [Toward Understanding In-context vs. In-weight Learning](https://mlsanity.com/paper/2410.23042/) (2024)
- [Enhancing In-context Learning via Linear Probe Calibration](https://mlsanity.com/paper/2401.12406/) (2024)
- [Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context](https://mlsanity.com/paper/2410.01774/) (2024)
- [MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning](https://mlsanity.com/paper/2505.16225/) (2025)
- [Token-based Decision Criteria Are Suboptimal in In-context Learning](https://mlsanity.com/paper/2406.16535/) (2024)
- [Rethinking Invariance in In-context Learning](https://mlsanity.com/paper/2505.04994/) (2025)
- [Stable Diffusion Models are Secretly Good at Visual In-Context Learning](https://mlsanity.com/paper/2508.09949/) (2025)
- [Transformers Meet In-Context Learning: A Universal Approximation Theory](https://mlsanity.com/paper/2506.05200/) (2025)
- [SMMILE: An Expert-Driven Benchmark for Multimodal Medical In-Context Learning](https://mlsanity.com/paper/2506.21355/) (2025)
- [Exposing and Mitigating Calibration Biases and Demographic Unfairness in MLLM Few-Shot In-Context Learning for Medical Image Classification](https://mlsanity.com/paper/2506.23298/) (2025)

## Built on by (most cited first, 2 of 2)

- [TEMA: Anchor the Image, Follow the Text for Multi-Modification Composed Image Retrieval](https://mlsanity.com/paper/2604.21806/) (2026)
- [MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution](https://mlsanity.com/paper/2604.26283/) (2026)

## Abstract (arXiv)

In-context learning (ICL) is a pivotal capability for the practical deployment of large-scale language models, yet its reliability can vary substantially with the number of demonstrations provided in the prompt. A central obstacle is that the target notion, distributional stability under demonstration resampling, is expensive to measure directly at scale, making prompt-length selection largely heuristic. We therefore study a computable sufficient condition based on a spectral-coverage proxy: the lower tail of the spectrum of a regularized empirical second-moment matrix formed from demonstration representations. Under sub-Gaussian representation assumptions, we derive a non-asymptotic sample-size requirement (a lower bound on K) that guarantees this proxy event with prescribed failure probability, yielding a conservative prompt-length recommendation produced by an observable two-stage estimator. In large-scale experiments, the resulting estimates consistently upper-bound empirical accuracy knee-points, which we treat only as a practical surrogate for the prompt-length transition rather than a definition of stability. On a smaller held-out subset, direct resampling-based distributional stability measurements further validate the intended stability interpretation. Finally, a validation-only calibration step tightens the conservatism (typically to about 1.03--1.20×) while preserving conservative ordering, providing practical and verifiable guidance for ICL prompt design.