Bigger Is Safer: Provable Robustness in In-Context Learning Scales with Capacity

In-context learning (ICL) allows large language models to adapt to new tasks from a few examples without updating their parameters. Existing theories explain ICL by assuming the test task distribution matches pretraining -- an assumption that breaks down under adversarial distribution shifts. We introduce a distributionally robust meta-learning framework that provides worst-case guarantees for ICL under Wasserstein-based distribution shifts. Focusing on linear self-attention Transformers, we derive a non-asymptotic bound connecting adversarial perturbation strength (ρ), model capacity (m), and the number of in-context examples (N). The analysis reveals that the maximum safe perturbation radius scales as ρmax propto √m, while maintaining performance under adversarial shift requires additional in-context examples with Nρ - N0 propto ρ2. Experiments on synthetic tasks confirm these scaling laws, and experiments on 21 real pretrained models (0.1B--7B parameters, 5 families) provide qualitative evidence consistent with the theory's predictions, while revealing that ICL capability is a prerequisite for robustness. These findings advance the theoretical understanding of ICL under adversarial conditions and formalize the sense in which larger models are safer under distributional shift.