# Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models

arXiv: https://arxiv.org/abs/2610.09709 · HTML: https://mlsanity.com/paper/2610.09709/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Builds on (most cited first, 11 of 11)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [Stable Rank Normalization for Improved Generalization in Neural Networks and GANs](https://mlsanity.com/paper/1906.04659/) (2019)
- [RITA: a Study on Scaling Up Generative Protein Sequence Models](https://mlsanity.com/paper/2205.05789/) (2022)
- [Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics](https://mlsanity.com/paper/2106.00734/) (2021)
- [Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology](https://mlsanity.com/paper/2605.14258/) (2026)
- [Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?](https://mlsanity.com/paper/2410.14602/) (2024)
- [Representation Geometry as a Diagnostic for Out-of-Distribution Robustness](https://mlsanity.com/paper/2602.03951/) (2026)
- [Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference](https://mlsanity.com/paper/2602.20449/) (2026)
- [Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth](https://mlsanity.com/paper/2607.14018/) (2026)
- [Domain Feature Collapse: Implications for Out-of-Distribution Detection and Solutions](https://mlsanity.com/paper/2512.04034/) (2025)

## Abstract (arXiv)

Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available? Existing diagnostics require source or target data, which rules them out before a target domain exists. Those that use the weights alone apply one statistic to every architecture, and do not separate robust models from fragile ones. We show the answer is encoded in the spectral structure of pretrained weights. Two forces shape that structure. Architecture determines how information is stored in weight matrices. Pretraining strategy determines what is rewarded. Together they set a spectral geometry that governs OOD robustness. We prove that the OOD accuracy gap is bounded by how tightly the source representations concentrate. A statistic computed from the pretrained weights alone serves as a proxy for that concentration. The direction of that proxy reverses between architecture families. We operationalize it: the direction is stable within one (architecture X strategy) combination, the finest grouping we test, which we call a cell. Pooled over 116 models spanning 7 modalities, a single statistic ranks OOD robustness weakly, because cells of opposite direction cancel. Within a cell, the statistic selected for it orders 92% of model pairs by OOD robustness in-sample. The selection does not leak the target: for each model family outside the matrix we logged the cell, metric and sign before running its OOD evaluation, and the predicted direction held in every case: EEG, genomic and protein. Acting on spectral concentration narrows the OOD gap by 24% at 87.5% ID retention. The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time, before data or compute is committed to a target domain.