Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models
Not on the map. Why?
Abstract (arXiv)
Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available? Existing diagnostics require source or target data, which rules them out before a target domain exists. Those that use the weights alone apply one statistic to every architecture, and do not separate robust models from fragile ones. We show the answer is encoded in the spectral structure of pretrained weights. Two forces shape that structure. Architecture determines how information is stored in weight matrices. Pretraining strategy determines what is rewarded. Together they set a spectral geometry that governs OOD robustness. We prove that the OOD accuracy gap is bounded by how tightly the source representations concentrate. A statistic computed from the pretrained weights alone serves as a proxy for that concentration. The direction of that proxy reverses between architecture families. We operationalize it: the direction is stable within one (architecture X strategy) combination, the finest grouping we test, which we call a cell. Pooled over 116 models spanning 7 modalities, a single statistic ranks OOD robustness weakly, because cells of opposite direction cancel. Within a cell, the statistic selected for it orders 92% of model pairs by OOD robustness in-sample. The selection does not leak the target: for each model family outside the matrix we logged the cell, metric and sign before running its OOD evaluation, and the predicted direction held in every case: EEG, genomic and protein. Acting on spectral concentration narrows the OOD gap by 24% at 87.5% ID retention. The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time, before data or compute is committed to a target domain.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 49 references that are in the map. First posted on arXiv 7 Oct 2026.
Builds on
11 of 49 references.
References · 11
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
- 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
- 2022RITA: a Study on Scaling Up Generative Protein Sequence Modelscited by 39
- 2021Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metricscited by 19
- 2026Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topologycited by 5
- 2024Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?cited by 4
- 2026Representation Geometry as a Diagnostic for Out-of-Distribution Robustnesscited by 3
- 2026Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inferencecited by 2
- 2026Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depthcited by 2
- 2025Domain Feature Collapse: Implications for Out-of-Distribution Detection and Solutionscited by 1
Built on by
Not cited yet.