# Prospective Prediction of OOD Degradation from Source-Side Training Dynamics

arXiv: https://arxiv.org/abs/2610.12397 · HTML: https://mlsanity.com/paper/2610.12397/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 16 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Builds on (most cited first, 15 of 15)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Learning from Failure: Training Debiased Classifier from Biased Classifier](https://mlsanity.com/paper/2007.02561/) (2020)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing](https://mlsanity.com/paper/2207.10384/) (2022)
- [Shortcut Learning in Medical Image Segmentation](https://mlsanity.com/paper/2403.06748/) (2024)
- [Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning](https://mlsanity.com/paper/2402.17510/) (2024)
- [Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlations](https://mlsanity.com/paper/2403.03375/) (2024)
- [Beyond Distribution Shift: Spurious Features Through the Lens of Training Dynamics](https://mlsanity.com/paper/2302.09344/) (2023)

## Abstract (arXiv)

We study whether persistent out-of-distribution (OOD) degradation can be predicted before it is directly observed using only source-side training dynamics. In a controlled shortcut-learning setting, a simple logistic regression predictor develops a clear prospective signal, while training time alone does not. Temporal summaries of the source-side quantities are substantially more informative than their current values. When transferred without additional training from a CNN to an MLP, confidence and entropy dynamics retain substantial predictive information. These results provide a proof of principle that source-side training dynamics can contain an early warning signal for future OOD failure.