# Towards out-of-distribution generalization in large-scale astronomical surveys: robust networks learn similar representations

arXiv: https://arxiv.org/abs/2311.18007 · HTML: https://mlsanity.com/paper/2311.18007/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 5 of 5)

- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [The CAMELS Multifield Dataset: Learning the Universe's Fundamental Parameters with Artificial Intelligence](https://mlsanity.com/paper/2109.10915/) (2021)
- [Multifield Cosmology with Artificial Intelligence](https://mlsanity.com/paper/2109.09747/) (2021)
- [Understanding Robust Learning through the Lens of Representation Similarities](https://mlsanity.com/paper/2206.09868/) (2022)

## Abstract (arXiv)

The generalization of machine learning (ML) models to out-of-distribution (OOD) examples remains a key challenge in extracting information from upcoming astronomical surveys. Interpretability approaches are a natural way to gain insights into the OOD generalization problem. We use Centered Kernel Alignment (CKA), a similarity measure metric of neural network representations, to examine the relationship between representation similarity and performance of pre-trained Convolutional Neural Networks (CNNs) on the CAMELS Multifield Dataset. We find that when models are robust to a distribution shift, they produce substantially different representations across their layers on OOD data. However, when they fail to generalize, these representations change less from layer to layer on OOD data. We discuss the potential application of similarity representation in guiding model design, training strategy, and mitigating the OOD problem by incorporating CKA as an inductive bias during training.