# Structure-based out-of-distribution (OOD) materials property prediction: a benchmark study

arXiv: https://arxiv.org/abs/2401.08032 · HTML: https://mlsanity.com/paper/2401.08032/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 80 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [A Survey of Unsupervised Deep Domain Adaptation](https://mlsanity.com/paper/1812.02849/) (2018)
- [GemNet: Universal Directional Graph Neural Networks for Molecules](https://mlsanity.com/paper/2106.08903/) (2021)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)
- [Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules](https://mlsanity.com/paper/2011.14115/) (2020)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [Graph neural networks for materials science and chemistry](https://mlsanity.com/paper/2208.09481/) (2022)
- [GOOD: A Graph Out-of-Distribution Benchmark](https://mlsanity.com/paper/2206.08452/) (2022)
- [Towards Deeper Graph Neural Networks with Differentiable Group Normalization](https://mlsanity.com/paper/2006.06972/) (2020)
- [The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts](https://mlsanity.com/paper/2206.08917/) (2022)
- [Periodic Graph Transformers for Crystal Material Property Prediction](https://mlsanity.com/paper/2209.11807/) (2022)
- [Predicting materials properties without crystal structure: Deep representation learning from stoichiometry](https://mlsanity.com/paper/1910.00617/) (2019)
- [Scalable Diffusion for Materials Generation](https://mlsanity.com/paper/2311.09235/) (2023)
- [Assaying Out-Of-Distribution Generalization in Transfer Learning](https://mlsanity.com/paper/2207.09239/) (2022)
- [Physics Guided Deep Learning for Generative Design of Crystal Materials with Symmetry Constraints](https://mlsanity.com/paper/2203.14352/) (2022)
- [Orbital Graph Convolutional Neural Network for Material Property Prediction](https://mlsanity.com/paper/2008.06415/) (2020)
- [Scalable deeper graph neural networks for high-performance materials property prediction](https://mlsanity.com/paper/2109.12283/) (2021)
- [Materials Property Prediction with Uncertainty Quantification: A Benchmark Study](https://mlsanity.com/paper/2211.02235/) (2022)
- [MaterialsAtlas.org: A Materials Informatics Web App Platform for Materials Discovery and Survey of State-of-the-Art](https://mlsanity.com/paper/2109.04007/) (2021)
- [Crystal structure prediction using neural network potential and age-fitness Pareto genetic algorithm](https://mlsanity.com/paper/2309.06710/) (2023)
- [Latent Conservative Objective Models for Data-Driven Crystal Structure Prediction](https://mlsanity.com/paper/2310.10056/) (2023)

## Built on by (most cited first, 17 of 17)

- [Known Unknowns: Out-of-Distribution Property Prediction in Materials and Molecules](https://mlsanity.com/paper/2502.05970/) (2025)
- [BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models](https://mlsanity.com/paper/2505.01912/) (2025)
- [Foundation Model for Composite Microstructures: Reconstruction, Stiffness, and Nonlinear Behavior Prediction](https://mlsanity.com/paper/2411.06565/) (2024)
- [Active Learning Enables Generation of Molecules that Advance the Known Pareto Front](https://mlsanity.com/paper/2501.02059/) (2025)
- [Out-of-distribution materials property prediction using adversarial learning based fine-tuning](https://mlsanity.com/paper/2408.09297/) (2024)
- [Benchmarking GNNs for OOD Materials Property Prediction with Uncertainty Quantification](https://mlsanity.com/paper/2511.11697/) (2025)
- [MatMind: A Structure-Activity Knowledge-Driven Generative Foundation Model for Materials Science](https://mlsanity.com/paper/2606.07712/) (2026)
- [Data-Efficient Machine Learning Potentials via Difference Vectors Based on Local Atomic Environments](https://mlsanity.com/paper/2501.16398/) (2025)
- [SA-GAT-SR: Self-Adaptable Graph Attention Networks with Symbolic Regression for high-fidelity material property prediction](https://mlsanity.com/paper/2505.00625/) (2025)
- [Symbolic regression for defect interactions in 2D materials](https://mlsanity.com/paper/2512.20785/) (2025)
- [A Critical Examination of Active Learning Workflows in Materials Science](https://mlsanity.com/paper/2601.05946/) (2026)
- [Inverse Design of Inorganic Compounds with Generative AI](https://mlsanity.com/paper/2604.11827/) (2026)
- [Benchmarking bandgap prediction in semiconductors under experimental and realistic evaluation settings](https://mlsanity.com/paper/2604.25568/) (2026)
- [Benchmarking Compositional Generalisation for Machine Learning Interatomic Potentials](https://mlsanity.com/paper/2605.08988/) (2026)
- [AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking](https://mlsanity.com/paper/2607.11526/) (2026)
- [Physics-residual machine learning predicts oxygen-evolution catalyst activity beyond the training range from sparse polarization measurements](https://mlsanity.com/paper/2609.23549/) (2026)
- [Measuring trainable degrees of freedom in materials graph neural networks: a random-subspace intrinsic dimension analysis](https://mlsanity.com/paper/2609.36084/) (2026)

## Abstract (arXiv)

In real-world material research, machine learning (ML) models are usually expected to predict and discover novel exceptional materials that deviate from the known materials. It is thus a pressing question to provide an objective evaluation of ML model performances in property prediction of out-of-distribution (OOD) materials that are different from the training set distribution. Traditional performance evaluation of materials property prediction models through random splitting of the dataset frequently results in artificially high performance assessments due to the inherent redundancy of typical material datasets. Here we present a comprehensive benchmark study of structure-based graph neural networks (GNNs) for extrapolative OOD materials property prediction. We formulate five different categories of OOD ML problems for three benchmark datasets from the MatBench study. Our extensive experiments show that current state-of-the-art GNN algorithms significantly underperform for the OOD property prediction tasks on average compared to their baselines in the MatBench study, demonstrating a crucial generalization gap in realistic material prediction tasks. We further examine the latent physical spaces of these GNN models and identify the sources of CGCNN, ALIGNN, and DeeperGATGNN's significantly more robust OOD performance than those of the current best models in the MatBench study (coGN and coNGN), and provide insights to improve their performance.