# Matbench Discovery -- A framework to evaluate machine learning crystal stability predictions

arXiv: https://arxiv.org/abs/2308.14920 · HTML: https://mlsanity.com/paper/2308.14920/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 95 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations](https://mlsanity.com/paper/2306.12059/) (2023)
- [Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs](https://mlsanity.com/paper/2302.03655/) (2023)
- [The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts](https://mlsanity.com/paper/2206.08917/) (2022)
- [Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations](https://mlsanity.com/paper/2210.07237/) (2022)
- [Predicting materials properties without crystal structure: Deep representation learning from stoichiometry](https://mlsanity.com/paper/1910.00617/) (2019)
- [Fast Uncertainty Estimates in Deep Learning Interatomic Potentials](https://mlsanity.com/paper/2211.09866/) (2022)

## Built on by (most cited first, 30 of 35)

- [UMA: A Family of Universal Models for Atoms](https://mlsanity.com/paper/2506.23971/) (2025)
- [A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems](https://mlsanity.com/paper/2312.07511/) (2023)
- [Orb: A Fast, Scalable Neural Network Potential](https://mlsanity.com/paper/2410.22570/) (2024)
- [SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models](https://mlsanity.com/paper/2502.03638/) (2025)
- [A practical guide to machine learning interatomic potentials -- Status and future](https://mlsanity.com/paper/2503.09814/) (2025)
- [PET-MAD, a lightweight universal interatomic potential for advanced materials modeling](https://mlsanity.com/paper/2503.14118/) (2025)
- [Reinforcement Fine-Tuning for Materials Design](https://mlsanity.com/paper/2504.02367/) (2025)
- [Pushing the limits of unconstrained machine-learned interatomic potentials](https://mlsanity.com/paper/2601.16195/) (2026)
- [EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers](https://mlsanity.com/paper/2604.09130/) (2026)
- [Toward Greater Autonomy in Materials Discovery Agents: Unifying Planning, Physics, and Scientists](https://mlsanity.com/paper/2506.05616/) (2025)
- [Establishing baselines for generative discovery of inorganic crystals](https://mlsanity.com/paper/2501.02144/) (2025)
- [MatterTune: An Integrated, User-Friendly Platform for Fine-Tuning Atomistic Foundation Models to Accelerate Materials Simulation and Discovery](https://mlsanity.com/paper/2504.10655/) (2025)
- [Automated Machine Learning Pipeline: Large Language Models-Assisted Automated Dataset Generation for Training Machine-Learned Interatomic Potentials](https://mlsanity.com/paper/2509.21647/) (2025)
- [Auto Research for Materials: Auditable AI-Scientist Workflows with Held-Out Transfer](https://mlsanity.com/paper/2607.17100/) (2026)
- [MiAD: Mirage Atom Diffusion for De Novo Crystal Generation](https://mlsanity.com/paper/2511.14426/) (2025)
- [MP-ALOE: An r2SCAN dataset for universal machine learning interatomic potentials](https://mlsanity.com/paper/2507.05559/) (2025)
- [Interpolation and differentiation of alchemical degrees of freedom in machine learning interatomic potentials](https://mlsanity.com/paper/2404.10746/) (2024)
- [AlphaNet: Scaling Up Local-frame-based Atomistic Interatomic Potential](https://mlsanity.com/paper/2501.07155/) (2025)
- [Enhancing Materials Discovery with Valence Constrained Design in Generative Modeling](https://mlsanity.com/paper/2507.19799/) (2025)
- [Spiers Memorial Lecture: How to do impactful research in artificial intelligence for chemistry and materials science](https://mlsanity.com/paper/2409.10304/) (2024)
- [Learning Ordering in Crystalline Materials with Symmetry-Aware Graph Neural Networks](https://mlsanity.com/paper/2409.13851/) (2024)
- [The OpenLAM Challenges](https://mlsanity.com/paper/2501.16358/) (2025)
- [Efficient Nudged Elastic Band Method using Neural Network Bayesian Algorithm Execution](https://mlsanity.com/paper/2512.14993/) (2025)
- [From Evaluation to Design: Using Potential Energy Surface Smoothness Metrics to Guide Machine Learning Interatomic Potential Architectures](https://mlsanity.com/paper/2602.04861/) (2026)
- [Proof-Carrying Materials: Falsifiable Safety Certificates for Machine-Learned Interatomic Potentials](https://mlsanity.com/paper/2603.12183/) (2026)
- [TriForces: Augmenting Atomistic GNNs for Transferable Representations](https://mlsanity.com/paper/2605.20581/) (2026)
- [MatMind: A Structure-Activity Knowledge-Driven Generative Foundation Model for Materials Science](https://mlsanity.com/paper/2606.07712/) (2026)
- [Predicting ionic conductivity in solids from the machine-learned potential energy landscape](https://mlsanity.com/paper/2411.06804/) (2024)
- [Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks](https://mlsanity.com/paper/2503.02147/) (2025)
- [Benchmarking Universal Interatomic Potentials on Zeolite Structures](https://mlsanity.com/paper/2509.07417/) (2025)

## Abstract (arXiv)

The rapid adoption of machine learning (ML) in domain sciences necessitates best practices and standardized benchmarking for performance evaluation. We present Matbench Discovery, an evaluation framework for ML energy models, applied as pre-filters for high-throughput searches of stable inorganic crystals. This framework addresses the disconnect between thermodynamic stability and formation energy, as well as retrospective vs. prospective benchmarking in materials discovery. We release a Python package to support model submissions and maintain an online leaderboard, offering insights into performance trade-offs. To identify the best-performing ML methodologies for materials discovery, we benchmarked various approaches, including random forests, graph neural networks (GNNs), one-shot predictors, iterative Bayesian optimizers, and universal interatomic potentials (UIP). Our initial results rank models by test set F1 scores for thermodynamic stability prediction: EquiformerV2 + DeNS > Orb > SevenNet > MACE > CHGNet > M3GNet > ALIGNN > MEGNet > CGCNN > CGCNN+P > Wrenformer > BOWSR > Voronoi fingerprint random forest. UIPs emerge as the top performers, achieving F1 scores of 0.57-0.82 and discovery acceleration factors (DAF) of up to 6x on the first 10k stable predictions compared to random selection. We also identify a misalignment between regression metrics and task-relevant classification metrics. Accurate regressors can yield high false-positive rates near the decision boundary at 0 eV/atom above the convex hull. Our results demonstrate UIPs' ability to optimize computational budget allocation for expanding materials databases. However, their limitations remain underexplored in traditional benchmarks. We advocate for task-based evaluation frameworks, as implemented here, to address these limitations and advance ML-guided materials discovery.