# Scalable deeper graph neural networks for high-performance materials property prediction

arXiv: https://arxiv.org/abs/2109.12283 · HTML: https://mlsanity.com/paper/2109.12283/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 76 references that are in the map.
- First posted on arXiv 25 Sep 2021.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Representation Learning on Graphs with Jumping Knowledge Networks](https://mlsanity.com/paper/1806.03536/) (2018)
- [Simple and Deep Graph Convolutional Networks](https://mlsanity.com/paper/2007.02133/) (2020)
- [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019)
- [DropEdge: Towards Deep Graph Convolutional Networks on Node Classification](https://mlsanity.com/paper/1907.10903/) (2019)
- [Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View](https://mlsanity.com/paper/1909.03211/) (2019)
- [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019)
- [PairNorm: Tackling Oversmoothing in GNNs](https://mlsanity.com/paper/1909.12223/) (2019)
- [DeeperGCN: All You Need to Train Deeper GCNs](https://mlsanity.com/paper/2006.07739/) (2020)
- [A Note on Over-Smoothing for Graph Neural Networks](https://mlsanity.com/paper/2006.13318/) (2020)
- [Training Graph Neural Networks with 1000 Layers](https://mlsanity.com/paper/2106.07476/) (2021)
- [GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs](https://mlsanity.com/paper/1803.07294/) (2018)
- [Towards Deeper Graph Neural Networks with Differentiable Group Normalization](https://mlsanity.com/paper/2006.06972/) (2020)
- [Syntax-Directed Variational Autoencoder for Structured Data](https://mlsanity.com/paper/1802.08786/) (2018)
- [DeepGCNs: Making GCNs Go as Deep as CNNs](https://mlsanity.com/paper/1910.06849/) (2019)
- [GeniePath: Graph Neural Networks with Adaptive Receptive Paths](https://mlsanity.com/paper/1802.00910/) (2018)
- [Break the Ceiling: Stronger Multi-scale Deep Graph Convolutional Networks](https://mlsanity.com/paper/1906.02174/) (2019)

## Built on by (most cited first, 26 of 26)

- [Efficient Approximations of Complete Interatomic Potentials for Crystal Property Prediction](https://mlsanity.com/paper/2306.10045/) (2023)
- [LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction](https://mlsanity.com/paper/2411.00177/) (2024)
- [Structure-based out-of-distribution (OOD) materials property prediction: a benchmark study](https://mlsanity.com/paper/2401.08032/) (2024)
- [Materials Property Prediction with Uncertainty Quantification: A Benchmark Study](https://mlsanity.com/paper/2211.02235/) (2022)
- [DenseGNN: universal and scalable deeper graph neural networks for high-performance property prediction in crystals and molecules](https://mlsanity.com/paper/2501.03278/) (2025)
- [Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materials](https://mlsanity.com/paper/2204.11953/) (2022)
- [Formula graph self-attention network for representation-domain independent materials discovery](https://mlsanity.com/paper/2201.05649/) (2022)
- [The Vendiscope: An Algorithmic Microscope For Data Collections](https://mlsanity.com/paper/2502.10828/) (2025)
- [Crystal structure prediction using neural network potential and age-fitness Pareto genetic algorithm](https://mlsanity.com/paper/2309.06710/) (2023)
- [Material Property Prediction using Graphs based on Generically Complete Isometry Invariants](https://mlsanity.com/paper/2212.11246/) (2022)
- [Polymorphism Crystal Structure Prediction with Adaptive Space Group Diversity Control](https://mlsanity.com/paper/2506.11332/) (2025)
- [Semi-supervised teacher-student deep neural network for materials discovery](https://mlsanity.com/paper/2112.06142/) (2021)
- [Materials Transformers Language Models for Generative Materials Design: a benchmark study](https://mlsanity.com/paper/2206.13578/) (2022)
- [Artificial Intelligence in Material Engineering: A review on applications of AI in Material Engineering](https://mlsanity.com/paper/2209.11234/) (2022)
- [Topological, or Non-topological? A Deep Learning Based Prediction](https://mlsanity.com/paper/2310.18907/) (2023)
- [Discovery of sustainable energy materials via the machine-learned material space](https://mlsanity.com/paper/2501.05903/) (2025)
- [Benchmarking GNNs for OOD Materials Property Prediction with Uncertainty Quantification](https://mlsanity.com/paper/2511.11697/) (2025)
- [Substitutional Alloying Using Crystal Graph Neural Networks](https://mlsanity.com/paper/2306.10766/) (2023)
- [MD-HIT: Machine learning for materials property prediction with dataset redundancy control](https://mlsanity.com/paper/2307.04351/) (2023)
- [Accelerating Material Property Prediction using Generically Complete Isometry Invariants](https://mlsanity.com/paper/2401.15089/) (2024)
- [AlphaCrystal-II: Distance matrix based crystal structure prediction using deep learning](https://mlsanity.com/paper/2404.04810/) (2024)
- [A Materials Map Integrating Experimental and Computational Data via Graph-Based Machine Learning for Enhanced Materials Discovery](https://mlsanity.com/paper/2503.07378/) (2025)
- [Facet: highly efficient E(3)-equivariant networks for interatomic potentials](https://mlsanity.com/paper/2509.08418/) (2025)
- [Element2Vec: Build Chemical Element Representation from Text for Property Prediction](https://mlsanity.com/paper/2510.13916/) (2025)
- [Optimizing Expert-Designed Crystal Graph Networks for Band-Gap Prediction with an Autonomous LLM Research Loop](https://mlsanity.com/paper/2606.29717/) (2026)
- [AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking](https://mlsanity.com/paper/2607.11526/) (2026)

## Abstract (arXiv)

Machine learning (ML) based materials discovery has emerged as one of the most promising approaches for breakthroughs in materials science. While heuristic knowledge based descriptors have been combined with ML algorithms to achieve good performance, the complexity of the physicochemical mechanisms makes it urgently needed to exploit representation learning from either compositions or structures for building highly effective materials machine learning models. Among these methods, the graph neural networks have shown the best performance by its capability to learn high-level features from crystal structures. However, all these models suffer from their inability to scale up the models due to the over-smoothing issue of their message-passing GNN architecture. Here we propose a novel graph attention neural network model DeeperGATGNN with differentiable group normalization and skip-connections, which allows to train very deep graph neural network models (e.g. 30 layers compared to 3-9 layers in previous works). Through systematic benchmark studies over six benchmark datasets for energy and band gap predictions, we show that our scalable DeeperGATGNN model needs little costly hyper-parameter tuning for different datasets and achieves the state-of-the-art prediction performances over five properties out of six with up to 10% improvement. Our work shows that to deal with the high complexity of mapping the crystal materials structures to their properties, large-scale very deep graph neural networks are needed to achieve robust performances.