# A Crystal-Specific Pre-Training Framework for Crystal Material Property Prediction

arXiv: https://arxiv.org/abs/2306.05344 · HTML: https://mlsanity.com/paper/2306.05344/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 8 Jun 2023.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Equivariant Diffusion for Molecule Generation in 3D](https://mlsanity.com/paper/2203.17003/) (2022)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [GPT-GNN: Generative Pre-Training of Graph Neural Networks](https://mlsanity.com/paper/2006.15437/) (2020)
- [Graph Contrastive Learning Automated](https://mlsanity.com/paper/2106.07594/) (2021)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules](https://mlsanity.com/paper/2011.14115/) (2020)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [E(n) Equivariant Normalizing Flows](https://mlsanity.com/paper/2105.09016/) (2021)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [Self-supervised Graph-level Representation Learning with Local and Global Structure](https://mlsanity.com/paper/2106.04113/) (2021)
- [Periodic Graph Transformers for Crystal Material Property Prediction](https://mlsanity.com/paper/2209.11807/) (2022)
- [Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching](https://mlsanity.com/paper/2206.13602/) (2022)
- [Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction](https://mlsanity.com/paper/2205.01893/) (2022)
- [Implementing graph neural networks with TensorFlow-Keras](https://mlsanity.com/paper/2103.04318/) (2021)

## Built on by (most cited first, 3 of 3)

- [A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applications](https://mlsanity.com/paper/2403.00485/) (2024)
- [AI-driven inverse design of materials: Past, present and future](https://mlsanity.com/paper/2411.09429/) (2024)
- [Siamese Foundation Models for Crystal Structure Prediction](https://mlsanity.com/paper/2503.10471/) (2025)

## Abstract (arXiv)

Crystal property prediction is a crucial aspect of developing novel materials. However, there are two technical challenges to be addressed for speeding up the investigation of crystals. First, labeling crystal properties is intrinsically difficult due to the high cost and time involved in physical simulations or lab experiments. Second, crystals adhere to a specific quantum chemical principle known as periodic invariance, which is often not captured by existing machine learning methods. To overcome these challenges, we propose the crystal-specific pre-training framework for learning crystal representations with self-supervision. The framework designs a mutex mask strategy for enhancing representation learning so as to alleviate the limited labels available for crystal property prediction. Moreover, we take into account the specific periodic invariance in crystal structures by developing a periodic invariance multi-graph module and periodic attribute learning within our framework. This framework has been tested on eight different tasks. The experimental results on these tasks show that the framework achieves promising prediction performance and is able to outperform recent strong baselines.