# CrysGNN : Distilling pre-trained knowledge to enhance property prediction for crystalline materials

arXiv: https://arxiv.org/abs/2301.05852 · HTML: https://mlsanity.com/paper/2301.05852/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 14 Jan 2023.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [GPT-GNN: Generative Pre-Training of Graph Neural Networks](https://mlsanity.com/paper/2006.15437/) (2020)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning](https://mlsanity.com/paper/2003.12862/) (2020)
- [Selfie: Self-supervised Pretraining for Image Embedding](https://mlsanity.com/paper/1906.02940/) (2019)

## Built on by (most cited first, 14 of 14)

- [CrysMMNet: Multimodal Representation for Crystal Property Prediction](https://mlsanity.com/paper/2307.05390/) (2023)
- [Periodic Materials Generation using Text-Guided Joint Diffusion Model](https://mlsanity.com/paper/2503.00522/) (2025)
- [LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generation](https://mlsanity.com/paper/2510.23040/) (2025)
- [M2Hub: Unlocking the Potential of Machine Learning for Materials Discovery](https://mlsanity.com/paper/2307.05378/) (2023)
- [Siamese Foundation Models for Crystal Structure Prediction](https://mlsanity.com/paper/2503.10471/) (2025)
- [TopoMAS: Large Language Model Driven Topological Materials Multiagent System](https://mlsanity.com/paper/2507.04053/) (2025)
- [CHILI: Chemically-Informed Large-scale Inorganic Nanomaterials Dataset for Advancing Graph Machine Learning](https://mlsanity.com/paper/2402.13221/) (2024)
- [CrysAtom: Distributed Representation of Atoms for Crystal Property Prediction](https://mlsanity.com/paper/2409.04737/) (2024)
- [Enhancing Large Language Models with Domain-Specific Knowledge: The Case in Topological Materials](https://mlsanity.com/paper/2409.13732/) (2024)
- [Latent Diffusion Pretraining for Crystal Property Prediction](https://mlsanity.com/paper/2606.00776/) (2026)
- [PDDFormer: Pairwise Distance Distribution Graph Transformer for Crystal Material Property Prediction](https://mlsanity.com/paper/2408.12984/) (2024)
- [Benchmarking bandgap prediction in semiconductors under experimental and realistic evaluation settings](https://mlsanity.com/paper/2604.25568/) (2026)
- [UNATE: UNsupervised ATomic Embedding for crystal structures property prediction](https://mlsanity.com/paper/2605.25866/) (2026)
- [Model Agnostic Graph Prompt Learning for Crystal Property Prediction](https://mlsanity.com/paper/2607.08996/) (2026)

## Abstract (arXiv)

In recent years, graph neural network (GNN) based approaches have emerged as a powerful technique to encode complex topological structure of crystal materials in an enriched representation space. These models are often supervised in nature and using the property-specific training data, learn relationship between crystal structure and different properties like formation energy, bandgap, bulk modulus, etc. Most of these methods require a huge amount of property-tagged data to train the system which may not be available for different properties. However, there is an availability of a huge amount of crystal data with its chemical composition and structural bonds. To leverage these untapped data, this paper presents CrysGNN, a new pre-trained GNN framework for crystalline materials, which captures both node and graph level structural information of crystal graphs using a huge amount of unlabelled material data. Further, we extract distilled knowledge from CrysGNN and inject into different state of the art property predictors to enhance their property prediction accuracy. We conduct extensive experiments to show that with distilled knowledge from the pre-trained model, all the SOTA algorithms are able to outperform their own vanilla version with good margins. We also observe that the distillation process provides a significant improvement over the conventional approach of finetuning the pre-trained model. We have released the pre-trained model along with the large dataset of 800K crystal graph which we carefully curated; so that the pretrained model can be plugged into any existing and upcoming models to enhance their prediction accuracy.