# Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction

arXiv: https://arxiv.org/abs/2205.01893 · HTML: https://mlsanity.com/paper/2205.01893/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 72 references that are in the map.
- First posted on arXiv 4 May 2022.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules](https://mlsanity.com/paper/2011.14115/) (2020)
- [Motif-based Graph Self-Supervised Learning for Molecular Property Prediction](https://mlsanity.com/paper/2110.00987/) (2021)
- [3D Infomax improves GNNs for Molecular Property Prediction](https://mlsanity.com/paper/2110.04126/) (2021)
- [N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules](https://mlsanity.com/paper/1806.09206/) (2018)
- [Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast](https://mlsanity.com/paper/2202.09346/) (2022)
- [Orbital Graph Convolutional Neural Network for Material Property Prediction](https://mlsanity.com/paper/2008.06415/) (2020)
- [Dual-view Molecule Pre-training](https://mlsanity.com/paper/2106.10234/) (2021)
- [Self-Supervised Graph Transformer on Large-Scale Molecular Data](https://mlsanity.com/paper/2007.02835/) (2020)
- [Self-Supervised Dialogue Learning](https://mlsanity.com/paper/1907.00448/) (2019)
- [AugLiChem: Data Augmentation Library of Chemical Structures for Machine Learning](https://mlsanity.com/paper/2111.15112/) (2021)
- [CRYSPNet: Crystal Structure Predictions via Neural Network](https://mlsanity.com/paper/2003.14328/) (2020)
- [Formula graph self-attention network for representation-domain independent materials discovery](https://mlsanity.com/paper/2201.05649/) (2022)

## Built on by (most cited first, 30 of 34)

- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Prediction](https://mlsanity.com/paper/2210.14188/) (2022)
- [A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applications](https://mlsanity.com/paper/2403.00485/) (2024)
- [Graph Neural Networks for Molecules](https://mlsanity.com/paper/2209.05582/) (2022)
- [GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction](https://mlsanity.com/paper/2310.03030/) (2023)
- [Capturing long-range interaction with reciprocal space neural network](https://mlsanity.com/paper/2211.16684/) (2022)
- [Denoise Pretraining on Nonequilibrium Molecules for Accurate and Transferable Neural Potentials](https://mlsanity.com/paper/2303.02216/) (2023)
- [Accelerating Molecular Graph Neural Networks via Knowledge Distillation](https://mlsanity.com/paper/2306.14818/) (2023)
- [MOFGPT: Generative Design of Metal-Organic Frameworks using Language Models](https://mlsanity.com/paper/2506.00198/) (2025)
- [AI-driven inverse design of materials: Past, present and future](https://mlsanity.com/paper/2411.09429/) (2024)
- [Siamese Foundation Models for Crystal Structure Prediction](https://mlsanity.com/paper/2503.10471/) (2025)
- [Materials Informatics Transformer: A Language Model for Interpretable Materials Properties Prediction](https://mlsanity.com/paper/2308.16259/) (2023)
- [AlloyBERT: Alloy Property Prediction with Large Language Models](https://mlsanity.com/paper/2403.19783/) (2024)
- [Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie](https://mlsanity.com/paper/2601.12554/) (2026)
- [Self-Conditioned Denoising for Atomistic Representation Learning](https://mlsanity.com/paper/2603.17196/) (2026)
- [A Crystal-Specific Pre-Training Framework for Crystal Material Property Prediction](https://mlsanity.com/paper/2306.05344/) (2023)
- [Self-supervised learning for crystal property prediction via denoising](https://mlsanity.com/paper/2408.17255/) (2024)
- [CrysAtom: Distributed Representation of Atoms for Crystal Property Prediction](https://mlsanity.com/paper/2409.04737/) (2024)
- [CLOUD: A Scalable and Physics-Informed Foundation Model for Crystal Representation Learning](https://mlsanity.com/paper/2506.17345/) (2025)
- [Leveraging neural network interatomic potentials for a foundation model of chemistry](https://mlsanity.com/paper/2506.18497/) (2025)
- [Enhancing material property prediction with ensemble deep graph convolutional networks](https://mlsanity.com/paper/2407.18847/) (2024)
- [Beyond Force Metrics: Pre-Training MLFFs for Stable MD Simulations](https://mlsanity.com/paper/2506.14850/) (2025)
- [Predicting CO2 Absorption in Ionic Liquids with Molecular Descriptors and Explainable Graph Neural Networks](https://mlsanity.com/paper/2210.01120/) (2022)
- [Contrastive Learning of English Language and Crystal Graphs for Multimodal Representation of Materials Knowledge](https://mlsanity.com/paper/2502.16451/) (2025)
- [Supervised Pretraining for Material Property Prediction](https://mlsanity.com/paper/2504.20112/) (2025)
- [A New Workflow for Materials Discovery Bridging the Gap Between Experimental Databases and Graph Neural Networks](https://mlsanity.com/paper/2602.00756/) (2026)
- [From Natural Language to Materials Discovery:The Materials Knowledge Navigation Agent](https://mlsanity.com/paper/2602.11123/) (2026)
- [TriForces: Augmenting Atomistic GNNs for Transferable Representations](https://mlsanity.com/paper/2605.20581/) (2026)
- [Accelerating Material Property Prediction using Generically Complete Isometry Invariants](https://mlsanity.com/paper/2401.15089/) (2024)
- [Pretraining Strategy for Neural Potentials](https://mlsanity.com/paper/2402.15921/) (2024)

## Abstract (arXiv)

Machine learning (ML) models have been widely successful in the prediction of material properties. However, large labeled datasets required for training accurate ML models are elusive and computationally expensive to generate. Recent advances in Self-Supervised Learning (SSL) frameworks capable of training ML models on unlabeled data have mitigated this problem and demonstrated superior performance in computer vision and natural language processing tasks. Drawing inspiration from the developments in SSL, we introduce Crystal Twins (CT): an SSL method for crystalline materials property prediction. Using a large unlabeled dataset, we pre-train a Graph Neural Network (GNN) by applying the redundancy reduction principle to the graph latent embeddings of augmented instances obtained from the same crystalline system. By sharing the pre-trained weights when fine-tuning the GNN for regression tasks, we significantly improve the performance for 7 challenging material property prediction benchmarks