# Periodic Graph Transformers for Crystal Material Property Prediction

arXiv: https://arxiv.org/abs/2209.11807 · HTML: https://mlsanity.com/paper/2209.11807/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 56 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [Graph U-Nets](https://mlsanity.com/paper/1905.05178/) (2019)
- [GemNet: Universal Directional Graph Neural Networks for Molecules](https://mlsanity.com/paper/2106.08903/) (2021)
- [Towards Deeper Graph Neural Networks](https://mlsanity.com/paper/2007.09296/) (2020)
- [Crystal Diffusion Variational Autoencoder for Periodic Material Generation](https://mlsanity.com/paper/2110.06197/) (2021)
- [Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules](https://mlsanity.com/paper/2011.14115/) (2020)
- [The Open Catalyst 2020 (OC20) Dataset and Community Challenges](https://mlsanity.com/paper/2010.09990/) (2020)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [Universal Invariant and Equivariant Graph Neural Networks](https://mlsanity.com/paper/1905.04943/) (2019)
- [GraphDF: A Discrete Flow Model for Molecular Graph Generation](https://mlsanity.com/paper/2102.01189/) (2021)
- [e3nn: Euclidean Neural Networks](https://mlsanity.com/paper/2207.09453/) (2022)
- [Spherical Message Passing for 3D Graph Networks](https://mlsanity.com/paper/2102.05013/) (2021)
- [E(n) Equivariant Normalizing Flows](https://mlsanity.com/paper/2105.09016/) (2021)
- [GraphEBM: Molecular Graph Generation with Energy-Based Models](https://mlsanity.com/paper/2102.00546/) (2021)
- [DIG: A Turnkey Library for Diving into Graph Deep Learning Research](https://mlsanity.com/paper/2103.12608/) (2021)
- [ComENet: Towards Complete and Efficient Message Passing for 3D Molecular Graphs](https://mlsanity.com/paper/2206.08515/) (2022)
- [Predicting materials properties without crystal structure: Deep representation learning from stoichiometry](https://mlsanity.com/paper/1910.00617/) (2019)
- [Advanced Graph and Sequence Neural Networks for Molecular Property Prediction and Drug Discovery](https://mlsanity.com/paper/2012.01981/) (2020)
- [Topology-Aware Graph Pooling Networks](https://mlsanity.com/paper/2010.09834/) (2020)
- [Deep Learning of High-Order Interactions for Protein Interface Prediction](https://mlsanity.com/paper/2007.09334/) (2020)
- [Deep Generative Model for Periodic Graphs](https://mlsanity.com/paper/2201.11932/) (2022)
- [Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent](https://mlsanity.com/paper/2106.07832/) (2021)
- [IRNet: A General Purpose Deep Residual Regression Framework for Materials Discovery](https://mlsanity.com/paper/1907.03222/) (2019)

## Built on by (most cited first, 30 of 87)

- [Crystal Structure Prediction by Joint Equivariant Diffusion](https://mlsanity.com/paper/2309.04475/) (2023)
- [Attending to Graph Transformers](https://mlsanity.com/paper/2302.04181/) (2023)
- [A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems](https://mlsanity.com/paper/2312.07511/) (2023)
- [SymmCD: Symmetry-Preserving Crystal Generation with Diffusion Models](https://mlsanity.com/paper/2502.03638/) (2025)
- [Space Group Informed Transformer for Crystalline Materials Generation](https://mlsanity.com/paper/2403.15734/) (2024)
- [Efficient and Equivariant Graph Networks for Predicting Quantum Hamiltonian](https://mlsanity.com/paper/2306.04922/) (2023)
- [Efficient Approximations of Complete Interatomic Potentials for Crystal Property Prediction](https://mlsanity.com/paper/2306.10045/) (2023)
- [Complete and Efficient Graph Transformers for Crystal Material Property Prediction](https://mlsanity.com/paper/2403.11857/) (2024)
- [LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptions](https://mlsanity.com/paper/2310.14029/) (2023)
- [A Latent Diffusion Model for Protein Structure Generation](https://mlsanity.com/paper/2305.04120/) (2023)
- [A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applications](https://mlsanity.com/paper/2403.00485/) (2024)
- [QH9: A Quantum Hamiltonian Prediction Benchmark for QM9 Molecules](https://mlsanity.com/paper/2306.09549/) (2023)
- [Towards Symmetry-Aware Generation of Periodic Materials](https://mlsanity.com/paper/2307.02707/) (2023)
- [Multimodal Foundation Models for Material Property Prediction and Discovery](https://mlsanity.com/paper/2312.00111/) (2023)
- [Equivariance via Minimal Frame Averaging for More Symmetries and Efficiency](https://mlsanity.com/paper/2406.07598/) (2024)
- [CrysMMNet: Multimodal Representation for Crystal Property Prediction](https://mlsanity.com/paper/2307.05390/) (2023)
- [Structure-based out-of-distribution (OOD) materials property prediction: a benchmark study](https://mlsanity.com/paper/2401.08032/) (2024)
- [A Periodic Bayesian Flow for Material Generation](https://mlsanity.com/paper/2502.02016/) (2025)
- [Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding](https://mlsanity.com/paper/2403.11686/) (2024)
- [Periodic Materials Generation using Text-Guided Joint Diffusion Model](https://mlsanity.com/paper/2503.00522/) (2025)
- [A Survey of Graph Transformers: Architectures, Theories and Applications](https://mlsanity.com/paper/2502.16533/) (2025)
- [LLM Meets Diffusion: A Hybrid Framework for Crystal Material Generation](https://mlsanity.com/paper/2510.23040/) (2025)
- [Invariant Tokenization of Crystalline Materials for Language Model Enabled Generation](https://mlsanity.com/paper/2503.00152/) (2025)
- [Connectivity Optimized Nested Graph Networks for Crystal Structures](https://mlsanity.com/paper/2302.14102/) (2023)
- [AI-driven inverse design of materials: Past, present and future](https://mlsanity.com/paper/2411.09429/) (2024)
- [Toward Greater Autonomy in Materials Discovery Agents: Unifying Planning, Physics, and Scientists](https://mlsanity.com/paper/2506.05616/) (2025)
- [Vector Field Oriented Diffusion Model for Crystal Material Generation](https://mlsanity.com/paper/2401.05402/) (2023)
- [Equivariant Diffusion for Crystal Structure Prediction](https://mlsanity.com/paper/2512.07289/) (2025)
- [M2Hub: Unlocking the Potential of Machine Learning for Materials Discovery](https://mlsanity.com/paper/2307.05378/) (2023)
- [A Space Group Symmetry Informed Network for O(3) Equivariant Crystal Tensor Prediction](https://mlsanity.com/paper/2406.12888/) (2024)

## Abstract (arXiv)

We consider representation learning on periodic graphs encoding crystal materials. Different from regular graphs, periodic graphs consist of a minimum unit cell repeating itself on a regular lattice in 3D space. How to effectively encode these periodic structures poses unique challenges not present in regular graph representation learning. In addition to being E(3) invariant, periodic graph representations need to be periodic invariant. That is, the learned representations should be invariant to shifts of cell boundaries as they are artificially imposed. Furthermore, the periodic repeating patterns need to be captured explicitly as lattices of different sizes and orientations may correspond to different materials. In this work, we propose a transformer architecture, known as Matformer, for periodic graph representation learning. Our Matformer is designed to be invariant to periodicity and can capture repeating patterns explicitly. In particular, Matformer encodes periodic patterns by efficient use of geometric distances between the same atoms in neighboring cells. Experimental results on multiple common benchmark datasets show that our Matformer outperforms baseline methods consistently. In addition, our results demonstrate the importance of periodic invariance and explicit repeating pattern encoding for crystal representation learning.