# Crystalformer: Infinitely Connected Attention for Periodic Structure Encoding

arXiv: https://arxiv.org/abs/2403.11686 · HTML: https://mlsanity.com/paper/2403.11686/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Geometric Transformer for Fast and Robust Point Cloud Registration](https://mlsanity.com/paper/2202.06688/) (2022)
- [Graph neural networks for materials science and chemistry](https://mlsanity.com/paper/2208.09481/) (2022)
- [Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems](https://mlsanity.com/paper/2307.08423/) (2023)
- [ShellNet: Efficient Point Cloud Convolutional Neural Networks using Concentric Shells Statistics](https://mlsanity.com/paper/1908.06295/) (2019)
- [Periodic Graph Transformers for Crystal Material Property Prediction](https://mlsanity.com/paper/2209.11807/) (2022)
- [FAENet: Frame Averaging Equivariant GNN for Materials Modeling](https://mlsanity.com/paper/2305.05577/) (2023)
- [Rotation-Invariant Transformer for Point Cloud Matching](https://mlsanity.com/paper/2303.08231/) (2023)
- [Ewald-based Long-Range Message Passing for Molecular Graphs](https://mlsanity.com/paper/2303.04791/) (2023)
- [Rotation-Invariant Local-to-Global Representation Learning for 3D Point Cloud](https://mlsanity.com/paper/2010.03318/) (2020)
- [Efficient Approximations of Complete Interatomic Potentials for Crystal Property Prediction](https://mlsanity.com/paper/2306.10045/) (2023)
- [Transformers Generalize DeepSets and Can be Extended to Graphs and Hypergraphs](https://mlsanity.com/paper/2110.14416/) (2021)
- [Incompleteness of graph neural networks for points clouds in three dimensions](https://mlsanity.com/paper/2201.07136/) (2022)
- [Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materials](https://mlsanity.com/paper/2204.11953/) (2022)

## Built on by (most cited first, 15 of 15)

- [ELECTRA: A Cartesian Network for 3D Charge Density Prediction with Floating Orbitals](https://mlsanity.com/paper/2503.08305/) (2025)
- [Bohrium + SciMaster: Building the Infrastructure and Ecosystem for Agentic Science at Scale](https://mlsanity.com/paper/2512.20469/) (2025)
- [Generative Inverse Design of Crystal Structures via Diffusion Models with Transformers](https://mlsanity.com/paper/2406.09263/) (2024)
- [Bridging Text and Crystal Structures: Literature-driven Contrastive Learning for Materials Science](https://mlsanity.com/paper/2501.12919/) (2025)
- [Beyond Structure: Invariant Crystal Property Prediction with Pseudo-Particle Ray Diffraction](https://mlsanity.com/paper/2509.21778/) (2025)
- [ContinuouSP: Generative Model for Crystal Structure Prediction with Invariance and Continuity](https://mlsanity.com/paper/2502.02026/) (2025)
- [ReciNet: Reciprocal Space-Aware Long-Range Modeling for Crystalline Property Prediction](https://mlsanity.com/paper/2502.02748/) (2025)
- [PDDFormer: Pairwise Distance Distribution Graph Transformer for Crystal Material Property Prediction](https://mlsanity.com/paper/2408.12984/) (2024)
- [Adaptable Method for Crystal Design across Diverse Constraints and Objectives with Pretrained Property Predictors](https://mlsanity.com/paper/2410.08562/) (2024)
- [MoMa: A Modular Deep Learning Framework for Material Property Prediction](https://mlsanity.com/paper/2502.15483/) (2025)
- [RecCrysFormer: Refined Protein Structural Prediction from 3D Patterson Maps via Recycling Training Runs](https://mlsanity.com/paper/2503.00143/) (2025)
- [CrystalFramer: Rethinking the Role of Frames for SE(3)-Invariant Crystal Structure Modeling](https://mlsanity.com/paper/2503.02209/) (2025)
- [Synthesizability Prediction of Crystalline Structures with Structure-Aware Feature Learning and Uncertainty Quantification](https://mlsanity.com/paper/2510.19251/) (2025)
- [Offline Materials Optimization with CliqueFlowmer](https://mlsanity.com/paper/2603.06082/) (2026)
- [MatBind: A Shared Embedding Space for Multimodal Materials Characterization](https://mlsanity.com/paper/2607.08470/) (2026)

## Abstract (arXiv)

Predicting physical properties of materials from their crystal structures is a fundamental problem in materials science. In peripheral areas such as the prediction of molecular properties, fully connected attention networks have been shown to be successful. However, unlike these finite atom arrangements, crystal structures are infinitely repeating, periodic arrangements of atoms, whose fully connected attention results in infinitely connected attention. In this work, we show that this infinitely connected attention can lead to a computationally tractable formulation, interpreted as neural potential summation, that performs infinite interatomic potential summations in a deeply learned feature space. We then propose a simple yet effective Transformer-based encoder architecture for crystal structures called Crystalformer. Compared to an existing Transformer-based model, the proposed model requires only 29.4% of the number of parameters, with minimal modifications to the original Transformer architecture. Despite the architectural simplicity, the proposed method outperforms state-of-the-art methods for various property regression tasks on the Materials Project and JARVIS-DFT datasets.