# MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Prediction

arXiv: https://arxiv.org/abs/2210.14188 · HTML: https://mlsanity.com/paper/2210.14188/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 73 references that are in the map.
- First posted on arXiv 25 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction](https://mlsanity.com/paper/1811.02633/) (2018)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Mapping the Space of Chemical Reactions Using Attention-Based Neural Networks](https://mlsanity.com/paper/2012.06051/) (2020)
- [DScribe: Library of Descriptors for Machine Learning in Materials Science](https://mlsanity.com/paper/1904.08875/) (2019)
- ["Found in Translation": Predicting Outcomes of Complex Organic Chemistry Reactions using Neural Sequence-to-Sequence Models](https://mlsanity.com/paper/1711.04810/) (2017)
- [TransPolymer: a Transformer-based language model for polymer property predictions](https://mlsanity.com/paper/2209.01307/) (2022)
- [Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast](https://mlsanity.com/paper/2202.09346/) (2022)
- [Crystal Twins: Self-supervised Learning for Crystalline Material Property Prediction](https://mlsanity.com/paper/2205.01893/) (2022)
- [Using Machine Learning and Data Mining to Leverage Community Knowledge for the Engineering of Stable Metal-Organic Frameworks](https://mlsanity.com/paper/2106.13327/) (2021)

## Built on by (most cited first, 28 of 28)

- [Developing ChemDFM as a large language foundation model for chemistry](https://mlsanity.com/paper/2401.14818/) (2024)
- [SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training](https://mlsanity.com/paper/2310.02227/) (2023)
- [From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction](https://mlsanity.com/paper/2310.16802/) (2023)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [An Interdisciplinary Outlook on Large Language Models for Scientific Research](https://mlsanity.com/paper/2311.04929/) (2023)
- [GPT-MolBERTa: GPT Molecular Features Language Model for molecular property prediction](https://mlsanity.com/paper/2310.03030/) (2023)
- [Strategies for Pretraining Neural Operators](https://mlsanity.com/paper/2406.08473/) (2024)
- [Denoise Pretraining on Nonequilibrium Molecules for Accurate and Transferable Neural Potentials](https://mlsanity.com/paper/2303.02216/) (2023)
- [MOFGPT: Generative Design of Metal-Organic Frameworks using Language Models](https://mlsanity.com/paper/2506.00198/) (2025)
- [34 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discovery](https://mlsanity.com/paper/2505.03049/) (2025)
- [Artificial Intelligence and Generative Models for Materials Discovery -- A Review](https://mlsanity.com/paper/2508.03278/) (2025)
- [Materials Informatics Transformer: A Language Model for Interpretable Materials Properties Prediction](https://mlsanity.com/paper/2308.16259/) (2023)
- [PolyCL: Contrastive Learning for Polymer Representation Learning via Explicit and Implicit Augmentations](https://mlsanity.com/paper/2408.07556/) (2024)
- [The Rise of Generative AI for Metal-Organic Framework Design and Synthesis](https://mlsanity.com/paper/2508.13197/) (2025)
- [Trillion Parameter AI Serving Infrastructure for Scientific Discovery: A Survey and Vision](https://mlsanity.com/paper/2402.03480/) (2024)
- [AlloyBERT: Alloy Property Prediction with Large Language Models](https://mlsanity.com/paper/2403.19783/) (2024)
- [Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie](https://mlsanity.com/paper/2601.12554/) (2026)
- [Building-Block Aware Generative Modeling for 3D Crystals of Metal Organic Frameworks](https://mlsanity.com/paper/2505.08531/) (2025)
- [Polymer-Agent: Large Language Model Agent for Polymer Design](https://mlsanity.com/paper/2601.16376/) (2026)
- [Machine Learning Based Prediction of Proton Conductivity in Metal-Organic Frameworks](https://mlsanity.com/paper/2407.09514/) (2024)
- [From Natural Language to Materials Discovery:The Materials Knowledge Navigation Agent](https://mlsanity.com/paper/2602.11123/) (2026)
- [BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models](https://mlsanity.com/paper/2605.08110/) (2026)
- [Accelerating Material Property Prediction using Generically Complete Isometry Invariants](https://mlsanity.com/paper/2401.15089/) (2024)
- [Pretraining Strategy for Neural Potentials](https://mlsanity.com/paper/2402.15921/) (2024)
- [Symmetry-Informed Graph Neural Networks for Carbon Dioxide Isotherm and Adsorption Prediction in Aluminum-Substituted Zeolites](https://mlsanity.com/paper/2503.22737/) (2025)
- [Multimodal Transformer for Sample-Aware Prediction of Metal-Organic Framework Properties](https://mlsanity.com/paper/2604.19383/) (2026)
- [Large language model agents accelerate inverse design of metal-organic frameworks for gas separation](https://mlsanity.com/paper/2607.10559/) (2026)
- [Atoms to Processes: The Role of Artificial Intelligence and Machine Learning in Chemical Engineering](https://mlsanity.com/paper/2610.02014/) (2026)

## Abstract (arXiv)

Metal-Organic Frameworks (MOFs) are materials with a high degree of porosity that can be used for applications in energy storage, water desalination, gas storage, and gas separation. However, the chemical space of MOFs is close to an infinite size due to the large variety of possible combinations of building blocks and topology. Discovering the optimal MOFs for specific applications requires an efficient and accurate search over an enormous number of potential candidates. Previous high-throughput screening methods using computational simulations like DFT can be time-consuming. Such methods also require optimizing 3D atomic structure of MOFs, which adds one extra step when evaluating hypothetical MOFs. In this work, we propose a structure-agnostic deep learning method based on the Transformer model, named as MOFormer, for property predictions of MOFs. The MOFormer takes a text string representation of MOF (MOFid) as input, thus circumventing the need of obtaining the 3D structure of hypothetical MOF and accelerating the screening process. Furthermore, we introduce a self-supervised learning framework that pretrains the MOFormer via maximizing the cross-correlation between its structure-agnostic representations and structure-based representations of crystal graph convolutional neural network (CGCNN) on >400k publicly available MOF data. Using self-supervised learning allows the MOFormer to intrinsically learn 3D structural information though it is not included in the input. Experiments show that pretraining improved the prediction accuracy of both models on various downstream prediction tasks. Furthermore, we revealed that MOFormer can be more data-efficient on quantum-chemical property prediction than structure-based CGCNN when training data is limited. Overall, MOFormer provides a novel perspective on efficient MOF design using deep learning.