# Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs

arXiv: https://arxiv.org/abs/2110.01191 · HTML: https://mlsanity.com/paper/2110.01191/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 125 references that are in the map.
- First posted on arXiv 4 Oct 2021.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Convolutional Networks on Graphs for Learning Molecular Fingerprints](https://mlsanity.com/paper/1509.09292/) (2015)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [Equivariant message passing for the prediction of tensorial properties and molecular spectra](https://mlsanity.com/paper/2102.03150/) (2021)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [Cormorant: Covariant Molecular Neural Networks](https://mlsanity.com/paper/1906.04015/) (2019)
- [Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules](https://mlsanity.com/paper/2011.14115/) (2020)
- [GEOM: Energy-annotated molecular conformations for property prediction and molecular generation](https://mlsanity.com/paper/2006.05531/) (2020)
- [Molecular Graph Convolutions: Moving Beyond Fingerprints](https://mlsanity.com/paper/1603.00856/) (2016)
- [Order Matters: Sequence to sequence for sets](https://mlsanity.com/paper/1511.06391/) (2015)
- [Hierarchical Generation of Molecular Graphs using Structural Motifs](https://mlsanity.com/paper/2002.03230/) (2020)
- [3D Object Detection with Pointformer](https://mlsanity.com/paper/2012.11409/) (2020)
- [DeepDTA: Deep Drug-Target Binding Affinity Prediction](https://mlsanity.com/paper/1801.10193/) (2018)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [Motif-based Graph Self-Supervised Learning for Molecular Property Prediction](https://mlsanity.com/paper/2110.00987/) (2021)

## Built on by (most cited first, 30 of 44)

- [Developing ChemDFM as a large language foundation model for chemistry](https://mlsanity.com/paper/2401.14818/) (2024)
- [Harnessing the Power of Neural Operators with Automatically Encoded Conservation Laws](https://mlsanity.com/paper/2312.11176/) (2023)
- [Relative Molecule Self-Attention Transformer](https://mlsanity.com/paper/2110.05841/) (2021)
- [Integration of Pre-trained Protein Language Models into Geometric Deep Learning Networks](https://mlsanity.com/paper/2212.03447/) (2022)
- [From Words to Molecules: A Survey of Large Language Models in Chemistry](https://mlsanity.com/paper/2402.01439/) (2024)
- [Transformers and Large Language Models for Chemistry and Drug Discovery](https://mlsanity.com/paper/2310.06083/) (2023)
- [A Hierarchical Training Paradigm for Antibody Structure-sequence Co-design](https://mlsanity.com/paper/2311.16126/) (2023)
- [Neural Atoms: Propagating Long-range Interaction in Molecular Graphs through Efficient Communication Channel](https://mlsanity.com/paper/2311.01276/) (2023)
- [Rethinking Explaining Graph Neural Networks via Non-parametric Subgraph Matching](https://mlsanity.com/paper/2301.02780/) (2023)
- [Instructor-inspired Machine Learning for Robust Molecular Property Prediction](https://mlsanity.com/paper/2304.03906/) (2023)
- [Revisiting Self-Supervised Heterogeneous Graph Learning from Spectral Clustering Perspective](https://mlsanity.com/paper/2412.00742/) (2024)
- [PoseX: AI Defeats Physics Approaches on Protein-Ligand Cross Docking](https://mlsanity.com/paper/2505.01700/) (2025)
- [Impact of Domain Knowledge and Multi-Modality on Intelligent Molecular Property Prediction: A Systematic Survey](https://mlsanity.com/paper/2402.07249/) (2024)
- [Towards DNA-Encoded Library Generation with GFlowNets](https://mlsanity.com/paper/2404.10094/) (2024)
- [Molecular Representations in Implicit Functional Space via Hyper-Networks](https://mlsanity.com/paper/2601.22327/) (2026)
- [Atom-Motif Contrastive Transformer for Molecular Property Prediction](https://mlsanity.com/paper/2310.07351/) (2023)
- [Question Rephrasing for Quantifying Uncertainty in Large Language Models: Applications in Molecular Chemistry Tasks](https://mlsanity.com/paper/2408.03732/) (2024)
- [A Semi-supervised Molecular Learning Framework for Activity Cliff Estimation](https://mlsanity.com/paper/2601.04507/) (2026)
- [Dynamics-inspired Structure Hallucination for Protein-protein Interaction Modeling](https://mlsanity.com/paper/2601.06214/) (2026)
- [Reducing Down(stream)time: Pretraining Molecular GNNs using Heterogeneous AI Accelerators](https://mlsanity.com/paper/2211.04598/) (2022)
- [Large language models, physics-based modeling, experimental measurements: the trinity of data-scarce learning of polymer properties](https://mlsanity.com/paper/2407.02770/) (2024)
- [FARM: Enhancing Molecular Representations with Functional Group Awareness](https://mlsanity.com/paper/2410.02082/) (2024)
- [MV-CLAM: Multi-View Molecular Interpretation with Cross-Modal Projection via Language Model](https://mlsanity.com/paper/2503.04780/) (2025)
- [Large Language Models Transform Organic Synthesis From Reaction Prediction to Automation](https://mlsanity.com/paper/2508.05427/) (2025)
- [ZapGPT: Free-form Language Prompting for Simulated Cellular Control](https://mlsanity.com/paper/2509.10660/) (2025)
- [Entropy-Guided Dynamic Tokens for Graph-LLM Alignment in Molecular Understanding](https://mlsanity.com/paper/2602.02742/) (2026)
- [Atomic and Subgraph-aware Bilateral Aggregation for Molecular Representation Learning](https://mlsanity.com/paper/2305.12618/) (2023)
- [CTAGE: Curvature-Based Topology-Aware Graph Embedding for Learning Molecular Representations](https://mlsanity.com/paper/2307.13275/) (2023)
- [Hierarchical Grammar-Induced Geometry for Data-Efficient Molecular Property Prediction](https://mlsanity.com/paper/2309.01788/) (2023)
- [Molecular Graph Representation Learning via Structural Similarity Information](https://mlsanity.com/paper/2409.08580/) (2024)

## Abstract (arXiv)

Procuring expressive molecular representations underpins AI-driven molecule design and scientific discovery. The research mainly focuses on atom-level homogeneous molecular graphs, ignoring the rich information in subgraphs or motifs. However, it has been widely accepted that substructures play a dominant role in identifying and determining molecular properties. To address such issues, we formulate heterogeneous molecular graphs (HMGs), and introduce a novel architecture to exploit both molecular motifs and 3D geometry. Precisely, we extract functional groups as motifs for small molecules and employ reinforcement learning to adaptively select quaternary amino acids as motif candidates for proteins. Then HMGs are constructed with both atom-level and motif-level nodes. To better accommodate those HMGs, we introduce a variant of Transformer named Molformer, which adopts a heterogeneous self-attention layer to distinguish the interactions between multi-level nodes. Besides, it is also coupled with a multi-scale mechanism to capture fine-grained local patterns with increasing contextual scales. An attentive farthest point sampling algorithm is also proposed to obtain the molecular representations. We validate Molformer across a broad range of domains, including quantum chemistry, physiology, and biophysics. Extensive experiments show that Molformer outperforms or achieves the comparable performance of several state-of-the-art baselines. Our work provides a promising way to utilize informative motifs from the perspective of multi-level graph construction.