# Chem-GMNet: A Sphere-Native Geometric Transformer for Molecular Property Prediction

arXiv: https://arxiv.org/abs/2605.13262 · HTML: https://mlsanity.com/paper/2605.13262/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 28 of 28)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [SchNet: A continuous-filter convolutional neural network for modeling quantum interactions](https://mlsanity.com/paper/1706.08566/) (2017)
- [Directional Message Passing for Molecular Graphs](https://mlsanity.com/paper/2003.03123/) (2020)
- [Gated Delta Networks: Improving Mamba2 with Delta Rule](https://mlsanity.com/paper/2412.06464/) (2024)
- [Gated Linear Attention Transformers with Hardware-Efficient Training](https://mlsanity.com/paper/2312.06635/) (2023)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields](https://mlsanity.com/paper/2206.07697/) (2022)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [Analyzing Learned Molecular Representations for Property Prediction](https://mlsanity.com/paper/1904.01561/) (2019)
- [ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction](https://mlsanity.com/paper/2010.09885/) (2020)
- [Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development](https://mlsanity.com/paper/2102.09548/) (2021)
- [Large-Scale Chemical Language Representations Capture Molecular Structure and Properties](https://mlsanity.com/paper/2106.09553/) (2021)
- [EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations](https://mlsanity.com/paper/2306.12059/) (2023)
- [ChemBERTa-2: Towards Chemical Foundation Models](https://mlsanity.com/paper/2209.01712/) (2022)
- [Molecular representation learning with language models and domain-relevant auxiliary tasks](https://mlsanity.com/paper/2011.13230/) (2020)
- [Molecule Attention Transformer](https://mlsanity.com/paper/2002.08264/) (2020)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [On the Scalability of GNNs for Molecular Graphs](https://mlsanity.com/paper/2404.11568/) (2024)
- [Evaluating Self-Supervised Learning for Molecular Graph Embeddings](https://mlsanity.com/paper/2206.08005/) (2022)
- [Attention on the Sphere](https://mlsanity.com/paper/2505.11157/) (2025)
- [Instructor-inspired Machine Learning for Robust Molecular Property Prediction](https://mlsanity.com/paper/2304.03906/) (2023)

## Abstract (arXiv)

Modern SMILES-based chemical language models obtain strong MoleculeNet performance by treating SMILES as generic text and compensating with multi-million-molecule self-supervised pretraining. We ask: when a domain carries structural priors as rich as chemistry's, does it warrant a domain-native transformer rather than a generic one rescued by scale? We answer affirmatively with GM-Net (Geometric Measure Network), a transformer family in which every module is replaced by a sphere-native counterpart, and instantiate it as Chem-GMNet. Three blocks follow: SH-Embedding (tokens as learnable directions on Sk-1 lifted through a Gegenbauer feature map); DualSKA (a per-head fusion of a linear-time gated Sphere-Flow recurrence whose persistent state we prove is the truncated multipole expansion of the input distribution, and a softmax Sphere-Kernel branch over the same Schoenberg-valid kernel); and SH-FFN (sphere projection → Gegenbauer lift → moment readout). On canonical DeepChem scaffold splits, against same-shape ChemBERTa-2 baselines under the chemberta3-faithful protocol: (i) random-initialised, Chem-GMNet wins on 7 of 10 MoleculeNet endpoints at ~!35% fewer parameters; (ii) pretrained on the same 10M-SMILES ZINC corpus as ChemBERTa-2 MLM-10M, it matches or beats the public release on 6 of 8 shared endpoints (5/7 excluding a known ClinTox release anomaly). A (k,L) ablation shows that increasing the sphere dimension from k!=!8 to k!=!10 at fixed L!=!3 lowers ESOL RMSE to 0.938 at scratch, beating pretrained ChemBERTa-2 MLM-10M on this endpoint without any pretraining at all.