mlsanityData to 7 Oct 2026

Chem-GMNet: A Sphere-Native Geometric Transformer for Molecular Property Prediction

arXiv 2605.13262 ↗First posted 13 May 2026cs.LG q-bio.QMcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern SMILES-based chemical language models obtain strong MoleculeNet performance by treating SMILES as generic text and compensating with multi-million-molecule self-supervised pretraining. We ask: when a domain carries structural priors as rich as chemistry's, does it warrant a domain-native transformer rather than a generic one rescued by scale? We answer affirmatively with GM-Net (Geometric Measure Network), a transformer family in which every module is replaced by a sphere-native counterpart, and instantiate it as Chem-GMNet. Three blocks follow: SH-Embedding (tokens as learnable directions on Sk-1 lifted through a Gegenbauer feature map); DualSKA (a per-head fusion of a linear-time gated Sphere-Flow recurrence whose persistent state we prove is the truncated multipole expansion of the input distribution, and a softmax Sphere-Kernel branch over the same Schoenberg-valid kernel); and SH-FFN (sphere projection → Gegenbauer lift → moment readout). On canonical DeepChem scaffold splits, against same-shape ChemBERTa-2 baselines under the chemberta3-faithful protocol: (i) random-initialised, Chem-GMNet wins on 7 of 10 MoleculeNet endpoints at ~!35% fewer parameters; (ii) pretrained on the same 10M-SMILES ZINC corpus as ChemBERTa-2 MLM-10M, it matches or beats the public release on 6 of 8 shared endpoints (5/7 excluding a known ClinTox release anomaly). A (k,L) ablation shows that increasing the sphere dimension from k!=!8 to k!=!10 at fixed L!=!3 lowers ESOL RMSE to 0.938 at scratch, beating pretrained ChemBERTa-2 MLM-10M on this endpoint without any pretraining at all.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

28 of 40 references.

References · 28

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  5. 2018Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point cloudscited by 735
  6. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 701
  7. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  9. 2017SchNet: A continuous-filter convolutional neural network for modeling quantum interactionscited by 569
  10. 2020Directional Message Passing for Molecular Graphscited by 556
  11. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  12. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
16 more
  1. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  2. 2022MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fieldscited by 467
  3. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  4. 2019Analyzing Learned Molecular Representations for Property Predictioncited by 373
  5. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  6. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 258
  7. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 203
  8. 2023EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representationscited by 201
  9. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 123
  10. 2020Molecule Attention Transformercited by 92
  11. 2020Molecular representation learning with language models and domain-relevant auxiliary taskscited by 91
  12. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  13. 2024On the Scalability of GNNs for Molecular Graphscited by 36
  14. 2022Evaluating Self-Supervised Learning for Molecular Graph Embeddingscited by 22
  15. 2025Attention on the Spherecited by 9
  16. 2023Instructor-inspired Machine Learning for Robust Molecular Property Predictioncited by 8

Built on by

Not cited yet.