mlsanityData to 7 Oct 2026

ESM All-Atom: Multi-scale Protein Language Model for Unified Molecular Modeling

arXiv 2403.12995 ↗First posted 5 Mar 2024q-bio.BM cs.CE cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Protein language models have demonstrated significant potential in the field of protein engineering. However, current protein language models primarily operate at the residue scale, which limits their ability to provide information at the atom level. This limitation prevents us from fully exploiting the capabilities of protein language models for applications involving both proteins and small molecules. In this paper, we propose ESM-AA (ESM All-Atom), a novel approach that enables atom-scale and residue-scale unified molecular modeling. ESM-AA achieves this by pre-training on multi-scale code-switch protein sequences and utilizing a multi-scale position encoding to capture relationships among residues and atoms. Experimental results indicate that ESM-AA surpasses previous methods in protein-molecule tasks, demonstrating the full utilization of protein language models. Further investigations reveal that through unified molecular modeling, ESM-AA not only gains molecular knowledge but also retains its understanding of proteins. The source codes of ESM-AA are publicly released at https://github.com/zhengkangjie/ESM-AA.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 38 of its 85 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

38 of 85 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  5. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  6. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 332
  7. 2020Learning from Protein Structure with Geometric Vector Perceptronscited by 308
  8. 2017Dilated Residual Networkscited by 293
  9. 2021Pre-training Molecular Graph Representation with 3D Geometrycited by 248
  10. 2019Evaluating Protein Transfer Learning with TAPEcited by 229
  11. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  12. 2018DeepDTA: Deep Drug-Target Binding Affinity Predictioncited by 171
26 more
  1. 2022Protein Representation Learning by Geometric Structure Pretrainingcited by 161
  2. 2021Motif-based Graph Self-Supervised Learning for Molecular Property Predictioncited by 159
  3. 2022ProGen2: Exploring the Boundaries of Protein Language Modelscited by 156
  4. 20213D Infomax improves GNNs for Molecular Property Predictioncited by 146
  5. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  6. 2019SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discoverycited by 89
  7. 2022Tranception: protein fitness prediction with autoregressive transformers and inference-time retrievalcited by 78
  8. 2022One Transformer Can Understand Both 2D & 3D Molecular Datacited by 77
  9. 2023Structure-informed Language Models Are Protein Designerscited by 68
  10. 2022Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matchingcited by 67
  11. 2021GeomGCL: Geometric Graph Contrastive Learning for Molecular Property Predictioncited by 61
  12. 2021Molecular Contrastive Learning with Chemical Element Knowledge Graphcited by 50
  13. 2022Unified 2D and 3D Pre-Training of Molecular Representationscited by 45
  14. 2020Code-switching pre-training for neural machine translationcited by 43
  15. 2019ProteinNet: a standardized data set for machine learning of protein structurecited by 39
  16. 2022Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrastcited by 37
  17. 2022Energy-Motivated Equivariant Pretraining for 3D Molecular Graphscited by 37
  18. 2022Contrastive Representation Learning for 3D Protein Structurescited by 30
  19. 2019Self-Attention Based Molecule Representation for Predicting Drug-Target Interactioncited by 22
  20. 2020Learn molecular representations from large-scale unlabeled molecules for drug discoverycited by 18
  21. 2023Generalist Equivariant Transformer Towards 3D Molecular Interaction Learningcited by 18
  22. 2020GEFA: Early Fusion Approach in Drug-Target Affinity Predictioncited by 16
  23. 2021Multilingual Molecular Representation Learning via Contrastive Pre-trainingcited by 12
  24. 2022Universal Conditional Masked Language Pre-training for Neural Machine Translationcited by 12
  25. 2021Toward Drug-Target Interaction Prediction via Ensemble Modeling and Transfer Learningcited by 4
  26. 2022EurNet: Efficient Multi-Range Relational Modeling of Spatial Multi-Relational Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2025UniMoMo: Unified Generative Modeling of 3D Molecules for De Novo Binder Designcited by 11
  2. 2024SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervisioncited by 3
  3. 2025ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Modelscited by 2
  4. 2025PharMolixFM: All-Atom Foundation Models for Molecular Modeling and Generationcited by 0
  5. 2026KEMP-PIP: A Feature-Fusion Based Approach for Pro-inflammatory Peptide Predictioncited by 0
  6. 2026From Snapshots to Symphonies: The Evolution of Protein Prediction from Static Structures to Generative Dynamics and Multimodal Interactionscited by 0
  7. 2026MI-PEFT: Mixture-of-Experts Integrated Parameter-Efficient Fine-Tuning Protein Language Models Improves Acidophilic Proteins Classificationcited by 0