mlsanityData to 7 Oct 2026

Molecular representation learning with language models and domain-relevant auxiliary tasks

arXiv 2011.13230 ↗First posted 26 Nov 2020cs.LG cs.AIcited by 91

Not on the map. Why?

Abstract (arXiv)

We apply a Transformer architecture, specifically BERT, to learn flexible and high quality molecular representations for drug discovery problems. We study the impact of using different combinations of self-supervised tasks for pre-training, and present our results for the established Virtual Screening and QSAR benchmarks. We show that: i) The selection of appropriate self-supervised task(s) for pre-training has a significant impact on performance in subsequent downstream tasks such as Virtual Screening. ii) Using auxiliary tasks with more domain relevance for Chemistry, such as learning to predict calculated molecular properties, increases the fidelity of our learnt representations. iii) Finally, we show that molecular representations learnt by our model `MolBert' improve upon the current state of the art on the benchmark datasets.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 13; 2023: 23; 2024: 18; 2025: 21; 2026: 12.

91 papers.

Sort:

By area:

  1. 2022Translation between Molecules and Natural Languagecited by 173
  2. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 123
  3. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  4. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  5. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  6. 2022Regression Transformer: Concurrent sequence regression and generation for molecular language modelingcited by 49
  7. 2023SELFormer: Molecular Representation Learning via SELFIES Language Modelscited by 40
  8. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 40
  9. 2022Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrastcited by 37
  10. 2021Chemical-Reaction-Aware Molecule Representation Learningcited by 36
  11. 2023Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformerscited by 32
  12. 2024ChemReasoner: Heuristic Search over a Large Language Model's Knowledge Space using Quantum-Chemical Feedbackcited by 32
38 more
  1. 2021Artificial Intelligence in Drug Discovery: Applications and Techniquescited by 28
  2. 2021Dual-view Molecule Pre-trainingcited by 27
  3. 2023nach0: Multimodal Natural and Chemical Languages Foundation Modelcited by 26
  4. 2023BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphscited by 24
  5. 2023ReLM: Leveraging Language Models for Enhanced Chemical Reaction Predictioncited by 24
  6. 2021Relative Molecule Self-Attention Transformercited by 21
  7. 2022A graph representation of molecular ensembles for polymer property predictioncited by 20
  8. 2024MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehensioncited by 19
  9. 2023Searching for High-Value Molecules Using Reinforcement Learning and Transformerscited by 18
  10. 2023Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Designcited by 16
  11. 2025Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Modelcited by 16
  12. 2025Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learningcited by 15
  13. 2023SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Designcited by 13
  14. 2023GPT-MolBERTa: GPT Molecular Features Language Model for molecular property predictioncited by 13
  15. 2022MolE: a molecular foundation model for drug discoverycited by 12
  16. 2022Exploiting Pretrained Biochemical Language Models for Targeted Drug Designcited by 10
  17. 2023Otter-Knowledge: benchmarks of multimodal knowledge graph representation learning from different sources for drug discoverycited by 10
  18. 2024Can Large Language Models Understand Molecules?cited by 10
  19. 2023Exploring QSAR Models for Activity-Cliff Predictioncited by 9
  20. 2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 9
  21. 2022SSM-DTA: Breaking the Barriers of Data Scarcity in Drug-Target Affinity Predictioncited by 8
  22. 2023ReactIE: Enhancing Chemical Reaction Extraction with Weak Supervisioncited by 7
  23. 2024When is an Embedding Model More Promising than Another?cited by 7
  24. 2023Explainability Techniques for Chemical Language Modelscited by 6
  25. 2023Structure to Property: Chemical Element Embeddings and a Deep Learning Approach for Accurate Prediction of Chemical Propertiescited by 6
  26. 2023Language models in molecular discoverycited by 6
  27. 2022Pre-training Transformers for Molecular Property Prediction Using Reaction Predictioncited by 5
  28. 2024Emerging Opportunities of Using Large Language Models for Translation Between Drug Molecules and Indicationscited by 5
  29. 2024An Evaluation of Large Language Models in Bioinformatics Researchcited by 5
  30. 2025Synergistic Benefits of Joint Molecule Generation and Property Predictioncited by 5
  31. 2023Uni-QSAR: an Auto-ML Tool for Molecular Property Predictioncited by 4
  32. 2023MolKD: Distilling Cross-Modal Knowledge in Chemical Reactions for Molecular Property Predictioncited by 4
  33. 2024Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learningcited by 4
  34. 2022Multimodal Model with Text and Drug Embeddings for Adverse Drug Reaction Classificationcited by 3
  35. 2023DYMAG: Rethinking Message Passing Using Dynamical-systems-based Waveformscited by 3
  36. 2024GNN-SKAN: Harnessing the Power of SwallowKAN to Advance Molecular Representation Learning with GNNscited by 3
  37. 2024Pretraining Graph Transformers with Atom-in-a-Molecule Quantum Properties for Improved ADMET Modelingcited by 3
  38. 2024AttackQA: Development and Adoption of a Dataset for Assisting Cybersecurity Operations using Fine-tuned and Open-Source LLMscited by 3

50 most cited of 91 All 91 (JSON)