mlsanityData to 14 Oct 2026

Transformers and Large Language Models for Chemistry and Drug Discovery

arXiv 2310.06083 ↗First posted 9 Oct 2023cs.LG physics.chem-phcited by 13

Not on the map. Why?

Abstract (arXiv)

Language modeling has seen impressive progress over the last years, mainly prompted by the invention of the Transformer architecture, sparking a revolution in many fields of machine learning, with breakthroughs in chemistry and biology. In this chapter, we explore how analogies between chemical and natural language have inspired the use of Transformers to tackle important bottlenecks in the drug discovery process, such as retrosynthetic planning and chemical space exploration. The revolution started with models able to perform particular tasks with a single type of data, like linearised molecular graphs, which then evolved to include other types of data, like spectra from analytical instruments, synthesis actions, and human language. A new trend leverages recent developments in large language models, giving rise to a wave of models capable of solving generic tasks in chemistry, all facilitated by the flexibility of natural language. As we continue to explore and harness these capabilities, we can look forward to a future where machine learning plays an even more integral role in accelerating scientific discovery.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 47 of its 95 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

47 of 95 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  11. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
35 more
  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,051
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  9. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 765
  10. 2015Convolutional Networks on Graphs for Learning Molecular Fingerprintscited by 754
  11. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  12. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  13. 2020ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Predictioncited by 337
  14. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  15. 2017Grammar Variational Autoencodercited by 285
  16. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 264
  17. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 229
  18. 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
  19. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  20. 2022Translation between Molecules and Natural Languagecited by 177
  21. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 156
  22. 2023Emergent autonomous scientific research capabilities of large language modelscited by 132
  23. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 127
  24. 2020Retrosynthesis Prediction with Conditional Graph Logic Networkcited by 104
  25. 2023Unifying Molecular and Textual Representations via Multi-task Language Modellingcited by 93
  26. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 86
  27. 2022SELFIES and the future of molecular string representationscited by 76
  28. 2020Mapping the Space of Chemical Reactions Using Attention-Based Neural Networkscited by 70
  29. 2021Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction predictioncited by 68
  30. 2023Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB filescited by 59
  31. 2022Regression Transformer: Concurrent sequence regression and generation for molecular language modelingcited by 50
  32. 2021Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphscited by 44
  33. 2020Exploring Chemical Space using Natural Language Processing Methodologies for Drug Discoverycited by 24
  34. 2023Bayesian Optimization of Catalysis With In-Context Learningcited by 22
  35. 2021C5T5: Controllable Generation of Organic Molecules with Transformerscited by 16

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 7; 2025: 5.

13 papers.

Sort:

By area:

  1. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  2. 2023Graph Prompt Learning: A Comprehensive Survey and Beyondcited by 61
  3. 2024Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequencescited by 13
  4. 2024Less can be more for predicting properties with large language modelscited by 11
  5. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 11
  6. 2024Geometry Informed Tokenization of Molecules for Language Model Generationcited by 10
  7. 2024Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiencycited by 7
  8. 2025DrugImproverGPT: A Large Language Model for Drug Optimization with Fine-Tuning via Structured Policy Optimizationcited by 6
  9. 2024Leveraging Chemistry Foundation Models to Facilitate Structure Focused Retrieval Augmented Generation in Multi-Agent Workflows for Catalyst and Materials Designcited by 3
  10. 2025The Catechol Benchmark: Time-series Solvent Selection Data for Few-shot Machine Learningcited by 3
  11. 2024Transformer models as an efficient replacement for statistical test suites to evaluate the quality of random numberscited by 1
  12. 2025From EMR Data to Clinical Insight: An LLM-Driven Framework for Automated Pre-Consultation Questionnaire Generationcited by 1
1 more
  1. 2025Transformers in Protein: A Surveycited by 0