mlsanityData to 7 Oct 2026

What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks

arXiv 2305.18365 ↗First posted 27 May 2023cs.CL cs.AIcited by 179

Not on the map. Why?

Cited by 25 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) with strong abilities in natural language processing tasks have emerged and have been applied in various kinds of areas such as science, finance and software engineering. However, the capability of LLMs to advance the field of chemistry remains unclear. In this paper, rather than pursuing state-of-the-art performance, we aim to evaluate capabilities of LLMs in a wide range of tasks across the chemistry domain. We identify three key chemistry-related capabilities including understanding, reasoning and explaining to explore in LLMs and establish a benchmark containing eight chemistry tasks. Our analysis draws on widely recognized datasets facilitating a broad exploration of the capacities of LLMs within the context of practical chemistry. Five LLMs (GPT-4, GPT-3.5, Davinci-003, Llama and Galactica) are evaluated for each chemistry task in zero-shot and few-shot in-context learning settings with carefully selected demonstration examples and specially crafted prompts. Our investigation found that GPT-4 outperformed other models and LLMs exhibit different competitive levels in eight chemistry tasks. In addition to the key findings from the comprehensive benchmark analysis, our work provides insights into the limitation of current LLMs and the impact of in-context learning settings on LLMs' performance across various chemistry tasks. The code and datasets used in this study are available at https://github.com/ChemFoundationModels/ChemLLMBench.

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Builds on 33 of its 76 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

33 of 76 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  10. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  11. 2022Galactica: A Large Language Model for Sciencecited by 736
  12. 2020Aligning AI With Shared Human Valuescited by 706
21 more
  1. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  2. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  3. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  4. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  5. 2023Mathematical Capabilities of ChatGPTcited by 263
  6. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 225
  7. 2018Fr'echet ChemNet Distance: A metric for generative models for molecules in drug discoverycited by 191
  8. 2022Translation between Molecules and Natural Languagecited by 173
  9. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  10. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  11. 2017Predicting Organic Reaction Outcomes with Weisfeiler-Lehman Networkcited by 102
  12. 2021Few-Shot Graph Learning for Molecular Property Predictioncited by 81
  13. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  14. 2023MolXPT: Wrapping Molecules with Text for Generative Pre-trainingcited by 81
  15. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  16. 2022Graph-based Molecular Representation Learningcited by 51
  17. 2021Chemical-Reaction-Aware Molecule Representation Learningcited by 36
  18. 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
  19. 2022Target-aware Abstractive Related Work Generation with Contrastive Learningcited by 20
  20. 2022Scientific Paper Extractive Summarization Enhanced by Citation Graphscited by 5
  21. 2022Few-shot News Recommendation via Cross-lingual Transfercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 58; 2025: 73; 2026: 43.

179 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  3. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 124
  4. 2024ChemLLM: A Chemical Large Language Modelcited by 100
  5. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  6. 2024CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligencecited by 61
  7. 2024Efficient Evolutionary Search Over Chemical Space with Large Language Modelscited by 59
  8. 2025LLM Agents for Education: Advances and Applicationscited by 59
  9. 2024Developing ChemDFM as a large language foundation model for chemistrycited by 51
  10. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  11. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoningcited by 41
  12. 2025SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integrationcited by 40
38 more
  1. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 37
  2. 2024Are large language models superhuman chemists?cited by 34
  3. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  4. 2024ChemReasoner: Heuristic Search over a Large Language Model's Knowledge Space using Quantum-Chemical Feedbackcited by 32
  5. 2024CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Masterycited by 30
  6. 2025Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agentscited by 30
  7. 2024Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planningcited by 29
  8. 2024Are Large Language Models Good Statisticians?cited by 26
  9. 2024Social Science Meets LLMs: How Reliable Are Large Language Models in Social Simulations?cited by 26
  10. 2024LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labscited by 24
  11. 2024Tx-LLM: A Large Language Model for Therapeuticscited by 23
  12. 2024MolTC: Towards Molecular Relational Modeling In Language Modelscited by 22
  13. 2024SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Modelscited by 19
  14. 2024The Impact of Large Language Models in Academia: from Writing to Speakingcited by 18
  15. 2024ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Modelscited by 18
  16. 2025DrugPilot: LLM-based Parameterized Reasoning Agent for Drug Discoverycited by 18
  17. 2024Benchmarking Large Language Models for Molecule Prediction Taskscited by 17
  18. 2024Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Modelscited by 17
  19. 2024Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Surveycited by 16
  20. 2024ChemDFM-X: Towards Large Multimodal Model for Chemistrycited by 16
  21. 2024Using Generative AI and Multi-Agents to Provide Automatic Feedbackcited by 15
  22. 2025SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Modelscited by 15
  23. 2025PaperArena: An Evaluation Benchmark for Tool-Augmented Agentic Reasoning on Scientific Literaturecited by 15
  24. 2024From Words to Molecules: A Survey of Large Language Models in Chemistrycited by 14
  25. 2024PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomescited by 14
  26. 2024ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domaincited by 14
  27. 2025Assessing the Chemical Intelligence of Large Language Modelscited by 14
  28. 2025Potemkin Understanding in Large Language Modelscited by 14
  29. 2025Optimizing Sequential Multi-Step Tasks with Parallel LLM Agentscited by 14
  30. 2025AOT*: Efficient Synthesis Planning via LLM-Empowered AND-OR Tree Searchcited by 13
  31. 2025Integrating Large Language Models for Automated Structural Analysiscited by 12
  32. 2024Knowledge Graph Enhanced Language Agents for Recommendationcited by 11
  33. 2024ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solvingcited by 11
  34. 2024From Generalist to Specialist: A Survey of Large Language Models for Chemistrycited by 11
  35. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  36. 2025MolReasoner: Toward Effective and Interpretable Reasoning for Molecular LLMscited by 11
  37. 2025ProtoMol: Enhancing Molecular Property Prediction via Prototype-Guided Multimodal Learningcited by 11
  38. 2024Can Large Language Models Understand Molecules?cited by 10

50 most cited of 179 All 179 (JSON)