mlsanityData to 7 Oct 2026

Are large language models superhuman chemists?

arXiv 2404.01475 ↗First posted 1 Apr 2024cs.LG cond-mat.mtrl-sci cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have gained widespread interest due to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here, we introduce "ChemBench," an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question-answer pairs, evaluated leading open- and closed-source LLMs, and found that the best models outperformed the best human chemists in our study on average. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs' impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 50 of its 95 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

50 of 95 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2017MoleculeNet: A Benchmark for Molecular Machine Learningcited by 1,037
  11. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  12. 2022Galactica: A Large Language Model for Sciencecited by 736
38 more
  1. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  2. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  3. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  4. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  5. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  6. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  7. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  8. 2021Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Developmentcited by 258
  9. 2021Trustworthy AI: From Principles to Practicescited by 206
  10. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  11. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  12. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  13. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  14. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  15. 2022ChemBERTa-2: Towards Chemical Foundation Modelscited by 123
  16. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
  17. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  18. 2024ChemLLM: A Chemical Large Language Modelcited by 100
  19. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  20. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 92
  21. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 85
  22. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Modelscited by 76
  23. 2024ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterizationcited by 60
  24. 2023Language models can generate molecules, materials, and protein binding sites directly in three dimensions as XYZ, CIF, and PDB filescited by 59
  25. 2023Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learningcited by 50
  26. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  27. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  28. 2024Are LLMs Ready for Real-World Materials Discovery?cited by 40
  29. 2023MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modelingcited by 35
  30. 2023Will releasing the weights of future large language models grant widespread access to pandemic agents?cited by 25
  31. 2023Bayesian Optimization of Catalysis With In-Context Learningcited by 20
  32. 2022DiSCoMaT: Distantly Supervised Composition Extraction from Tables in Materials Science Articlescited by 17
  33. 2024Benchmarking Large Language Models for Molecule Prediction Taskscited by 17
  34. 2024From Words to Molecules: A Survey of Large Language Models in Chemistrycited by 14
  35. 2023Image and Data Mining in Reticular Chemistry Using GPT-4Vcited by 6
  36. 2023Towards Responsible Governance of Biological Design Toolscited by 5
  37. 2023Empirical assessment of ChatGPT's answering capabilities in natural science and engineeringcited by 4
  38. 2023Censoring chemical data to mitigate dual use riskcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 13.

34 papers.

Sort:

By area:

  1. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  2. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
  3. 2025Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operationscited by 46
  4. 2025TxGemma: Efficient and Agentic LLMs for Therapeuticscited by 40
  5. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  6. 2024Aviary: training language agents on challenging scientific taskscited by 23
  7. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
  8. 2025Chemical reasoning in LLMs unlocks strategy-aware synthesis planning and reaction mechanism elucidationcited by 17
  9. 2025Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameterscited by 11
  10. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 10
  11. 2024Reflections from the 2024 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistrycited by 8
  12. 2025FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Modelscited by 8
22 more
  1. 2025ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Modelscited by 7
  2. 2025EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistantscited by 5
  3. 2025SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generated LLMscited by 5
  4. 2026Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoningcited by 5
  5. 2025MADD: Multi-Agent Drug Discovery Orchestracited by 4
  6. 2025HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discoverycited by 3
  7. 2026Evaluating the Progression of Large Language Model Capabilities for Small-Molecule Drug Designcited by 3
  8. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  9. 2025MiST: Understanding the Role of Mid-Stage Scientific Training in Developing Chemical Reasoning Modelscited by 2
  10. 2025Large Language Models Transform Organic Synthesis From Reaction Prediction to Automationcited by 1
  11. 2026MolQuest: A Benchmark for Agentic Evaluation of Abductive Reasoning in Chemical Structure Elucidationcited by 1
  12. 2026From Knowledge to Action: Outcomes of the 2025 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistrycited by 1
  13. 2026SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verificationcited by 1
  14. 2026Accurate, Interdisciplinary and Transparent Structure-property Understanding with Deep Native Structural Reasoningcited by 1
  15. 2025Assay2Mol: large language model-based drug design using BioAssay contextcited by 0
  16. 2026Teaching and Evaluating LLMs to Reason About Polymer Design Related Taskscited by 0
  17. 2026MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Modelscited by 0
  18. 2026Rethinking Molecular Text Representations for LLMs: An Empirical Studycited by 0
  19. 2026From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Modelscited by 0
  20. 2026A Task-Centric Ontology and Deterministic Domain Rules as a Verifiable Core for AI-Assisted Chemistry Problem Solvingcited by 0
  21. 2026Evaluating LLM-based AI agents integrated with materials synthesis tools: the case of atomic layer depositioncited by 0
  22. 2026Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Assessing the Chemical Intelligence of Large Language Models
  2. 2026ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation
  3. 2026ChemPro: A Progressive Chemistry Benchmark for Large Language Models
  4. 2025ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation
  5. 2025Training a Scientific Reasoning Model for Chemistry
  6. 2025ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions
  7. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area
  8. 2024ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solving