mlsanityData to 7 Oct 2026

ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solving

arXiv 2411.07228 ↗First posted 11 Nov 2024cs.AI cs.CEcited by 11

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

To enhance large language models (LLMs) for chemistry problem solving, several LLM-based agents augmented with tools have been proposed, such as ChemCrow and Coscientist. However, their evaluations are narrow in scope, leaving a large gap in understanding the benefits of tools across diverse chemistry tasks. To bridge this gap, we develop ChemToolAgent, an enhanced chemistry agent over ChemCrow, and conduct a comprehensive evaluation of its performance on both specialized chemistry tasks and general chemistry questions. Surprisingly, ChemToolAgent does not consistently outperform its base LLMs without tools. Our error analysis with a chemistry expert suggests that: For specialized chemistry tasks, such as synthesis prediction, we should augment agents with specialized tools; however, for general chemistry questions like those in exams, agents' ability to reason correctly with chemistry knowledge matters more, and tool augmentation does not always help.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 49 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

26 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024A Review of Large Language Models and Autonomous Agents in Chemistry26 Jun
  2. 2024CACTUS: Chemistry Agent Connecting Tool-Usage to Science2 May
  3. 2024CRISPR-GPT for Agentic Automation of Gene-editing Experiments27 Apr
  4. 2023Chemist-X: Large Language Model-empowered Agent for Reaction Condition Recommendation in Chemical Synthesis16 Nov

Other references · 22

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  11. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  12. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
10 more
  1. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  2. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 225
  3. 2024Tool Learning with Large Language Models: A Surveycited by 221
  4. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 179
  5. 2024Are We Done with MMLU?cited by 178
  6. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 153
  7. 2024LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Datasetcited by 124
  8. 2023BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biologycited by 42
  9. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 41
  10. 2019Self-Referencing Embedded Strings (SELFIES): A 100% robust molecular string representationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  4. 2025PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaborationcited by 25
  5. 2025AutoLabs: Cognitive Multi-Agent Systems with Self-Correction for Autonomous Chemical Experimentationcited by 13
  6. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  7. 2025LARC: Towards Human-level Constrained Retrosynthesis Planning through an Agentic Frameworkcited by 9
  8. 2025MADD: Multi-Agent Drug Discovery Orchestracited by 4
  9. 2026BrainAgent: A Large Language Model-Driven Multi-Agent Framework for Autonomous Brain Signal Understandingcited by 2
  10. 2026Bolek: A Multimodal Language Model for Molecular Reasoningcited by 0
  11. 2026Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Are large language models superhuman chemists?
  2. 2025CheMatAgent: Enhancing LLMs for Chemistry and Materials Science through Tree-Search Based Tool Learning
  3. 2026ChemPro: A Progressive Chemistry Benchmark for Large Language Models
  4. 2025Assessing the Chemical Intelligence of Large Language Models
  5. 2026TMCS: Tool-Grounded Multi-Agent Reasoning for Compositional Chemical Problem Solving
  6. 2025ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning
  7. 2026ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation
  8. 2025ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation