# Structured Chemistry Reasoning with Large Language Models

arXiv: https://arxiv.org/abs/2311.09656 · HTML: https://mlsanity.com/paper/2311.09656/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 11 of its 45 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 11 of 11)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023)
- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models](https://mlsanity.com/paper/2306.08018/) (2023)
- [GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems](https://mlsanity.com/paper/2310.12397/) (2023)

## Built on by (most cited first, 28 of 28)

- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph](https://mlsanity.com/paper/2410.14684/) (2024)
- [Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents](https://mlsanity.com/paper/2311.11797/) (2023)
- [LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models](https://mlsanity.com/paper/2404.05221/) (2024)
- [MedCalc-Bench: Evaluating Large Language Models for Medical Calculations](https://mlsanity.com/paper/2406.12036/) (2024)
- [Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning](https://mlsanity.com/paper/2312.05230/) (2023)
- [ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows](https://mlsanity.com/paper/2505.19897/) (2025)
- [Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations](https://mlsanity.com/paper/2505.21318/) (2025)
- [ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning](https://mlsanity.com/paper/2501.06590/) (2025)
- [Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra](https://mlsanity.com/paper/2404.03647/) (2024)
- [LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation](https://mlsanity.com/paper/2401.17244/) (2024)
- [Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents](https://mlsanity.com/paper/2412.00821/) (2024)
- [DyFlow: Dynamic Workflow Framework for Agentic Reasoning](https://mlsanity.com/paper/2509.26062/) (2025)
- [ControlAgent: Automating Control System Design via Novel Integration of LLM Agents and Domain Expertise](https://mlsanity.com/paper/2410.19811/) (2024)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Leveraging Biomolecule and Natural Language through Multi-Modal Learning: A Survey](https://mlsanity.com/paper/2403.01528/) (2024)
- [Benchmarking the Capabilities of Large Language Models in Transportation System Engineering: Accuracy, Consistency, and Reasoning Behaviors](https://mlsanity.com/paper/2408.08302/) (2024)
- [FGBench: A Dataset and Benchmark for Molecular Property Reasoning at Functional Group-Level in Large Language Models](https://mlsanity.com/paper/2508.01055/) (2025)
- [Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesis](https://mlsanity.com/paper/2601.17687/) (2026)
- [Normative Conflicts and Shallow AI Alignment](https://mlsanity.com/paper/2506.04679/) (2025)
- [CoTox: Chain-of-Thought-Based Molecular Toxicity Reasoning and Prediction](https://mlsanity.com/paper/2508.03159/) (2025)
- [Bridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule](https://mlsanity.com/paper/2403.13830/) (2024)
- [Atom-anchored LLMs speak Chemistry: A Retrosynthesis Demonstration](https://mlsanity.com/paper/2510.16590/) (2025)
- [Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction](https://mlsanity.com/paper/2603.23550/) (2026)
- [PepThink-R1: LLM for Interpretable Cyclic Peptide Optimization with CoT SFT and Reinforcement Learning](https://mlsanity.com/paper/2508.14765/) (2025)
- [Full-Stack Domain Enhancement for Combustion LLMs: Construction and Optimization](https://mlsanity.com/paper/2603.19268/) (2026)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)
- [Masked Diffusion Language Agents for Tool-Integrated Chemical Reasoning](https://mlsanity.com/paper/2610.13330/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) excel in diverse areas, yet struggle with complex scientific reasoning, especially in the field of chemistry. Different from the simple chemistry tasks (e.g., molecule classification) addressed in previous studies, complex chemistry problems require not only vast knowledge and precise calculation, but also compositional reasoning about rich dynamic interactions of different concepts (e.g., temperature changes). Our study shows that even advanced LLMs, like GPT-4, can fail easily in different ways. Interestingly, the errors often stem not from a lack of domain knowledge within the LLMs, but rather from the absence of an effective reasoning structure that guides the LLMs to elicit the right knowledge, incorporate the knowledge in step-by-step reasoning, and iteratively refine results for further improved quality. On this basis, we introduce StructChem, a simple yet effective prompting strategy that offers the desired guidance and substantially boosts the LLMs' chemical reasoning capability. Testing across four chemistry areas -- quantum chemistry, mechanics, physical chemistry, and kinetics -- StructChem substantially enhances GPT-4's performance, with up to 30% peak improvement. Our analysis also underscores the unique difficulties of precise grounded reasoning in science with LLMs, highlighting a need for more research in this area. Code is available at urlhttps://github.com/ozyyshr/StructChem.