# ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models

arXiv: https://arxiv.org/abs/2305.14323 · HTML: https://mlsanity.com/paper/2305.14323/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 3 of its 48 references that are in the map.
- First posted on arXiv 23 May 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 3 of 3)

- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023)
- [StructGPT: A General Framework for Large Language Model to Reason over Structured Data](https://mlsanity.com/paper/2305.09645/) (2023)
- [ART: Automatic multi-step reasoning and tool-use for large language models](https://mlsanity.com/paper/2303.09014/) (2023)

## Built on by (most cited first, 30 of 43)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Agentic Reinforced Policy Optimization](https://mlsanity.com/paper/2507.19849/) (2025)
- [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [A Review of Large Language Models and Autonomous Agents in Chemistry](https://mlsanity.com/paper/2407.01603/) (2024)
- [Demystifying Chains, Trees, and Graphs of Thoughts](https://mlsanity.com/paper/2401.14295/) (2024)
- [LLM With Tools: A Survey](https://mlsanity.com/paper/2409.18807/) (2024)
- [TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage](https://mlsanity.com/paper/2308.03427/) (2023)
- [Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning](https://mlsanity.com/paper/2306.02408/) (2023)
- [Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live](https://mlsanity.com/paper/2511.02230/) (2025)
- [TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systems](https://mlsanity.com/paper/2311.11315/) (2023)
- [Generative to Agentic AI: Survey, Conceptualization, and Challenges](https://mlsanity.com/paper/2504.18875/) (2025)
- [Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models](https://mlsanity.com/paper/2505.16122/) (2025)
- [An In-depth Survey of Large Language Model-based Artificial Intelligence Agents](https://mlsanity.com/paper/2309.14365/) (2023)
- [TOOLVERIFIER: Generalization to New Tools via Self-Verification](https://mlsanity.com/paper/2402.14158/) (2024)
- [Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyond](https://mlsanity.com/paper/2402.14259/) (2024)
- [GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution](https://mlsanity.com/paper/2307.08775/) (2023)
- [CausalEval: Towards Better Causal Reasoning in Language Models](https://mlsanity.com/paper/2410.16676/) (2024)
- [Uncertainty Estimation of Large Language Models in Medical Question Answering](https://mlsanity.com/paper/2407.08662/) (2024)
- [MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems](https://mlsanity.com/paper/2503.16549/) (2025)
- [LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems](https://mlsanity.com/paper/2504.01963/) (2025)
- [A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions](https://mlsanity.com/paper/2504.16939/) (2025)
- [Hell or High Water: Evaluating Agentic Recovery from External Failures](https://mlsanity.com/paper/2508.11027/) (2025)
- [Conformal Information Pursuit for Interactively Guiding Large Language Models](https://mlsanity.com/paper/2507.03279/) (2025)
- [DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text](https://mlsanity.com/paper/2310.20170/) (2023)
- [DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMs](https://mlsanity.com/paper/2401.05190/) (2024)
- [GraphChain: Large Language Models for Large-scale Graph Analysis via Tool Chaining](https://mlsanity.com/paper/2511.00457/) (2025)

## Abstract (arXiv)

Although large language models (LLMs) have achieved excellent performance in a variety of evaluation benchmarks, they still struggle in complex reasoning tasks which require specific knowledge and multi-hop reasoning. To improve the reasoning abilities, we propose ChatCoT, a tool-augmented chain-of-thought reasoning framework for chat-based LLMs (e.g., ChatGPT). In ChatCoT, we model the chain-of-thought (CoT) reasoning as multi-turn conversations, to utilize tools in a more natural way through chatting. At each turn, LLMs can either interact with tools or perform the reasoning. Our approach can effectively leverage the multi-turn conversation ability of chat-based LLMs, and integrate the thought chain following and tools manipulation in a unified way. Specially, we initialize the early turns of the conversation by the knowledge about tools, tasks, and reasoning format, and propose an iterative tool-augmented reasoning step to perform step-by-step tool-augmented reasoning. The experiment results on two complex reasoning datasets (MATH and HotpotQA) have shown the effectiveness of ChatCoT on complex reasoning tasks, achieving a 7.9% relative improvement over the state-of-the-art baseline. Our code and data are available at: urlhttps://github.com/RUCAIBOX/ChatCoT.