# The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models

arXiv: https://arxiv.org/abs/2305.14999 · HTML: https://mlsanity.com/paper/2305.14999/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 24 May 2023.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 22 of 22)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023)
- [Multi-hop Reading Comprehension through Question Decomposition and Rescoring](https://mlsanity.com/paper/1906.02916/) (2019)
- [Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings](https://mlsanity.com/paper/2305.02317/) (2023)
- [Towards Data Distillation for End-to-end Spoken Conversational Question Answering](https://mlsanity.com/paper/2010.08923/) (2020)

## Built on by (most cited first, 22 of 22)

- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling](https://mlsanity.com/paper/2407.09887/) (2024)
- [Demystifying Chains, Trees, and Graphs of Thoughts](https://mlsanity.com/paper/2401.14295/) (2024)
- [LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints](https://mlsanity.com/paper/2410.06458/) (2024)
- [Mathematical Language Models: A Survey](https://mlsanity.com/paper/2312.07622/) (2023)
- [Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)](https://mlsanity.com/paper/2407.12858/) (2024)
- [AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web](https://mlsanity.com/paper/2505.17978/) (2025)
- [Boosting Large Language Models with Socratic Method for Conversational Mathematics Teaching](https://mlsanity.com/paper/2407.17349/) (2024)
- [Wrong-of-Thought: An Integrated Reasoning Framework with Multi-Perspective Verification and Wrong Information](https://mlsanity.com/paper/2410.04463/) (2024)
- [Which Type of Students can LLMs Act? Investigating Authentic Simulation with Graph-based Human-AI Collaborative System](https://mlsanity.com/paper/2502.11678/) (2025)
- [Conditions for Length Generalization in Learning Reasoning Skills](https://mlsanity.com/paper/2311.16173/) (2023)
- [Navigating the Landscape of Hint Generation Research: From the Past to the Future](https://mlsanity.com/paper/2404.04728/) (2024)
- [Advancing Chart Question Answering with Robust Chart Component Recognition](https://mlsanity.com/paper/2407.21038/) (2024)
- [POQD: Performance-Oriented Query Decomposer for Multi-vector retrieval](https://mlsanity.com/paper/2505.19189/) (2025)
- [Socratic-PRMBench: Benchmarking Process Reward Models with Systematic Reasoning Patterns](https://mlsanity.com/paper/2505.23474/) (2025)
- [Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition](https://mlsanity.com/paper/2504.20946/) (2025)
- [Multi-Conditional Ranking with Large Language Models](https://mlsanity.com/paper/2404.00211/) (2024)
- [A New Query Expansion Approach via Agent-Mediated Dialogic Inquiry](https://mlsanity.com/paper/2502.08557/) (2025)
- [The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM Reasoning](https://mlsanity.com/paper/2605.01704/) (2026)
- [Framework of Thoughts: A Foundation Framework for Dynamic and Optimized Reasoning based on Chains, Trees, and Graphs](https://mlsanity.com/paper/2602.16512/) (2026)
- [Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.03790/) (2026)
- [Personalized Object Identification and Localization via In-Context Inference with Vision-Language Models](https://mlsanity.com/paper/2607.00357/) (2026)

## Abstract (arXiv)

Chain-of-Thought (CoT) prompting enables large language models to solve complex reasoning problems by generating intermediate steps. However, confined by its inherent single-pass and sequential generation process, CoT heavily relies on the initial decisions, causing errors in early steps to accumulate and impact the final answers. In contrast, humans adopt recursive thinking when tackling complex reasoning problems, i.e., iteratively breaking the original problem into approachable sub-problems and aggregating their answers to resolve the original one. Inspired by the human cognitive process, we propose SOCRATIC QUESTIONING, a divide-and-conquer style algorithm that mimics the recursive thinking process. Specifically, SOCRATIC QUESTIONING leverages large language models to raise and answer sub-questions until collecting enough information to tackle the original question. Unlike CoT, SOCRATIC QUESTIONING explicitly navigates the thinking space, stimulates effective recursive thinking, and is more robust towards errors in the thinking process. Extensive experiments on several complex reasoning tasks, including MMLU, MATH, LogiQA, and visual question-answering demonstrate significant performance improvements over the state-of-the-art prompting methods, such as CoT, and Tree-of-Thought. The qualitative analysis clearly shows that the intermediate reasoning steps elicited by SOCRATIC QUESTIONING are similar to humans' recursively thinking process of complex reasoning problems.