# LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning

arXiv: https://arxiv.org/abs/2505.18744 · HTML: https://mlsanity.com/paper/2505.18744/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 48 references that are in the map.
- First posted on arXiv 24 May 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale](https://mlsanity.com/paper/2503.02240/) (2025)
- [Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQL](https://mlsanity.com/paper/2503.23157/) (2025)
- [From LLM to NMT: Advancing Low-Resource Machine Translation with Claude](https://mlsanity.com/paper/2404.13813/) (2024)
- [OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment](https://mlsanity.com/paper/2502.14913/) (2025)
- [A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions](https://mlsanity.com/paper/2208.13629/) (2022)
- [A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL](https://mlsanity.com/paper/2411.08599/) (2024)
- [Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark](https://mlsanity.com/paper/2402.12243/) (2024)
- [Towards Knowledge-Intensive Text-to-SQL Semantic Parsing with Formulaic Knowledge](https://mlsanity.com/paper/2301.01067/) (2023)
- [UNITE: A Unified Benchmark for Text-to-SQL Evaluation](https://mlsanity.com/paper/2305.16265/) (2023)

## Built on by (most cited first, 7 of 7)

- [Text2Mem: A Unified Memory Operation Language for Memory Operating System](https://mlsanity.com/paper/2509.11145/) (2025)
- [OraPlan-SQL: A Planning-Centric Framework for Complex Bilingual NL2SQL Reasoning](https://mlsanity.com/paper/2510.23870/) (2025)
- [Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models](https://mlsanity.com/paper/2604.25149/) (2026)
- [SteinerSQL: Graph-Guided Mathematical Reasoning for Text-to-SQL Generation](https://mlsanity.com/paper/2509.19623/) (2025)
- [IESR:Efficient MCTS-Based Modular Reasoning for Text-to-SQL with Large Language Models](https://mlsanity.com/paper/2602.05385/) (2026)
- [ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language](https://mlsanity.com/paper/2609.06527/) (2026)
- [TQTS-Bench: A Multi-Syntax Benchmark for Text-to-Query over Time-Series Databases](https://mlsanity.com/paper/2609.34783/) (2026)

## Abstract (arXiv)

Text-to-SQL is a critical task in natural language processing that aims to transform natural language questions into accurate and executable SQL queries. In real-world scenarios, these reasoning tasks are often accompanied by complex mathematical computations, domain knowledge, and hypothetical reasoning scenarios. However, existing large-scale Text-to-SQL datasets typically focus on business logic and task logic, neglecting critical factors such as vertical domain knowledge, complex mathematical reasoning, and hypothetical reasoning, which are essential for realistically reflecting the reasoning demands in practical applications and completing data querying and analysis. To bridge this gap, we introduce LogicCat, the first Text-to-SQL benchmark dataset specifically designed for complex reasoning and chain-of-thought parsing, encompassing physics, arithmetic, commonsense, and hypothetical reasoning scenarios. LogicCat comprises 4,038 English questions paired 12,114 detailed chain-of-thought reasoning steps, spanning 45 databases across diverse domains, significantly surpassing existing datasets in complexity. Experimental results demonstrate that LogicCat substantially increases the task difficulty for current state-of-the-art models to at most 33.20% execution accuracy, indicating that this task remains exceptionally challenging. The advancement of LogicCat represents a crucial step toward developing systems suitable for real-world enterprise data analysis and autonomous query generation. We have released our dataset code at https://github.com/Ffunkytao/LogicCat.