# MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL

arXiv: https://arxiv.org/abs/2406.12692 · HTML: https://mlsanity.com/paper/2406.12692/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 18 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation](https://mlsanity.com/paper/1905.08205/) (2019)
- [CodeS: Towards Building Open-source Language Models for Text-to-SQL](https://mlsanity.com/paper/2402.16347/) (2024)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [C3: Zero-shot Text-to-SQL with ChatGPT](https://mlsanity.com/paper/2307.07306/) (2023)
- [OpenAgents: An Open Platform for Language Agents in the Wild](https://mlsanity.com/paper/2310.10634/) (2023)
- [MCS-SQL: Leveraging Multiple Prompts and Multiple-Choice Selection For Text-to-SQL Generation](https://mlsanity.com/paper/2405.07467/) (2024)
- [Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation](https://mlsanity.com/paper/2405.15307/) (2024)
- [SQLBench: A Comprehensive Evaluation for Text-to-SQL Capabilities of Large Language Models](https://mlsanity.com/paper/2403.02951/) (2024)

## Built on by (most cited first, 30 of 35)

- [The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models](https://mlsanity.com/paper/2408.07702/) (2024)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimization](https://mlsanity.com/paper/2502.04295/) (2025)
- [SHARE: An SLM-based Hierarchical Action CorREction Assistant for Text-to-SQL](https://mlsanity.com/paper/2506.00391/) (2025)
- [ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation](https://mlsanity.com/paper/2507.14201/) (2025)
- [Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors](https://mlsanity.com/paper/2501.09310/) (2025)
- [SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL](https://mlsanity.com/paper/2506.04494/) (2025)
- [Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQL](https://mlsanity.com/paper/2502.11656/) (2025)
- [PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL](https://mlsanity.com/paper/2409.14082/) (2024)
- [DAC: Decomposed Automation Correction for Text-to-SQL](https://mlsanity.com/paper/2408.08779/) (2024)
- [Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL](https://mlsanity.com/paper/2411.13244/) (2024)
- [PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents](https://mlsanity.com/paper/2604.17653/) (2026)
- [SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL](https://mlsanity.com/paper/2502.11438/) (2025)
- [Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities](https://mlsanity.com/paper/2505.23838/) (2025)
- [Memo-SQL: Structured Decomposition and Experience-Driven Self-Correction for Training-Free NL2SQL](https://mlsanity.com/paper/2601.10011/) (2026)
- [Context-Aware SQL Error Correction Using Few-Shot Learning -- A Novel Approach Based on NLQ, Error, and SQL Similarity](https://mlsanity.com/paper/2410.09174/) (2024)
- [Dynamic Context-Aware Prompt Recommendation for Domain-Specific AI Applications](https://mlsanity.com/paper/2506.20815/) (2025)
- [Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence](https://mlsanity.com/paper/2507.10281/) (2025)
- [End-to-End Text-to-SQL with Dataset Selection: Leveraging LLMs for Adaptive Query Generation](https://mlsanity.com/paper/2508.06387/) (2025)
- [ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement](https://mlsanity.com/paper/2603.03742/) (2026)
- [NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions](https://mlsanity.com/paper/2604.16493/) (2026)
- [Knowing When to Stop: Predicting Execution-Consistency Convergence in Text-to-SQL](https://mlsanity.com/paper/2607.03991/) (2026)
- [The Role of Accuracy and Validation Effectiveness in Conversational Business Analytics](https://mlsanity.com/paper/2411.12128/) (2024)
- [SQLCritic: Correcting Text-to-SQL Generation via Clause-wise Critic](https://mlsanity.com/paper/2503.07996/) (2025)
- [GradeSQL: Test-Time Inference with Outcome Reward Models for Text-to-SQL Generation from Large Language Models](https://mlsanity.com/paper/2509.01308/) (2025)
- [GEMMA-SQL: A Novel Text-to-SQL Model Based on Large Language Models](https://mlsanity.com/paper/2511.04710/) (2025)
- [SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answering](https://mlsanity.com/paper/2511.17559/) (2025)
- [Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQL](https://mlsanity.com/paper/2512.22744/) (2025)
- [SQL Query Engine: A Self-Healing LLM Pipeline for Natural Language to PostgreSQL Translation](https://mlsanity.com/paper/2604.16511/) (2026)
- [ZAS-SQL: Distilling Rules from Failures for Zero-Shot Text-to-SQL](https://mlsanity.com/paper/2606.08245/) (2026)

## Abstract (arXiv)

Self-correction in text-to-SQL is the process of prompting large language model (LLM) to revise its previously incorrectly generated SQL, and commonly relies on manually crafted self-correction guidelines by human experts that are not only labor-intensive to produce but also limited by the human ability in identifying all potential error patterns in LLM responses. We introduce MAGIC, a novel multi-agent method that automates the creation of the self-correction guideline. MAGIC uses three specialized agents: a manager, a correction, and a feedback agent. These agents collaborate on the failures of an LLM-based method on the training set to iteratively generate and refine a self-correction guideline tailored to LLM mistakes, mirroring human processes but without human involvement. Our extensive experiments show that MAGIC's guideline outperforms expert human's created ones. We empirically find out that the guideline produced by MAGIC enhances the interpretability of the corrections made, providing insights in analyzing the reason behind the failures and successes of LLMs in self-correction. All agent interactions are publicly available at https://huggingface.co/datasets/microsoft/MAGIC.