# Beyond Query-Level Comparison: Fine-Grained Reinforcement Learning for Text-to-SQL with Automated Interpretable Critiques

arXiv: https://arxiv.org/abs/2511.22258 · HTML: https://mlsanity.com/paper/2511.22258/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 27 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 27 of 27)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation](https://mlsanity.com/paper/1905.08205/) (2019)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [RM-R1: Reward Modeling as Reasoning](https://mlsanity.com/paper/2505.02387/) (2025)
- [CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL](https://mlsanity.com/paper/2410.01943/) (2024)
- [Semantic Evaluation for Text-to-SQL with Distilled Test Suites](https://mlsanity.com/paper/2010.02840/) (2020)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization](https://mlsanity.com/paper/2109.05157/) (2021)
- [OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale](https://mlsanity.com/paper/2503.02240/) (2025)
- [JudgeLRM: Large Reasoning Models as a Judge](https://mlsanity.com/paper/2504.00050/) (2025)
- [SQL-R1: Training Natural Language to SQL Reasoning Model By Reinforcement Learning](https://mlsanity.com/paper/2504.08600/) (2025)
- [Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey](https://mlsanity.com/paper/2310.17894/) (2023)
- [Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance](https://mlsanity.com/paper/2404.08817/) (2024)
- [Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation](https://mlsanity.com/paper/2410.14251/) (2024)
- [Text-to-SQL Error Correction with Language Models of Code](https://mlsanity.com/paper/2305.13073/) (2023)
- [Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQL](https://mlsanity.com/paper/2504.15077/) (2025)
- [MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification](https://mlsanity.com/paper/2412.04494/) (2024)
- [Graph-Reward-SQL: Execution-Free Reinforcement Learning for Text-to-SQL via Graph Matching and Stepwise Reward](https://mlsanity.com/paper/2505.12380/) (2025)
- [Fundamental Challenges in Evaluating Text2SQL Solutions and Detecting Their Limitations](https://mlsanity.com/paper/2501.18197/) (2025)

## Built on by (most cited first, 2 of 2)

- [OptiSQL: Executable SQL Generation from Optical Tokens](https://mlsanity.com/paper/2601.13695/) (2026)
- [GLEAN: Grounded Lightweight Evaluation Anchors for Contamination-Aware Tabular Reasoning](https://mlsanity.com/paper/2603.02212/) (2026)

## Abstract (arXiv)

Text-to-SQL, a pivotal natural language processing (NLP) task that converts textual queries into executable SQL, has seen substantial progress in recent years. However, existing evaluation and reward mechanisms used to train and assess the text-to-SQL models remain a critical bottleneck. Current approaches heavily rely on manually annotated gold SQL queries, which are costly to produce and impractical for large-scale evaluation. More importantly, most reinforcement learning (RL) methods in text-to-SQL leverage only the final binary execution outcome as the reward signal, a coarse-grained supervision that overlooks detailed structural and semantic errors from the perspective of rubrics. To address these challenges, we propose RuCo-C, a novel generative judge model for fine-grained, query-specific automatic evaluation using interpretable critiques without human intervention. Our framework first automatically generates query-specific evaluation rubrics for human-free annotation, linking them to interpretable critiques. Subsequently, it integrates densified reward feedback through a "progressive exploration" strategy during the RL training process, which dynamically adjusts the rewards to enhance the model's performance. Comprehensive experiments demonstrate that RuCo-C outperforms existing methods in text-to-SQL evaluation, yielding significant performance gains.