# Uncovering the Impact of Chain-of-Thought Reasoning for Direct Preference Optimization: Lessons from Text-to-SQL

arXiv: https://arxiv.org/abs/2502.11656 · HTML: https://mlsanity.com/paper/2502.11656/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 36 of its 66 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 30 of 36)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [KTO: Model Alignment as Prospect Theoretic Optimization](https://mlsanity.com/paper/2402.01306/) (2024)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [From r to Q^*: Your Language Model is Secretly a Q-Function](https://mlsanity.com/paper/2404.12358/) (2024)
- [V-STaR: Training Verifiers for Self-Taught Reasoners](https://mlsanity.com/paper/2402.06457/) (2024)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [C3: Zero-shot Text-to-SQL with ChatGPT](https://mlsanity.com/paper/2307.07306/) (2023)
- [MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL](https://mlsanity.com/paper/2312.11242/) (2023)
- [Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://mlsanity.com/paper/2402.14830/) (2024)
- [CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL](https://mlsanity.com/paper/2410.01943/) (2024)
- [DPO Meets PPO: Reinforced Token Optimization for RLHF](https://mlsanity.com/paper/2404.18922/) (2024)
- [CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences](https://mlsanity.com/paper/2403.09032/) (2024)
- [Bootstrapping Language Models with DPO Implicit Rewards](https://mlsanity.com/paper/2406.09760/) (2024)

## Built on by (most cited first, 9 of 9)

- [LearNAT: Learning NL2SQL with AST-guided Task Decomposition for Large Language Models](https://mlsanity.com/paper/2504.02327/) (2025)
- [ReEx-SQL: Reasoning with Execution-Aware Reinforcement Learning for Text-to-SQL](https://mlsanity.com/paper/2505.12768/) (2025)
- [SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces](https://mlsanity.com/paper/2511.06778/) (2025)
- [See or Say Graphs: Agent-Driven Scalable Graph Structure Understanding with Vision-Language Models](https://mlsanity.com/paper/2510.16769/) (2025)
- [Knowledge Distillation with Structured Chain-of-Thought for Text-to-SQL](https://mlsanity.com/paper/2512.17053/) (2025)
- [EvolSQL: Structure-Aware Evolution for Scalable Text-to-SQL Data Synthesis](https://mlsanity.com/paper/2601.04875/) (2026)
- [EXPO-SQL: Execution-based Clause-level Policy Optimization for Text-to-SQL](https://mlsanity.com/paper/2606.23693/) (2026)
- [Learning When to Reason for Text-to-SQL via SFT and DPO](https://mlsanity.com/paper/2607.22622/) (2026)
- [BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations](https://mlsanity.com/paper/2608.29345/) (2026)

## Abstract (arXiv)

Direct Preference Optimization (DPO) has proven effective in complex reasoning tasks like math word problems and code generation. However, when applied to Text-to-SQL datasets, it often fails to improve performance and can even degrade it. Our investigation reveals the root cause: unlike math and code tasks, which naturally integrate Chain-of-Thought (CoT) reasoning with DPO, Text-to-SQL datasets typically include only final answers (gold SQL queries) without detailed CoT solutions. By augmenting Text-to-SQL datasets with synthetic CoT solutions, we achieve, for the first time, consistent and significant performance improvements using DPO. Our analysis shows that CoT reasoning is crucial for unlocking DPO's potential, as it mitigates reward hacking, strengthens discriminative capabilities, and improves scalability. These findings offer valuable insights for building more robust Text-to-SQL models. To support further research, we publicly release the code and CoT-enhanced datasets.