Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline
Not on the map. Why?
Abstract (arXiv)
Production text-to-SQL pipelines often end with an LLM-as-judge whose agreement with human annotators has never actually been measured. When we checked ours, the deployed gpt-4o-mini judge agreed with two-author gold at only Cohen's kappa = 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot-check, over-flagging 77.1% of the human-FAITHFUL cases in the enriched set. Most of its over-flags trace back to a single mechanism we call GRADE-HALLUCINATION. A self-hosted Qwen3.6-27B replacement (kappa = 0.72) lands in the same range as Claude Opus 4.7 (kappa = 0.71); the head-to-head is underpowered at n = 96, but for the deployment decision that hardly matters, since Qwen costs roughly 1/300 as much per call. Ensembling does not help for free. Pairing the weak judge with a stronger one degrades agreement, whereas three strong judges under unanimity routing reach kappa = 0.79 at 89.7% auto-coverage. Applied out-of-domain, the same audit recipe flags 25.5% of BIRD-financial's expert-authored gold SQLs as candidate gold-SQL issues under our annotation protocol. Code and pre-registration are at https://github.com/JamesL404/synca-audit.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 9 Sep 2026.
Builds on
27 of 34 references.
References · 27
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
- 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
- 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
- 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
- 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 143
15 more
- 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
- 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
- 2024Can LLM be a Personalized Judge?cited by 100
- 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
- 2022Recent Advances in Text-to-SQL: A Survey of What We Have and What We Expectcited by 47
- 2024Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmarkcited by 34
- 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
- 2025Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreementcited by 25
- 2025Leveraging Uncertainty Estimation for Efficient LLM Routingcited by 21
- 2025Cost-Saving LLM Cascades with Early Abstentioncited by 14
- 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
- 2025Semantic Agreement Enables Efficient Open-Ended LLM Cascadescited by 7
- 2025Auto-Prompt Ensemble for LLM Judgecited by 5
- 2025Taming SQL Complexity: LLM-Based Equivalence Evaluation for Text-to-SQLcited by 2
- 2024To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotationcited by 1
Built on by
Not cited yet.