ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi-scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution-based benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver-User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 6 Sep 2026.
Builds on
22 of 46 references.
References · 22
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2024Qwen2.5-Coder Technical Reportcited by 1,203
- 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
- 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
- 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
- 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
- 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232
- 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
- 2023RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLcited by 195
10 more
- 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
- 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 164
- 2021LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relationscited by 119
- 2023Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for Text-to-SQL Parsingcited by 93
- 2021Natural SQL: Making SQL Easier to Infer from Natural Language Specificationscited by 68
- 2024A Survey on Employing Large Language Models for Text-to-SQL Taskscited by 67
- 2025OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignmentcited by 53
- 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
- 2024Structure Guided Large Language Model for SQL Generationcited by 9
- 2025LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoningcited by 7
Built on by
Not cited yet.