mlsanityData to 7 Oct 2026

LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings

arXiv 2603.17017 ↗First posted 17 Mar 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Robustness evaluation for Natural Language to SQL (NL2SQL) systems is essential because real-world database environments are dynamic, noisy, and continuously evolving, whereas conventional benchmark evaluations typically assume static schemas and well-formed user inputs. In this work, we introduce a robustness evaluation benchmark containing approximately ten types of perturbations and conduct evaluations under both traditional and agentic settings. We assess multiple state-of-the-art large language models (LLMs), including Grok-4.1, Gemini-3-Pro, Claude-Opus-4.6, and GPT-5.2. Our results show that these models generally maintain strong performance under several perturbations; however, notable performance degradation is observed for surface-level noise (e.g., character-level corruption) and linguistic variation that preserves semantics while altering lexical or syntactic forms. Furthermore, we observe that surface-level noise causes larger performance drops in traditional pipelines, whereas linguistic variation presents greater challenges in agentic settings. These findings highlight the remaining challenges in achieving robust NL2SQL systems, particularly in handling linguistic variability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 18 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

10 of 18 references.

References · 10

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  3. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
  4. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  5. 2023C3: Zero-shot Text-to-SQL with ChatGPTcited by 159
  6. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  7. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  8. 2020It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbationscited by 64
  9. 2024Solid-SQL: Enhanced Schema-linking based In-context Learning for Robust Text-to-SQLcited by 11
  10. 2024Improving Generalization in Semantic Parsing by Increasing Natural Language Variationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimizing Small Language Models for NL2SQL via Chain-of-Thought Fine-Tuning
  2. 2026An Agentic System for Schema Aware NL2SQL Generation
  3. 2026LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling
  4. 2025LearNAT: Learning NL2SQL with AST-guided Task Decomposition for Large Language Models
  5. 2024Is Large Language Model Good at Database Knob Tuning? A Comprehensive Experimental Evaluation
  6. 2026BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
  7. 2025FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation
  8. 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models