# Insights into Natural Language Database Query Errors: From Attention Misalignment to User Handling Strategies

arXiv: https://arxiv.org/abs/2402.07304 · HTML: https://mlsanity.com/paper/2402.07304/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2024.

## Built on by (most cited first, 12 of 12)

- [Selective Prompt Anchoring for Code Generation](https://mlsanity.com/paper/2408.09121/) (2024)
- [Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors](https://mlsanity.com/paper/2501.09310/) (2025)
- [Text-to-SQL Domain Adaptation via Human-LLM Collaborative Data Annotation](https://mlsanity.com/paper/2502.15980/) (2025)
- [PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents](https://mlsanity.com/paper/2604.17653/) (2026)
- [Mis-prompt: Benchmarking Large Language Models for Proactive Error Handling](https://mlsanity.com/paper/2506.00064/) (2025)
- [IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection](https://mlsanity.com/paper/2512.14792/) (2025)
- [ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement](https://mlsanity.com/paper/2603.03742/) (2026)
- [KeyInst: Keyword Instruction for Improving SQL Formulation in Text-to-SQL](https://mlsanity.com/paper/2411.00788/) (2024)
- [How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair](https://mlsanity.com/paper/2607.25873/) (2026)
- [ISEE: Interactive Semantic Enrichment for Database Fields](https://mlsanity.com/paper/2608.02604/) (2026)
- [SafeQL: Search-based Refinement for Safe and Efficient LLM-based Text-to-SQL](https://mlsanity.com/paper/2608.09260/) (2026)
- [DataWeave: Deploying Human-LLM Analytics for Exploratory Structured Data Analysis](https://mlsanity.com/paper/2610.02679/) (2026)

## Abstract (arXiv)

Querying structured databases with natural language (NL2SQL) has remained a difficult problem for years. Recently, the advancement of machine learning (ML), natural language processing (NLP), and large language models (LLM) have led to significant improvements in performance, with the best model achieving ~85% percent accuracy on the benchmark Spider dataset. However, there is a lack of a systematic understanding of the types, causes, and effectiveness of error-handling mechanisms of errors for erroneous queries nowadays. To bridge the gap, a taxonomy of errors made by four representative NL2SQL models was built in this work, along with an in-depth analysis of the errors. Second, the causes of model errors were explored by analyzing the model-human attention alignment to the natural language query. Last, a within-subjects user study with 26 participants was conducted to investigate the effectiveness of three interactive error-handling mechanisms in NL2SQL. Findings from this paper shed light on the design of model structure and error discovery and repair strategies for natural language data query interfaces in the future.