# TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis

arXiv: https://arxiv.org/abs/2608.24145 · HTML: https://mlsanity.com/paper/2608.24145/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 83 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Builds on (most cited first, 25 of 25)

- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation](https://mlsanity.com/paper/2406.14991/) (2024)
- [Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness](https://mlsanity.com/paper/2301.08881/) (2023)
- [DeepAnalyze: Agentic Large Language Models for Autonomous Data Science](https://mlsanity.com/paper/2510.16872/) (2025)
- [Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbation](https://mlsanity.com/paper/2212.09994/) (2022)
- [PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries](https://mlsanity.com/paper/2410.11076/) (2024)
- [Interpretable LLM-based Table Question Answering](https://mlsanity.com/paper/2412.12386/) (2024)
- [TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring](https://mlsanity.com/paper/2403.15879/) (2024)
- [Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL](https://mlsanity.com/paper/2010.12634/) (2020)
- [Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents](https://mlsanity.com/paper/2403.05307/) (2024)
- [Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries](https://mlsanity.com/paper/2402.08349/) (2024)
- [Confidence Estimation for Error Detection in Text-to-SQL Systems](https://mlsanity.com/paper/2501.09527/) (2025)
- [Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysis](https://mlsanity.com/paper/2406.12719/) (2024)
- [Hallucination Detection for LLM-based Text-to-SQL Generation via Two-Stage Metamorphic Testing](https://mlsanity.com/paper/2512.22250/) (2025)
- [Towards Fair In-Context Learning with Tabular Foundation Models](https://mlsanity.com/paper/2505.09503/) (2025)
- [Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilities](https://mlsanity.com/paper/2411.16742/) (2024)
- [Know What I don't Know: Handling Ambiguous and Unanswerable Questions for Text-to-SQL](https://mlsanity.com/paper/2212.08902/) (2022)
- [Unmasking Database Vulnerabilities: Zero-Knowledge Schema Inference Attacks in Text-to-SQL Systems](https://mlsanity.com/paper/2406.14545/) (2024)
- [Are Your LLM-based Text-to-SQL Models Secure? Exploring SQL Injection via Backdoor Attacks](https://mlsanity.com/paper/2503.05445/) (2025)
- [RADAR: Benchmarking Language Models on Imperfect Tabular Data](https://mlsanity.com/paper/2506.08249/) (2025)
- [SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfaces](https://mlsanity.com/paper/2511.06778/) (2025)
- [EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolution](https://mlsanity.com/paper/2603.10697/) (2026)
- [When Tables Leak: Attacking String Memorization in LLM-Based Tabular Data Generation](https://mlsanity.com/paper/2512.08875/) (2025)
- [AIDABench: AI Data Analytics Benchmark](https://mlsanity.com/paper/2603.15636/) (2026)
- [Same Data, Different Schemas: Robustness of LLM-based Text-to-SQL](https://mlsanity.com/paper/2605.25838/) (2026)

## Abstract (arXiv)

LLMs are increasingly used to analyze spreadsheets, CSV files, and other structured data, but producing a correct-looking answer is not the same as producing a trustworthy analysis. A trustworthy result should be supported by a valid path from the user question to the relevant data evidence. This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms. We introduce TrustDABench, a benchmark that operationalizes these questions as reliability and robustness. Starting from the evidence-path view, we derive 19 perturbation operators and instantiate them through an Agentic-LLM-based generation framework. TrustDABench contains 2,340 human-verified perturbed instances, and we evaluate eight representative LLMs. The results show substantial headroom: the best reliability result is only 24.21% average MRS, achieved by GPT-5.5, while the best robustness result still has 9.10% average ASR, achieved by Claude-Sonnet-5. The failures are systematic: models rarely detect conflicting evidence, often continue along executable but unsupported analysis paths, and remain sensitive to perturbations that change observation boundaries or cross-table relations. These findings suggest that stronger evidence-boundary recognition and representation-invariant reasoning are still needed for reliable structured-data analysis.