mlsanityData to 14 Oct 2026

TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis

arXiv 2608.24145 ↗First posted 25 Aug 2026cs.CL cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

LLMs are increasingly used to analyze spreadsheets, CSV files, and other structured data, but producing a correct-looking answer is not the same as producing a trustworthy analysis. A trustworthy result should be supported by a valid path from the user question to the relevant data evidence. This requirement creates two diagnostic questions: whether an LLM can refuse to answer or ask for clarification when such a path does not exist, and whether it can preserve the correct analysis when the same evidence is expressed in different table forms. We introduce TrustDABench, a benchmark that operationalizes these questions as reliability and robustness. Starting from the evidence-path view, we derive 19 perturbation operators and instantiate them through an Agentic-LLM-based generation framework. TrustDABench contains 2,340 human-verified perturbed instances, and we evaluate eight representative LLMs. The results show substantial headroom: the best reliability result is only 24.21% average MRS, achieved by GPT-5.5, while the best robustness result still has 9.10% average ASR, achieved by Claude-Sonnet-5. The failures are systematic: models rarely detect conflicting evidence, often continue along executable but unsupported analysis paths, and remain sensitive to perturbations that change observation boundaries or cross-table relations. These findings suggest that stronger evidence-boundary recognition and representation-invariant reasoning are still needed for reliable structured-data analysis.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 83 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

25 of 83 references.

References · 25

  1. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 140
  2. 2024SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulationcited by 111
  3. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 53
  4. 2025DeepAnalyze: Agentic Large Language Models for Autonomous Data Sciencecited by 50
  5. 2022Towards Robustness of Text-to-SQL Models Against Natural and Realistic Adversarial Table Perturbationcited by 28
  6. 2024PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queriescited by 19
  7. 2024Interpretable LLM-based Table Question Answeringcited by 18
  8. 2024Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agentscited by 17
  9. 2024TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoringcited by 17
  10. 2020Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQLcited by 16
  11. 2024Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queriescited by 12
  12. 2025Confidence Estimation for Error Detection in Text-to-SQL Systemscited by 11
13 more
  1. 2024Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysiscited by 9
  2. 2025Hallucination Detection for LLM-based Text-to-SQL Generation via Two-Stage Metamorphic Testingcited by 9
  3. 2025Towards Fair In-Context Learning with Tabular Foundation Modelscited by 8
  4. 2024Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilitiescited by 7
  5. 2022Know What I don't Know: Handling Ambiguous and Unanswerable Questions for Text-to-SQLcited by 6
  6. 2024Unmasking Database Vulnerabilities: Zero-Knowledge Schema Inference Attacks in Text-to-SQL Systemscited by 6
  7. 2025Are Your LLM-based Text-to-SQL Models Secure? Exploring SQL Injection via Backdoor Attackscited by 6
  8. 2025RADAR: Benchmarking Language Models on Imperfect Tabular Datacited by 6
  9. 2026EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolutioncited by 5
  10. 2025SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfacescited by 4
  11. 2025When Tables Leak: Attacking String Memorization in LLM-Based Tabular Data Generationcited by 3
  12. 2026AIDABench: AI Data Analytics Benchmarkcited by 2
  13. 2026Same Data, Different Schemas: Robustness of LLM-based Text-to-SQLcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks
  2. 2025SafeLawBench: Towards Safe Alignment of Large Language Models
  3. 2026VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
  4. 2026ReliableTableQA:How Much Supervision Does Reliability Annotation Need?
  5. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
  6. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
  7. 2026All Verdicts are Not Equal: Rethinking LLM Judge Reliability
  8. 2025DataSciBench: An LLM Agent Benchmark for Data Science