# Fail Loudly: An Auditable Runtime for Agentic Data Analysis

arXiv: https://arxiv.org/abs/2609.32528 · HTML: https://mlsanity.com/paper/2609.32528/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 36 of its 46 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 36)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [TableBench: A Comprehensive and Complex Benchmark for Table Question Answering](https://mlsanity.com/paper/2408.09174/) (2024)
- [DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning](https://mlsanity.com/paper/2402.17453/) (2024)
- [DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?](https://mlsanity.com/paper/2409.07703/) (2024)
- [MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization](https://mlsanity.com/paper/2402.11453/) (2024)
- [Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow](https://mlsanity.com/paper/2306.07209/) (2023)
- [DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models](https://mlsanity.com/paper/2410.07331/) (2024)
- [DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing](https://mlsanity.com/paper/2410.12189/) (2024)
- [DeepAnalyze: Agentic Large Language Models for Autonomous Data Science](https://mlsanity.com/paper/2510.16872/) (2025)
- [DABstep: Data Agent Benchmark for Multi-step Reasoning](https://mlsanity.com/paper/2506.23719/) (2025)
- [KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes](https://mlsanity.com/paper/2506.06541/) (2025)
- [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025)
- [AgenticData: An Agentic Data Analytics System for Heterogeneous Data](https://mlsanity.com/paper/2508.05002/) (2025)
- [Automating Data Science: Prospects and Challenges](https://mlsanity.com/paper/2105.05699/) (2021)
- [LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science](https://mlsanity.com/paper/2510.01285/) (2025)
- [DAgent: A Relational Database-Driven Data Analysis Report Generation Agent](https://mlsanity.com/paper/2503.13269/) (2025)
- [Abacus: A Cost-Based Optimizer for Semantic Operator Systems](https://mlsanity.com/paper/2505.14661/) (2025)
- [Scaling Generalist Data-Analytic Agents](https://mlsanity.com/paper/2509.25084/) (2025)
- [DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automation](https://mlsanity.com/paper/2503.07044/) (2025)
- [QUITE: A Query Rewrite System Beyond Rules with LLM Agents](https://mlsanity.com/paper/2506.07675/) (2025)
- [Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Study](https://mlsanity.com/paper/2506.19794/) (2025)
- [Deep Research is the New Analytics System: Towards Building the Runtime for AI-Driven Analytics](https://mlsanity.com/paper/2509.02751/) (2025)
- [DataCross: A Unified Benchmark and Agent Framework for Cross-Modal Heterogeneous Data Analysis](https://mlsanity.com/paper/2601.21403/) (2026)
- [DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytelling](https://mlsanity.com/paper/2604.06474/) (2026)
- [AgentAda: Skill-Adaptive Data Analytics for Tailored Insight Discovery](https://mlsanity.com/paper/2504.07421/) (2025)

## Abstract (arXiv)

Large language models (LLMs) have enabled data-science agents to automate multi-step analyses over heterogeneous files. However, incorrect choices regarding data sources, scope, or statistical definitions often lead to silent errors: computations execute successfully but produce plausible yet incorrect outputs that fail to answer the intended question. To mitigate this, we present RADAR, an auditable runtime that makes an agent's analytical choices inspectable and supports their revision through execution feedback. RADAR operates through three core mechanisms. First, an evidence-preserving exploration module retrieves task-relevant content while retaining source locations and observation coverage. Next, the runtime uses typed operators to record the agent's declared inputs, operation arguments, and resulting observations. Finally, runtime validation checks proposed operations against these observations. When a conflict is detected, the runtime rejects the operation or provides diagnostic feedback, allowing the agent to revise its choices before errors propagate. This design enables agents to fail loudly while leaving semantic interpretation to the LLM. On KramaBench, RADAR achieves overall scores of 0.723 with full source retrieval and 0.747 with gold sources supplied, corresponding to relative gains of 35.9% and 28.8% over the strongest baselines. Beyond KramaBench, RADAR achieves relative performance gains of 14.0% on DA-Code and 59.3% on DABStep, demonstrating its applicability across diverse agentic data-analysis workflows.