# Business Utility of Large Language Models as Exploratory Data Analysis Agents

arXiv: https://arxiv.org/abs/2606.00051 · HTML: https://mlsanity.com/paper/2606.00051/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/)

## Builds on (most cited first, 15 of 15)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code](https://mlsanity.com/paper/2311.09835/) (2023)
- [DataSciBench: An LLM Agent Benchmark for Data Science](https://mlsanity.com/paper/2502.13897/) (2025)
- [Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?](https://mlsanity.com/paper/2407.10956/) (2024)
- [DABstep: Data Agent Benchmark for Multi-step Reasoning](https://mlsanity.com/paper/2506.23719/) (2025)
- [NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation](https://mlsanity.com/paper/2503.11984/) (2025)
- [nvBench 2.0: Resolving Ambiguity in Text-to-Visualization through Stepwise Reasoning](https://mlsanity.com/paper/2503.12880/) (2025)
- [TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents](https://mlsanity.com/paper/2505.13291/) (2025)
- [FDABench: A Benchmark for Data Agents on Analytical Queries over Heterogeneous Data](https://mlsanity.com/paper/2509.02473/) (2025)
- [Text2Vis: A Challenging and Diverse Benchmark for Generating Multimodal Visualizations from Text](https://mlsanity.com/paper/2507.19969/) (2025)
- [IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis](https://mlsanity.com/paper/2505.18223/) (2025)
- [ML-Tool-Bench: Tool-Augmented Planning for ML Tasks](https://mlsanity.com/paper/2512.00672/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used in analytical workflows, but their suitability as exploratory data analysis (EDA) agents in business settings remains uncertain. In practice, a deployable EDA agent must provide not only useful average performance but also sufficient repeatability to support trust in its outputs. We evaluate this requirement in a controlled, business-relevant benchmark built on an agent-based supply chain simulation. The task is to identify supplier-product combinations responsible for low quality and downstream sales loss by reasoning from indirect operational traces rather than from explicit labels. Fifteen model-variant configurations from eight model families were evaluated under four experimental conditions that varied data representation, prompt clarity, and signal strength, with five trajectories per condition. Outputs were scored against deterministic ground truth using the Jaccard index and assessed through a framework that combines mean score (ms), coefficient of variation (CV), exploratory cross-condition significance tests, and Business utility, a risk-adjusted metric that we propose to summarise quality and repeatability in a single operational measure. The results show that most configurations are not reliable enough for autonomous EDA use, even when their average scores appear acceptable. GPT-5.4 with extra-high reasoning effort achieved the strongest overall profile, with an experiment-averaged ms of 0.8748 and an experiment-averaged Business utility of 0.6952, while the next-best configurations lost substantially more utility after variability discounting. Our findings suggest that evaluation of EDA agents should treat average quality, repeatability, and condition sensitivity as complementary dimensions of operational trustworthiness.