# A Framework for Assessing AI Agent Decisions and Outcomes in AutoML Pipelines

arXiv: https://arxiv.org/abs/2602.22442 · HTML: https://mlsanity.com/paper/2602.22442/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 26 of 26)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML](https://mlsanity.com/paper/2410.02958/) (2024)
- [MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks](https://mlsanity.com/paper/2304.14979/) (2023)
- [AutoML for Deep Recommender Systems: A Survey](https://mlsanity.com/paper/2203.13922/) (2022)
- [ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering](https://mlsanity.com/paper/2505.23723/) (2025)
- [SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning](https://mlsanity.com/paper/2410.17238/) (2024)
- [MLZero: A Multi-Agent System for End-to-end Machine Learning Automation](https://mlsanity.com/paper/2505.13941/) (2025)
- [I-MCTS: Enhancing Agentic AutoML via Introspective Monte Carlo Tree Search](https://mlsanity.com/paper/2502.14693/) (2025)
- [AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models](https://mlsanity.com/paper/2408.00665/) (2024)
- [Large Language Models Synergize with Automated Machine Learning](https://mlsanity.com/paper/2405.03727/) (2024)
- [Empowering Time Series Forecasting with LLM-Agents](https://mlsanity.com/paper/2508.04231/) (2025)
- [Structured Agentic Workflows for Financial Time-Series Modeling with LLMs and Reflective Feedback](https://mlsanity.com/paper/2508.13915/) (2025)
- [AutoML in The Wild: Obstacles, Workarounds, and Expectations](https://mlsanity.com/paper/2302.10827/) (2023)
- [KompeteAI: Accelerated Autonomous Multi-Agent System for End-to-End Pipeline Generation for Machine Learning Problems](https://mlsanity.com/paper/2508.10177/) (2025)
- [AutoML-GPT: Large Language Model for AutoML](https://mlsanity.com/paper/2309.01125/) (2023)
- [Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization](https://mlsanity.com/paper/2509.09321/) (2025)
- [AutoProteinEngine: A Large Language Model Driven Agent Framework for Multimodal AutoML in Protein Engineering](https://mlsanity.com/paper/2411.04440/) (2024)
- [Multi-Agent Automated Machine Learning](https://mlsanity.com/paper/2210.09084/) (2022)
- [Scalable End-to-End ML Platforms: from AutoML to Self-serve](https://mlsanity.com/paper/2302.14139/) (2023)
- [OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System](https://mlsanity.com/paper/2303.00501/) (2023)
- [LightAutoDS-Tab: Multi-AutoML Agentic System for Tabular Data](https://mlsanity.com/paper/2507.13413/) (2025)

## Abstract (arXiv)

Agent-based AutoML systems rely on large language models to make complex, multi-stage decisions across data processing, model selection, and evaluation. However, existing evaluation practices remain outcome-centric, focusing primarily on final task performance. Through a review of prior work, we find that none of the surveyed agentic AutoML systems report structured, decision-level evaluation metrics intended for post-hoc assessment of intermediate decision quality. To address this limitation, we propose an Evaluation Agent (EA) that performs decision-centric assessment of AutoML agents without interfering with their execution. The EA is designed as an observer that evaluates intermediate decisions along four dimensions: decision validity, reasoning consistency, model quality risks beyond accuracy, and counterfactual decision impact. Across four proof-of-concept experiments, we demonstrate that the EA can (i) detect faulty decisions with an F1 score of 0.919, (ii) identify reasoning inconsistencies independent of final outcomes, and (iii) attribute downstream performance changes to agent decisions, revealing impacts ranging from -4.9% to +8.3% in final metrics. These results illustrate how decision-centric evaluation exposes failure modes that are invisible to outcome-only metrics. Our work reframes the evaluation of agentic AutoML systems from an outcome-based perspective to one that audits agent decisions, offering a foundation for reliable, interpretable, and governable autonomous ML systems.