# A Computational Theory for Efficient Mini Agent Evaluation with Causal Guarantees

arXiv: https://arxiv.org/abs/2503.21138 · HTML: https://mlsanity.com/paper/2503.21138/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 101 references that are in the map.
- First posted on arXiv 27 Mar 2025.

## Builds on (most cited first, 30 of 33)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Causal Effect Inference with Deep Latent-Variable Models](https://mlsanity.com/paper/1705.08821/) (2017)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [Causal-learn: Causal Discovery in Python](https://mlsanity.com/paper/2307.16405/) (2023)
- [The Blessings of Multiple Causes](https://mlsanity.com/paper/1805.06826/) (2018)
- [AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator](https://mlsanity.com/paper/2402.09742/) (2024)
- [The Causal-Neural Connection: Expressiveness, Learnability, and Inference](https://mlsanity.com/paper/2107.00793/) (2021)
- [DoWhy: An End-to-End Library for Causal Inference](https://mlsanity.com/paper/2011.04216/) (2020)
- [Learning Counterfactual Representations for Estimating Individual Dose-Response Curves](https://mlsanity.com/paper/1902.00981/) (2019)
- [Time Series Deconfounder: Estimating Treatment Effects over Time in the Presence of Hidden Confounders](https://mlsanity.com/paper/1902.00450/) (2019)
- [Generalization Bounds and Representation Learning for Estimation of Potential Outcomes and Causal Effects](https://mlsanity.com/paper/2001.07426/) (2020)
- [VCNet and Functional Targeted Regularization For Learning Causal Effects of Continuous Treatments](https://mlsanity.com/paper/2103.07861/) (2021)
- [gCastle: A Python Toolbox for Causal Discovery](https://mlsanity.com/paper/2111.15155/) (2021)
- [SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories](https://mlsanity.com/paper/2409.07440/) (2024)
- [DeepMatch: Balancing Deep Covariate Representations for Causal Inference Using Adversarial Training](https://mlsanity.com/paper/1802.05664/) (2018)
- [DoWhy-GCM: An extension of DoWhy for causal inference in graphical causal models](https://mlsanity.com/paper/2206.06821/) (2022)
- [Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context](https://mlsanity.com/paper/2406.05972/) (2024)
- [Benchmarking Framework for Performance-Evaluation of Causal Inference Analysis](https://mlsanity.com/paper/1802.05046/) (2018)
- [Evaluation Methods and Measures for Causal Learning Algorithms](https://mlsanity.com/paper/2202.02896/) (2022)
- [Bounds on Causal Effects and Application to High Dimensional Data](https://mlsanity.com/paper/2106.12121/) (2021)
- [Direct Heterogeneous Causal Learning for Resource Allocation Problems in Marketing](https://mlsanity.com/paper/2211.15728/) (2022)
- [Identifying Causal Effects with the R Package causaleffect](https://mlsanity.com/paper/1806.07161/) (2018)
- [LBCF: A Large-Scale Budget-Constrained Causal Forest Algorithm](https://mlsanity.com/paper/2201.12585/) (2022)
- [Causal Effect Identification from Multiple Incomplete Data Sources: A General Search-based Approach](https://mlsanity.com/paper/1902.01073/) (2019)
- [How and Why to Use Experimental Data to Evaluate Methods for Observational Causal Inference](https://mlsanity.com/paper/2010.03051/) (2020)
- [Decision Focused Causal Learning for Direct Counterfactual Marketing Optimization](https://mlsanity.com/paper/2407.13664/) (2024)
- [Ananke: A Python Package For Causal Inference Using Graphical Models](https://mlsanity.com/paper/2301.11477/) (2023)

## Abstract (arXiv)

In order to reduce the cost of experimental evaluation for agents, we introduce a computational theory of evaluation for mini agents: build evaluation model to accelerate the evaluation procedures. We prove upper bounds of generalized error and generalized causal effect error of given evaluation models for infinite agents. We also prove efficiency, and consistency to estimated causal effect from deployed agents to evaluation metric by prediction. To learn evaluation models, we propose a meta-learner to handle heterogeneous agents space problem. Comparing with existed evaluation approaches, our (conditional) evaluation model reduced 24.1% to 99.0% evaluation errors across 12 scenes, including individual medicine, scientific simulation, social experiment, business activity, and quantum trade. The evaluation time is reduced 3 to 7 order of magnitude per subject comparing with experiments or simulations.