mlsanityData to 7 Oct 2026

A Computational Theory for Efficient Mini Agent Evaluation with Causal Guarantees

arXiv 2503.21138 ↗First posted 27 Mar 2025cs.AI cs.LG math.STcited by 0

Not on the map. Why?

Abstract (arXiv)

In order to reduce the cost of experimental evaluation for agents, we introduce a computational theory of evaluation for mini agents: build evaluation model to accelerate the evaluation procedures. We prove upper bounds of generalized error and generalized causal effect error of given evaluation models for infinite agents. We also prove efficiency, and consistency to estimated causal effect from deployed agents to evaluation metric by prediction. To learn evaluation models, we propose a meta-learner to handle heterogeneous agents space problem. Comparing with existed evaluation approaches, our (conditional) evaluation model reduced 24.1% to 99.0% evaluation errors across 12 scenes, including individual medicine, scientific simulation, social experiment, business activity, and quantum trade. The evaluation time is reduced 3 to 7 order of magnitude per subject comparing with experiments or simulations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 101 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

33 of 101 references.

References · 33

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  3. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 408
  4. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  5. 2016Learning Representations for Counterfactual Inferencecited by 327
  6. 2019Adapting Neural Networks for the Estimation of Treatment Effectscited by 285
  7. 2023Causal-learn: Causal Discovery in Pythoncited by 108
  8. 2018The Blessings of Multiple Causescited by 103
  9. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 99
  10. 2021The Causal-Neural Connection: Expressiveness, Learnability, and Inferencecited by 97
  11. 2020DoWhy: An End-to-End Library for Causal Inferencecited by 92
  12. 2019Learning Counterfactual Representations for Estimating Individual Dose-Response Curvescited by 90
21 more
  1. 2019Time Series Deconfounder: Estimating Treatment Effects over Time in the Presence of Hidden Confounderscited by 74
  2. 2020Generalization Bounds and Representation Learning for Estimation of Potential Outcomes and Causal Effectscited by 72
  3. 2021gCastle: A Python Toolbox for Causal Discoverycited by 60
  4. 2021VCNet and Functional Targeted Regularization For Learning Causal Effects of Continuous Treatmentscited by 59
  5. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  6. 2018DeepMatch: Balancing Deep Covariate Representations for Causal Inference Using Adversarial Trainingcited by 42
  7. 2024Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Contextcited by 41
  8. 2022DoWhy-GCM: An extension of DoWhy for causal inference in graphical causal modelscited by 40
  9. 2018Benchmarking Framework for Performance-Evaluation of Causal Inference Analysiscited by 33
  10. 2022Evaluation Methods and Measures for Causal Learning Algorithmscited by 23
  11. 2021Bounds on Causal Effects and Application to High Dimensional Datacited by 22
  12. 2022Direct Heterogeneous Causal Learning for Resource Allocation Problems in Marketingcited by 21
  13. 2018Identifying Causal Effects with the R Package causaleffectcited by 18
  14. 2022LBCF: A Large-Scale Budget-Constrained Causal Forest Algorithmcited by 17
  15. 2019Causal Effect Identification from Multiple Incomplete Data Sources: A General Search-based Approachcited by 16
  16. 2020How and Why to Use Experimental Data to Evaluate Methods for Observational Causal Inferencecited by 13
  17. 2024Decision Focused Causal Learning for Direct Counterfactual Marketing Optimizationcited by 9
  18. 2023Ananke: A Python Package For Causal Inference Using Graphical Modelscited by 8
  19. 2024Continuous Treatment Effect Estimation Using Gradient Interpolation and Kernel Smoothingcited by 7
  20. 2024Contrastive Balancing Representation Learning for Heterogeneous Dose-Response Curves Estimationcited by 6
  21. 2024Evaluatology: The Science and Engineering of Evaluationcited by 5

Built on by

Not cited yet.