# Analyzing Probabilistic Methods for Evaluating Agent Capabilities

arXiv: https://arxiv.org/abs/2409.16125 · HTML: https://mlsanity.com/paper/2409.16125/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Builds on (most cited first, 13 of 13)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [Evaluating Language-Model Agents on Realistic Autonomous Tasks](https://mlsanity.com/paper/2312.11671/) (2023)
- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)

## Built on by (most cited first, 4 of 4)

- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [Estimating the Probabilities of Rare Outputs in Language Models](https://mlsanity.com/paper/2410.13211/) (2024)
- [Quantifying detection rates for dangerous capabilities: a theoretical model of dangerous capability evaluations](https://mlsanity.com/paper/2412.15433/) (2024)
- [Measuring temporal effects of agent knowledge by date-controlled tool use](https://mlsanity.com/paper/2503.04188/) (2025)

## Abstract (arXiv)

To mitigate risks from AI systems, we need to assess their capabilities accurately. This is especially difficult in cases where capabilities are only rarely displayed. Phuong et al. propose two methods that aim to obtain better estimates of the probability of an AI agent successfully completing a given task. The milestone method decomposes tasks into subtasks, aiming to improve overall success rate estimation, while the expert best-of-N method leverages human guidance as a proxy for the model's independent performance. Our analysis of these methods as Monte Carlo estimators reveals that while both effectively reduce variance compared to naive Monte Carlo sampling, they also introduce bias. Experimental results demonstrate that the milestone method underestimates true solve rates for many real-world tasks due to its constraining assumptions. The expert best-of-N method exhibits even more severe underestimation across all tasks, attributed to an inherently flawed re-weighting factor. To enhance the accuracy of capability estimates of AI agents on difficult tasks, we suggest future work should leverage the rich literature on Monte Carlo Estimators.