# HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statistics

arXiv: https://arxiv.org/abs/2505.05602 · HTML: https://mlsanity.com/paper/2505.05602/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 8 May 2025.

## Builds on (most cited first, 17 of 17)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro](https://mlsanity.com/paper/1912.11554/) (2019)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [Clio: Privacy-Preserving Insights into Real-World AI Use](https://mlsanity.com/paper/2412.13678/) (2024)
- [HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation](https://mlsanity.com/paper/2412.21199/) (2024)
- [HCAST: Human-Calibrated Autonomy Software Tasks](https://mlsanity.com/paper/2503.17354/) (2025)

## Built on by (most cited first, 18 of 18)

- [Measuring what Matters: Construct Validity in Large Language Model Benchmarks](https://mlsanity.com/paper/2511.04703/) (2025)
- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [Do Repetitions Matter? Strengthening Reliability in LLM Evaluations](https://mlsanity.com/paper/2509.24086/) (2025)
- [Ask don't tell: Reducing sycophancy in large language models](https://mlsanity.com/paper/2602.23971/) (2026)
- [Skewed Score: A statistical framework to assess autograders](https://mlsanity.com/paper/2507.03772/) (2025)
- [Propensity Inference: Environmental Contributors to LLM Behaviour](https://mlsanity.com/paper/2604.21098/) (2026)
- [Seven simple steps for log analysis in AI systems](https://mlsanity.com/paper/2604.09563/) (2026)
- [Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors](https://mlsanity.com/paper/2607.07368/) (2026)
- [A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models](https://mlsanity.com/paper/2511.00527/) (2025)
- [Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations](https://mlsanity.com/paper/2608.14425/) (2026)
- [Sequential Bayesian Evaluation of Large Language Model Behavior](https://mlsanity.com/paper/2511.10661/) (2025)
- [CAFE: A Compound-AI Factorial Evaluation Framework](https://mlsanity.com/paper/2607.10380/) (2026)
- [Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering](https://mlsanity.com/paper/2510.05709/) (2025)
- [A Multi-Turn Framework for Evaluating AI Misuse in Fraud and Cybercrime Scenarios](https://mlsanity.com/paper/2602.21831/) (2026)
- [Generative Responsible AI Data Evaluation Schema (GRAIDES) for AI Assurance in Local Government](https://mlsanity.com/paper/2606.20963/) (2026)
- [Modeling Memory-Dependent Reliability of LLMs: A Hidden Markov Model](https://mlsanity.com/paper/2607.22951/) (2026)
- [The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering](https://mlsanity.com/paper/2608.21262/) (2026)
- [Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations](https://mlsanity.com/paper/2610.08364/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) and other AI systems evolve, robustly estimating their capabilities from inherently stochastic outputs while systematically quantifying uncertainty in these estimates becomes increasingly important. Further, advanced AI evaluations often have a nested hierarchical structure, exhibit high levels of complexity, and come with high costs in testing the most advanced AI systems. To address these challenges, we introduce HiBayES, a generalizable Hierarchical Bayesian modeling framework for AI Evaluation Statistics. HiBayES supports robust inferences in classical question-answer benchmarks and advanced agentic evaluations, particularly in low-data scenarios (e.g., < 20 data points per evaluation). Built on Generalized Linear Models (GLMs), Bayesian data analysis, and formal model comparison, HiBayES provides principled uncertainty quantification and robust parameter estimation. This paper offers a comprehensive introduction to HiBayES, including illustrative examples, comparisons to conventional statistical methods, and practical guidance for implementing multilevel Bayesian GLMs. Additionally, we provide a HiBayES software package [4] (Beta version) for out-of-the-box implementation.