# Bounded Difference Concentration for Infinitely Exchangeable Sequences with Applications to AI Benchmark Uncertainty

arXiv: https://arxiv.org/abs/2606.17426 · HTML: https://mlsanity.com/paper/2606.17426/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Are We Done with MMLU?](https://mlsanity.com/paper/2406.04127/) (2024)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)
- [Quantifying Variance in Evaluation Benchmarks](https://mlsanity.com/paper/2406.10229/) (2024)
- [Transductive Rademacher Complexity and its Applications](https://mlsanity.com/paper/1401.3441/) (2014)
- [Concentration Inequalities for Exchangeable Tensors and Matrix-valued Data](https://mlsanity.com/paper/2601.20152/) (2026)
- [Hoeffding-Type Concentration Bounds for Exchangeable Random Variables](https://mlsanity.com/paper/2603.10190/) (2026)

## Abstract (arXiv)

We consider the concentration properties of functions of infinitely exchangeable random variables. By conditioning on the de Finetti directing measure, we show that the deviation of any function with bounded-difference constants c1, dots, cn decomposes into a conditional sampling fluctuation and a latent mixture fluctuation. When this latent mixture is σmix2-subgaussian, we establish a concentration inequality with an effective variance proxy of frac14sumi ci2 + σmix2. Crucially, we demonstrate that for zero-sum linear contrasts, such as the difference between a subsample mean and a full population mean, the latent mixture term cancels exactly. This cancellation yields a tight, mixture-free Hoeffding-type bound that provides a direct de Finetti mechanism for the infinite-extendibility limit of recent finite-exchangeable concentration results. We apply this framework to quantify uncertainty in composite AI benchmarks, such as MMLU, where question items naturally exhibit exchangeable dependence across domains. Our results provide both a domain-stratified hierarchical model for bounding the uncertainty of accuracy scores, and a distribution-free, cost-saving statistical guarantee for accurately estimating full benchmark scores from random subsets.