mlsanityData to 14 Oct 2026

The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering

arXiv 2608.21262 ↗First posted 21 Aug 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Suppose we want a cutoff that 90% of a population falls below. We estimate it from a sample, and another sample would give a different cutoff and a different fraction below it. We ask how much that fraction varies when observations come in independent groups, such as pupils in classrooms or sentences in news articles. We prove that grouping multiplies its large-sample variance by 1+(m-1)ρI(p), where m is the group size, p is the target fraction, and ρI(p) measures whether two members of a group fall on the same side of the cutoff. That correlation can differ from the correlation between the scores themselves, and it changes with the target. We give a direct proof, a counterexample to using score correlation, and an extension to unequal group sizes. A dataset therefore does not have one effective sample size. How much information it contains depends on the question you ask. In our document experiment, the same 1,000 rows carried about 217 independent observations' worth of information at the median. At the 95th percentile, they carried about 621. Nothing about the dataset changed. We asked it a different question. The number of rows is a property of the dataset. The effective sample size belongs to the analysis.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 115 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

39 of 115 references.

References · 39

  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  2. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  3. 2019Conformal Prediction Under Covariate Shiftcited by 519
  4. 2022Conformal prediction beyond exchangeabilitycited by 355
  5. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 148
  6. 2023Class-Conditional Conformal Prediction with Many Classescited by 109
  7. 2018Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Datacited by 66
  8. 2021Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approachcited by 62
  9. 2021Conformalized Survival Analysiscited by 53
  10. 2021Conformal Sensitivity Analysis for Individual Treatment Effectscited by 40
  11. 2018Distribution-Free Prediction Sets for Two-Layer Hierarchical Modelscited by 35
  12. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 26
27 more
  1. 2023Adaptive conformal classification with noisy labelscited by 25
  2. 2025Know What You Don't Know: Uncertainty Calibration of Process Reward Modelscited by 21
  3. 2025HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statisticscited by 18
  4. 2023Conformal Autoregressive Generation: Beam Search with Coverage Guaranteescited by 16
  5. 2023Distribution-free inference with hierarchical datacited by 15
  6. 2024Group-Weighted Conformal Predictioncited by 14
  7. 2022Robust Design and Evaluation of Predictive Algorithms under Unobserved Confoundingcited by 8
  8. 2026PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelinescited by 7
  9. 2025On the Provable Performance Guarantee of Efficient Reasoning Modelscited by 6
  10. 2026When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scalingcited by 6
  11. 2024Privacy-preserving Quantile Treatment Effect Estimation for Randomized Controlled Trialscited by 4
  12. 2024Robust Conformal Volume Estimation in 3D Medical Imagescited by 4
  13. 2026When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generationcited by 4
  14. 2023Design-based conformal predictioncited by 2
  15. 2026Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3cited by 2
  16. 2026Conformal Prediction via Transported Beta Lawscited by 2
  17. 2026Conformal Certification of Reasoning Trace Prefixescited by 2
  18. 2026Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Datacited by 2
  19. 2026State-dependent error correlations shape voting thresholds in committees of AI agentscited by 2
  20. 2021Nonparametric estimator of the tail dependence coefficient: balancing bias and variancecited by 1
  21. 2025Reliable Statistical Guarantees for Conformal Predictors with Small Datasetscited by 1
  22. 2026Weighted Bayesian Conformal Predictioncited by 1
  23. 2026Conformal Path Reasoning: Trustworthy Knowledge Graph Question Answering via Path-Level Calibrationcited by 1
  24. 2026Multi-Agent Conformal Prediction with Personalized Statistical Validitycited by 1
  25. 2026Predicting Current Outcomes From Historical Survey Data With Weighted Conformal Predictioncited by 1
  26. 2026Conformal Prediction for Dyadic Regression Under Complex Missingnesscited by 1
  27. 2026False Sense of Safety in Selective Signal Classification: Auditing Bound Tightness and Exchangeability for Risk Controlcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting the Dataset Bias Problem from a Statistical Perspective
  2. 2026Available Guardrails: Certifying Selective Prediction across ML Systems
  3. 2022Partial Identification with Noisy Covariates: A Robust Optimization Approach
  4. 2026What Fixed-Rollout pass@k Evaluations Can Identify
  5. 2024Do we need rebalancing strategies? A theoretical and empirical study around SMOTE and its variants
  6. 2026Rethinking Representativeness and Diversity in Dynamic Data Selection
  7. 2026Scaling Laws for Precision in High-Dimensional Linear Regression
  8. 2025Rescaled Influence Functions: Accurate Data Attribution in High Dimension