# Post-Selection Distributional Model Evaluation

arXiv: https://arxiv.org/abs/2603.23055 · HTML: https://mlsanity.com/paper/2603.23055/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Testing for Outliers with Conformal p-values](https://mlsanity.com/paper/2104.08279/) (2021)
- [Exact post-selection inference, with application to the lasso](https://mlsanity.com/paper/1311.6238/) (2013)
- [Hypothesis testing with e-values](https://mlsanity.com/paper/2410.23614/) (2024)
- [Understanding O-RAN: Architecture, Interfaces, Algorithms, Security, and Research Challenges](https://mlsanity.com/paper/2202.01032/) (2022)
- [Intelligence and Learning in O-RAN for Data-driven NextG Cellular Networks](https://mlsanity.com/paper/2012.01263/) (2020)
- [5G Wireless Network Slicing for eMBB, URLLC, and mMTC: A Communication-Theoretic View](https://mlsanity.com/paper/1804.05057/) (2018)
- [Post-selection inference for e-value based confidence intervals](https://mlsanity.com/paper/2203.12572/) (2022)
- [Beyond Neyman-Pearson: e-values enable hypothesis testing with a data-driven alpha](https://mlsanity.com/paper/2205.00901/) (2022)
- [Post-Hoc Large-Sample Statistical Inference](https://mlsanity.com/paper/2603.08002/) (2026)
- [Ensuring Reliability via Hyperparameter Selection: Review and Advances](https://mlsanity.com/paper/2502.04206/) (2025)
- [Hypothesis testing for community structure in temporal networks using e-values](https://mlsanity.com/paper/2507.23034/) (2025)
- [Equivalence testing with data-dependent and post-hoc equivalence margins](https://mlsanity.com/paper/2603.16213/) (2026)

## Abstract (arXiv)

Formal model evaluation methods typically certify that a model satisfies a prescribed target key performance indicator (KPI) level. However, in many applications, the relevant target KPI level may not be known a priori, and the user may instead wish to compare candidate models by analyzing the full trade-offs between performance and reliability achievable at test time by the models. This task, requiring the reliable estimate of the test-time KPI distributions, is made more complicated by the fact that the same data must often be used both to pre-select a subset of candidate models and to estimate their KPI distributions, causing a potential post-selection bias. In this work, we introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent model pre-selection. Building on e-values, PS-DME controls post-selection false coverage rate (FCR) for the distributional KPI estimates and we establish explicit conditions under which it is provably more sample efficient than a baseline method based on sample splitting. Experiments on synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation demonstrate that PS-DME enables reliable comparison of candidate configurations across a range of reliability levels, supporting the statistically reliable exploration of performance--reliability trade-offs.