# Prioritizing Repeated LLM Evaluation for Hidden Failure Discovery

arXiv: https://arxiv.org/abs/2609.32547 · HTML: https://mlsanity.com/paper/2609.32547/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 13 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [Demystifying Prompts in Language Models via Perplexity Estimation](https://mlsanity.com/paper/2212.04037/) (2022)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [AcTracer: Active Testing of Large Language Model via Multi-Stage Sampling](https://mlsanity.com/paper/2408.03573/) (2024)
- [Scaling Up Active Testing to Large Language Models](https://mlsanity.com/paper/2508.09093/) (2025)
- [Estimating Tail Risks in Language Model Output Distributions](https://mlsanity.com/paper/2604.22167/) (2026)
- [Conformal Tail Risk Control for Large Language Model Alignment](https://mlsanity.com/paper/2502.20285/) (2025)
- [Rare Event Analysis of Large Language Models](https://mlsanity.com/paper/2602.06791/) (2026)
- [Active Testing of Large Language Models via Approximate Neyman Allocation](https://mlsanity.com/paper/2605.10075/) (2026)
- [Quantifying the Risk of Transferred Black Box Attacks](https://mlsanity.com/paper/2511.05102/) (2025)
- [Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling](https://mlsanity.com/paper/2604.09606/) (2026)
- [Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering](https://mlsanity.com/paper/2608.27621/) (2026)

## Abstract (arXiv)

Large language models are commonly evaluated by generating a small number of stochastic responses for each prompt in a benchmark. Because inference budgets are limited, this shallow evaluation may fail to observe low-probability but operationally important failures. A prompt that produces no failures in a small sample may therefore appear reliable despite having a nonzero latent probability of failure under repeated inference. We formulate LLM reliability evaluation as a budget-constrained discovery problem in which each prompt is associated with an unknown per-generation failure probability. We propose a budgeted discovery framework that first performs shallow evaluation across the prompt set and then uses trial-level failure outcomes together with prompt-derived representations to learn a feature-based ranking of failure propensity. The resulting scores prioritize prompts with zero observed shallow failures for deeper evaluation, concentrating the deep-evaluation budget where hidden failures are more likely to be discovered. We evaluate this approach on AIRBench and StrongREJECT across multiple model and system-prompt conditions. The central empirical test asks whether models fit without access to deep-evaluation outcomes can rank prompts with zero observed shallow failures according to their likelihood of producing failures under deeper evaluation. On AIRBench, the highest-ranked 10% of unresolved prompts achieves 2.54x hidden-failure lift for Qwen 2.5 7B and 1.87x for Gemma 3n E4B, recovering 25.4% and 18.7% of subsequently observed hidden failures, respectively, compared with 10% expected under random allocation. Semantic-neighborhood and feature-ablation analyses further show that this predictive signal can be recovered from multiple representations of prompt content and relationships.