# MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration

arXiv: https://arxiv.org/abs/2606.22826 · HTML: https://mlsanity.com/paper/2606.22826/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 8 of 8)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization](https://mlsanity.com/paper/2602.07909/) (2026)

## Built on by (most cited first, 1 of 1)

- [Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models](https://mlsanity.com/paper/2610.18248/) (2026)

## Abstract (arXiv)

Evaluating LLMs across many model variants---quantized, fine-tuned, or deployment-specific---requires running large benchmarks repeatedly, a process that can take tens of hours per model on edge hardware such as NPUs. Existing subset selection methods reduce this cost but depend on large calibration pools or learned prediction layers. We introduce MINCE (Monte Carlo Informed N-sizing for Compact Evaluation), which uses Monte Carlo simulation over per-item logs from a small set of calibration models to find the minimum subset size that bounds accuracy drift and then fixes a randomly sampled subset at that size, with no prediction layer needed. MINCE reduces IFEVAL by 54%, MMLU by 89%, GSM8K by 70%, and MMLU-Pro by 88% with maximum drift ≤2.62,pp on BF16 calibration models. The frozen subsets generalize to held-out GPU models with mean drift ≤1.40,pp and to INT4 NPU models with mean drift of 0.77--3.59,pp, while delivering evaluation speedups of up to 8.1× on the GPU models and evaluation speedups of 1.7--3.4× on the NPU models. The method is robust to calibration pool size and achieves lower drift than tinyBenchmarks (12× lower on MMLU, 3.3× on GSM8K) while using 42× fewer calibration models.