Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit

This is a planning-method note with an unpaired pilot audit. We adapt the classical paired-binary sample-size calculation (Miettinen, 1968) to quantization benchmarks, giving a conservative minimum detectable effect (MDE) bound δ* ≤ (z1-α/2+z1-β)√ρd/m in the paired item count m and the FP16-NF4 disagreement rate ρd. The bound turns "how reliable is my quantization claim?" into a one-line budget a benchmark designer can commit to before running. We illustrate the bound on four models and four benchmarks (k=5 splits of n=100), and add a parallel MMLU prompt-template study to put the bound's quantization-noise scale alongside the prompt-noise scale. Assuming ρd=0.10 (an unmeasured planning value), all observed NF4-FP16 deltas fall below the implied MDE, and most cross-split SDs lie within ± 1.5 pp of the binomial reference √p(1-p)/n, so much of the variance reported as "benchmark unreliability" on n=100 subsamples is binomial sampling noise. The single borderline cell (OPT-WinoGrande, |Delta|=3.2 pp) is below the implied MDE at ρd=0.10 but above it at ρd=0.05, illustrating the planning trade-off the bound makes explicit. On MMLU, prompt-template ranges of 2-10 pp meet or exceed the largest observed quantization delta (3.2 pp), so a quantization audit that does not first fix the prompt template absorbs template variance into its noise floor. We complement the bound with a five-line pre-registration template.