Bag of Coins: A Statistical Probe into Neural Confidence Structures

Modern neural networks often produce miscalibrated confidence scores and struggle to detect out-of-distribution (OOD) inputs, while most existing methods post-process outputs without testing internal consistency. We introduce the Bag-of-Coins (BoC) probe, a non-parametric diagnostic of logit coherence that compares softmax confidence hat p to an aggregate of pairwise Luce-style dominance probabilities bar q, yielding a deterministic coherence score and a p-value-based structural score. Across ViT, ResNet, and RoBERTa with ID/OOD test sets, the coherence gap Delta=bar q-hat p reveals clear ID/OOD separation for ViT (ID ~0.1-0.2, OOD ~0.5-0.6) but substantial overlap for ResNet and RoBERTa (both ~0), indicating architecture-dependent uncertainty geometry. As a practical method, BoC improves calibration only when the base model is poorly calibrated (ViT: ECE 0.024 vs. 0.180) and underperforms standard calibrators (ECE ~0.005), while for OOD detection it fails across architectures (AUROC 0.020-0.253) compared to standard scores (0.75-0.99). We position BoC as a research diagnostic for interrogating how architectures encode uncertainty in logit geometry rather than a production calibration or OOD detection method.