Instance-Level Costs for Nuanced Classifier Evaluation

Standard classification treats all errors equally, but in applications such as content moderation and medical screening, mistakes on clear-cut cases are more costly than errors on ambiguous ones. From a contextual bandit framework, we propose normalized excess cost (NEC), a metric that weighs classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate: models with 5% error rate can achieve 1.8% NEC, revealing that most mistakes concentrate on ambiguous, low-cost examples. We also find that incorporating costs into training via loss weighting, sampling strategies, or regression yields inconsistent benefits. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs, even if cost-sensitive training offers limited benefit.