mlsanityData to 7 Oct 2026

Counterfactual Fragility Certificates: Exposing High-Confidence Brittleness under Structured Evidence Failure

arXiv 2609.00366 ↗First posted 29 Jun 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

High test accuracy and good aggregate calibration do not show whether an individual prediction is structurally supported by its evidence. In tabular decision systems, failures often occur when a feature family becomes unavailable, delayed, noisy, stale, or low-trust while the model remains highly confident. Existing calibration, uncertainty, selective-prediction, explanation, and perturbation methods provide scalar scores or attribution maps, but not a recomputable audit object answering: under a declared evidence-failure protocol, what trajectory makes this prediction lose support? We introduce Counterfactual Fragility Certificates (CFC), a model-agnostic protocol-level audit certificate-not a formal robustness certificate-that maps each prediction into an ordered evidence-failure trajectory summarized by greedy flip budget, normalized margin-collapse area, degradation thresholds, and fragility dominance score. Across seven tabular benchmarks and strong linear, tree-based, boosting, and neural baselines, CFC-FDS identifies independently brittle high-confidence cases with 0.915 AUROC, improving over the strongest non-certificate score by +0.405. The advantage persists across perturbation, permutation-importance, group-SHAP, baseline-choice, seed-variance, budgeted-review, and naturalistic field-unavailability checks. Under a 20% review budget, CFC-FDS captures 88.9% of brittle high-confidence cases, compared with 31.8-37.4% for confidence and energy scores. We also evaluate fragility-aware regularization and brittleness-aware temperature correction as secondary uses. CFC provides a concrete reliability framework for exposing high-confidence brittleness missed by ordinary score-centric evaluation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

29 of 38 references.

References · 29

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  6. 2017Selective Classification for Deep Neural Networkscited by 799
  7. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  8. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  9. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
  10. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  11. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  12. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 333
17 more
  1. 2018To Trust Or Not To Trust A Classifiercited by 205
  2. 2019Model-Agnostic Counterfactual Explanations for Consequential Decisionscited by 204
  3. 2019Addressing Failure Prediction by Learning Model Confidencecited by 183
  4. 2019Feature relevance quantification in explainable AI: A causal problemcited by 164
  5. 2020Confidence-Aware Learning for Deep Neural Networkscited by 113
  6. 2019Combating Label Noise in Deep Learning Using Abstentioncited by 86
  7. 2021CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithmscited by 85
  8. 2019Calibration tests in multi-class classification: A unifying frameworkcited by 79
  9. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 64
  10. 2021Exploring Counterfactual Explanations Through the Lens of Adversarial Examples: A Theoretical and Empirical Analysiscited by 52
  11. 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
  12. 2018What made you do this? Understanding black-box decisions with sufficient input subsetscited by 31
  13. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 31
  14. 2022Calibrated Selective Classificationcited by 30
  15. 2023Beyond In-Domain Scenarios: Robust Density-Aware Calibrationcited by 14
  16. 2024TabularBench: Benchmarking Adversarial Robustness for Tabular Deep Learning in Real-world Use-casescited by 10
  17. 2024Confidence-aware Contrastive Learning for Selective Classificationcited by 7

Built on by

Not cited yet.