# Accurate Layerwise Interpretable Competence Estimation

arXiv: https://arxiv.org/abs/1910.11363 · HTML: https://mlsanity.com/paper/1910.11363/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2019.

## Built on by (most cited first, 10 of 10)

- [A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift](https://mlsanity.com/paper/2311.14743/) (2023)
- [Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models](https://mlsanity.com/paper/2303.12748/) (2023)
- [To Trust or Not to Trust a Regressor: Estimating and Explaining Trustworthiness of Regression Predictions](https://mlsanity.com/paper/2104.06982/) (2021)
- [RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement](https://mlsanity.com/paper/2605.09730/) (2026)
- [Understanding the Dependence of Perception Model Competency on Regions in an Image](https://mlsanity.com/paper/2407.10543/) (2024)
- [A Decision-driven Methodology for Designing Uncertainty-aware AI Self-Assessment](https://mlsanity.com/paper/2408.01301/) (2024)
- [PaRCE: Probabilistic and Reconstruction-based Competency Estimation for CNN-based Image Classification](https://mlsanity.com/paper/2411.16715/) (2024)
- [Persistent Convolution: A Topological Framework for AI Alignment Testing and Semantic Space Characterization](https://mlsanity.com/paper/2607.29008/) (2026)
- [Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energy](https://mlsanity.com/paper/2401.12129/) (2024)
- [Explaining Low Perception Model Competency with High-Competency Counterfactuals](https://mlsanity.com/paper/2504.05254/) (2025)

## Abstract (arXiv)

Estimating machine learning performance 'in the wild' is both an important and unsolved problem. In this paper, we seek to examine, understand, and predict the pointwise competence of classification models. Our contributions are twofold: First, we establish a statistically rigorous definition of competence that generalizes the common notion of classifier confidence; second, we present the ALICE (Accurate Layerwise Interpretable Competence Estimation) Score, a pointwise competence estimator for any classifier. By considering distributional, data, and model uncertainty, ALICE empirically shows accurate competence estimation in common failure situations such as class-imbalanced datasets, out-of-distribution datasets, and poorly trained models. Our contributions allow us to accurately predict the competence of any classification model given any input and error function. We compare our score with state-of-the-art confidence estimators such as model confidence and Trust Score, and show significant improvements in competence prediction over these methods on datasets such as DIGITS, CIFAR10, and CIFAR100.