# Birds of a Feather Trust Together: Knowing When to Trust a Classifier via Adaptive Neighborhood Aggregation

arXiv: https://arxiv.org/abs/2211.16466 · HTML: https://mlsanity.com/paper/2211.16466/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [Probabilistic Face Embeddings](https://mlsanity.com/paper/1904.09658/) (2019)
- [Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts](https://mlsanity.com/paper/2006.09239/) (2020)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Data Uncertainty Learning in Face Recognition](https://mlsanity.com/paper/2003.11339/) (2020)
- [POEM: Out-of-Distribution Detection with Posterior Sampling](https://mlsanity.com/paper/2206.13687/) (2022)
- [Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration](https://mlsanity.com/paper/2109.14285/) (2021)
- [On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products](https://mlsanity.com/paper/1610.01256/) (2016)
- [Distribution-free calibration guarantees for histogram binning without sample splitting](https://mlsanity.com/paper/2105.04656/) (2021)

## Built on by (most cited first, 7 of 7)

- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [SteerConf: Steering LLMs for Confidence Elicitation](https://mlsanity.com/paper/2503.02863/) (2025)
- [Proximity-Informed Calibration for Deep Neural Networks](https://mlsanity.com/paper/2306.04590/) (2023)
- [Thinking Out Loud: Do Reasoning Models Know When They're Right?](https://mlsanity.com/paper/2504.06564/) (2025)
- [Great Models Think Alike: Improving Model Reliability via Inter-Model Latent Agreement](https://mlsanity.com/paper/2305.01481/) (2023)
- [GraphCleaner: Detecting Mislabelled Samples in Popular Graph Learning Benchmarks](https://mlsanity.com/paper/2306.00015/) (2023)
- [A Trustworthy Framework for Medical Image Analysis with Deep Learning](https://mlsanity.com/paper/2212.02764/) (2022)

## Abstract (arXiv)

How do we know when the predictions made by a classifier can be trusted? This is a fundamental problem that also has immense practical applicability, especially in safety-critical areas such as medicine and autonomous driving. The de facto approach of using the classifier's softmax outputs as a proxy for trustworthiness suffers from the over-confidence issue; while the most recent works incur problems such as additional retraining cost and accuracy versus trustworthiness trade-off. In this work, we argue that the trustworthiness of a classifier's prediction for a sample is highly associated with two factors: the sample's neighborhood information and the classifier's output. To combine the best of both worlds, we design a model-agnostic post-hoc approach NeighborAgg to leverage the two essential information via an adaptive neighborhood aggregation. Theoretically, we show that NeighborAgg is a generalized version of a one-hop graph convolutional network, inheriting the powerful modeling ability to capture the varying similarity between samples within each class. We also extend our approach to the closely related task of mislabel detection and provide a theoretical coverage guarantee to bound the false negative. Empirically, extensive experiments on image and tabular benchmarks verify our theory and suggest that NeighborAgg outperforms other methods, achieving state-of-the-art trustworthiness performance.