# Meta Pattern Concern Score: A Novel Evaluation Measure with Human Values for Multi-classifiers

arXiv: https://arxiv.org/abs/2209.06408 · HTML: https://mlsanity.com/paper/2209.06408/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 24 references that are in the map.
- First posted on arXiv 14 Sep 2022.

## Builds on (most cited first, 4 of 4)

- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors](https://mlsanity.com/paper/2005.02000/) (2020)
- [Scalable Quantitative Verification For Deep Neural Networks](https://mlsanity.com/paper/2002.06864/) (2020)
- [On the Separability of Classes with the Cross-Entropy Loss Function](https://mlsanity.com/paper/1909.06930/) (2019)

## Abstract (arXiv)

While advanced classifiers have been increasingly used in real-world safety-critical applications, how to properly evaluate the black-box models given specific human values remains a concern in the community. Such human values include punishing error cases of different severity in varying degrees and making compromises in general performance to reduce specific dangerous cases. In this paper, we propose a novel evaluation measure named Meta Pattern Concern Score based on the abstract representation of probabilistic prediction and the adjustable threshold for the concession in prediction confidence, to introduce the human values into multi-classifiers. Technically, we learn from the advantages and disadvantages of two kinds of common metrics, namely the confusion matrix-based evaluation measures and the loss values, so that our measure is effective as them even under general tasks, and the cross entropy loss becomes a special case of our measure in the limit. Besides, our measure can also be used to refine the model training by dynamically adjusting the learning rate. The experiments on four kinds of models and six datasets confirm the effectiveness and efficiency of our measure. And a case study shows it can not only find the ideal model reducing 0.53% of dangerous cases by only sacrificing 0.04% of training accuracy, but also refine the learning rate to train a new model averagely outperforming the original one with a 1.62% lower value of itself and 0.36% fewer number of dangerous cases.