# Incorporating Experts' Judgment into Machine Learning Models

arXiv: https://arxiv.org/abs/2304.11870 · HTML: https://mlsanity.com/paper/2304.11870/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 100 references that are in the map.
- First posted on arXiv 24 Apr 2023.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Detecting Adversarial Samples from Artifacts](https://mlsanity.com/paper/1703.00410/) (2017)
- [Compositional Fairness Constraints for Graph Embeddings](https://mlsanity.com/paper/1905.10674/) (2019)
- [Data Valuation using Reinforcement Learning](https://mlsanity.com/paper/1909.11671/) (2019)
- [Label-Free Supervision of Neural Networks with Physics and Domain Knowledge](https://mlsanity.com/paper/1609.05566/) (2016)
- [Label Efficient Learning of Transferable Representations across Domains and Tasks](https://mlsanity.com/paper/1712.00123/) (2017)
- [The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems](https://mlsanity.com/paper/2008.09100/) (2020)
- [ModelHub: Towards Unified Data and Lifecycle Management for Deep Learning](https://mlsanity.com/paper/1611.06224/) (2016)
- [Fast Training of Triplet-based Deep Binary Embedding Networks](https://mlsanity.com/paper/1603.02844/) (2016)
- [The Data Representativeness Criterion: Predicting the Performance of Supervised Classification Based on Data Set Similarity](https://mlsanity.com/paper/2002.12105/) (2020)
- [Pattern Decomposition with Complex Combinatorial Constraints: Application to Materials Discovery](https://mlsanity.com/paper/1411.7441/) (2014)
- [A Machine Learning Framework Towards Transparency in Experts' Decision Quality](https://mlsanity.com/paper/2110.11425/) (2021)

## Abstract (arXiv)

Machine learning (ML) models have been quite successful in predicting outcomes in many applications. However, in some cases, domain experts might have a judgment about the expected outcome that might conflict with the prediction of ML models. One main reason for this is that the training data might not be totally representative of the population. In this paper, we present a novel framework that aims at leveraging experts' judgment to mitigate the conflict. The underlying idea behind our framework is that we first determine, using a generative adversarial network, the degree of representation of an unlabeled data point in the training data. Then, based on such degree, we correct the textcolorblackmachine learning model's prediction by incorporating the experts' judgment into it, where the higher that aforementioned degree of representation, the less the weight we put on the expert intuition that we add to our corrected output, and vice-versa. We perform multiple numerical experiments on synthetic data as well as two real-world case studies (one from the IT services industry and the other from the financial industry). All results show the effectiveness of our framework; it yields much higher closeness to the experts' judgment with minimal sacrifice in the prediction accuracy, when compared to multiple baseline methods. We also develop a new evaluation metric that combines prediction accuracy with the closeness to experts' judgment. Our framework yields statistically significant results when evaluated on that metric.