# Measuring Competency of Machine Learning Systems and Enforcing Reliability

arXiv: https://arxiv.org/abs/2212.01415 · HTML: https://mlsanity.com/paper/2212.01415/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 8 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 3 of 3)

- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks](https://mlsanity.com/paper/2009.11848/) (2020)
- [Principles and Practice of Explainable Machine Learning](https://mlsanity.com/paper/2009.11698/) (2020)

## Abstract (arXiv)

We explore the impact of environmental conditions on the competency of machine learning agents and how real-time competency assessments improve the reliability of ML agents. We learn a representation of conditions which impact the strategies and performance of the ML agent enabling determination of actions the agent can make to maintain operator expectations in the case of a convolutional neural network that leverages visual imagery to aid in the obstacle avoidance task of a simulated self-driving vehicle.